I started moving all of our hosting, regardless of package, to RAID-5 early this year. It was a no brainer because I was buying the hardware, not leasing it, which made it cheaper in the long run. The PowerEdge servers I bought had RAID-5 + Hotspare built in.

I am now adding a few new servers at SoftLayer and am deciding whether to go RAID-1 or nothing. This is for standard $10/mo hosting accounts, so it's hard to justify spending ~$60/mo for an extra drive + RAID card on a new server. Sounds cheap but that's six extra customers you had to sell to (never easy, we all know) that still demand support, use space, etc. and you're not making a penny.

I'm also trying to figure out just how often drives crash. I understand MTTF etc but have never personally experienced it in ~11 years in hosting on > 100 servers when all is said and done.

What's the conventional wisdom? I understand that for bigger clients (like 10GB ecommerce sites paying $75/mo, for example) that having RAID is just due diligence. But what about regular accounts?

Regs,

Drew:peace:

That $60 a month could be the difference between loosing all your customers when a harddrive fails or staying up and keeping all your clients.

Really its your choice but is it worth the risk?

When you look at it like that, the whole thread seems petty. Here's the rub: with good backups, you can restore a server to pretty much par condition in 4-6 hours MAX for, say, 30 GB worth of Web sites.

Assuming a drive fails once every 2 years, which is way more than in my experience, you've spent $60 x 24 = $1,440 to save 6 hours of downtime, which isn't likely to lose you many clients if you've had a good track record.

If you can assume failure once every 4 years, that's ~$3k for 6 hours of downtime. $500/hour.

My economics degree (Clemson '05!) is worthless in hosting, but it's these types of conceptual things that make for good discussion and I'm riding the fence, so convince me.


Drew

then why go lease from SL? you can still buy your own server and colocate some where.

i personally prefer Raid system, purchased own hardware and colo.

Your 4-6 hour restore is dependent on the DC personnel jumping on this problem right away, reinstalling a clean OS, handing it off to you, and you being ready to restore. Reality is DC takes some time figuring out what failed, a minor issue comes up at the same time your drive fails, so add a few hours to your estimate. Of course the failure happened 6 days away from your full backup, so this causes loss of data even with daily incrementals, and restoration of files that should not exist, etc...

With RAID-1 and a good backup system, that one drive failure gives you time to schedule a replacement, resulting in minutes of downtime coupled with no data loss. You also get the added bonus of faster reads, thereby improving system performance. If you prevent the loss of 7 clients because you eliminated 6-10 hours of downtime (since you have less loyal penny pinching clients who are ready to host hop at the first sign of trouble), you come out ahead in the long run (assuming the drive fails).

But, for $10/month clients, every penny counts, so just having a good backup system (and luck) may be "good enough" for your situation. :)

- John C.
(who has learned to embrace SCSI RAID-1)

Honestly, RAID-1 should be a MINIMUM for a production server.

$10/month clients? what is your cost per server (plus overhead cost ie. support, etc) and how much account per server? I heard there somebody put about 1,000 account per server (depend on server load) So, if you put about 500 users per server, you can get $5k/month for 1 server. I think your overhead not exceed 2, 3k per server. :) I don't know. Just my stupid calculation. So, if this is your situation, I advice you to use the RAID-1

With RAID-1 and a good backup system, that one drive failure gives you time to schedule a replacement, resulting in minutes of downtime coupled with no data loss. You also get the added bonus of faster reads, thereby improving system performance.

Well, I don't think Raid1 improves read/write speeds. However RAID5 definately does.

I definately am a strong supporter of RAID too. We use both RAID1 and RAID5 SCSI disk arrays and they have definately protected us against some hard disk failures.

Also, a 30GB restore in 6 hours is optimistic. I'd double that estimate especially with the time needed to put new disks in, reinstall OS.. reinstal software, recreate resources then finally restore data

Honestly, RAID-1 should be a MINIMUM for a production server.

Couldn't agree more. We won't provision a server without at least *hardware* RAID 1.

Well, I don't think Raid1 improves read/write speeds. However RAID5 definately does....
Raid-1 actually has a little bit slower write speed due to writing to both disks each time, but on reads there is a small speed improvement being able to read from either disk. At least that's what my reading and empirical testing tell me. :)

- John C.

$10/month clients?

Yeah - just because $10 is a nice even number. I try to keep less than 200 sites on each server because they just run better that way. Plus, if somebody gets slashdotted there's a chance you may be able to stay up. And *that* is how you get loyal clients.

RAID1 has saved us from extended downtime several times. You can tell when a drive is failing generally before it actually fails. With RAID1, you can realistically do something about it without the customers experiencing more than a couple minutes downtime. As John said, downtime in the event of a hard crash is hugely dependant on how quickly the datacenter helps you. I'm not willing to be unneccessarily dependant on any datacenter.

Our average customer pays around $10 a month and RAID1 doesn't hurt our bottom line at all. I think the extra $$$ is justified...if nothing more than to save our admin the time and headache of having to deal with the datacenter in an emergency situation and restore accounts from backup, etc.

--Tina

Honestly, RAID-1 should be a MINIMUM for a production server.

Agree 100%. I wouldn't even dream of putting paying clients on a server without at least RAID-1.

If there's one thing you learn in the IT industry, it's make backups, then don't trust the backups. RAID-1 + a backup solution gives you two fail-safes. Both harddrives could fail simultaneously (or a drive and the raid array), and then you would only incur 6 hours worth of downtime restoring. Or a hard drive could fail, and the backups could be corrupt, and you'd still only have minutes of downtime swapping in the new drive and rebuilding the array. But in no circumstance would I put paying customers on a server without raid-1. For me it's considered standard operating cost.

When you look at it like that, the whole thread seems petty. Here's the rub: with good backups, you can restore a server to pretty much par condition in 4-6 hours MAX for, say, 30 GB worth of Web sites.

Assuming a drive fails once every 2 years, which is way more than in my experience, you've spent $60 x 24 = $1,440 to save 6 hours of downtime, which isn't likely to lose you many clients if you've had a good track record.

If you can assume failure once every 4 years, that's ~$3k for 6 hours of downtime. $500/hour.

My economics degree (Clemson '05!) is worthless in hosting, but it's these types of conceptual things that make for good discussion and I'm riding the fence, so convince me.


Drew

Another Clemson grad in the house ('93 i my case) :-)

Anyways...

It's also the reputation you are looking for. Are you looking to be reliable hosting provider or just low cost? If reliable, the RAID is well worth it. We just had 2 DRIVES fail on the same machine at the same time (that's not a typo) that were only 30 days old. If anything the MTBF is getting worse than better, at least from what we are seeing. Hard drives are the #1 item that fails, memory second (esp if not ECC), power supply a distant third.

All it takes is one dead server down for 8 hours to piss off a lot of customers who also tell their friends. Since we get many of our customers through referrals it's a no brainer for us. The cost for referals is priceless IMHO.

I'd get it just for redundancy, it's a disservice to your customers NOT to have reliable backups.

Raid 1 has saved my butt several times in the past 4 years. New servers are all raid 5, as I find even an hour of downtime to rebuild the raid1 array is unacceptable.

Raid5 means 0 downtime for hd failure (if they are hotswap drives with hotspare)

I'm encouraged that so many of you are also using RAID. I slept better after moving to RAID-5 as I mentioned at the onset of the thread and have decided to continue with RAID-1 on the new server. Unfortunately, I like the datacenter but RAID-5 is not an option. :-/

Raid is an importnt part of my services. But I much prefer raid10 or raid1 to raid5. I primarily only use raid5 on backup servers at this point.

I have had far to many issues with 5 or fewer drive raid5 arrays. They are redundant, and they offer less of a space loss than others, but raid5 writes are SO much slower it can be painfull. This is really noticable on write intensive apps like Databases.

Als, a good raid1 card will let you read differant data from each drive simultaniously (sp) and the writes, on a good controler, are faster as they are buffered in the controllers cach and can be writen in at more efficient times.

Raid-1 actually has a little bit slower write speed due to writing to both disks each time, but on reads there is a small speed improvement being able to read from either disk. At least that's what my reading and empirical testing tell me. :)

- John C.

Yes, that is the case, and as a web server generally reads much more than it writes, RAID1 generally offers a decent speed improvement.

vantage255, as for RAID5, we use RAID5 for our web servers and it works great, as there aren't that many writes. The speed is right on par with RAID10 for less money. Definitely don't go with RAID5 for DB though, which is why we use RAID10 for our database servers. Use the right tool for the right job.

I have had far to many issues with 5 or fewer drive raid5 arrays. They are redundant, and they offer less of a space loss than others, but raid5 writes are SO much slower it can be painfull. This is really noticable on write intensive apps like Databases.


We not experienced this issue. We are a big user of RAID5 on SCSI systems and are using high-end RAID cards as well. Write performance is on par with our expectations.

Reads are quite fast

raid5 reads are better than single disk. Much better. and the more disks in your array the better.

Writes, on an array of less that 6 disks are a good deal slower. This is mainly due to having to write 4 pieces of data simultaniously and parity calculations. (not going into raid theory here..)

On web servers you dont notice this so much, as there arent as many writes. But on DB servers and heavily used mail servers it makes a huge differance.

Yash, On the default Hsphere mail server (Really odd instalation of qmail) install there are a minimum of 5 writes to disk for each incomming mail. I have seen HUGE performance hits when running raid5. On the other hand, I am running a 12 drive raid10 on a mail server and am seeing a HUGE performance increase over the same 12 drives in raid5.

We are running 3 large Mail servers with a 73GBx3 RAID5 config on a high-end raid card. These are Dual Opteron servers and have 20 to 30K mail boxes each. The last I heard from my admin is that we were getting good read/write speeds of the array.

However since you mentioned it, I am going to check the difference in write performance between our RAID1 and RAID5 arrays.

Greetings:

We've been in business since June 1995, and since that time we've see RAID (we currently use hardware-based RAID-5) save us time and time again over the years.

On the other side, we've seen a number of our clients go through pain when a hard drive fails, and then they are migrating clients from one server to use that server for parts, and then migrating back when they get more parts.

So the bottom line, at least in over a decade of experience, is RAID (specifically hardware-based RAID) is worth every single penny.

Thank you.

RAID is worth it. In looking at the benefits vs the cost you have to remember that customers do not like downtime, that a restore from backups would result in customer changes that occured after the backup being lost, and that unhappy people tend to be very vocal. I wouldn't enter the hosting arena without first having reliable backups and the proper RAID solution for the job.

Why not spend the money to buy servers and co-locate them? The per month costs are significantly lower over the long term when buying the hardware.

If you have a problem, you have more control over how quickly faulty hardware is diagnosed and replaced. A 3Ware 2-drive RAID card is ~$150, the extra drive ~$100. You can also depreciate the hardware costs on your tax returns.

Its not always cheaper, and it depends on your luck with hardware and proximity to the DC you are using.

Its not always easy to get someone to work on a server that you own. If you are to far away from a decent DC to do the work yourself then it may take a long time to get a hardware failure fixed. But if it is a dedicated then the DC is doign the hardware maint.

Luck has nothing to do with it. If you buy high quality hardware, designed for servers, you should have few problems.

I believe the OP already stated that they have built out servers previously with RAID-5 for their own racks, so that's why I was asking.

I was simply stating that it is not always cheaper and that, depending on the situation, dedicated often makes more sense.

As for luck having nothing to do with it.... I have seen $25K HP servers die within 2 days of being brought up. Out of a recent shipment of Dell 2650s We had 3 DOA servers out of 15. Remember the IBM harddrive issues a few years ago..? They were considered high quality hardware until that point.

There is no way to tell when you are going to have hardware failure. It can happen at any time. Part of any good business is DR (Disaster Recovery) and part of any good DR plan is what to do in case of unexpected hardware failure.

Most hardware failures come within the first few days of operation. This is why you burn in hardware before you put customers on it.

The IBM drives were related to one model, the DeskStar 75GXP, IIRC. This model was a desktop IDE drive. Just about every manufacturer has released a bad batch of hardware. It's a fact of life in this business. However, if you have good backups, and spare parts on hand, the pain hardware failures can be minimized.

That is true. With good Backups and hardware on hand.. AKA good DR planning.
and the IBM issues we not only related to one drive. Thats just the one that was most prevelant. and YES every manufacturer mhas released bad production runs. but they do not always fail within the first few days. Sometimes it takes a few months.

The point being, one good, valid DR plan, is to go with dedicated hardware. Most reputable DCs that offer dedicated hardware have good techs on hand with the ability to fix your dedicated hardware in case of failure. Unexprected hardware failure can drive a small to mid sized hosting business out of business. Its always wise to plan for it.