I've read threads here and kind of shiver at the thought of one drive holding client data, but I guess I'm paranoid. Others here have a serious hate of SATA and prefer SCSI only, so at least I'm not the worst (or at least I'm not alone...)
I've got a new web hosting venture, and I'm interested to know how y'all handle backups. I've got every server running mirrored SCSI (most hardware, one software under Linux) and I've got a control panel applet to make backups easy for clients (both web/ftp data and MySQL databases), but I still feel a little exposed. I'm thinking I'd like to have a server that does nothing but rsync data from the other servers every few hours, with nightly scripted MySQL dumps to grab that data as well.
What do y'all do? What's too paranoid? And what do most find "good enough?"
Yea, our managed services provider backs up our servers nightly on external tapes (includes mysql dbs, etc)...i would suggest having your terms of service include details about having the end user backup their own data as well just to be safe.
Originally posted by dzeanah I've read threads here and kind of shiver at the thought of one drive holding client data, but I guess I'm paranoid.
You're not being paranoid at all. Drives crash all the time, and in certain cases, you can lose every single drive in a server at the same time. Mirrored drives are nice, but are no substitute for a proper backup.
On that note, you might designate one of your servers as a backup server, and use Rsync to perform nightly backups to it. This does not need to be a dedicated server, but could just be some extra space on an idle server.
On our shared hosting platforms we run the following configurations
3ware 9500s 256mb cache controller
Main Drive Array: 4 x 200GB Raid-5
Backup Array: 2 x 400GB Raid-0
Then on top of that backups are rscyned to a network backup server for secondary storage.
We find it's best to maintain in-box backups as well as off-box backups to ensure the quickest restore time possible in the event of a failure that requires a restore.
Imagine having to restore 400GB of data accross a 100mbps line?
even at a full 100mbps thats a 9.1hour restore time just to place the data back onto the machine, not even counting the time to actually restore accounts (cpanel)
If you have the capital, what I would do is setup a Windows 2000 server at home and attach a nice tape drive, SDLT or LTO 3 that can handle a lot of information. Then purchase ARCserve for Windows. Install it, then install the Linux client and you can do a backup over the network right to your external media. Other then that, disk to disk backup is not a bad solution. I have never in my 10 years ever see all the drives in one system go defunt at the same time. I come from 8 years of system development test at IBM. But, not trying to say it can not happen just I have not seen it making the % of all drives defunt is probally slim to none. But then again, if your datacenter floods and you are doing a disk to disk backup there is not much you can do to save that data if the server becomes damaged from flood or fire. Another option is to setup another server in a different datacenter and do resync or some type of disk to disk backup.
Carolina Hosting Services http://www.carolinahosting.net | @CarolinaHosting | Facebook.com/CarolinaHosting
VMware | cPanel | Linux | Softaculous | 16 Years Hosting Experience
Well, I've got an unused DLT sitting here, and a dual-proc P3 sitting idle. I could always reformat to Linux and Rsync to home I guess...
Re: multiple drive failures. I've twice seen multiple drive failures in a RAID-5 array. Both were at the same client site, and had been installed by the same idiot. The client had been told that RAID-5 negated the need for the tape backups they had been performing, and the SCSI drives in question were (according to the Seagate web site) designed for entry-level Mac workstations.
I'm probably being paranoid, but that isn't always a bad thing. For now I probably will install Linux locally and begin to RSYNC the data so I'm protected from site failures.
Of course, I will still likely rsync multiple times daily. What I've seen indicates there's little overhead, and the hack in the Linux Hacks book is just too cool.
Definitely not being paranoid. Any good host should have a backup plan, even if they say that they don't offer it. Most hosts DO stress that it's the clients responsibility in the end to make sure they have backups locally, it still a good idea to have a backup system in place for those special occassions like when a drive goes belly up or rogue software decides to make a mess of the filesystem.
Worst case I ever seen was having teh filesystem go belly up DURING the backups and not get caught till it was too late and half the backups became unusable..*shudder*. (another good reason for clients to ensure they have local copies....Murphy's Law can strike anytime).
I presume you are doing "normal" hosting (hosting for a major company or big e-commerce would require more elaborate solutions, many have already been discussed).
We backup all sites nightly, our CP software makes it easy for a client to create and download their own if they want to.
We also offer clients who want to pay for it off-site backups of their sites on whatever schedule they are willing to pay for and for the most paranoid both the above and periodic backup to CD or DVD stored in a fire-data safe at yet another location. (possible but very unlike all those would get wiped off the map at once, if it did we would have bigger things to worry about than web sites).
Basically we offer what the customer is willing to pay for. The majority of our folks are happy with standard backups and whatever they may do themselves. If they want to pay more they get more protection.
BUT if we had every site on mirrored clusters with full off-site daily backups (probably the most secure) the cost would be far more than the average customer is willing to spend. It's all the market, I'm sure e-bay or Amazon have quite amazing back-up procedures. But the average small business or customer not dealing with big e-commerce or lots of dynamic data doesn't want to pay for that type protection.
I agree with the above though. We make it VERY clear what we do, what we offer as extras, and what the customer can do themselves so there is a clear understanding. That solves most problems, just making sure everyone is "on the same page".
New Idea HostingNO Overselling-Business-Grade, Shared Only! New-In House Design Team. High Speed & Uptime; , DIY Pro-Site Builder-Daily Backups-Custom Plans, All Dual Xeon Quad Intel servers w/ ECC DDR3 RAM SCSI RAID minimums. We Concentrate on Shared Hosting ...doing one thing and doing it VERY well
At the moment, I use RAID-10 SATA configuration for the main drives, with a 230 GB backup IDE drive for dailies. I rsync weeklies off once a week to another server of mine in a different datacenter (that doesn't host websites). Weekly instead of daily mainly just because of the data transfer it uses.
However, I currently don't guarantee backups to my clients. I tell them to make backups of their own sites to be safe.
As far as drives failing, I haven't experienced multiple drive failures at once, but I have experienced RAID problems from faulty RAID controllers that can cause you to corrupt partitions or even lose entire RAID containers.
Last edited by BigGorilla; 10-17-2005 at 03:23 PM.
I personally have a raid controller setup on my main drive for the first step in safety. Then for the second step I have a backup drive that makes backups nightly through cpanel. Then for extra safety, I have a script copy those cpanel backups every morning to an offsite ftp server. I also have a script that puts everything in /var/lib/mysql (all database files) into a tar and dumps it onto the backup offsite server every night just in case the datacenter catches on fire in the middle of the night. I also have everything copied weekly to a personal computer I have in my office with 4X250GB hard drives in it specifically for backups. Also, of course, I stress like no tomorrow that clients should make their own backups and store them on their PC as often as possible.
This seems a bit extreme to some people, but the old saying says it best, "better safe than sorry".
• Imeanwebhosting.com - Shared cpanel hosting, 99.9% uptime. • 10 min average ticket responses, softaculous, rvsitebuilder, and more! • Reliable, affordable shared hosting. I Mean Web Hosting!
I would want to tar, and gzip it up first like Cpanel does then store it locally and or on Rsync/FTP backup like at gnax. Daily backup are the way to go. If you are really paranoid you can go with Raid 10 configurations which can take a loss of multiple drives. I have seen a few instances where more than one drive has crashed at the same time. I normally go with RAID 5, with a hot standby, + you can have a large IDE drive in there just for the backup also.
The big thing to do is to have a plan written down for backup recovery, and as a webhost explain, to some degree what and how often you backup. With some legal info there, that you are not liable and so on and so forth.