hosted by liquidweb


Go Back   Web Hosting Talk : Web Hosting Main Forums : Web Hosting Talk Tutorials : Hosting Security and Technology Tutorials : Tutorial: simple backup rotation scripts
Reply

Forum Jump

Tutorial: simple backup rotation scripts

Reply Post New Thread In Hosting Security and Technology Tutorials Subscription
 
Send news tip View All Posts Thread Tools Search this Thread Display Modes
  #1  
Old 08-14-2005, 07:26 AM
Orc Webhosting Orc Webhosting is offline
Devoted Supporter
 
Join Date: Aug 2004
Location: Zurich, Switzerland
Posts: 762

Tutorial: simple backup rotation scripts


I received several requests to post my backup scripts. Before I start, I think some background info is needed (i.e. what their purpose is): I've written them as part of the "Enterprise Class Backup" system we offer for webhosting accounts. Following rotation is used:

1) Full account backups:
a) on-server (dedicated hard drive used only for storing backups)
- daily backups, 7 days rotation
- weekly backups, 4 weeks rotation
- the standard daily/weekly/monthly cPanel backups are also available on this drive (1 day/1 week/1 month rotation)
b) off-server (another server in the same datacenter)
- weekly backups, 12 weeks rotation
c) off-site (stored on DVDs in a Swiss bank vault)
- monthly backups, 24 months rotation


2) MySQL database backups:
a) on-server (dedicated hard drive used only for storing backups)
- hourly backups, 24 hours rotation
b) off-server (another server in the same datacenter)
- daily backups, 7 days rotation
c) off-site
- weekly backups, stored on the monthly DVDs


Step by step guide

First of all, I'll describe the environment that's needed to run these scripts 1:1. This is a cPanel server running GNU/Linux, and while cPanel is not neccessary (you could use any other backups instead of the daily cPanel account backups, just modify the scripts), I can't say on what non-GNU/Linux OSes it would work without changes. Since we're talking about simple shell commands, they might work on other *Nix OSes like *BSD or Solaris, but I haven't tested it. You're welcome to post your findings.


a) on-server backups

You'll need a certain directory structure so the scripts work without modifications. The backup hard drive is mounted as /backup and has the following directories:

/backup/cpbackup : here are the cPanel backups. The three subdirs will be created automatically by cPanel/WHM.

/backup/rotation: this is where my scripts write their output. You need to create following subdirs: hourly, daily, weekly, monthly. You don't need to create any directories under these subdirs, the scripts will take care of that.


Let's start with the hourly rotation for MySQL backups. These aren't compressed because their primary purpose is if a client suddenly calls you in tears saying they just destroyed/corrupted/whatever their big and important DB 5 minutes ago, you can just copy back their files from the backup location to the /var/lib/mysql subdir and they'll be happy. This is my script in /etc/cron.hourly (file permissions 7xx):

Code:
#!/bin/sh
rm -rf /backup/rotation/hourly/$(date +"%H")
mkdir /backup/rotation/hourly/$(date +"%H")
cp -R /var/lib/mysql/* /backup/rotation/hourly/$(date +"%H")

The daily rotation runs a couple of hours after the cPanel nightly backup and copies the latest account backups into the directory that corresponds to the current day and keeps them for 7 days before overwriting them:

Code:
#!/bin/sh
rm -rf /backup/rotation/daily/$(date +"%u")
mkdir /backup/rotation/daily/$(date +"%u")
cp -R /backup/cpbackup/daily/* /backup/rotation/daily/$(date +"%u")
It's in /etc/cron.daily of course and needs to be owner executable (7xx).


The weekly rotation script does the same. Note that this one doesn't have an automatic delete mechanism (well a 52-week one), because I prefer to monitor available disc space and as long as there are sufficient reserves I keep the files, longer than advertised (after all, advertising and SLA is in my eyes the minimum, more is always OK). The script is located in /etc/cron.weekly and needs to be chmoded to 7xx.

Code:
#!/bin/sh
rm -rf /backup/rotation/weekly/$(date +"%V")
mkdir /backup/rotation/weekly/$(date +"%V")
cp -R /backup/cpbackup/daily/* /backup/rotation/weekly/$(date +"%V")

The monthly script adds some new twists for still another layer of security: there have been reports again and again that cPanel/WHM corrupted a certain account's backup and all backup files were useless. In this case, the following (mind you, storage space wasting) addition makes sure that in this worst case you still have the majority of data for the customer. While it needs you to setup the account from scratch and copy lots of things into it and still do some of the steps manually (like setting up the MySQL DBs and users in cPanel), at least all website data, MySQL DBs, e-mails stored on the server, webstat outputs etc is still available. The script goes into /etc/cron.monthly and has a 7xx permission.

Code:
#!/bin/sh
rm -rf /backup/rotation/monthly/$(date +"%m")
mkdir /backup/rotation/monthly/$(date +"%m")
cp -R /backup/cpbackup/daily /backup/rotation/monthly/$(date +"%m")
cp -R /var/lib/mysql /backup/rotation/monthly/$(date +"%m")
cp -R /home /backup/rotation/monthly/$(date +"%m")


b) off-server backups

These are stored on an external backup server to which I have only FTP access (thus no fancy stuff). The first is the MySQL daily backup, located in /etc/cron.daily (7xx as always):

Code:
#!/bin/sh
cd /backup/rotation
tar -zcf mysql_$(date +"%F").tar.gz /var/lib/mysql/*
ftp -in <<EOF
open 127.0.0.1
user yourusername yourpassword
bin
hash
prompt
put mysql_$(date +"%F").tar.gz
bye
rm -f mysql_$(date +"%F").tar.gz
You need to modify three variables: IP address (or hostname) where you connect to, username and password for the FTP access. I have to admit that the last line doesn't work (not yet sure why), the files are put but then they aren't deleted from the local place. Since these are relatively small and I've got ample storage space, I didn't look into this further (yet).


Next is the weekly FTP backup. This is a rather crude job, taking lots of CPU horsepower if you've got lots of accounts. It creates a giant .tar.gz of everything in the daily cPanel backup dir and pushes this on to the FTP backup server. Since by now we're speaking about disaster recovery backups (which you should not need in a normal case), I prefer having them in one big file instead of many small ones (if you prefer hundreds of small files every week in your FTP space, you can always use the mput command):

Code:
#!/bin/sh
cd /backup/rotation
tar -zcf cpbackup_$(date +"%F").tar.gz /backup/cpbackup/daily/*
ftp -in <<EOF
open 127.0.0.1
user yourusername yourpassword
bin
hash
prompt
put cpbackup_$(date +"%F").tar.gz
bye
rm -f cpbackup_$(date +"%F").tar.gz
Same comments apply as to the daily job.


There is no monthly FTP job because in its place is the monthly off-site backup job.



c) off-site backups

The off-site part is done manually, I first copy the contents of /backup/cpbackup/daily to a directory with FTP access, then use an FTP client to download the files to a local machine and shovel them on a DVD. For the weekly MySQL part, I prefer to use the most recent state, so I create a .tar.gz directly into an FTP-accessible directory and download it.



Notes

* You shouldn't even try to set up this system on an overloaded or low-end server.
* You cannot sell even remotely all the space on your primary hard drive (unless your backup drive is several times larger).
* I am aware that doing full backups every time is the most space-wasting kind of backup, but I prefer it over more sophisticated methods because with a differential or incremental backup system once you lose the starting point, the rest is so much dead bytes. With full backups, any single one stands on its own.
* Offering such a sophisticated solution on a per client basis (like a paid addon) needs only small modifications: all you need is to specify the account backup files and the subdirs in /var/lib/mysql that are to be part of the process. You could e.g. solve it by creating a dir where you first copy over the needed files from the original places, and then run the backup scripts on that directory.
* Any questions/feedback is welcome.

__________________
Orc Webhosting: professional support with a personal touch since 2002
Premium hosting and feature rich reseller accounts at www.orcwebhosting.com
cPanel, WHM, Fantastico, SohoLaunch, RVskin; 64bit GNU/Linux
Intel Core i5 servers with 8 GB RAM and Hardware RAID 1, redundant Tier 1 connectivity



Sponsored Links
  #2  
Old 08-14-2005, 07:30 AM
PTNHosting PTNHosting is offline
Aspiring Evangelist
 
Join Date: Jan 2005
Posts: 361
Very good.

Thanks for sharing.

__________________
Portugal Networks
Shared and Reseller cPanel Accounts. NEW! Windows 2003 with Plesk, ASP.NET 2.0 and MSSQL 2005.

  #3  
Old 08-14-2005, 07:41 AM
Dacsoft Dacsoft is offline
Web Hosting Master
 
Join Date: May 2003
Location: Florida
Posts: 877
RambOrc,
Great instructions. Thank you for providing this.

Sponsored Links
  #4  
Old 08-14-2005, 10:13 AM
activelobby4u activelobby4u is offline
Your support partner
 
Join Date: Apr 2005
Location: Queen of Arabian Sea
Posts: 2,408
hey good work man

  #5  
Old 08-18-2005, 10:04 PM
IMeanWebHosting IMeanWebHosting is offline
Web Hosting Master
 
Join Date: Sep 2004
Location: Fairborn, ohio
Posts: 916
Would anyone care to walk a newbie through this? Do I just stick those lines of code in a text file, and rename it cron.daily and whatnot? Yeah, I'm an idiot

__________________
Imeanwebhosting.com - Shared cpanel hosting, 99.9% uptime.
10 min average ticket responses, softaculous, rvsitebuilder, and more!
Reliable, affordable shared hosting. I Mean Web Hosting!


  #6  
Old 08-18-2005, 10:26 PM
Dacsoft Dacsoft is offline
Web Hosting Master
 
Join Date: May 2003
Location: Florida
Posts: 877
Quote:
Originally posted by Bohica
Would anyone care to walk a newbie through this? Do I just stick those lines of code in a text file, and rename it cron.daily and whatnot? Yeah, I'm an idiot
Basically you put the lines of code in a text file, change the permissions on the file, and put the file into the related directory. As an example, for the HOURLY backup:

1 - go to the /etc/cron.hourly directory
-- cd /etc/cron.hourly

2- create a file called hourlySql.sh and put this code in it:
-- use the editor of choice (like vi)
Quote:
#!/bin/sh
rm -rf /backup/rotation/hourly/$(date +"%H")
mkdir /backup/rotation/hourly/$(date +"%H")
cp -R /var/lib/mysql/* /backup/rotation/hourly/$(date +"%H")
3 - change the permissions to 7xx
-- chmod 700 hourlySql.sh

4 - create a directory for the backup files. This script is coded to use:
/backup/rotation/hourly

You can create this by using:
mkdir /backup/rotation/hourly

Now do the same for the other scripts.

Hope I didn't miss a big step. Good luck.

  #7  
Old 08-19-2005, 12:58 PM
IMeanWebHosting IMeanWebHosting is offline
Web Hosting Master
 
Join Date: Sep 2004
Location: Fairborn, ohio
Posts: 916
Can I name the files anything I want as long as they're in the right folder? Like would thisismymysqlcron.sh work? (dumb example, but you get what I'm asking).

__________________
Imeanwebhosting.com - Shared cpanel hosting, 99.9% uptime.
10 min average ticket responses, softaculous, rvsitebuilder, and more!
Reliable, affordable shared hosting. I Mean Web Hosting!


  #8  
Old 08-19-2005, 01:08 PM
Orc Webhosting Orc Webhosting is offline
Devoted Supporter
 
Join Date: Aug 2004
Location: Zurich, Switzerland
Posts: 762
We aren't talking about something dumb like Winblows, so you can name the files anything you want to and don't even need to give them a file extension (mines don't have any, I usually don't use file extensions on GNU/Linux unless it's an Apache handler).

Actually, you don't even have to put them into /etc/cron.daily and the like, you can actually put them anywhere you want. The comfortable thing about the predefined crondirs on RedHat is only that you don't need to specify every command/script in the crontab manually, they'll automatically run at predefined times.

__________________
Orc Webhosting: professional support with a personal touch since 2002
Premium hosting and feature rich reseller accounts at www.orcwebhosting.com
cPanel, WHM, Fantastico, SohoLaunch, RVskin; 64bit GNU/Linux
Intel Core i5 servers with 8 GB RAM and Hardware RAID 1, redundant Tier 1 connectivity

  #9  
Old 08-19-2005, 01:35 PM
IMeanWebHosting IMeanWebHosting is offline
Web Hosting Master
 
Join Date: Sep 2004
Location: Fairborn, ohio
Posts: 916
That's pretty sweet! One more "newb" question.

The one that tars everything in daily and throws it in my ftp backup, will it overwrite the old file every time? I mean, I would like it to only be 1 rotation because my ftp backup is only 50GB. Just curious, thanks!

__________________
Imeanwebhosting.com - Shared cpanel hosting, 99.9% uptime.
10 min average ticket responses, softaculous, rvsitebuilder, and more!
Reliable, affordable shared hosting. I Mean Web Hosting!


  #10  
Old 08-19-2005, 01:47 PM
Orc Webhosting Orc Webhosting is offline
Devoted Supporter
 
Join Date: Aug 2004
Location: Zurich, Switzerland
Posts: 762
Since the filename includes the date, of course it won't, but if you want it to overwrite the previous file every day, that's easy, just give the file a static name, leaving away the whole dynamic "date" part, i.e. instead of mysql_$(date +"%F").tar.gz you can just name it something like mysql_daily.tar.gz. Make sure you change all 3 instances in the script.

__________________
Orc Webhosting: professional support with a personal touch since 2002
Premium hosting and feature rich reseller accounts at www.orcwebhosting.com
cPanel, WHM, Fantastico, SohoLaunch, RVskin; 64bit GNU/Linux
Intel Core i5 servers with 8 GB RAM and Hardware RAID 1, redundant Tier 1 connectivity

  #11  
Old 08-19-2005, 02:20 PM
IMeanWebHosting IMeanWebHosting is offline
Web Hosting Master
 
Join Date: Sep 2004
Location: Fairborn, ohio
Posts: 916
Excellent. Sorry for being a newb.

Thanks RambOrc!

__________________
Imeanwebhosting.com - Shared cpanel hosting, 99.9% uptime.
10 min average ticket responses, softaculous, rvsitebuilder, and more!
Reliable, affordable shared hosting. I Mean Web Hosting!


  #12  
Old 01-23-2006, 02:45 PM
Johnburk Johnburk is offline
Web Hosting Master
 
Join Date: Nov 2004
Posts: 673
So this scripts automaticly would make a full backup in cpanel?

__________________
Just imagine, if the world was like an online community. Where people help each other, just because they can.

  #13  
Old 01-24-2006, 06:32 AM
Orc Webhosting Orc Webhosting is offline
Devoted Supporter
 
Join Date: Aug 2004
Location: Zurich, Switzerland
Posts: 762
Nope, what it does is copies the cPanel-created full backup files to other locations so that they aren't overwritten next time backup runs.



BTW I've modified the last two scripts in the meantime so they now delete the local backup files as they are supposed to:

Code:
#!/bin/sh
cd /backup/rotation
rm -f mysql_*.tar.gz
cd /var/lib
tar -zcf /backup/rotation/mysql_$(date +"%F").tar.gz mysql
cd /backup/rotation
ftp -in <<EOF
open 127.0.0.1
user yourusername yourpassword
bin
hash
prompt
put mysql_$(date +"%F").tar.gz
bye
Code:
#!/bin/sh
cd /backup/rotation
rm -f cpbackup_*.tar.gz
cd /backup/cpbackup
tar -zcf /backup/rotation/cpbackup_$(date +"%F").tar.gz daily
cd /backup/rotation
ftp -in <<EOF
open 127.0.0.1
user yourusername yourpassword
bin
hash
prompt
put cpbackup_$(date +"%F").tar.gz
bye

__________________
Orc Webhosting: professional support with a personal touch since 2002
Premium hosting and feature rich reseller accounts at www.orcwebhosting.com
cPanel, WHM, Fantastico, SohoLaunch, RVskin; 64bit GNU/Linux
Intel Core i5 servers with 8 GB RAM and Hardware RAID 1, redundant Tier 1 connectivity

  #14  
Old 01-12-2007, 09:26 AM
edkay edkay is offline
New Member
 
Join Date: Mar 2006
Posts: 1
A minor modification

These are great, simple backup scripts. Thanks for sharing.

I've made a simple modification in my implmentation however, by setting a variable to the date part instead of calling date each time.

e.g.

Code:
#!/bin/sh
rm -rf /backup/rotation/monthly/$(date +"%m") mkdir /backup/rotation/monthly/$(date +"%m") cp -R /backup/cpbackup/daily /backup/rotation/monthly/$(date +"%m") cp -R /var/lib/mysql /backup/rotation/monthly/$(date +"%m") cp -R /home /backup/rotation/monthly/$(date +"%m")
becomes

Code:
#!/bin/sh
month=`date +"%m"` rm -rf /backup/rotation/monthly/$month mkdir /backup/rotation/monthly/$month cp -R /backup/cpbackup/daily /backup/rotation/monthly/$month cp -R /var/lib/mysql /backup/rotation/monthly/$month cp -R /home /backup/rotation/monthly/$month
This will prevent any problems if the month/day/hour/whatever changes whilst the script is executing.

  #15  
Old 01-12-2007, 09:48 AM
JorgeBR JorgeBR is offline
Newbie
 
Join Date: Jan 2007
Posts: 10
Great Work man, thanks a lot

Reply

Related posts from TheWhir.com
Title Type Date Posted
Symantec to Discontinue Backup Exec.Cloud; Customers Presented with Next-Step Options Web Hosting News 2013-12-03 14:49:46
Dimension Data to Launch Three-Tier Cloud Backup Add-On Service Web Hosting News 2013-08-06 15:55:24
DemoWolf Changes Web Hosting Video Tutorials Pricing Structure Web Hosting News 2012-11-09 15:00:08
New BackupAgent Version Simplifies Channel Delivery of Cloud Backup Solution Web Hosting News 2012-06-26 16:57:11
CloudBerry Launches Version 2.9 of Cloud Backup for Amazon S3, Azure Web Hosting News 2012-06-25 16:38:04


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes
Postbit Selector

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump
Login:
Log in with your username and password
Username:
Password:



Forgot Password?
Advertisement:
Web Hosting News:



 

X

Welcome to WebHostingTalk.com

Create your username to jump into the discussion!

WebHostingTalk.com is the largest, most influentual web hosting community on the Internet. Join us by filling in the form below.


(4 digit year)

Already a member?