Results 1 to 25 of 41
Thread: incremental rsync
-
01-17-2008, 09:55 PM #1Web Hosting Master
- Join Date
- Apr 2005
- Posts
- 697
incremental rsync
Hi guys,
I have two folders:
/misc
/misc-backup
how do I go about making /misc-backup a mirror of /misc correctly?
-
01-17-2008, 10:57 PM #2Aspiring Evangelist
- Join Date
- Mar 2005
- Posts
- 402
-
01-17-2008, 11:02 PM #3Web Hosting Master
- Join Date
- Apr 2005
- Posts
- 697
right, but does it run in the background? what if /misc a file gets overwrite on /misc which is a new file, would it update /misc-backup? also, how can we make this auto start when the server gets rebooted.
-
01-17-2008, 11:05 PM #4Aspiring Evangelist
- Join Date
- Mar 2005
- Posts
- 402
Since you're using it to make a backup of a directory, I'm guessing you're going to just be making a cron out of it.
The rsync command would delete any files that are not there anymore, and replace any files that have changed.
As to your auto start when server gets rebooted.. Again I'm assuming you'd put this as a cron.
-
01-18-2008, 12:57 AM #5Retired Moderator
- Join Date
- Feb 2005
- Location
- Australia
- Posts
- 5,849
Bear in mind though that mirrors don't make good backups. What if you accidentally delete a file in /misc... Oops, it's gone from misc-backup too. And are the two directories on the same drive? Oops, a hard-drive failure just destroyed them both. Or even if they're on two different drives, what if a kiddie gets into your server and deletes everything? What was that rm -rf command again?
With a little extra setup rsync is great for remote backups and keeping multiple snapshots, which will protect you from all of these problems. Check out rsnapshot or Mike Rubel's tutorial for more info.Chris
"Some problems are so complex that you have to be highly intelligent and well informed just to be undecided about them." - Laurence J. Peter
-
01-18-2008, 01:15 AM #6Web Hosting Evangelist
- Join Date
- Apr 2003
- Location
- Melbourne, AU
- Posts
- 539
ti_nhatrang, what situations do you expect your backups (done your way) to cover?
WK Woon
CTO | http://www.aflexi.net - A flexible Network
Building the next generation CDN platform - DEMO .... coming soon
-
01-18-2008, 02:12 AM #7Web Hosting Master
- Join Date
- Apr 2005
- Posts
- 697
One folder is mounted so that's fine. I'm just looking for things to be mirrored so I can have the 2nd server bring up the files directly rather than going off the mounted folder.
-
01-18-2008, 01:46 PM #8Newbie
- Join Date
- Jun 2006
- Location
- Tucson
- Posts
- 23
to run in the background, place a nohup in front of the command and & at the end, that way you can log off and it'll still run:
nohup rsync -av --delete /misc/ /misc-backup/ &
-
01-18-2008, 06:11 PM #9Web Hosting Master
- Join Date
- Apr 2005
- Posts
- 697
I try doing that in cron but it doesn't start, can you please help me figure this out on how to make this run in /etc/crontab?
Thank you.
-
01-18-2008, 06:57 PM #10Newbie
- Join Date
- Jun 2006
- Location
- Tucson
- Posts
- 23
You may have to specify the full path to the commands in the script or cron, usually located under:
/usr/local/bin/rsync
or
/usr/bin/rsync
you can find the location by keying in:
which rsync
and it'll display the path.
You may also have to restart crond. The output of cron is normally sent somewhere like: /var/spool/mail/root (or whichever user you are executing the cron as)
-
01-18-2008, 07:00 PM #11Web Hosting Master
- Join Date
- Apr 2005
- Posts
- 697
/usr/bin/rsync nohup rsync -av --delete /misc/ /misc-backup/ &
That's what I have in crontab but when I restart the server, it doesn't seem to run.
-
01-18-2008, 07:09 PM #12Web Hosting Master
- Join Date
- Jan 2006
- Location
- Athens, Greece
- Posts
- 1,481
*/10 * * * * /usr/bin/rsync -av /misc/ /misc-backup/ > /dev/null 2>&1
Every 10 minutes.
-
01-18-2008, 08:16 PM #13Newbie
- Join Date
- Mar 2007
- Location
- Dublin
- Posts
- 17
Use something like rsnapshot or rdiff-backup to make your life a little easier.
Want me for contracting? Please read
http://www.brassy.net/contracting/
-
01-19-2008, 12:25 AM #14Web Hosting Master
- Join Date
- Apr 2005
- Posts
- 697
-
01-19-2008, 12:27 AM #15Web Hosting Master
- Join Date
- Apr 2005
- Posts
- 697
-
01-19-2008, 02:53 AM #16Web Hosting Master
- Join Date
- Aug 2003
- Location
- Gods Own Country
- Posts
- 892
Hope this link will be of some help
http://gentoo-wiki.com/HOWTO_BackupBlessen Cherian
Follow me on twitter.com/blessenonly
Two decade in Web Hosting Industry
-
01-19-2008, 10:41 PM #17Web Hosting Master
- Join Date
- Apr 2005
- Posts
- 697
Also, im rsync'ing 1TB of DATA, is this a good way to do it with this amount in size?
-
01-19-2008, 11:21 PM #18Web Hosting Master
- Join Date
- Apr 2005
- Posts
- 697
Also, nohup is creating a logs which is getting pretty huge, how can we not have this log created?
-
01-20-2008, 03:02 AM #19Retired Moderator
- Join Date
- Feb 2005
- Location
- Australia
- Posts
- 5,849
Leave off the v flag (verbose) ie. rsync -a source destination.
But you should be running it as a cron job.Chris
"Some problems are so complex that you have to be highly intelligent and well informed just to be undecided about them." - Laurence J. Peter
-
01-20-2008, 07:11 AM #20Newbie
- Join Date
- Oct 2006
- Posts
- 9
http://www.cis.upenn.edu/~bcpierce/unison/
Should be already on your system, command called unison.
Good luck,
Pedram M.
-
01-20-2008, 07:58 AM #21Web Hosting Master
- Join Date
- Apr 2005
- Posts
- 697
wow, so many tools out there... But my main question and hopefully to get a good recommendation on which tool I should use to sync 700GB-1TB of data?
-
01-20-2008, 08:54 AM #22Web Hosting Master
- Join Date
- Nov 2004
- Location
- Australia
- Posts
- 1,737
rsync is fine with that. It does block checksums on large files and only transfers the parts of the files that have changed. However it does need to read and checksum the entire file to do that.
Of course, optimally, you should have your 1Tb of data spread across many files. And yes, rsync will work beautifully, and optimally for that.
I'd live with the logfiles getting large (but do remove -v) as you'll need them for debugging in the future when something goes wrong.
-
01-20-2008, 09:20 AM #23Web Hosting Master
- Join Date
- Apr 2005
- Posts
- 697
I had to reformat my server because it through off my volggroup0... I'm trying again without the logs or -v, just the -a by itself.
-
01-23-2008, 06:15 AM #24Web Hosting Master
- Join Date
- Apr 2005
- Posts
- 697
-
01-23-2008, 06:49 AM #25Junior Guru
- Join Date
- Aug 2006
- Location
- Duluth, GA
- Posts
- 201
less /var/log/cron
fred b.
www.broce.net