Page 1 of 2 12 LastLast
Results 1 to 40 of 41
  1. #1
    Join Date
    Apr 2005
    Posts
    694

    incremental rsync

    Hi guys,

    I have two folders:

    /misc
    /misc-backup

    how do I go about making /misc-backup a mirror of /misc correctly?

  2. #2
    You could do it like this:

    Code:
    rsync -av --delete /misc/ /misc-backup/
    || Dennis Liang,
    || ServaxNet LLC

  3. #3
    Join Date
    Apr 2005
    Posts
    694
    right, but does it run in the background? what if /misc a file gets overwrite on /misc which is a new file, would it update /misc-backup? also, how can we make this auto start when the server gets rebooted.

  4. #4
    Since you're using it to make a backup of a directory, I'm guessing you're going to just be making a cron out of it.

    The rsync command would delete any files that are not there anymore, and replace any files that have changed.

    As to your auto start when server gets rebooted.. Again I'm assuming you'd put this as a cron.
    || Dennis Liang,
    || ServaxNet LLC

  5. #5
    Join Date
    Feb 2005
    Location
    Australia
    Posts
    5,842
    Bear in mind though that mirrors don't make good backups. What if you accidentally delete a file in /misc... Oops, it's gone from misc-backup too. And are the two directories on the same drive? Oops, a hard-drive failure just destroyed them both. Or even if they're on two different drives, what if a kiddie gets into your server and deletes everything? What was that rm -rf command again?

    With a little extra setup rsync is great for remote backups and keeping multiple snapshots, which will protect you from all of these problems. Check out rsnapshot or Mike Rubel's tutorial for more info.
    Chris

    "Some problems are so complex that you have to be highly intelligent and well informed just to be undecided about them." - Laurence J. Peter

  6. #6
    Join Date
    Apr 2003
    Location
    Melbourne, AU
    Posts
    539
    ti_nhatrang, what situations do you expect your backups (done your way) to cover?
    WK Woon
    CTO | http://www.aflexi.net - A flexible Network
    Building the next generation CDN platform - DEMO .... coming soon

  7. #7
    Join Date
    Apr 2005
    Posts
    694
    One folder is mounted so that's fine. I'm just looking for things to be mirrored so I can have the 2nd server bring up the files directly rather than going off the mounted folder.

  8. #8
    Join Date
    Jun 2006
    Location
    Tucson
    Posts
    23
    to run in the background, place a nohup in front of the command and & at the end, that way you can log off and it'll still run:

    nohup rsync -av --delete /misc/ /misc-backup/ &

  9. #9
    Join Date
    Apr 2005
    Posts
    694
    I try doing that in cron but it doesn't start, can you please help me figure this out on how to make this run in /etc/crontab?

    Thank you.

  10. #10
    Join Date
    Jun 2006
    Location
    Tucson
    Posts
    23
    You may have to specify the full path to the commands in the script or cron, usually located under:

    /usr/local/bin/rsync
    or
    /usr/bin/rsync

    you can find the location by keying in:

    which rsync

    and it'll display the path.

    You may also have to restart crond. The output of cron is normally sent somewhere like: /var/spool/mail/root (or whichever user you are executing the cron as)

  11. #11
    Join Date
    Apr 2005
    Posts
    694
    /usr/bin/rsync nohup rsync -av --delete /misc/ /misc-backup/ &

    That's what I have in crontab but when I restart the server, it doesn't seem to run.

  12. #12
    Join Date
    Jan 2006
    Location
    Athens, Greece
    Posts
    1,479
    */10 * * * * /usr/bin/rsync -av /misc/ /misc-backup/ > /dev/null 2>&1

    Every 10 minutes.

  13. #13
    Join Date
    Mar 2007
    Location
    Dublin
    Posts
    17
    Use something like rsnapshot or rdiff-backup to make your life a little easier.
    Want me for contracting? Please read

    http://www.brassy.net/contracting/

  14. #14
    Join Date
    Apr 2005
    Posts
    694
    Quote Originally Posted by Steve_Arm View Post
    */10 * * * * /usr/bin/rsync -av /misc/ /misc-backup/ > /dev/null 2>&1

    Every 10 minutes.
    Hmm, Wow, excuse my newbie thinking but I didn't know you actually have to run it multiple times. I thought it does it automatically and run in the background as a process. I thought it sits there and wait to fetch new/delete files.

  15. #15
    Join Date
    Apr 2005
    Posts
    694
    Quote Originally Posted by taec View Post
    Use something like rsnapshot or rdiff-backup to make your life a little easier.
    I would love to make my life a little easier. Can you please tell me how those will change?

    Bottom line is I want both folders to have the identical files/folders at all times.

  16. #16
    Join Date
    Aug 2003
    Location
    Gods Own Country
    Posts
    869
    Hope this link will be of some help
    http://gentoo-wiki.com/HOWTO_Backup
    Blessen Cherian
    Follow me on twitter.com/blessenonly
    Over a decade plus in the Hosting Industry

  17. #17
    Join Date
    Apr 2005
    Posts
    694
    Also, im rsync'ing 1TB of DATA, is this a good way to do it with this amount in size?

  18. #18
    Join Date
    Apr 2005
    Posts
    694
    Also, nohup is creating a logs which is getting pretty huge, how can we not have this log created?

  19. #19
    Join Date
    Feb 2005
    Location
    Australia
    Posts
    5,842
    Leave off the v flag (verbose) ie. rsync -a source destination.

    But you should be running it as a cron job.
    Chris

    "Some problems are so complex that you have to be highly intelligent and well informed just to be undecided about them." - Laurence J. Peter

  20. #20
    http://www.cis.upenn.edu/~bcpierce/unison/

    Should be already on your system, command called unison.

    Good luck,
    Pedram M.

  21. #21
    Join Date
    Apr 2005
    Posts
    694
    wow, so many tools out there... But my main question and hopefully to get a good recommendation on which tool I should use to sync 700GB-1TB of data?

  22. #22
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,683
    rsync is fine with that. It does block checksums on large files and only transfers the parts of the files that have changed. However it does need to read and checksum the entire file to do that.

    Of course, optimally, you should have your 1Tb of data spread across many files. And yes, rsync will work beautifully, and optimally for that.

    I'd live with the logfiles getting large (but do remove -v) as you'll need them for debugging in the future when something goes wrong.

  23. #23
    Join Date
    Apr 2005
    Posts
    694
    I had to reformat my server because it through off my volggroup0... I'm trying again without the logs or -v, just the -a by itself.

  24. #24
    Join Date
    Apr 2005
    Posts
    694
    Quote Originally Posted by Steve_Arm View Post
    */10 * * * * /usr/bin/rsync -av /misc/ /misc-backup/ > /dev/null 2>&1

    Every 10 minutes.
    I don't think this is working for me. How do I check to see if the cron ran or not?

  25. #25
    Join Date
    Aug 2006
    Location
    Duluth, GA
    Posts
    201
    less /var/log/cron

  26. #26
    Join Date
    Apr 2005
    Posts
    694
    Quote Originally Posted by Steve_Arm View Post
    */10 * * * * /usr/bin/rsync -av /misc/ /misc-backup/ > /dev/null 2>&1

    Every 10 minutes.
    I have nearly a TB of DATA, how come every time it runs it takes forever? Even if only like 5 files were added? Does it have to go through the 1TB or something?

  27. #27
    Join Date
    Aug 2006
    Location
    Duluth, GA
    Posts
    201
    I think it does a checksum on the files to see if they were changed. It doesn't actually d/l the files unless there is a change.

    I used rsync to d/l a new linux distro and updated it a couple of times and I noticed the same thing.

    Someone correct me if I am wrong.

  28. #28
    Join Date
    Apr 2005
    Posts
    694
    that's painful... if it goes through 1tb every 10/20 minutes...

  29. #29
    Join Date
    May 2000
    Posts
    486
    (I think) rsync works in memory for file comparison, so 1TB of small files would probably cause swapping, and massively slow things down. You can speed it up by running smaller rsyncs on subdirs rather than one process at the top level.

  30. #30
    Join Date
    Feb 2005
    Location
    Australia
    Posts
    5,842
    By default rsync will skip any kind of file comparison if the timestamp and size match.

    ti_nhatrang - have you ever actually let it complete the transfer? The first time transferring data will take a very long time, subsequent runs with just a few file changes should be fast.
    Chris

    "Some problems are so complex that you have to be highly intelligent and well informed just to be undecided about them." - Laurence J. Peter

  31. #31
    Join Date
    Apr 2005
    Posts
    694
    I have. I notice that if I set it to say 10 minutes, if the first rysnc run is not completed, the second 10 minute one starts. Is there anyway we can prevent this?

  32. #32
    Join Date
    Feb 2005
    Location
    Australia
    Posts
    5,842
    Perhaps a little bash script? This is untested, take it for what it's worth:

    Code:
    #!/bin/bash
    
    running=`ps ux |grep rsync |grep -v grep`
    
    if [ !"$running" ]; then
      rsync -a source destination
    fi
    Chris

    "Some problems are so complex that you have to be highly intelligent and well informed just to be undecided about them." - Laurence J. Peter

  33. #33
    Join Date
    Apr 2005
    Posts
    694
    can we make it not hog all the resources?

  34. #34
    Join Date
    Feb 2005
    Location
    Australia
    Posts
    5,842
    man nice
    ...
    Chris

    "Some problems are so complex that you have to be highly intelligent and well informed just to be undecided about them." - Laurence J. Peter

  35. #35
    Join Date
    Apr 2005
    Posts
    694
    I have:

    nice -10 rsync and it hogs up 4 cpu....

  36. #36
    Join Date
    Feb 2005
    Location
    Australia
    Posts
    5,842
    Well, nice will let it use whatever resources are available - the only difference should be that other jobs can still run at normal speed.

    But rsync shouldn't be using that much resource anyway - you've got something weird going on there. What kind of storage is it?
    Chris

    "Some problems are so complex that you have to be highly intelligent and well informed just to be undecided about them." - Laurence J. Peter

  37. #37
    Join Date
    Apr 2005
    Posts
    694
    nfs mounted folder... I basically want rsync to run at the lowest and yield to other, is this possible?

  38. #38
    Join Date
    Apr 2005
    Posts
    694
    Quote Originally Posted by foobic View Post
    Perhaps a little bash script? This is untested, take it for what it's worth:

    Code:
    #!/bin/bash
    
    running=`ps ux |grep rsync |grep -v grep`
    
    if [ !"$running" ]; then
      rsync -a source destination
    fi
    this doesn't seem to work, I tried running one after the other with both ssh session open and it loads two rsync.

  39. #39
    Join Date
    Apr 2005
    Posts
    694
    I did "$running" = "" instead and it works like a charm!

    This has been the best solution so far since rsync takes more than 20+ mins to run each time.

  40. #40
    Join Date
    Apr 2005
    Posts
    694
    I've used rsync -a, however, on the backup location, there are a few files that was deleted in the source that the backup location didn't delete itself, does anyone know why this is? please help.

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •