Results 1 to 10 of 10
  1. #1
    Join Date
    Jul 2004
    Posts
    868

    Talking daily backup + rsync ssh + large number of files

    Hi ,
    i just wana know is it safe to do remote daily backup for about 70,000 files ?
    file sizes is about 200kb and every day i have about 1000 new file ,
    so rsync first should check old files becouse i am deleting about 30-50 of them daily and them backup new 1000 files ,
    so how much it will take every time to compare that 70,000 files ?


    i have 2 option now :
    1-using second hdd and raid 1
    2-using rsync and backuping to my second server , so i can save about $70 each month !

    thanks !

  2. #2
    Have you looked into rdiff?

    http://www.nongnu.org/rdiff-backup/

  3. #3
    Join Date
    Nov 2004
    Posts
    49
    Don't see why Rync is a problem with that amount of files. I use it to do my backups of e-mail server and many more files daily at night and it works fine.

    You might want to consider the bandwidth usage which for you would be minimum 200kb x 1000 = 200mb every day.

    You can also setup a script (like I have done) to check if Rsync completes or not by doing additional checks before and after rsync finishes. Things like checking folder size etc. If the folder size is different than make it send you an e-mail.

  4. #4
    Join Date
    Nov 2005
    Location
    Minneapolis, MN
    Posts
    1,648
    Quote Originally Posted by FirmbIT
    Have you looked into rdiff?

    http://www.nongnu.org/rdiff-backup/
    I second that recommendation. rdiff-backup makes the process of performing incremental backups remarkably simple.
    Eric Spaeth
    Enterprise Network Engineer :: Hosting Hobbyist :: Master of Procrastination
    "The really cool thing about facts is they remain true regardless of who states them."

  5. #5
    Join Date
    Jul 2004
    Posts
    868
    Quote Originally Posted by FirmbIT
    Have you looked into rdiff?

    http://www.nongnu.org/rdiff-backup/
    what is benefits of using rdiff ? is it faster than rsync ?

    Quote Originally Posted by UkWebsiteFan
    Don't see why Rync is a problem with that amount of files. I use it to do my backups of e-mail server and many more files daily at night and it works fine.

    You might want to consider the bandwidth usage which for you would be minimum 200kb x 1000 = 200mb every day.

    You can also setup a script (like I have done) to check if Rsync completes or not by doing additional checks before and after rsync finishes. Things like checking folder size etc. If the folder size is different than make it send you an e-mail.
    i dont have problem with that 200mb daily traffic ! , i just wana know how much will take to every day check and compare 70-80k files every day ? it will take more than 10 minutes ? ( only checking ) ?

    also if a file was removed in source server , rsync will remove it in target ?

  6. #6
    Join Date
    Nov 2005
    Location
    Minneapolis, MN
    Posts
    1,648
    Quote Originally Posted by goolex
    what is benefits of using rdiff ? is it faster than rsync ?
    Rsync, by default, will only give you a single point-in-time copy for the data from the source. Rdiff-backup will retain an incremental history for as long as you want. So lets say a file gets changed and then 3 days later you find out you need the original file back. With rsync the file is gone (again, assuming default without fancy scripts) but with rdiff-backup you can specify to restore a file as it was x amount of time ago. ie "rdiff-backup -r 4d /path/to/file [email protected]::/path/to/file" will restore a file as it was 4 days ago.
    Quote Originally Posted by goolex
    i dont have problem with that 200mb daily traffic ! , i just wana know how much will take to every day check and compare 70-80k files every day ? it will take more than 10 minutes ? ( only checking ) ?
    That depends on several factors, including the performance of the server, performance of the file system, latency between the server being backed up and the backup destination, etc. You won't know how long it's going to take unless you actually run it.
    Quote Originally Posted by goolex
    also if a file was removed in source server , rsync will remove it in target ?
    If you use the "--delete" option, yes.
    Eric Spaeth
    Enterprise Network Engineer :: Hosting Hobbyist :: Master of Procrastination
    "The really cool thing about facts is they remain true regardless of who states them."

  7. #7
    Sorry guys, I got to ask, is there any open source solution like Rsync which I can install on Windows?

  8. #8
    Join Date
    Apr 2004
    Location
    Singapore
    Posts
    1,506
    tanfwc
    Singapore Managed Colocation
    Singapore BGP Announcement

  9. #9
    Robocopy looks good, but does it do incremental copy ?
    Thanks
    Joshua

  10. #10
    Join Date
    Feb 2005
    Location
    Australia
    Posts
    5,842
    You can use rsync itself on windows with cygwin. Unix-style hard links (the normal method for incremental / snapshot backups) also work under XP. You might also look for cwrsync but personally I prefer the full cygwin.

    HTH
    Chris

    "Some problems are so complex that you have to be highly intelligent and well informed just to be undecided about them." - Laurence J. Peter

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •