Results 1 to 6 of 6
  1. #1
    Join Date
    Sep 2010

    Incremental Server Backup

    I have a dedicated server with a /home/ dir of about 250GB. I purchased an additional offsite dedicated server with a 1TB hard drive and configured automatic daily incremental backups between the two. All of the data was successfully backed up and was running fine without any issues.

    However, the 1TB HDD (the backup location) started filling up very quickly and has now reached full capacity despite there only being about 250GB of actual data on the main server.

    Surely this isn't right? The only idea I have is that rsync is being very inefficient as there are a couple hundred thousand files involved. I know that rsync creates symbolic links, could they fill up the disk that quickly?

    Here's the bash script I wrote to handle the daily backups, does this look right?

    DAY0=`date -I`
    DAY1=`date -I -d "1 day ago"`
    SRC="[email protected]:/home/"
    #Execute the backup
    rsync -a -v -h -e "ssh -i /home/backup/.ssh/backup_rsa" --progress --stats --delete \
    --exclude='cache' --exclude='backup' --exclude='thumbcache' --exclude='cached' --exclude='cachedfiles' --exclude='_versionbackups' --exclude='error_log' \
    --exclude /.cpcpan --exclude /cpeasyapache --exclude /src --exclude /cPanelInstall --exclude /virtfs --exclude /.cpan --exclude /.cpanm \
    --exclude '/*/.*' --exclude '/*/mail' --exclude '/*/etc' --exclude '/*/perl5' --exclude '/*/tmp' \
    --link-dest=$LNK $SRC $TRG
    Any suggestions or tips would be greatly appreciated!


  2. #2
    Impossible, even if you're dealing with millions of files. Symbolic link as the name suggest is just small little config that says 'this' related to 'that', nothing more.

    What to watch
    - Check your cron job. Does the rsync finish before invoking another?
    - Make sure you don't create multiple similar backup on different directory
    - Make sure your old files is actually being deleted(or incremented) by your script

  3. #3
    Join Date
    May 2013
    Just to clarify, are you backing up the data to the same directory? if so rsync should sync only the changed data and the total usage should be equal or less than your /home . if you take multiple copies, yes it can easily fill your remote HDD as rsync do not create any links between them. Rather than using your own script, use available backup utilities like rsnapshot, dirvish, rdiff-backup, etc. That can create hard-links between backup points and can save some space.

  4. #4
    try rsnapshot - a snapshot-generating backup-tool using Rsync and hard links

  5. #5
    I haven't used --link-dest with rsync before so I may be misunderstanding what it's doing, but I think what's going on is that you're creating a new directory each day, so (roughly) every 4 days you'd fill the 1TB drive. You can check using:

    du -sch /home/backup/servers/*
    on the backup target. If I'm correct, you'll have multiple directories with different dates each using roughly 250GB. I could easily be wrong since I cba to research exactly what --link-dir does, but I think $LNK needs to be a static directory, not a directory named after yesterday's date.

    You may be able to do what you're trying to do by making $LNK a directory named "weekly" and running a separate cron job to move the "weekly" directory to a different location such as "lastweek" before the next week's backup is run, and removing "lastweek" prior to moving "weekly".

    Another idea is that --link-dir is creating symlinks, so your script only be removing the symlinks when a file is deleted with --delete since it only deletes files from the previous day's directory. The directory manipulation I suggested above may resolve the space issue, but I can also see it breaking symlinks for the previous week's backups, so please proceed with caution.

  6. #6
    Join Date
    Feb 2005
    --link-dest should do what you want, which is to create hard-links (not symlinks) of unchanged files. But there is a warning in the manual:
    If fileís arenít linking, double-check their attributes. Also check if some attributes are getting
    forced outside of rsyncís control, such a mount option that squishes root to a single user, or
    mounts a removable drive with generic ownership (such as OS Xís "Ignore ownership on this volume"
    Personally I've always used the simpler alternative of cp -al to prepare a full hardlinked copy of the previous version, and then rsync to there without link-dest.

    "Some problems are so complex that you have to be highly intelligent and well informed just to be undecided about them." - Laurence J. Peter

Similar Threads

  1. Replies: 11
    Last Post: 04-09-2012, 10:48 AM
  2. incremental backup or full backup - which to use?
    By Compworld in forum Hosting Software and Control Panels
    Replies: 10
    Last Post: 12-26-2011, 01:38 PM
  3. Incremental backup
    By king_himself in forum Web Hosting
    Replies: 17
    Last Post: 11-18-2011, 04:35 PM
  4. Incremental MySql Backup
    By WishIwasntH1 in forum Hosting Security and Technology
    Replies: 6
    Last Post: 04-14-2009, 02:33 PM
  5. Rsyn Question on --backup --backup-dir Switches For Incremental Backs
    By mrbt in forum Hosting Security and Technology
    Replies: 0
    Last Post: 01-26-2007, 09:16 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts