Results 1 to 9 of 9
  1. #1
    Join Date
    May 2004
    Location
    Corona, CA USA
    Posts
    386

    What Method of HD Cloning Uses Least CPU

    I have a live production server running my most important client and own sites that has developed critical errors on the third hard drive used for backups only (/dev/sdc). The machine is running CentOS 5.6 with WHM/cPanel, and of course I can haz r00t. It's in a 4 bay SuperMicro chassis that supports hot swapping. The 4th bay is currently unused. The machine has a Core 2 Quad Q9550. The failing drive is a WDC WD1002FAEX-00Z3A0 1TB drive.

    My competent and reliable provider for this dedicated server has suggested that they can add an identical replacement 1TB drive for me to partition/format and then copy my retrievable backups. Since the fstab uses labels it should be trivial to do this.

    Here finally is my question: what method of copying an entire hard drive will produce the least CPU load? I'm thinking tar might be the best choice. I expect using dd would be unwise with a drive producing read errors, and I'm guessing rsync would use more cpu than tar.

    Suggestions?

    Bonus question: what switches would you use with tar to clone a hard drive?
    Skeptic Hosting
    Promoting a reality-based lifestyle choice
    Hosting by invitation only, nothing for sale

  2. #2
    Join Date
    May 2011
    Location
    London, UK
    Posts
    32
    Hi srider,
    I would choose Rsync.
    Remember that with TAR, it will be a two-step process, you'll need to archive then extract the data.

    With Rsync you can limit the IO/CPU load by adjusting the transfer rate.
    Also use the verbose switch to get useful output which will help you identify any transfer errors, since the source is unreliable.
    Obviously, don't use compression.

    Veselin
    Last edited by kantsev; 06-01-2011 at 12:47 PM.
    Kantsev Linux Laboratories
    t: 02071832945
    w: kantsev.com
    Linux server management, London UK

  3. #3
    Join Date
    Mar 2002
    Location
    Philadelphia, PA
    Posts
    2,517
    dd (disk dump).

  4. #4
    Join Date
    May 2004
    Location
    Corona, CA USA
    Posts
    386
    Quote Originally Posted by kantsev View Post
    Hi srider,
    I would choose Rsync.
    Remember that with TAR, it will be a two-step process, you'll need to archive then extract the data.

    With Rsync you can limit the IO/CPU load by adjusting the transfer rate.
    Also use the verbose switch to get useful output which will help you identify any transfer errors, since the source is unreliable.
    Obviously, don't use compression.

    Veselin
    Thank you, TAR is out. Assuming that the failing drive is mounted at /backup and the replacement at /backup2 would you use something like

    rsync -rvpt --bwlimit=18000 /backup/* /backup2/ > /tmp/cloneresults.txt &


    Quote Originally Posted by viGeek View Post
    dd (disk dump).
    I wondered if it would be unwise to use dd with a drive that is failing. Would it handle read errors in a way that is better than rsync?

    Thanks to both of you good people for your replies.
    Skeptic Hosting
    Promoting a reality-based lifestyle choice
    Hosting by invitation only, nothing for sale

  5. #5
    Join Date
    Oct 2002
    Location
    Vancouver, B.C.
    Posts
    2,699
    Quote Originally Posted by kantsev View Post
    Remember that with TAR, it will be a two-step process, you'll need to archive then extract the data.

    With Rsync you can limit the IO/CPU load by adjusting the transfer rate.
    Also use the verbose switch to get useful output which will help you identify any transfer errors, since the source is unreliable.
    Obviously, don't use compression.

    Veselin
    All the TAR format does is concatenate files together, so computationally and effectively speaking, it would not be a two-step process; you'd just be copying the files over. The only overhead will be from piping the data, which is pretty minimal.

    rsync on the other hand is always going to be computationally intensive because it has to compare data in multiple combinations. Regardless of how much you limit the IO/CPU, it's still taking up the same number of clock cycles; you're just spreading it out more. If anything, you'll use up even more clock cycles because you'll need more context switches between rsync and other processes over the course of the rsync run. Unless you're cloning to a drive that already has most of the data already, it's pointless to use rsync.

    The best way to reduce CPU is to keep things simple. Use dump with -0 for a full backup, as this is what dump is specifically designed for. dd is a bit lower level than necessary. tar can be useful if you need some extra flexibility. If you don't need to preserve anything on the filesystem level (like permissions or modification times), even a recursive cp will work.
    ASTUTE INTERNET: Advanced, customized, and scalable solutions with AS54527 Premium Performance and Canadian Optimized Network (Level3, Shaw, CogecoPeer1, GTT/Tinet),
    AS63213 Cost Effective High Performance Network (Cogent, HE, GTT/Tinet)
    Dedicated Hosting, Colo, Bandwidth, and Fiber out of Vancouver, Seattle, LA, Toronto, NYC, and Miami

  6. #6
    If you haven't lost your primary data, then are the backups really important? On a failing drive, you might not be able to trust any of the data, first of all, and secondly, it can hang the system to try to recover from read errors. The cpu usage of many copying methods is reasonably low, but a server can behave unpredictably when reading a lot of data from a questionable drive.
    IOFLOOD.com -- We Love Servers
    Phoenix, AZ Dedicated Servers in under an hour
    ★ Ryzen 9: 7950x3D ★ Dual E5-2680v4 Xeon ★
    Contact Us: sales@ioflood.com

  7. #7
    Join Date
    May 2011
    Location
    London, UK
    Posts
    32
    Quote Originally Posted by srider View Post
    Thank you, TAR is out. Assuming that the failing drive is mounted at /backup and the replacement at /backup2 would you use something like

    rsync -rvpt --bwlimit=18000 /backup/* /backup2/ > /tmp/cloneresults.txt &




    I wondered if it would be unwise to use dd with a drive that is failing. Would it handle read errors in a way that is better than rsync?

    Thanks to both of you good people for your replies.
    Hi srider,
    hhw's comment states some useful facts on resource efficiency, so if you are looking for best utilization, it'd be beneficial to check the tools mentioned there.


    Otherwise if you are still interested in Rsync, you could use:

    rsync -av --bwlimit=18000 /backup/ /backup2/ > /tmp/cloneresults.txt

    - you dont need the *, a trailing / is enough
    - it will be better to leave the process in the foreground, so skip the &
    - with this redirection, standard output will go to the TXT file and error will print on the console
    - you could capture error output to a file too if you want, in case later you decide to review which files/folders failed to transfer and for what reason

    rsync .... >standard.log 2>error.log
    Kantsev Linux Laboratories
    t: 02071832945
    w: kantsev.com
    Linux server management, London UK

  8. #8
    Join Date
    May 2004
    Location
    Corona, CA USA
    Posts
    386
    Thanks to all of you for helping a 34% senile old man.

    We discovered that the 4th bay in my server was not cabled. So we put the new drive in bay 3 (sdc) and after I did fdisk and mkfs.ext3 with it I tried to use
    dump -0u -f /dev/sdd1 /dev/sdc1
    while the failing drive was mounted (and mounted) in an external USB enclosure.

    A few minutes into that process it appears the failing drive gave up the ghost, as CentOS is no longer able to find an ext3 filesystem on it.

    So the new drive will simply get filled up as new backups are made.

    As funkywizard pointed out, nothing critical has been lost.
    Skeptic Hosting
    Promoting a reality-based lifestyle choice
    Hosting by invitation only, nothing for sale

  9. #9
    Join Date
    Aug 2003
    Location
    PA
    Posts
    1,914
    Quote Originally Posted by srider
    Thanks to all of you for helping a 34% senile old man.

    We discovered that the 4th bay in my server was not cabled. So we put the new drive in bay 3 (sdc) and after I did fdisk and mkfs.ext3 with it I tried to use
    dump -0u -f /dev/sdd1 /dev/sdc1
    while the failing drive was mounted (and mounted) in an external USB enclosure.

    A few minutes into that process it appears the failing drive gave up the ghost, as CentOS is no longer able to find an ext3 filesystem on it.

    So the new drive will simply get filled up as new backups are made.

    As funkywizard pointed out, nothing critical has been lost.
    Glad to see you've gotten it all straightened out.

    Quote Originally Posted by funkywizard
    If you haven't lost your primary data, then are the backups really important? On a failing drive, you might not be able to trust any of the data, first of all, and secondly, it can hang the system to try to recover from read errors. The cpu usage of many copying methods is reasonably low, but a server can behave unpredictably when reading a lot of data from a questionable drive.
    Well said, I'd tend to agree 100% on this. srider in the future your best bet is to continue to be very diligent and cautious (as you've already exhibited coming here for advice and information ahead of time) when recovering known failing drives. Just remember technology tends to have surprise on it's side more often than not, so just be prepared

Similar Threads

  1. Website Cloning
    By whitecollar in forum Other Offers & Requests
    Replies: 16
    Last Post: 11-15-2006, 01:02 AM
  2. New CPU performance measuring method
    By Opteron4P in forum Dedicated Server
    Replies: 0
    Last Post: 08-22-2005, 06:51 AM
  3. Cloning - What is it?
    By HelpQuick in forum Web Design and Content
    Replies: 6
    Last Post: 10-10-2004, 04:13 AM
  4. Cloning
    By Artashes in forum Web Hosting Lounge
    Replies: 10
    Last Post: 03-10-2004, 01:02 PM
  5. Method to check each user's CPU/RAM usage?
    By wheimeng in forum Hosting Security and Technology
    Replies: 12
    Last Post: 10-12-2003, 11:33 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •