Results 1 to 15 of 15
  1. #1
    Join Date
    Mar 2009
    Posts
    245

    Best way for transfer tons of files?

    I have directory of 90 gb with tons of files which i need to transfer to another server.I usually tar directory with command tar -pczf bla.tar.gz directory,but in this case it will take ages first to archive files,then to transfer,and then to extract.Is there faster way which doesn't involve archiving?

  2. #2
    Join Date
    Oct 2006
    Location
    /usr/src/linux/
    Posts
    700
    I'd use rsync and renice its pid to avoid hogging resources, but it will still take a while.
    VPSnoc.com offers high quality Xen® OpenVZ & Windows® Virtual Private Servers at affordable prices.
    99.95% Uptime | 24/7/365 Support | Unmetered bandwidth.
    Follow us: twitter.com/VPSnoc

  3. #3
    Join Date
    Aug 2009
    Location
    Orlando, FL
    Posts
    1,063
    I agree. Rsync is your best bet. At least that way if the transfer gets interrupted you won't have to start all over again.

  4. #4
    Join Date
    Mar 2009
    Posts
    245
    Well,it would be nice if someone give a example command.Also what about scp?I read how that is also good.

  5. #5
    Yea Rsync it`s by far the best way to transfer such amount of files.
    FROGOST - Cheap Web Hosting

  6. #6
    Quote Originally Posted by roguehosting View Post
    Well,it would be nice if someone give a example command.Also what about scp?I read how that is also good.
    these days rsync is tunnelled over ssh so is the same as scp really.

    use something like this (as root) from the server with the files on it, to a remote server, assuming you want to backup the whole /home/website/ directory preserving permissions/uid's and using ssh compression:

    Code:
    rsync -avzp --progress --delete-after --numeric-ids /home/website/ root@newserver:/home/website/
    note this will delete everything on the remote server that already exists in /home/website/ so be sure you know what you are doing, or maybe rsync to a temp directory instead, then move the files into place over ssh.

    or use the -n flag to just show you what it will do without actually doing it.

    you have to be root on both machines to preserve the permissions, unless you only want to read/write a temp directory or $home.

    usual disclaimer applies, try it on a dev box first or a test directory of unimportant files, also read up on rsync.
    Last edited by sej7278; 09-15-2009 at 06:35 PM.

  7. #7
    Join Date
    Mar 2009
    Posts
    245
    I used following command:
    scp -r directory root@ipadress:/home/bla
    And it works fine.
    Now there is following problem:i need to transfer 90 gb of files.I transfering it over LAN(both machines are at same datacenter)and that gives me max speed.But can i speed it up by repeating scp on another subdirectory?And also how do i put password without question which shows after i type command.I need that so i can put proccess into background.

  8. #8
    Join Date
    Apr 2009
    Location
    USA / UK
    Posts
    4,577
    Quote Originally Posted by roguehosting View Post
    I used following command:
    scp -r directory root@ipadress:/home/bla
    And it works fine.
    Now there is following problem:i need to transfer 90 gb of files.I transfering it over LAN(both machines are at same datacenter)and that gives me max speed.But can i speed it up by repeating scp on another subdirectory?And also how do i put password without question which shows after i type command.I need that so i can put proccess into background.
    you could just use screen to background it:

    screen scp -r directory root@ipadress:/home/bla

    then once it get's going press CTRL+A and then D.

    "screen -r" to foreground it again.
    RAM Host -- USA Premium & Budget Linux Hosting
    █ Featuring Powerful cPanel Shared Hosting
    █ & Premium Virtual Dedicated Servers
    Follow us on Twitter

  9. #9
    Join Date
    Apr 2009
    Location
    Nevada
    Posts
    662
    You can start multiple threads of scp in screen depending on the load on your server and drive busyness.
    Also you need to use ssh keys for password less scp or use rsync where you have the option of specifying the password in a file.

    Quote Originally Posted by roguehosting View Post
    But can i speed it up by repeating scp on another subdirectory?And also how do i put password without question which shows after i type command.I need that so i can put proccess into background.
    James B
    WWW.EZEELOGIN.COM |Setup your Secure Linux SSH GatewayMEET PCI DSS & ISO 27001 Compliance|Manage & Administer Multiple Linux Servers Quickly & Securely.

  10. #10
    Join Date
    Jul 2009
    Location
    Manila
    Posts
    301
    Or you can use the bg command. You can check on this tutorial:

    http://nolimitshostdesk.luv.ph/knowl...hp?article=113

    I believe there is no way to "remember" passwords for scp. you need to have keys or use one scp to transfer the entire directory

  11. #11
    Join Date
    May 2009
    Location
    L'viv Ukraine
    Posts
    21
    scp is suxx - doesnt keep file attributes
    use rsync - its really rocks and you can restart it
    from where its stopped every time its crashed ;-)

  12. #12
    Rsync is freely available and it is the fastest best mode to transfer the files. You will be happy to use it.
    SUPPORT FACILITY | 24/7 TECH SUPPORT
    SERVER MANAGEMENT | WEB HOSTING SUPPORT | WP EXPERTS

  13. #13
    You can skip password prompt by using SSH Public Key Authentication.

    Check if public key is already present in the source server.
    Code:
    # cat /root/.ssh/id_rsa.pub
    If it exists, you can simply copy it to the destination server.
    Code:
    scp /root/.ssh/id_dsa.pub root@destination.com:/root/.ssh/authorized_keys2
    If the public key is not available, you can create one using the ssh-keygen command.
    Code:
    $cd /root/.ssh/
    $ssh-keygen -t rsa
    Whether or not you use a passphrase or not depends on your system standards. If you use a passphrase, be sure to make a note of it someplace safe. If you do not use a passphrase, just hit enter when it asks for one.

    Then copy the public key to destination server as explained above.

  14. #14
    Join Date
    May 2009
    Location
    L'viv Ukraine
    Posts
    21
    hmmm...

    ssh-keygen -t rsa

    for dsa keys? weird...

  15. #15
    You can use “dsa” instead of the “rsa” after the -t to generate a DSA key.
    Code:
    $ssh-keygen -t dsa

Similar Threads

  1. Ftp files transfer
    By decybel in forum Hosting Software and Control Panels
    Replies: 1
    Last Post: 09-26-2007, 01:16 PM
  2. Transfer files to new vps
    By ltaylor in forum VPS Hosting
    Replies: 4
    Last Post: 01-07-2007, 03:39 AM
  3. How can i Transfer files from one PC to another?
    By Mike235 in forum Web Hosting Lounge
    Replies: 8
    Last Post: 11-26-2005, 07:38 PM
  4. Replies: 2
    Last Post: 02-24-2003, 11:22 AM
  5. tons and tons and tons of fraud
    By ebo in forum Web Hosting Lounge
    Replies: 1
    Last Post: 03-29-2002, 03:04 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •