Results 1 to 11 of 11
  1. #1

    Transfer 50GIG Folder to New Server

    How long should it take to transfer a 50GIG folder containing 5300 files to another server? My server management company has been working on this for over 1 week now. The problem was that I did not have enough room to make a tar file since my server space was only 73GIG's But we added an extra 73 GIG drive a few days ago and the folder has still not been transfered. Is there anyway to transfer this faster?

  2. #2
    Join Date
    Oct 2005
    United States
    Do you have root access at 2 servers?
    Tommy Tran - tommy @ ::: VINAX, LLC ::: ::: Since 2004
    Premium Dedicated Servers and Colocation in downtown Chicago (350 E. Cermak Rd)
    Premium Bandwidth, 100% Network & Power Uptime SLA, 24/7 Prompt Tech Support

  3. #3
    Quote Originally Posted by ServerVina
    Do you have root access at 2 servers?
    Yes I have root access to the servers

  4. #4
    Join Date
    Apr 2001
    use rsync scp or lftp

  5. #5
    Quote Originally Posted by Dilhole
    use rsync scp or lftp
    Yes.. but how long should it take?

  6. #6
    Join Date
    Jun 2004
    depends on the network connection between two servers

  7. #7
    May be trying out tar command over ssh which will save your disk space and time both. Just google and first link was:

    No need to have root access but disk space needed on 2nd box.

    Hope this helps.
    <<Please see rules for signature setup>>

  8. #8
    Join Date
    Jan 2004
    York, UK
    For that size of data I would definitely use rsync rather than ftp/scp or tar+ssh - it's ability to easily restart from where it left off will be useful if anything interrrupts the transfer.

    You will need to muck about with file ownership at the other end - but you'll need to do that with the other methods also.

    How long should it take? How long a peice of string do you have over there?
    To transfer 50Gb of data at a flat 1Mb/sec, ignoring all overheads/complications, you are looking at over 14.2 hours. Add to that some time to account for network latency and times when your CPU is the bottleneck, and take off the time saved if the content is significanlty compressable[1] which it will be if it is mainly text/html/similar.

    I plucked the 1Mb/s figure from the air - if both servers have a 10Mb port and there is decent connectivity between the two providers then it should be possibe but don't be too disapointed if you get less on average. If you do a quick test of transfering data from one server to the other, you'll get a better idea - remember to test with a single stream as that is how you are about to transfer the stuff.

    [1] rsync has an option to gzip the transferred data, and the tar+gz+ssh method above will do similar[2] which it will be if it contains a lot of text/html/similar.

    [2] in fact tar+ssh would do the trick, if you ask ssh to compress for you, though you'll get a slightly better compression rate using gzip and an uncompressed SSH link.

  9. #9
    Join Date
    Mar 2004
    Tokyo / Japan
    The problems you are looking at with this amount of data is:

    - chances are you dont have a copy of ALL the data from the same timestamp (so data might change while you are doing the backup)
    - chances are that you want to retry the files that failed the first time transfer
    - Your average filesize seems like 10MB which is no problem for Rsync
    - I am not sure if you can "easily" compress 50GB of data. The processing may take a loooong time - especially if you want to tar or zip it - depending on the type of data this might need quite some processing power, memory etc.

    I do think rsync might be your best option here, especially if you want to resync the data a second time - rsync will be your friend.

    I hope this helps.

    Best regards
    __--- Nils Valentin ---
    ☆★☆ Speedtest One: Tokyo / Japan ☆★☆ / Feedback
    ☆★☆ Speedtest Two: Tokyo / Japan ☆★☆ / Feedback

  10. #10

    What OS is running ? If its windows then there is a MS tool called Robocopy which is what we do.
    The other way is if you have access to the hardware, add 2nd HDD and copy the data to that. Then insert it in the new server. Affordable, Fast and Reliable UK Hosting.
    Shared, Multidomain and Reseller Packages. Fast UK Servers.
    Windows + Linux, ASP.Net, PHP, MySQL inside our own datacenter onsite - AS48825

  11. #11
    Thanks Guys
    I have started the transfer via scp. It will take me 10 hours at 1.4Mbs.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts