Results 1 to 24 of 24
  1. #1

    Moving a huge amount of data

    Hi

    I have 6 servers which on each one of them I have about 10TB of information (in total it's 55TB), I want to move all of these information to another data center and sending the hard disks physically is no possible for me so the only way is to send them over internet connectivity of the servers, both servers have good uplink BUT the problem is when I rsync the data from my current server to the new server that I want to move the data to I only get 2-3Mbps speed top (I tried wget which is single connection and I get the same speed), I was wondering is there anyway that I can move these data with multi connection

    PS - There is about 120000 file on my servers so sending them one by one won't work and I can't compress them and move them because my servers hard disks are full so no space to do that

    Regards
    Parham

  2. #2
    Join Date
    Nov 2009
    Location
    /etc/my.cnf
    Posts
    10,041
    Quote Originally Posted by Mr_Parham View Post
    Hi

    I have 6 servers which on each one of them I have about 10TB of information (in total it's 55TB), I want to move all of these information to another data center and sending the hard disks physically is no possible for me so the only way is to send them over internet connectivity of the servers, both servers have good uplink BUT the problem is when I rsync the data from my current server to the new server that I want to move the data to I only get 2-3Mbps speed top (I tried wget which is single connection and I get the same speed), I was wondering is there anyway that I can move these data with multi connection

    PS - There is about 120000 file on my servers so sending them one by one won't work and I can't compress them and move them because my servers hard disks are full so no space to do that

    Regards
    Parham
    Simple solution would be to contact your server provider(s) and increase the connectivity all round if possible. The only solution I can think of other than physically sending the disks would be Rsync or another transfer method which is secure over SSH.

    EDIT: What specs are the new servers do they have multiple disks which give a decent write speed same with the servers your moving from do they offer decent read speeds?

  3. #3
    Quote Originally Posted by cd/home View Post
    Simple solution would be to contact your server provider(s) and increase the connectivity all round if possible. The only solution I can think of other than physically sending the disks would be Rsync or another transfer method which is secure over SSH.

    EDIT: What specs are the new servers do they have multiple disks which give a decent write speed same with the servers your moving from do they offer decent read speeds?
    Any though about this

    http://monalisa.cern.ch/FDT/

    and about the server spec that's not the problem, the main server usually hit 1Gbps without problem in peak time and the other server there is 16 enterprise disks in raid 50 which if we push it hit 2Gbps easily

  4. #4
    Join Date
    Nov 2005
    Location
    Nassau, The Bahamas
    Posts
    74
    Quote Originally Posted by Mr_Parham View Post
    Any though about this

    http://monalisa.cern.ch/FDT/

    and about the server spec that's not the problem, the main server usually hit 1Gbps without problem in peak time and the other server there is 16 enterprise disks in raid 50 which if we push it hit 2Gbps easily
    If you have Gige ports on the servers on both sides you should be able to move the data.

    Have you investigate why you're getting 2-3Mbps on the new server, when you have Gige connectivity?
    Secure Hosting | Premium Cloud & Dedicated hosting solutions since 2001
    email: sales(at)securehost(dot)com • +1-242-502-8700 • 24/7 support by phone/helpdesk
    Located in The Bahamas & Bermuda: Redundant Cooling, Power & Network, PCI DSS compliant, 7ms from USA

  5. #5
    Join Date
    Nov 2009
    Location
    /etc/my.cnf
    Posts
    10,041
    Quote Originally Posted by Mr_Parham View Post
    Any though about this

    http://monalisa.cern.ch/FDT/

    and about the server spec that's not the problem, the main server usually hit 1Gbps without problem in peak time and the other server there is 16 enterprise disks in raid 50 which if we push it hit 2Gbps easily
    I would focus on the speed issue first before selecting a transfer method clearly something wrong if your getting a low speed like you are. Have you done a test between the servers to if somethings going on at one of the hops or something.

  6. #6
    Join Date
    Feb 2003
    Location
    Dallas, TX
    Posts
    1,125

  7. #7
    Join Date
    May 2007
    Location
    Bhopal - India
    Posts
    331
    FDT is heavily threaded and works perfectly for long distance transfers. We have been using it to move raw uncompressed disk images from one DC to another.
    LoopByte
    India Based Dedicated Servers, VPS & Hosting Services

  8. #8
    The data are being moved over extremely long distances and that's the reason for slow connection speed

  9. #9
    Join Date
    Sep 2008
    Location
    Seattle, WA
    Posts
    1,268
    Quote Originally Posted by Mr_Parham View Post
    The data are being moved over extremely long distances and that's the reason for slow connection speed
    Your speed seems slow for gig ports even for opposite ends of the globe.
    █ Brian Kearney, Stealthy Hosting Inc. Seattle, WA [AS54931] Skype: StealthyHosting
    Affordable Dedicated Servers
    Remote Hands Colocation

    █ Email: [email protected] Phone: 253-880-1233

  10. #10
    Join Date
    Aug 2003
    Posts
    597
    120000 files is probably your issue. You will probably need to create some kind of image of the file system and then transfer it with multiple connections.

  11. #11
    Join Date
    Feb 2005
    Location
    UK
    Posts
    553
    Even on a smaller scale, I find that transferring large volumes of data in a compressed file or image takes infinitely less time than doing it file-by-file — especially if you're not multi-threading the transfer (e.g. transferring the files 50 at a time).

  12. #12
    Hello,

    I would check with my providers why the connection is so slow. They should know exactly why and give you more information. If wget, scp, or nfs share give you the same speeds when transferring single large file then there is something wrong with your connection.

    As for the transfer method FDT sounds good.

  13. #13
    Join Date
    Nov 2009
    Location
    /etc/my.cnf
    Posts
    10,041
    Quote Originally Posted by Mr_Parham View Post
    The data are being moved over extremely long distances and that's the reason for slow connection speed
    You done a tracert between the 2 servers?

  14. #14
    Join Date
    Mar 2012
    Posts
    48
    There will be nothing wrong with the OP's gig ports !

    The issue will be latency, and then the affect the latency has on throughput 'bandwidth delay product' is the technical name for this.

    FDT is an excellent solution to the issue, as to get higher thoughput parallelisation (eg multiple tcp sockets / streams) will be needed.

  15. #15
    Quote Originally Posted by ifastnet View Post
    There will be nothing wrong with the OP's gig ports !

    The issue will be latency, and then the affect the latency has on throughput bandwidth delay product is the technical name for this.

    FDT is an excellent solution to the issue, as to get higher thoughput parallelisation (eg multiple tcp sockets / streams) will be needed.
    It is not always the latency. Here is a short example:

    HTTP request sent, awaiting response... 200 OK
    Length: 524298240 (500M) [application/x-tar]
    Saving to: “testfile-500mb.tar”

    15% [=================> ] 81,723,984 4.52M/s eta 2m 32s

    I am downloading a file located on the other part of the world (well sort of, EU/US), latency is over 150ms, yet I am using pretty much the full capacity of my connection at the office which is around 50Mbps. Our provider doesn't utilize any kind of traffic shaping or whatsoever. It doesn't change if I test a large, small file or lots of small files. We are using similar backup methods for huge amount of data over gigabit dedicated channels and latency was rarely an issue for us. Most of the time we are limited by the backup medium writing speed, not the actual connection.

    This is the reason I suggested OP to contact his providers and ask them for an explanation, they will know what is happening and why.

  16. #16
    Join Date
    Apr 2009
    Posts
    1,143
    We did a single threaded (vmware storage vmotion) transfer over a 1g wave once. Wasnt Real pretty.. I Think it did 50Mbit. The distance was ~7ms roundtrip.. That wasnt really optimal. So either pack up your data or use simultanious threaded transfer. Filezilla Can do it between servers.

    Or, try with an udp/connectionless oriented transfer protocol

  17. #17
    Join Date
    Mar 2012
    Posts
    48
    It is not always the latency. Here is a short example:

    HTTP request sent, awaiting response... 200 OK
    Length: 524298240 (500M) [application/x-tar]
    Saving to: “testfile-500mb.tar”

    15% [=================> ] 81,723,984 4.52M/s eta 2m 32s

    I am downloading a file located on the other part of the world (well sort of, EU/US), latency is over 150ms, yet I am using pretty much the full capacity of my connection at the office which is around 50Mbps
    Ok , I did not give the full picture.

    The bandwidth delay product is key, however also in tcp we have what is called 'window scaling' , some applications (eg ssh/scp ) lock the window size to 65,535 bytes causing window scaling to not be possible and you then see the full affect of the bandwidth delay product.

    for example on a 150ms link, with a window size of 65,535 you can calculate best case throughput using the following formula.

    window size in bits / latency (rtt) in seconds = bits per second ..

    so

    65536 Bytes converts to 524288 bits

    150ms in seconds = 0.15

    we then divide the 524288 bits by the round trip time in seconds (0.15)

    this gives 3495253.33333 bits per second,,, then we divide this by 8 to get bytes per second ( 8 bits in a byte ! ) .

    3495253.33333 / 8 = 436906.666666

    then we convert bytes to Mb

    436906.666666 /1000000

    == 0.43MB's

    so max theoretical throughput for a single connection on a long fat network with tcp and latency of 150ms and a 65536 byte window size is 0.43mb/s

    Hope this helps explain the underlying issue how latency can affect throughput for single connections.

  18. #18

    Use SCP

    Do you have root access? If yes, have you thought of using scp, if not try searching for it in Google.

  19. #19
    Join Date
    Mar 2012
    Posts
    48
    Do you have root access? If yes, have you thought of using scp, if not try searching for it in Google.
    scp fundamentally uses ssh to transport, this has a locked window size and will give low transfer rates over a latent connection

    http://www.psc.edu/index.php/hpn-ssh is a good article (and patches to ssh ) to fix this in ssh,, however the OP's origional thought of using http://monalisa.cern.ch/FDT/ is probably the best solution.

  20. #20
    Ok sorry I couldn't update this threat earlier but we tested FDT and it's simply amazing (!), it actually managed to use the full port on both side and it's quite simple to use, I would recommend it to anyone who want to make a large transfer

  21. #21
    Mr. Parham,

    One thing you may want to look at is spinning up a temporary Appassure Core Node at the other DC location. You should be able to sign up for 30 day trial. This way you can install the agent on the other Dedicated nodes that you want to migrate. They will start with an initial image base line image transfer which will obviously take some time however, after the first synch of the image you will only send the delta. The good news is the agents work well even on larger latency hauls which could increase your throughput performance. At which point you can then locally restore the image snap to the new machine or temporary mount the image and copy the files off which ever best fits your use case.

    That might be your best case for this. Since you don't have the space locally to image or compress the files for transfer. Another option is to migrate them using sftp or similar which is more forgiving of latency and do full file copies. Regardless of your choice it's going to take a long time to migrate that volume of data.
    PCLHS | SAS70 Datacenters in New Jersey/Texas
    100TB Dedicated Servers 1U - Full Cab Colocation Complex Hosting Horizontally Scalable Hosting DR/HA Hosting Public and Private Clouds Web Farms Innovative, Reliable, and Responsive
    Contact Us E: mark [at] pclhs.net | W: www.pclhs.net

  22. #22
    Join Date
    Jan 2014
    Posts
    92
    It's Simple 1st buy Window VPS for this having 2 Gb Ram & then ask your 1st company to increase port up to 100MBPS to 1 GBP for one to two days and then select good company who have also have same port or higher port, then migrate all this through window Vps, Winodw VPS will do every thing for you, you don't need to open your laptop/computer 24 hours.
    Note: Choose Best winodow Vps.
    If you think that's complicated & will take a lot of time then check both companies in same country if yes then ask from both companies for hard drive exchange..
    Its pretty easy and time saving but for this you have best speaking skills.

  23. #23
    Join Date
    Feb 2005
    Location
    UK
    Posts
    553
    He already solved the problem guys.

  24. #24
    Join Date
    Jul 2008
    Location
    New Zealand
    Posts
    1,208
    We had to move a similar amount of data,and rsync did not cut it. We ended up using LFTP (with 30 threads) for an initial sync, and then did a rsync to transfer any changed files.

    LFTP will normally have you max your pipe. You can just increase the amount of threads/connections for faster speeds.

Similar Threads

  1. HUGE traffic amount mySQL - how do I upgrade?
    By zedaxy in forum Dedicated Server
    Replies: 16
    Last Post: 12-02-2012, 07:19 PM
  2. Huge amount of fraud attempts from Italy
    By backtogeek in forum Fraud and Abuse
    Replies: 1
    Last Post: 06-28-2012, 07:29 PM
  3. Moving accounts with smallest amount of downtime?
    By build-a-host in forum Hosting Security and Technology
    Replies: 4
    Last Post: 02-08-2011, 06:13 AM
  4. Replies: 0
    Last Post: 06-29-2005, 11:55 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •