Results 1 to 17 of 17

Thread: SFTP Woes..

  1. #1
    Join Date
    Aug 2001
    Location
    Denver, CO
    Posts
    273

    SFTP Woes..

    I'm trying to upload about 200 Gb of data and it became clear to me to route it out a specific connection sftp was the way to go.. (the ssh2 kind)

    The problems I'm now having is that the 2 applications I rely on cannot get the job done.. CuteFTP Pro 8 can't even begin to get it done it crashes, errors out etc. and for while there it looked like the free program WinSCP was going to get it done but now it errors out/crash's and while it actually did get about 1/4 of the data done, it seems to think when it reported it done.. it got it all (that is when it doesn't crash for no reason, these crashes only occurred mainly when I tried to get it do 3/4's or the rest of the data)

    So anyway.. recommend me a good client app if you can.. (that does real folder sync transfers) preferably ssh2/sftp, I'm uploading to dreamhost if that makes a difference I assume it doesn't though..

  2. #2
    Join Date
    Aug 2007
    Posts
    39
    I use FileZilla: http://filezilla-project.org/

    Its a really good piece of software.
    Edward Tobia
    Administrator/Founder
    EuroSRV Hosting Solutions
    http://www.eurosrv.com

  3. #3
    Join Date
    Feb 2005
    Location
    Australia
    Posts
    5,842
    I'd use rsync. If you only get a partial transfer, just use the same command again and it will pick up where it left off.

    For a Windows binary, Google cwRsync or just install the full cygwin.
    Chris

    "Some problems are so complex that you have to be highly intelligent and well informed just to be undecided about them." - Laurence J. Peter

  4. #4
    Join Date
    Aug 2000
    Location
    Sheffield, South Yorks
    Posts
    3,480
    SFTP isn't going to be much good for all that data, it'll take forever as SFTP isn't the fastest protocol in the world, a lot slower than plain old FTP that's for sure. rsync over ssh would definitely be a better option IMHO.
    Karl Austin :: KDA Web Services Ltd.
    UK Business Hosting and Managed Servers - Hosting for Business Users :: 0800 5429 764
    Call us today and ask about our hosting solutions.

  5. #5
    Join Date
    Aug 2001
    Location
    Denver, CO
    Posts
    273
    Thanks very much.. think cwrysnc might just do it.. got it started just wish it was showing status.. probally just need a command line parameter for that.. (it's actually topped out my upload at 1.5 mbit.. I didn't even know that was possible)

    If not I'll try filezilla, thanks!

  6. #6
    Join Date
    Jul 2006
    Location
    Detroit, MI
    Posts
    1,955
    Use FTPS and save yourself the headaches.



    Regards,

  7. #7
    Join Date
    Dec 2002
    Location
    chica go go
    Posts
    11,858
    I'd split it up into multi-part rars, and then use the unrar e whatever.rar to extract them all properly.

  8. #8
    Join Date
    Aug 2001
    Location
    Denver, CO
    Posts
    273
    Rar's aren't gonna work.. 1st I don't have that much space.. 2nd it's just over the top.. (though I certainly see the bonus of having it resume to the one file)

    cwrsync seems to have timed itself out/stopped after only a few hours.. and it seems pointless to relaunch it without naming folder parameters 1st.. and it spends a long time just skipping past data already uploaded..

    Trying filezilla but I think I've tried this program before and had issues...

    Karl did you mean use regular ftp w/ ssl? At this point with all these problems I'm willing to do that.. (I don't even care about security) it seems the upload headache just goes on and on.. I'm 200 Gb of pictures, the only real question is can any normal ftp client handle this load..

  9. #9
    Join Date
    Jul 2006
    Location
    Detroit, MI
    Posts
    1,955
    Quote Originally Posted by Ophelus View Post

    Karl did you mean use regular ftp w/ ssl?


    Yep.










    Kind Regards,

  10. #10
    Join Date
    Feb 2002
    Location
    New York, NY
    Posts
    4,612
    If that 200GB is made up of a large number of files, definitely use Rsync if it is available. The Rsync protocol packs the data in an efficient way, compared to FTP or SFTP. For a typical server with 200GB of data, Rsync can easily complete in half the time.
    Scott Burns, President
    BQ Internet Corporation
    Remote Rsync and FTP backup solutions
    *** http://www.bqbackup.com/ ***

  11. #11
    Join Date
    Feb 2005
    Location
    Australia
    Posts
    5,842
    Quote Originally Posted by Ophelus View Post
    cwrsync seems to have timed itself out/stopped after only a few hours.. and it seems pointless to relaunch it without naming folder parameters 1st.. and it spends a long time just skipping past data already uploaded..
    It may be that your ISP is restricting your connection. Still, you can just run the same command repeatedly - it should have transferred a fair amount in a few hours. Use the v flag (verbose) to see what it's doing. (Or as a general rule use -avz to transfer entire directories, tell you what it's doing and compress the data for transfer.)
    Chris

    "Some problems are so complex that you have to be highly intelligent and well informed just to be undecided about them." - Laurence J. Peter

  12. #12
    Join Date
    Aug 2000
    Location
    Sheffield, South Yorks
    Posts
    3,480
    If rsync stopped, start it again - it'll pickup where it left off, it won't look like it, as it'll do a comparisson of files already done, but it'll fly through those, then start uploading from where it left off. No need to specify a list of folders to skip.
    Karl Austin :: KDA Web Services Ltd.
    UK Business Hosting and Managed Servers - Hosting for Business Users :: 0800 5429 764
    Call us today and ask about our hosting solutions.

  13. #13
    Join Date
    Oct 2006
    Location
    uk
    Posts
    448
    Find an old version (they start with 3.) of winscp. The new "beta" versions suck in terms of reliabily in my experience. Version 3 is rock solid though.

  14. #14
    Join Date
    Aug 2003
    Location
    Chicago, IL USA
    Posts
    2,781
    Use Rsync as bqinternet suggested, makes things a lot simpler.
    CloudRck.com - Host on Cloudrck
    Unmetered VPS Solutions at it's finest

  15. #15
    Join Date
    Mar 2008
    Location
    Miami, FL
    Posts
    40
    Quote Originally Posted by doc_flabby View Post
    Find an old version (they start with 3.) of winscp. The new "beta" versions suck in terms of reliabily in my experience. Version 3 is rock solid though.
    I agree the new versions of WinSCP tend to choke on large file transfers. I'm still using v3.0 myself.

    Btw you can find it here:

    http://www.oldversion.com/program.php?n=winscp
    <<please see forum rules for signature formatting guidelines>>

  16. #16
    Join Date
    Aug 2001
    Location
    Denver, CO
    Posts
    273
    I'm giving rysnc another try and being specific about folders the data is in rather then not.. it's taking 20+ minutes to right now verify about 1/3 of the data being upped for one folder.. then it should start uploading the 2/3

    Your winscp suggestion makes a lot of sense.. I was using Beta 4.1, I'll give that another try if rsync gives me continued headaches..

    It appears my initial figures of 1/4 of the data being done was wrong.. it appears to be above 1/3 now give or take.. most of that was done by winscp

  17. #17
    Join Date
    Aug 2001
    Location
    Denver, CO
    Posts
    273
    I tried winscp 3 (the old one wasn't labeled beta but says beta in about) beta and I didn't get any kind of status or mbit much showing on my router.. so I think that is a failure..

    I'm loving rsync because it is resuming without haveing to go over the files over and over again.. but it's still pausing on me..

    I was wondering if people knew the correct syntax for using cwrysnc seems that the only 2 things I'm looking for..
    is remembering the password (saw a few password commands but was confused) and being able to restart it; if it stops (saw some timeout commands but was also confused on if those would help)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •