Results 1 to 5 of 5
  1. #1
    Join Date
    Jun 2012
    Posts
    399

    Rackspace Cloudfiles API Problems

    Anyone using Rackspace CloudFiles? I am trying to upload about a half million images using their Pyrax python package (folder upload and folder sync methods of a container object), but it is incredibly slow. After a few hours just 160MB was uploaded.

    I opened a ticket and the support suggested I should try some other API bindings that are multithreaded (Pyrax apparently isn't) but I'm getting the same results with those unofficial packages too, can't upload more than one (up to 200kB) image per second, more or less. I tried to delete (because you can't delete a non-empty container) only to find out it does at the same pace.

    Single file upload bandwidth is okay, I can upload a single big file at up to 200Mbps (varying a lot, but it's acceptable).

    Is this normal? Anyone else seeing this?

    Their support is useless, not reading the tickets properly and I had to repeat myself in several posts until I explained the matter, after which they stopped posting, heh. I know there are limits in their API, they allow up to 100 API requests per second after which they throttle but this is nowhere near that limit.

  2. #2
    Join Date
    Jan 2011
    Location
    Canada
    Posts
    934
    It might be the dns lookups that are slowing down the api calls. Try temporarily putting them in hosts to see if that helps speed it up.

  3. #3
    Join Date
    Dec 2011
    Location
    Netherlands
    Posts
    831
    I've been working a lot with Rackspace CloudFiles, I used their CloudFiles python module tho, speeds is great, but I was using the IPs directly, because I know DNS lookup takes time, no problem uploading a few terabyte storage.

  4. #4
    Join Date
    Jun 2012
    Posts
    399
    It's not DNS, that was my first thought.

    But the problem here is pretty specific, I just can't tell on which end it is. Speed is great here too if I upload a single large file. I managed to put a 40GB tarball @ 200Mbps!

    The problem is if I try to upload half a million small (50kb - 200kb JPEG) files. I tried multithreaded upload with the hypercf package (that the support suggested I should try) (https://github.com/rackerroush/hypercf), and regardless of the number of threads it uploads a few files then holds for a few seconds, then uploads few files etc...


    Quote Originally Posted by Hosting4Real View Post
    I've been working a lot with Rackspace CloudFiles, I used their CloudFiles python module tho, speeds is great, but I was using the IPs directly, because I know DNS lookup takes time, no problem uploading a few terabyte storage.
    Which python module is that? I tried the official Pyrax package, it's folder upload and folder sync methods more specifically, and teh support blamed poor performance on the fact that it's single threaded... Still, even that single threaded performance should be much better than few thousand JPEG images in a few hours time...

  5. #5
    Join Date
    Dec 2011
    Location
    Netherlands
    Posts
    831
    Quote Originally Posted by HaronMedia View Post
    It's not DNS, that was my first thought.

    But the problem here is pretty specific, I just can't tell on which end it is. Speed is great here too if I upload a single large file. I managed to put a 40GB tarball @ 200Mbps!

    The problem is if I try to upload half a million small (50kb - 200kb JPEG) files. I tried multithreaded upload with the hypercf package (that the support suggested I should try) (https://github.com/rackerroush/hypercf), and regardless of the number of threads it uploads a few files then holds for a few seconds, then uploads few files etc...




    Which python module is that? I tried the official Pyrax package, it's folder upload and folder sync methods more specifically, and teh support blamed poor performance on the fact that it's single threaded... Still, even that single threaded performance should be much better than few thousand JPEG images in a few hours time...
    https://github.com/rackerlabs/python-cloudfiles

    This is the one I'm using, also an official one.

    I would somehow assume, the reason why yours is waiting for 1-2 secs for every upload, can be because it tries to reauthenticate all the time maybe?

Similar Threads

  1. Amazon S3 + Cloudfront VS Rackspace CloudFiles
    By bujji1 in forum Web Hosting
    Replies: 4
    Last Post: 06-05-2013, 06:58 AM
  2. AWS s3 with cloudfront vs rackspace cloudfiles
    By jonniejoejonson in forum Cloud Hosting
    Replies: 3
    Last Post: 02-25-2012, 09:27 PM
  3. API Problems on VPS
    By amnaa in forum VPS Hosting
    Replies: 5
    Last Post: 05-13-2011, 06:08 AM
  4. Rackspace Cloud Server API
    By netpicker9 in forum Cloud Hosting
    Replies: 2
    Last Post: 08-17-2009, 03:57 PM
  5. Rackspace Problems
    By GibbBoy in forum Dedicated Server
    Replies: 27
    Last Post: 10-07-2004, 11:01 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •