Results 1 to 10 of 10
  1. #1

    cloud storage bandwidth distribution

    Hello,

    I have visited some website of different cloud providers but I am still unsure if there is a company that offers that kind of storage service that I am searching for. I would like to start with a short description of the impression I got of how a cloud storage works. Please feel free to correct me if I got something wrong.

    If a file gets uploaded to the "cloud", it will be stored on a special storage server, e.g. a SAN. When this file gets downloaded, a server where my node is running on, will get the file from the SAN and forward it to the destination/user that requested the file. The download speed is therefore limited by this server.

    Is there a cloud storage that will distribute the bandwidth among several servers so the actual speed is not limited by the server my cloud node is on, WITHOUT the help of an actual CDN ?

    If not, is there a software that can do something like this ?

    Thanks for your help in advance.

    Best regards,
    alex18

  2. #2
    Hello,

    Did I get something wrong with my impression on how cloud storage works or is it that nobody knows a provider that fits my description ?

    best regards
    Alex18

  3. #3
    Join Date
    Aug 2007
    Posts
    352
    Let me address your questions about cloud storage:

    1. "...If a file gets uploaded to the "cloud", it will be stored on a special storage server, e.g. a SAN. When this file gets downloaded, a server where my node is running on, will get the file from the SAN and forward it to the destination/user that requested the file. The download speed is therefore limited by this server."

    Cloud storage can be something complicated like a cluster backend SANs or something less exotic like a bunch of linux machines with a shared distributed file system.

    Depending on what kind of system is implemented, there might already be redundancy and copies of your data floating around in their machine pool. Most distributed file systems divide data up so you data is found in pieces, parts and in multiple places plus a RAID like copy just in case something goes wrong. You need to ask your potential providers about what sort of cloud storage they actually have and how the front end nodes connect to the backend storage.

    No matter what kind, there will be bottlenecks somewhere. Most users are way beneath the bottleneck limits due to cost typically. You internet access to the remote storage is always the bottleneck most obvious. If you are using a solution like mentioned to serve files to users online directly then your bottleneck will their connectivity followed by the performance and load of the front end machines serving the documents to the users.

    2. "Is there a cloud storage that will distribute the bandwidth among several servers so the actual speed is not limited by the server my cloud node is on"

    Well depending again on your providers model we can formulate a response that is accurate. You at minimum are wanting a solution that sounds like multiple front end access machines to multiple storage blocks. Typically the front ends are going to be served via iSCSI, TCPIP, or similar network based storage methods. So you limit will be the NIC speed (1 Gbit), overhead losses and load on both the front end and backend storage.

    What sort of data are you intending on storing and how are folks going to access it?

    Provide some more information and other folks will chime in on this.

  4. #4
    Thank you for your excellent reply ! You helped me a lot !

    I am looking for a provider that gives me the opportunity to resell their cloud storage that I can offer as "file server" to customers who need space for their files that will be downloaded by others. Basically a rapidshare like service but with no file size limitation. Also, a traffic limit feature would be nice !

    Thanks again !

    regards
    Alex18

  5. #5
    Join Date
    Aug 2007
    Posts
    352
    Well in theory all providers should be able to handle your needs. Might create lots of problems for them though (huge file sizes, type of content, storage needs, etc.)

    Couple of gotchas I think however:

    1. Everything has limitations.
    2. No file size limitations is unurealistic. All storage has limits and to be honest gigantic files are a crush to IO and you are talking about jamming internet pipes moving stuff like that.

    This sort of project will likely be cost prohibitive using cloud technology or even a CDN solution. We use both for smaller scale stuff.

    Most sites at scale like this are using clustered dedicated servers with their own internal storage (due to high IO requirements and piling up of front end requests via web).

    I'd recommend looking for a dedicated server provider who can bundle many TB's of drives in a server and look at the cost of say two of those to start. You are going to probably near term need that provider to sell you front end server nodes to deal with web customers downloading these files, a VLAN switch and wiring of the front end nodes to your big 24TB server storage machine.

    SwiftNoc has a 24TB offering with different large bandwidth chunks:
    http://www.webhostingtalk.com/showth...highlight=24TB

    This isn't going to be cheap

    24TB server x 2:
    INTEL XEON X3450 2660MHz 8MB quadcore , 8GB RAM, 12 x 1.5 GB HDD
    5 usable IP's in own VLAN and SUBNET - Install your own OS
    Location: Netherlands. Server with IPMI, KVM/IP and Remote media / reboot
    - 500Mbps - €699/month w/€699 setup
    - 750Mbps - €899/month w/€899 setup
    - 1000Mbps - €999/month w/€999 setup

    + whatever front end nodes soon to scale the front side...

  6. #6
    Join Date
    Aug 2007
    Posts
    352
    Guess I should mention this also

    You should really consider front ending your solution as described with a cache solution (like Squid or Varnish).

    Your front end cache in something like this would be best utilizing dedicated SSD drive(s). That will / can deal with commonly repetitive downloaded items without smashing your overtaxed backend spinning disk storage.

  7. #7
    Thank you for your really informative answers !
    Actually I am looking for a service that could offer all this but I did not know that there were actually "that many limitations" within a cloud as the cloud always gets marketed as infinitive scalable. Although I knew that there are limitations, I thought these are much far beyond then just serving files at e.g. 1Gbps .

    Thanks again for your great answers !

    Best regards
    Alex18

  8. #8
    Join Date
    Aug 2007
    Posts
    352
    Don't believe the hype

    Everything has limitations - EVERYTHING. A jet plane, a race car, even a server cluster super computer.

    Clouds can scale just like clusters have for a decade or two. Very similar and most clouds are just really clusters with some fancier software on top.

    What you are trying to do *CAN* be done inside a cloud solution. However, the part about unlimited file size (assuming large files) will create scale issues for you and any cloud provider. Put that together with what seems like a large amount of disk space required, and it's a multiple issue.

    Dedicated servers = best cost for you. Colo would work also, but you have a good amount of hardware costs up front and then trying to find a place with the pipe and plan to handle that server ~ might prove to be time consuming and nearly same cost as the dedicated.

    Here's another provider with a similar large storage server and a good chunk of bandwidth - unfamiliar with them so ask the tough question of them:
    oloGigabit.es Stock Clearance 24TB HDD/50TB bw/2x 1000Mbps 499/m *Promotion*
    *** OWN DATACENTER IN VALENCIA ESPAŅA ***
    • Gigabit Express Servers: online within 30 min / 7 days a week

    Gigabit Express Storage Server 24TB
    - IC Desingcase 2U - 13 bays
    - Intel Core2Quad 4x2.50Ghz 8MB fsb: 1333Mhz
    - 4096MB RAM DDR2-800
    - 30GB SSD
    - 12x2000GB HDD
    - 50TB datatraffic
    - 2 x 1Gigabit Connectivity (bonding/trunk)
    - OS: Debian 5.05 32/64bits CentOS 5.5 32/64bits
    - KVM over IP (Dedicated Portal 24/7 - BIOS Access Boot Process)
    - APC over IP (Dedicated Portal 24/7 Hard-Reboot outlet)
    - SysRescueCD over IP (Dedicated Portal 24/7)

    € 499/month >> order here

    Bandwidth
    0-10 TB : € 35,00 / TB
    11-20 TB : € 33,00 / TB
    21-30 TB : € 30,00 / TB
    31-50 TB : € 25,00 / TB
    51-100TB : € 20,00 / TB
    100+ TB : Please contact us, we will make a special price.

  9. #9
    Thanks again ! I was hoping to set it up without much or worries about hardware etc. But it seems like this is not possible at the moment !

    I really appreciate all your help !

    Best regards,
    Alex18

  10. #10
    Join Date
    Aug 2007
    Posts
    352
    Yep, feel free to private message me if you have any questions or need pointed in the right direction.

    Dedicated servers are nice because you avoid worrying about the hardware. However, you need the technological proficiency to set the solution up on the hardware - or the financials to pay someone else to do this.

    There are lots of knowledgeable folks here on these forums. If you someone post in the help wanted related forum(s).

Similar Threads

  1. Replies: 11
    Last Post: 08-19-2008, 02:35 PM
  2. High-bandwidth for content distribution - ideas?
    By nafrance in forum Dedicated Server
    Replies: 26
    Last Post: 04-05-2007, 12:37 AM
  3. Bandwidth Distribution Setup
    By Genesis5 in forum Colocation, Data Centers, IP Space and Networks
    Replies: 5
    Last Post: 01-20-2005, 10:20 PM
  4. Replies: 0
    Last Post: 06-25-2003, 10:33 AM
  5. Distribution of bandwidth...
    By jesus in forum Web Hosting
    Replies: 3
    Last Post: 02-14-2003, 04:43 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •