Results 1 to 11 of 11
  1. #1

    Coping with extreme bandwidth spikes.

    One of the sites I manage and administrate is looking at receiving some major traffic in the next 6 months, which is expected to far surpass what we anticipated originally, and also beyond what was expected for growth in the next 1-3 months.

    The problem comes in that the site is being heavily promoted across the internet, and is going to be featured on a major cable television station. The backstory behind this and exact details are irrelvant; lets just suffice to say that it will be receiving a lot of hits, and that these hits are not uniformly distributed across the day.

    My question to you is this: how do you ensure sufficient bandwidth to keep a site online when it is under extremely heavy traffic, which tends to come in bursts?

    As a whole, the limiting factor is not page rendering - the site is load balanced among two servers in two different datacenters using round robin DNS. The frontend can handle 122 pgs/sec and 88 pgs/sec among the two servers, respectively, as determined by apachebench. This test was made before the new caching system was put into place, which, although untested, will generate pages on the fly without a database, but will regenerate pages when they become updated (as determined by a CRON job). Thus, while the origins are based on a PHP/MySQL user interface, it would be a falsehood to represent the site as being generated dynamically, since Apache is really only serving static HTML.

    What will bring the servers to their knees is the amount of data that is projected to be transferred in such a short period of time. The featured item is a 8 MB flash file, and numerous other older files occur. So, as projected, each user will be loading at minimum of 8MB, plus additional traffic for pageloads, site graphics, and older flash/video files. Currently both servers are 100Mbit FDX.

    What I am interested in is a cost effective solution to handle the bandwidth spikes. This will not involve getting a 100Mbit unmetered, or anything crazy like that, since the bandwidth usage comes in spikes.


    My current plan of action includes the following, please feel free to evaluate it and offer suggestions/criticism:
    • I will be getting a third server - a 20Mbit unmetered box from OrangeFiber. It won't be exclusively for this task alone, but for media serving in general. Its certainly not the solution.
    • I have two other webmasters who have offered to colocate the newest file. They also have 100Mbit FDX pipes.
    • I am planning on purchasing three shared hosting accounts, all from different providers. (Verio, Dreamhost, and LiquidWeb). It is worth noting that all these providers are also on 100Mbit FDX connections.
    • I will be mirroring it on my development box, connected at 10Mbit FDX.
    • All of the shared/borrowed mirrors will be monitored once every five minutes with a CRON job checking for proper connectivity and operational status. Nonfunctioning mirrors will be removed automatically.

    This will mirror the files across 9 machines, with 730Mbit of theoretical upstream capacity, with the psuedorandom mirroring favoring pulling the files from the mirrors, not the two boxes that the site itself runs on. The mirrors are also weighted by their connection speed, with each machine having a chance in 730 equal to its connection speed in megabits. This, I hope, will prevent connection saturation.

    It is worth noting that the all the servers, with the exception of my secondary content server and one of the borrowed servers (both at ThePlanet), all the servers reside in different datacenters.

    BiTTorrent is too complex for the users, and will only cause more confusion. Please don't suggest it.

    This is an interesting problem, and I look forward to hearing your input on it.

  2. #2
    Join Date
    Jan 2004
    Location
    Texas
    Posts
    1,556
    Just one tidbit to think about, the shared hosting accounts may very well get a bit annoyed if you start to use alot of bandwidth off of them, and most will prohibit you from doing what you are planning on doing, also, they are not 100 FDX just for you, but that is shared among many other shared accounts that will be competing for that bandwidth and server resources, so I wouldn't really count on those too much to alleviate any of the burden from the other servers..

    One other thing, do you have any idea as to how much bandwidth you will be using, or how many viewers you will be recieving?

    Personal opinion, and I'm sure I'll be corrected by someone, but if the flash is the main bandwidth eater and the main resource eater, than if you have your front end sites serving the actual pages, and a bunch of other servers serving the flash, there shouldn't be alot of load or bandwidth on the front end servers. A few unmetered (20 Mb/s would be sufficient) spread around should be able to server your bandwidth needs for the flash. I didn't gleen from your post what the database was used for exactly, but if it is going to be used heavily, then you would benefit by putting it on it's own machine.

    I do believe that this solution would be sufficient for you.
    Last edited by lumbyjj; 05-04-2005 at 03:52 AM.
    James Lumby

  3. #3
    Join Date
    Feb 2004
    Posts
    634
    This is the perfect kind of situation to consider a Content Distribution Network (CDN). There are several vendors that could handle this at a variety of price points. Some to consider include "pureplay" CDN vendors such as Akamai/SpeedEra, MirrorImage & CacheNetworks, or network operators that offer CDN services like Globix, Peer1, Limelight, AT&T & NaviSite.

  4. #4
    Originally posted by lumbyjj
    Just one tidbit to think about, the shared hosting accounts may very well get a bit annoyed if you start to use alot of bandwidth off of them, and most will prohibit you from doing what you are planning on doing
    Not really - these hosts offer a monthly bandwidth quota, which I am free to use any way I choose. The account will be disabled once it hits quota, but then again, these exist to cope with spikes, not as a primary method of distribution. I can just renew the accounts as I need, since the low cost per GB Xfer makes this an economical way to buy bandwidth.

    It is also worth noting that the Verio account is an unlimited business account, and another webmaster I know (provider of mirror #2, but on a different box) has done this with them twice before, and not heard a thing from them. Their TOS doesn't prohibit this.


    also, they are not 100 FDX just for you, but that is shared among many other shared accounts that will be competing for that bandwidth and server resources, so I wouldn't really count on those too much to alleviate any of the burden from the other servers..
    None of these hosts have a problem with saturated upstream, which is ususual among huge hosts. While I still would have to compete for resources among others on the same server, its nowhere near what one typically expects from shared performance - large sustained transfers utilizing many sockets from a large number of remote machines is not swamped by "typical" traffic. I've done this before, just not on this scale, and it is an effective solution.


    One other thing, do you have any idea as to how much bandwidth you will be using, or how many viewers you will be recieving?
    We estimate 130000 per day.


    Personal opinion, and I'm sure I'll be corrected by someone, but if the flash is the main bandwidth eater and the main resource eater, than if you have your front end sites serving the actual pages, and a bunch of other servers serving the flash, there shouldn't be alot of load or bandwidth on the front end servers. A few unmetered (20 Mb/s would be sufficient) spread around should be able to server your bandwidth needs for the flash. I didn't gleen from your post what the database was used for exactly, but if it is going to be used heavily, then you would benefit by putting it on it's own machine.

    I do believe that this solution would be sufficient for you.
    I think you misunderstood what I was trying to say - the only resource issue is the large files. The database is only used for generating the pages once - my caching system ensures that high-traffic pages are not re-built from SQL on every page load.

    The problem with this being so sporadic makes the use of unmetered lines extremely wasteful. We won't need all that bandwidth all the time, so why have unmetered?

    CDNs were talking several thousand for this. I don't want to pay that much for some ultrafancy service to cache flash, especially when I can do it myself.

  5. #5
    Join Date
    Oct 2004
    Location
    Nevada
    Posts
    887
    We estimate 130,000 per day.
    .....
    CDNs were talking several thousand for this. I don't want to pay that much for some ultrafancy service to cache flash, especially when I can do it myself.
    The cost of the CDNs should be minimal compared to what you are charging your client for a project of this size.

    Also, you might want to contact different CDNs that have lower pricing per GB transferred

  6. #6
    Originally posted by Dennis Nugent
    The cost of the CDNs should be minimal compared to what you are charging your client for a project of this size.

    Also, you might want to contact different CDNs that have lower pricing per GB transferred
    I can do it for significantly cheaper than they can, so it is worth my while to do so. This is the same reason I buy unmanaged servers from datacenters - I'm a systems administrator and designer, not merely a coordinator of other people performing the services for me.

  7. #7
    Join Date
    Feb 2004
    Posts
    634
    If price is the primary concern, you might want to contact CacheNetworks and Peer1. They seem to have the lowest pricing among CDN vendors; Akamai/SpeedEra is definitely at the high end, with MirrorImage in the middle. CacheNetworks might be a good fit; they have a product specifically geared towards single file downloads, though I have no experience with them. A CDN will definitely offer end users better performance than your setup, but if that cost isn't worth it to you then stick with what you have.

    Have you projected or analyzed existing bandwidth data to figure out what 95% billing and/or GBs delivered would be on this project?

  8. Even multi-billion dollar companies use CDN's for content distribution - they could certainly do it themselves, but the companies that are dedicated to CDN know how to do it better, easier.
    TrapZero Hosting
    Web, Forum, E-mail hosting and more for business and personal users. Accepting PayPal.

    FMVperformance Automotive Communities

  9. #9
    Join Date
    Jun 2001
    Location
    Denver, CO
    Posts
    3,301
    I concur with others. Let a CDN network worry about distributing your 8MB flash file. It may cost some, but the solution you've proposed above seems hodge-podge, overly complex, and poorly designed - particularly the shared hosting and the 10Mb/sec bits.
    Jay Sudowski // Handy Networks LLC // Co-Founder & CTO
    AS30475 - Level(3), HE, Telia, XO and Cogent. Noction optimized network.
    Offering Dedicated Server and Colocation Hosting from our SSAE 16 SOC 2, Type 2 Certified Data Center.
    Current specials here. Check them out.

  10. #10
    Join Date
    Oct 2002
    Posts
    226
    As stated above, you should contact Peer1. They're selling their CDN product very cheaply right now. They'll even let you test it for free. Personally, I would side step the DYI route. For just a bit more you can get a proven solution.

    Ryan

  11. #11
    Shared accounts is not the way to go, most hosts will suspend your account if you abuse your account (i.e. go over a certain amount of server resources). Using a CDN product would the best way to, especially considering they have whole networks designed for this purpose. If you wish to do it yourself, I would recommend using cheap dedicated servers across a variety of locations, NOT shared hosting accounts.
    crucialparadigm - Affordable, Reliable, Professional :
    Web Hosting
    24/7 Support Web Hosting Reseller Hosting Cloud/VPS Plans Dedicated Servers

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •