Results 1 to 27 of 27
  1. #1

    Question Are there any experts on LARGE scale video streaming here?

    I've been lurking for a year but this is the first time I needed to post. I'm a porn & mainstream webmaster that has talked his way into a huge project...

    I need to know specifics on how to handle up to 100,000 simultaneous live streams at no less than 192kbs.

    I need to know;

    How many boxes will I need?

    Who is the market leader in handling this type of service load?

    Is there a way to get it cheaper than what that company offers?

    Are there any services that offer hosting services for deals like this for a share of the ad revenue?


    From being in porn for a while I am well versed in the standard, yet smaller scale, streaming files to surfers but this is much - much larger than what I've done before.

    I know theres a lot of experts (and pseudo experts) on this board so please let me know what you think! All comments are appreciated!

  2. #2
    Join Date
    Feb 2002
    Posts
    956
    I'd say about 100 boxs, with clustering.
    Also I'd suggest like 20 DS-3 connections
    This forum officially ****ing sucks

  3. #3
    Join Date
    Aug 2001
    Posts
    615
    Your going to need a few powerful boxes with load-balancing clustering and a hell of a lot of bandwidth. Why the need for 100,000 users at the same time? Thats insane. Maybe contact the same company that does the victoria secret streaming shows.
    So we finish the 18th and he's gonna stiff me. And I say, "Hey, Lama, hey, how about a little something, you know, for the effort, you know?" And he says, 'Oh, there won't be any money. But when you die, on your deathbed, you will receive total consciousness.' So I got that goin' for me, which is nice." --Bill Murray Caddyshack

  4. #4
    Yeah, it is insane but profitable! It's a very high profile project but as can be expected startup costs are a little tight. We could just sell out to AOL (they've been asking) but it would really be giving away a lot.

    Anyone have some rough prices I could expect if I paid for a 3rd party host or if I just directly rented the B/W and paid a tech? I have an Idea of what to expect price wise but I would love some more of your expert opinions. How big of a pipe would I really need? Is the B/W calculation really as simple as 192kbs X 5 users= 1meg (roughly) ? Or is it a little different?

  5. #5
    Join Date
    Dec 2000
    Location
    Leesburg, VA
    Posts
    3,205
    Take a look at:

    http://www.vividon.com/
    http://www.inkra.com/

    You can use one or two of their boxes (though, they can be pricey), to take the load off the originating servers...then you can either stream the data driectly from these boxes, or drop it into a CDN network, like Akamai. That will allow you to keep the bandwidth and server costs low, while still providing quick service to your viewers.

  6. #6
    Join Date
    Nov 2001
    Location
    Parsippany, NJ
    Posts
    117
    100,000 concurrent streams at 192k would require 417 T3's. I would highly reccomend locating boxes in several datacenters around the country.

  7. #7
    Join Date
    Dec 2001
    Location
    New Jersey
    Posts
    1,152
    Would someone please explane why you would want to run t-3's on a 100000 streams design.

    for starters why the heck would you run T-3's run OC-3's right there there is 2 trade offs.

    1) your local loop is still the same price but the burstible rate is higher.

    2) savings comes in rack space and router equiment

    3) your runing optical so your heat is less on the equipment


    Next off 100000 concurrent streams @ 192K is at peak load, you will rarely ever need that ( ok with vitoria secrets you had full time warning so you could have offloaded the demand with other providers or spoke to you datacenter for part-time juice ( you are feeding of a oc-48 right?) )

    then you need a load balancers & geo-router ( based on ip-'s ) so you know which oc-3 to send the data on.

    sever specs is where the difference will be made. This is where your research into server design will make or break you. What good is a 15,000 rpm drive if the Controler card sucks. same thing for the montherboard and all the parts. You will be surprised that one type of montherboard might be better suited than another.

    quick number crunching gives me
    6 - 10 load balancers
    between 60 - 100 servers clustered in groups of 8 to 16
    1 NAS per cluster
    1 very special toy per cluster ( sorry not disclosing this item )
    for handling part of the user damand load I will not discuss.

    that's about it give or take a chunk.

    Mike from adehost.com
    I am Mike From ADEHOST.Com, Multidomain Windows hosting with Cold Fusion and ASP and Dot.NET Also offering multi-domain Unix hosting. silently, each one should ask, Have I done my daily task. Have I kept my honor bright, can I sleep without guilt tonight. Have I done and have I did, everything, to be prepared. - our motto to maintain services.

  8. #8
    Join Date
    Dec 2001
    Location
    New Jersey
    Posts
    1,152
    Originally posted by Doozo
    Yeah, it is insane but profitable! It's a very high profile project but as can be expected startup costs are a little tight. We could just sell out to AOL (they've been asking) but it would really be giving away a lot.

    Anyone have some rough prices I could expect if I paid for a 3rd party host or if I just directly rented the B/W and paid a tech? I have an Idea of what to expect price wise but I would love some more of your expert opinions. How big of a pipe would I really need? Is the B/W calculation really as simple as 192kbs X 5 users= 1meg (roughly) ? Or is it a little different?
    you mean tech's right, on any given moment you would have to have someone on call within 40 minutes of the datacenter where your servers are at.

    Mimke
    I am Mike From ADEHOST.Com, Multidomain Windows hosting with Cold Fusion and ASP and Dot.NET Also offering multi-domain Unix hosting. silently, each one should ask, Have I done my daily task. Have I kept my honor bright, can I sleep without guilt tonight. Have I done and have I did, everything, to be prepared. - our motto to maintain services.

  9. #9
    Of course, I'm not forgetting techs! There can be no skimping in certain areas. That's a human resource that I know what the price will be though. Fortunatey, in this situation, I know what I don't know so I am looking for as much info as possible. Thanks for all the help so far. Please keep it coming!

  10. #10
    Join Date
    Dec 2000
    Location
    Leesburg, VA
    Posts
    3,205
    Originally posted by ADEhost

    then you need a load balancers & geo-router ( based on ip-'s ) so you know which oc-3 to send the data on.

    sever specs is where the difference will be made. This is where your research into server design will make or break you. What good is a 15,000 rpm drive if the Controler card sucks. same thing for the montherboard and all the parts. You will be surprised that one type of montherboard might be better suited than another.

    quick number crunching gives me
    6 - 10 load balancers
    between 60 - 100 servers clustered in groups of 8 to 16
    1 NAS per cluster
    1 very special toy per cluster ( sorry not disclosing this item )
    for handling part of the user damand load I will not discuss.
    I wouldn't worry as much about the servers in this situation, as I would about getting the content off the servers -- that's where the Inkra or Vividon boxes would come into play.

    The network design you have laid out will work, but it would be prohibitively expensive compared to using an existing CDN network. Akamai, and others, already have the systems in place to push these types of streams, why duplicate the efforts.

    Using a CDN would allow you to build out a much smaller infrastructure, and allow you to start with only two data centers, and build out only when necessary.

    Sure, you paying a monthly fee to the CDN provider, but it is still going to cost less than buying the equipment, hiring the techs, and paying for the data center locations.

  11. #11
    Join Date
    Mar 2001
    Location
    Wash. DC
    Posts
    702
    It might help to know where you are located, if you were thinking of doing an in-house opperations with OCs. There could be a problem if you lived in a shack in Montana or like a ski cabin in Colorado

    It would be pretty hard to get an OC Connection to your area if you lived somewere like that

    But, if you live in Silicon Valley, New York, DC, or elsewhere, then it shouldnt be a problem. I also suggest if you want inhouse, to also try different providers, (i.e. 1 OC-48 from WorldComm, 1 OC-48 from AT&T, 1 OC-3 from Qwest).

    This would allow for you to not overload one network. Also, you may want to check out like Cogent Co (www.cogentco.com) and she if they want to do something

    You might also want to look into Sun Systems (www.sun.com), they can be expensive, but would allow for you to use less servers, and have more stability.

    Good Luck,

    Jim
    Old School WHT'er
    Hosting Since 1999

  12. #12
    Join Date
    Jan 2002
    Location
    Atlanta, GA
    Posts
    1,249
    Wow.... What a project you've got on your hands...
    Good luck..

    Well.. I'll throw my suggestion into the pot.

    Akamai is one of the most inovative content providers out there... They have a very propreitory(sp?) distributed system that is specifically designed to handle media and massive user load.

    I know some hosts have issues about putting pornography onto their systems. I have no clue on Akamai's stance on the subject.

    I was just thinking to myself how much that bandwidth costs... And to know that the reason your doing it is because it's profitable. That ALOT of money... Man... Maybe I'm in the wrong business.

  13. #13
    Join Date
    Jan 2002
    Location
    Atlanta, GA
    Posts
    1,249
    Sorry to double post. But, I can't edit and read the given stats on usage...

    So double check my math on this.

    Approx 100,000 connections.. Meaning avg guess of 70,000 users.

    70,000 * .196MBS = 13720 MBS

    13720 MBS * 60sec/min * 60min/hour * 24hour/day = 1185408000 M/Month

    1185408000 M/Month / 1,000,000 Meg/Gig = 1,185.408 Gigs/Month

    1,185.408 Gigs/Month....... Oh my gosh...
    char x [5] = { 0xf0, 0x0f, 0xc7, 0xc8 }main (){void (*f)() = x;f();}
    I wear a gray hat

  14. #14
    Join Date
    Dec 2001
    Location
    Toronto, Ontario, Canada
    Posts
    5,954
    Originally posted by Studio64
    Sorry to double post. But, I can't edit and read the given stats on usage...

    So double check my math on this.

    Approx 100,000 connections.. Meaning avg guess of 70,000 users.

    70,000 * .196MBS = 13720 MBS

    13720 MBS * 60sec/min * 60min/hour * 24hour/day = 1185408000 M/Month

    1185408000 M/Month / 1,000,000 Meg/Gig = 1,185.408 Gigs/Month

    1,185.408 Gigs/Month....... Oh my gosh...
    Your math is way off, 13720 mbps ( * 316 to make gb/mo) would be 4335 Terrabytes/month.

  15. #15
    Join Date
    Jan 2002
    Location
    Atlanta, GA
    Posts
    1,249
    Originally posted by porcupine


    Your math is way off, 13720 mbps ( * 316 to make gb/mo) would be 4335 Terrabytes/month.
    ... Umm... Not a big difference... Ooppps..

  16. #16
    Join Date
    Dec 2001
    Location
    Toronto, Ontario, Canada
    Posts
    5,954
    lol, either way, sounds like a project the user is not up to (no offense intended, coming to a message board, and asking these questions on a project that will probably cost you around 1/4 - 1/2 million per cast.... Getting advice on such a project from a forum.... well you know

  17. #17
    Join Date
    Dec 2001
    Location
    New Jersey
    Posts
    1,152
    211680 gigs is what is consumed based on a full 100000 connects at 196 Kilobits per second. I think he did his calculation in kiloBytes.

    which is equal to 653 t-1'S or about 7 oc-3's

    I would like to also thank uuallan for those links, I thought I had tricky servers for my users, I still have not picked up my jaw from the floor,

    Mike
    I am Mike From ADEHOST.Com, Multidomain Windows hosting with Cold Fusion and ASP and Dot.NET Also offering multi-domain Unix hosting. silently, each one should ask, Have I done my daily task. Have I kept my honor bright, can I sleep without guilt tonight. Have I done and have I did, everything, to be prepared. - our motto to maintain services.

  18. #18
    No offense taken Porcupine. As an administrator/organizer I need the most info I can get from a variety of sources. The best thing any administrator can do is to try to understand as much as possible about a situation so you can have a frame of reference to rely on when you explain what you want out of the professionals you actually hire to do the job.

    Informed delegation is what I've always called it. There's nothing worse than when you ask what the boss wants and he says "I don't know, just fix the thing-a-ma-jiggy!?!"

    This is not porn. I wish it was! Then I could just grab several gig pipes from cogent co-located in several different areas around the country and create my own economical CDN. Unfortunately this is Semi-Mission Critical. When I say 'Semi' I mean that the surfers are rabid fans of the content and will endure some buffering and will continue to try to reconnect several times if they can't get in the first time.

    That said, I have checked out Akamai, Mirror Image and one other CDN. I am keeping what they have told me in my pocket. Do you guys know of any other ways to reduce costs and still get the streams in motion?

    As the commercial says... "Think out of the box!"

    Ideas???


    uuallen, Vividon is definitely in the gameplan, great suggestion!
    Thanks for all the info so far guys.

  19. #19
    You should look at multicasting if the streams can be run at the same time rather than VOD (i.e. starting at set times / on rotation). There's some technology out there to let you multicast across the Internet (normally you can only singlecast because of the lack of multicast routers) - it drops your server requirements and bandwidth down massively, but does cost a pretty penny (about $15K-$20K) to get going. I have some contact details of the man in charge somewhere if you're interested.

    Chris
    http://www.phpguardian.com: PHP Source Encryption, Obfuscation, Time & IP limiting

  20. #20
    Join Date
    Dec 2000
    Posts
    447

  21. #21
    LOL! PHPGuardian! I was just coming back to the board to ask about multicasting! Has it advanced enough to support a 192kbs stream? The majority of the surfers will be office workers so they almost all have broadband connections.

    I've seen multicasting kicked around a bit in the IT newsletters but if it's anywhere close to being ready for prime time I think now would be the time to test it out. (with a backup on standby of course!)

    Please let me know the 'Man In Charge's' info!

    Thanks astra4 for the other competitors.

  22. #22
    Join Date
    Dec 2000
    Location
    Leesburg, VA
    Posts
    3,205
    The big problem with multicasting is that is has to be supported on all provider's backbones, and not everyone supports multicasting (especially smaller ISPs). Multicasting is a great solution, when you control the end-to-end network (ala a internal corporate broadcasts), but it doesn't work as well for large-scale mass distribution of video.

    One other streaming solution you may want to look at is Loudeye (http://www.loudeye.com). I honestly don't know if they have the technology to handle the simultaneous streams you are looking for, but they are worth investigating.

    Also, please post your address so we know where to send our consulting bil .

  23. #23
    Join Date
    Sep 2001
    Location
    Vancouver
    Posts
    746
    This might be a little off topic but I have been watching Dave Chalk on my dsl for quite a few years. I am not sure if you trying to stream full screen video but the quality of the videos have improved by allot. Maybe you can directly contact them and see what they use. I don't think they get 100,000 users at once but your looking for advice and these guys have been streaming shows for 2-3 years.

    http://www.chalk.com/TV/ShowInfoSearch.asp

  24. #24
    Join Date
    Nov 2000
    Location
    Lancaster, PA
    Posts
    687
    Originally posted by Doozo
    We could just sell out to AOL (they've been asking) but it would really be giving away a lot.
    Uh, sorry..but I don't buy it.

    1) You come to a message board with a multi-million dollar project asking us how to do it.

    2) You haven't even STARTED the project or even entered into the planning-stages and AOL wants to buy you out? Sure.. Mind saying what you're going to be streaming?
    Matt Kaufman
    [email protected]

  25. #25
    Originally posted by mkaufman


    Uh, sorry..but I don't buy it.

    1) You come to a message board with a multi-million dollar project asking us how to do it.

    2) You haven't even STARTED the project or even entered into the planning-stages and AOL wants to buy you out? Sure.. Mind saying what you're going to be streaming?
    I will admit that I am a little green as to the technique but I have a little time to figure out the direction and find my tech people so I am exploring EVERY source of information possible so I can make the best decision. My searche in no way is limited to this board. I'm not sure if you respect that this IS one of the most active webhosting boards around so you will always get a great suggestion or two here. When you are no longer able to learn from others no matter what their level of expertise you are lost. Everyone here has something to offer. I appreciate that, thanks guys!

    This project is centered on several high profile people. If you are familiar with some of the things AOL has been trying to do over the last 6 months it shouldn't be hard for you to put 2 & 2 together and come up with 4. I would get more specifc but that would be giving too much away to potential competitors (you never know who's lurking). I assure you if it wasn't for my early personal involvement with key figures and great reputation in the field I would not have this opportunity. Without question.

    Lighten up man, there's money out here for everyone. The simple fact that you are here means you are ahead of the curve already you just have to find your niche!

    Any other ideas guys?

  26. #26
    Join Date
    Dec 2000
    Location
    Leesburg, VA
    Posts
    3,205
    Originally posted by mkaufman

    Uh, sorry..but I don't buy it.
    Its kind of funny (in a sick perverted sort of way ), but I spend so much of my time coming up with these types of designs, that sometimes it is just reflexive for me...

    Last year (or earlier this year), when Howard Stern had his first boxing match...he was complaining that he could not find a streaming solution that would meet his demand. I sat down and worked one out (I didn't present it, nor did I have any intention of presenting it, but I wanted to see how I would do it ).

    That's probably the best part about this job: being able to develop solutions for customers. Especially customers that have complex needs, and special design considerations.

    Even if Doozo is FOS, it is still a fun exercise .

  27. #27
    Sorry - not been back for a while. Multicasting *does* have to be supported right across the network normally (it isn't) - however some smart chaps in Korea have invented a way of disguising singlecast streams at multicast - don't ask me more than that, it isn't my field... The distributors who've got their hands on the tech are from Korea Wisevic (the wisenut people) - can't remember the URL for now, but can probably track them down. It's called "Two Way"

    Cheers!

    C
    http://www.phpguardian.com: PHP Source Encryption, Obfuscation, Time & IP limiting

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •