Results 1 to 21 of 21
  1. #1
    Join Date
    Apr 2002
    Location
    Houston, Texas
    Posts
    842

    Routers and Switchs O my…

    Hello,

    I am interested as in the type of switches and routers dedicated server providers use. Here is how I think they do it, but someone please correct me if I am wrong:

    The have one big router I am sure with multiple Gig-e uplink ports on it where they plug in their Gig-E lines from their many bandwidth providers into. Then I am guessing this router has many Gig-E output ports on it as well, so the provider can plug their switches into it?

    Then each rack or section of a datacenter has its own switch with many 10/100 ports, and a few Gig-E uplink ports which plug into the big fancy router?


    Is this how providers do it? I am sure there are a million ways to do it, but is there a most common way?

    Like, take EV1 for instance with their 20,000 servers, what kind of switches do they use?

    What about a small provider with a cage, a few racks, and a few Ethernet bandwidth uplinks to different carriers? One who can’t afford millions to spend on switches, and is trying to be cost effective, but wants to easily be able to expand his network?

    Can anyone recommend a brand for a good but not to expensive BGP4 router?

    Thanks, Kyle

  2. #2
    You had me at "big fancy router".

    I hate to break it to you, but you're in *way* over your head.

    The carrier class routers on a typical datacenter border (Cisco GSRs for example) will run you six figures for a set, even with a nice discount (keeping in mind you need at least two, and you need to buy the cards and the support).

    A pair of core switches typically used in a datacenter, such as a Cisco 6500 series, can also set you back six figures depending on the chassis size (for instance a 6509) and how stocked each unit is (how many populated slots).

    If you want to expand your network, lease that cage you mentioned at an existing datacenter, get a nice Cisco VXR or any number of mid-range routers, pick up a Cisco 4003 or 4006 on eBay, hire someone who knows how to set it up, and worry about your local network.

    You probably won't need a dynamic routing protocol for a one-cage straightforward hosting environment. You can run static routes and do simple load balancing.

    Building enterprise networks is for experts, especially those participating in global BGP and advertising their own routes. At best you'd be running OSPF or even, seriously, RIPv2 to your upstream provider simply to allow for ease of configuration on your side, and let them worry about handing traffic off to a BGP peer. Don't even think about getting involved in global BGP unless you know exactly what you're doing.

    Not to sound mean or disrespectful, but from the content of your posting, you're about 1-2 years away from being able to do something like this on your own without making a significant number of potentially devastating mistakes. You should look into hiring someone to handle the technical aspect for you and worry about the business side of things if you're going to move forward.

  3. #3
    Join Date
    Aug 2002
    Location
    Atlanta, GA
    Posts
    1,114
    Most larger or even mid sized facilites operate mutiple routers, we run a pair of Ciscos, but I know of one that runs several routers, it's not uncommon.

    These primary routers may then go to subrouters or they may go directly to switches.

    Most companies use Cisco, Foundry or Juniper for the larger routers. Switces are usually Cisco.

    Routers are a science of their own and should be left to people who have experiece with them. If you have little or no experience contract an experienced router tech.
    Last edited by mgphoto; 03-24-2004 at 11:14 PM.
    SiteSouth
    Atlanta, GA and Las Vegas, NV. Colocation

  4. #4
    Join Date
    Apr 2002
    Location
    Houston, Texas
    Posts
    842
    Angelvalley,

    I knew I was in over my head way before I even started

    I don't plan on worrying about the technical side of my web hosting business if I ever decide to start one; I will have many certified technicians on my staff to do that.

    I was just more curious as to how the "big boys" do it, verse actually wanting to design a network all by my self If you left it up to me to install a network for you, I would go buy you one of those all in one Linksys wireless routers with an Ethernet switch built in at Radio Shack for $99, will that do the trick for my BGP4 gigabit network uplink multi IP wireless multi frequency routing across the internet network?

    I am just joking, I know a little more about the technical side of things, but I like to stay out of it and let the technicians handle it. I am more of a business man just trying to understand what is involved in running a web hosting provider and the cost involved in doing it on a large scale.

    Thanks for the help,
    Kyle

  5. #5
    Originally posted by KyleLC23
    I don't plan on worrying about the technical side of my web hosting business if I ever decide to start one; I will have many certified technicians on my staff to do that.

    ...

    I am more of a business man just trying to understand what is involved in running a web hosting provider and the cost involved in doing it on a large scale.
    Good answers - it's obviously a Good Thing(tm) to understand as much as you can about the business. You should definitely be asking these kinds of questions. You'll probably want to find a technically savvy partner before moving ahead, however.

  6. #6
    Join Date
    Apr 2002
    Location
    Houston, Texas
    Posts
    842
    angelvalley,

    Would a person such as yourself be considered a technically savvy partner

    Thanks, Kyle

  7. #7
    Join Date
    Jan 2003
    Location
    Chicago, IL
    Posts
    6,889
    You'd normally run a conenction into dual routers, such as Juniper M20's, that would then connect to dual switches, such as 6509's. Those would output GigE to each rack which would have say a Cisco 2948. That's a fairly typical setup.
    Karl Zimmerman - Steadfast: Managed Dedicated Servers and Premium Colocation
    karl @ steadfast.net - Sales/Support: 312-602-2689
    Cloud Hosting, Managed Dedicated Servers, Chicago Colocation, and New Jersey Colocation
    Now Open in New Jersey! - Contact us for New Jersey colocation or dedicated servers

  8. #8
    Join Date
    Feb 2004
    Location
    Louisville, Kentucky
    Posts
    1,083
    Originally posted by angelvalley
    The carrier class routers on a typical datacenter border (Cisco GSRs for example) will run you six figures for a set, even with a nice discount (keeping in mind you need at least two, and you need to buy the cards and the support).
    These boxes are getting less expensive every quarter. Most folks agree these days that GSR is substantially inferior to Juniper, and you can pick up an M5 - M40, depending on needs, for way less than $100k on the aftermarket. The new boxes are becoming reasonably priced as well. Same for switches, as you mentioned.

    Building enterprise networks is for experts, especially those participating in global BGP and advertising their own routes. At best you'd be running OSPF or even, seriously, RIPv2 to your upstream provider simply to allow for ease of configuration on your side, and let them worry about handing traffic off to a BGP peer. Don't even think about getting involved in global BGP unless you know exactly what you're doing.
    I agree BGP is for experts, however, being one myself I can state with confidence that there are a lot of us around who moonlight for small ISPs and businesses who wish to multihome but don't have the necessary in-house talent.

    I don't think it's remotely safe to speak OSPF to a transit provider. The best choice is to run private-as eBGP, and if you can't get a consultant to set up your equipment; just pay your ISP to do it, and hope they do it correctly.

    You should look into hiring someone to handle the technical aspect for you and worry about the business side of things if you're going to move forward.
    Of course, the poster was referring only to the network engineering aspects of the business, I believe. Many folks are very qualified system administrators, programmers, or database admins, for example, but I wouldn't expect persons in those professions to possess dynamic routing expertiese.
    Jeff at Innovative Network Concepts / 212-981-0607 x8579 / AIM: jeffsw6
    Expert IP network consultation and operation at affordable rates
    95th Percentile Explained Rate-Limiting on Cisco IOS switches

  9. #9
    Join Date
    Feb 2003
    Location
    NY
    Posts
    586
    EV1 Im pretty sure uses Juniper for routing a bigiron 8k's for distribution.

  10. #10
    Join Date
    Dec 2001
    Location
    Toronto, Ontario, Canada
    Posts
    5,954
    Originally posted by KyleLC23
    [B]Angelvalley,

    I knew I was in over my head way before I even started

    I don't plan on worrying about the technical side of my web hosting business if I ever decide to start one; I will have many certified technicians on my staff to do that.

    ...
    ..... And when you have "many certified technicians" on your staff, they'll know exactly what to do and walk you through it far better then the WHT people ever will.

    Now back to that bit about "when I win the lottery; I will have many ladies, cars, and sportbikes at my disposal 24x7". Do you see the problem with that quote? You should . Dont forget, everyone starts big!

  11. #11
    Join Date
    Dec 2003
    Location
    Canton, Ohio
    Posts
    62
    If all your doing is hosting servers and not providing any Internet access to clients I doubt you need any BGP. I recommend running BGP on the edge routers and running OSPF inside your network. OSPF is probably overkill, but since your talking about growing quickly who knows what you might need. No one wants to change protocols.

    Leave BGP to the big boys. Back in the day I worked at an ISP and we run static routes everywhere. Looking back I have no idea why but it worked. Lots of printed out tables kept everything well documented.

    There are some good books on both OSPF and BGP which are required reading. Know them both before deciding.

  12. #12
    Originally posted by jsw6
    I don't think it's remotely safe to speak OSPF to a transit provider. The best choice is to run private-as eBGP, and if you can't get a consultant to set up your equipment; just pay your ISP to do it, and hope they do it correctly.
    [/B]
    I think I was misunderstood. I was saying that at best he'd be running OSPF or RIPv2 or heck, EIGRP, between himself and his colocation facility's network (i.e., he rents a cage, runs a routing protocol between his network and the upstream network, in this case the facility's equipment), and let the facility worry about the transit -- not to run OSPF to/between directly connected transit providers. That would be silly.

  13. #13
    Join Date
    Feb 2004
    Location
    Louisville, Kentucky
    Posts
    1,083
    Originally posted by angelvalley
    I think I was misunderstood. I was saying that at best he'd be running OSPF or RIPv2 or heck, EIGRP, between himself and his colocation facility's network (i.e., he rents a cage, runs a routing protocol between his network and the upstream network, in this case the facility's equipment), and let the facility worry about the transit -- not to run OSPF to/between directly connected transit providers. That would be silly.
    I didn't misunderstand your suggestion. The routing information protocols you've listed are not intended for inter-domain use. In addition, the level of expertiese required to operate a multi-area OSPF network, even if your equipment only participates in a single area, is in my opinion substantially higher than the level of expertiese required to run private-AS eBGP.

    Private AS eBGP is simple, easy, and above all, safe for both the ISP and the customer. I'm not advocating taking full BGP routes for a situation where two default routes will do; certainly the poster can accept 0/0 via BGP and leave it at that. What I'm saying is, using an IGP as a bastardized EGP is neither necessary nor advantageous in the context of this thread.
    Jeff at Innovative Network Concepts / 212-981-0607 x8579 / AIM: jeffsw6
    Expert IP network consultation and operation at affordable rates
    95th Percentile Explained Rate-Limiting on Cisco IOS switches

  14. #14
    Originally posted by KyleLC23
    angelvalley,

    Would a person such as yourself be considered a technically savvy partner
    I would, but I'm unofficially retired from this business. I started doing this about 10 years ago, and in the past 5 years alone I've done 3 major ground-up datacenter buildouts from the tiles to the router configs. I've had everything on my borders from Nexabits to GSRs to Riverstone and beyond. After the last one, where we designed a voice/video/data network with a lot of really bleeding-edge stuff like CableTV and VOD over multicast to a set-top box in the mix, I all but swore it off. I still run a ton of my own colo, but it's for my own company now, and someone else gets to come in at midnight and do the change controls.

    Feel free to drop me a PM if you really want to discuss it, I'm always willing to keep my ear to the track for interesting projects. It doesn't sound like something I'd be interested in, but I have plenty of my old consultants that probably would be.

    Whatever you do, my advice is to not just hire some network engineer who knows what BGP is and how to set up a network. Hire someone who has done it *all*, they understand all of the intricacies of datacenter design and are at the very least CCNPs (Cisco Certified Network Professionals). They should be able to do everything from wiring the DC power to setting up your dynamic routing protocols. The reason I say you should find a CCNP, is because there's no other way for me to tell you how to properly interview the person you're going to be dealing with. I could tell you to ask him "How are you going to configure our spanning tree setup for optimum redundancy and link utilization?" but since you don't know the answer, and a 15 minute tirade about PVST+ would bore you to tears, stick with the certifications as your baseline.

    I could rant on this topic endlessly, as datacenter and colo design is something that I have strong feelings about after seeing a lot of very poorly designed datacenters. But there's just too much to say. Definitely find someone extremely qualified to partner with before you do anything. Expect to spend money doing so.

  15. #15
    would each rack really need a GigE?

    Not to sound like a little kid but my father has worked in the networking industry mostly competiting with cisco for most of his career so when i want to get some answers to questions like this i usaully get a few hour lecture from him.

    A "good" biuld should be out of the datacenter in <1ms if im not mistaken.
    Last edited by webephex; 03-26-2004 at 09:02 PM.

  16. #16

  17. #17
    Join Date
    Apr 2001
    Location
    Pittsburgh, PA
    Posts
    1,304
    Originally posted by webephex
    an interesting read: http://servermatrix.com/networkOverview.pdf
    That's odd, last I heard, the Cisco Certified program required that you use exclusively Cisco equipment where possible - ie, no Juniper border routers. I suppose it's changed since then.

    Kevin

  18. #18
    Join Date
    Dec 2002
    Location
    acton, MASS
    Posts
    229
    Originally posted by angelvalley
    You had me at "big fancy router".

    I hate to break it to you, but you're in *way* over your head.

    The carrier class routers on a typical datacenter border (Cisco GSRs for example) will run you six figures for a set, even with a nice discount (keeping in mind you need at least two, and you need to buy the cards and the support).

    A pair of core switches typically used in a datacenter, such as a Cisco 6500 series, can also set you back six figures depending on the chassis size (for instance a 6509) and how stocked each unit is (how many populated slots).

    If you want to expand your network, lease that cage you mentioned at an existing datacenter, get a nice Cisco VXR or any number of mid-range routers, pick up a Cisco 4003 or 4006 on eBay, hire someone who knows how to set it up, and worry about your local network.

    You probably won't need a dynamic routing protocol for a one-cage straightforward hosting environment. You can run static routes and do simple load balancing.

    Building enterprise networks is for experts, especially those participating in global BGP and advertising their own routes. At best you'd be running OSPF or even, seriously, RIPv2 to your upstream provider simply to allow for ease of configuration on your side, and let them worry about handing traffic off to a BGP peer. Don't even think about getting involved in global BGP unless you know exactly what you're doing.

    Not to sound mean or disrespectful, but from the content of your posting, you're about 1-2 years away from being able to do something like this on your own without making a significant number of potentially devastating mistakes. You should look into hiring someone to handle the technical aspect for you and worry about the business side of things if you're going to move forward.
    Not to sound mean or disrespectful by any means, but you, yourself are 1-2 years away from being able to setup a hosting network infrastructure by any means.

    First of all, "cisco 4003, 4006" are not available products, unless of course you are talking about Cisco Catalyst 4006?

    And as jsw6 mentioned, Juniper M series is an ideal solution for your core for most hosting centers and can run less than six figures in general.

    And OSPF and RIPv2 to your upstream? What are you thinking? You do _NOT_ _NOT_ _NOT_ _EVER_ _NEVER_ run an IGP with your upstream. If you do, you and your upstream should be beaten in the head countlessly with a clue bat. BGP is specfically designed to exchange routing information between two different autonomous systems, in which in this case, one can be you, one can be your upstream. BGP is the defacto standard of EGP, or Exterior Gateway Protocol, where you run it between two distinctive and administrative controlled networks (or technically different; either way called AS). RIPv1/v2, OSPF, E/IGRP, ISIS are called IGP or Interior Gateway Protocol, where you run it WITHIN your AS. I hope this clarifies your knowledge.

    Setting up BGP is not that hard, and I can guarantee you there are more people in the WHT community than you know who know how to configure a simple BGP session with an upstream, however technical competence in maintaining BGP is a different story, and I'll have to agree with you on that part.

    -J
    Last edited by haesu; 03-30-2004 at 02:11 AM.
    TowardEX Technologies, Inc.
    Metro-Boston Affordable Colocation, Dedicated Servers & IP Consulting
    http://www.towardex.com
    [email protected]

  19. #19
    Join Date
    Feb 2003
    Location
    NY
    Posts
    586
    if you want to learn quick and dirt routing stuff with cisco get the oriely cisco cookbook.

  20. #20
    Good routers: Juniper
    Good switches: Foundry and Extreme

    But you should consult someone who has done this before !

  21. #21
    Join Date
    Oct 2002
    Location
    New York
    Posts
    1,063
    Networks is all about topology, not MASSIVE switches and HUUUUUGE routers. The idea is to efficiently spread the network out, using branches created by switches, so it CAN go from

    BIG SWITCH -> Smaller Switch -> small switch..

    Yeah not what you were asking, but 20,000 servers has nothing to do with the switches they use, they could use the same switches to power a single server and obtain the same efficiency
    The Hostworks:: Offering Managed and Unmanaged VPS
    Shared Web Hosting and Managed E-Mail security services.
    24/7 Support :: 10 year anniversary coming soon!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •