Results 1 to 9 of 9

Thread: Network Design

  1. #1
    Join Date
    Apr 2004
    Posts
    33

    Question Network Design

    Hello,

    I'm designing a network that should be able to support both dedicated servers and also offer colocation (full rack and half rack). Things will be relatively small at start but the design must be able to scale. The colo area will be secured from the dedicated server area.

    Right now, this is what I'm thinking:

    CORE/DISTRIBUTION combined:
    - Two cisco 6509's running HSRP
    - Each 6509 is connected to the same two upstream ISP's via BGP (so that means 4 links in total)
    - Each 6509 is connected to the access switches (described below)
    - The 6509's will have a single gigabit crossconnect between them so that they can talk to each other for HSRP, iBGP, etc.

    ACCESS
    - Each of our racks will have an L3 switch at the top of the rack for all the servers in the rack to plug into
    - there will be ~30 servers per rack
    - these switches will use 4 x 1 gbit ports for trunking to the core. They will be configured in two 802.3ad link aggregation groups (ie 2 x 1gbit links per LAG group). One LAG group will connect to the first 6509, the second LAG group will connect to the second 6509.
    - The default gateway on these L3 switches will be configured with the HSRP virtual IP address of the 6509's.
    -spanning tree will have to be enabled to avoid loops since the two 6509's are connected directly also


    QUESTIONS:
    1) Each dedicated server will get 5 usable IP addresses, so I'd have to subnet my IP address space accordingly (into a bunch of /29's). This also means that I'd have to setup 1 VLAN per subnet (wich means 1 VLAN per server!). Is this a scalable design? If I have 1000 servers I'd need 1000 VLAN's! Isn't there a limit to the number of VLANS that can be created/handled by my core switches? How do larger providers do it that have thousands of servers?

    2) Is this design scalable/redundant? The only single point of failure that I see is my access switch (if it dies, it could take out a rack's worth of servers). I guess I would have to live with this and would have spares on hand.

    3) I'm a little confused with the interaction of BGP and HSRP. What happens, for example, if one of the links to one of the ISP's goes down on the active HSRP router? I dont want it to fail over to the inactive HSRP router, because the router is still good, its just a link that went down. Would the active HSRP router be smart enough to realize (maybe via iBGP) that the inactive HSRP router can still route to that ISP and thus just ROUTE the traffic to that inactive 6509 and then have that router send it out to the ISP? I'm assuming this traffic would travel across the 1gbit xconnect between the two 6509's so I may have to consider increasing that capacity using link aggregation aswell?

    4) Which cisco switch would be good as my L3 access switch? It would have to support ~30 servers in the rack plus have at least 4x1gbit ports that I can config into two lag groups to uplink to the core.


    thanks and sorry for the noob questions

    SoN][c

  2. #2
    Join Date
    Nov 2006
    Posts
    64
    You might want to consider if you really need L3 switches. You don't look like you really need them right now and you'll save a bundle.

  3. #3
    Join Date
    Nov 2005
    Location
    Minneapolis, MN
    Posts
    1,648
    Couple things:

    - Spanning tree is evil. It was designed for a much more simple scenario of being able to inject a redundant learning bridge into a simple network topology. On average, with modern networks spanning tree will cause you at least one major outage per year when a link state event causes it to not be able to converge on a topology.

    - If you're using L3 switches in the racks, don't run HSRP on the 6500s. HSRP is designed for edge devices that can only use a fixed default gateway (ie servers & workstations). You should never be running HSRP between network devices capable of running an IGP. OSPF, EIGRP, heck even RIP would provide a better solution. There is no interaction between HSRP and any routing protocol, nor is there any awareness about upstream connectivity. You have a pair/group of router interfaces and they are configured to serve a VIP (the default gateway) and the active router is chosen based on its configured priority. It only fails over when the interfaces can no longer see each other to heartbeat.

    As for the VLANs, on IOS-based switches you can configure individual ports as routed interfaces. Or if you're using layer3 uplinks you can re-use the same set of vlans on every switch. The more of a layer3 hierarchy that you build the larger the design will be able to scale.
    Eric Spaeth
    Enterprise Network Engineer :: Hosting Hobbyist :: Master of Procrastination
    "The really cool thing about facts is they remain true regardless of who states them."

  4. #4
    Join Date
    Apr 2004
    Posts
    33
    flaggg,

    Thanks for your reply.

    I was initially thinking about using L2 switches in each rack, however, I figured for scalability reasons it would be better to go with L3.

    What do you think the cons of L3 would be? (aside from price)

    -SoN][c



    Quote Originally Posted by flaggg
    You might want to consider if you really need L3 switches. You don't look like you really need them right now and you'll save a bundle.

  5. #5
    Join Date
    Apr 2004
    Posts
    33
    Eric,

    Thanks very much for your response. I would like to (please) pick your brain with a few more questions.

    -Thanks for your advice about spanning tree. If I shouldnt use it due to its instability, how would I get around the loop caused since the L3 switch is connected to each 6509, and the 6509's are cross connected?? If I use routed interfaces on the L3 for the 'uplink' does that mean the connection between each L3 switch and the 6509 needs its own subnet?

    - I was thinking (hoping) for simplicity that I could get away with having all my L3 switches and the 6509's all connected to the same subnet. Each 6509 and all the L3's would get their own IP address (x.x.x.1, x.x.x.2, x.x.x.3, etc). OSPF would run on that network. How well would that scale? I guess eventually I would have to create more subnets to split the traffic if I were to have, say, hundreds of racks?

    -I understand what you're saying about not running HSRP. Do you think it would be better instead to run OSPF on the network (per above) between the rack switches and the core 6509's? If one of the 6509's were to go down, how long would it take OSPF to converge and update the routing tables so that traffic would go to the other 6509? Also, if just one link to one of the ISP's were to go down, would the 6509 be smart enough to change its routing table and have OSPF pick that up and notify all the L3 switches so that they can send traffic to that ISP to the other 6509?

    -Regarding the VLAN's and scalability. If I were to reuse the same VLAN numbers in each rack, I would have to UNTAG the egress traffic from the L3 uplink's to the 6509's, correct? And the INGRESS traffic will find its way to the correct L3 switch beause I'd be running OSPF?


    Thanks for your time, and sorry for all the noob questions. I cant wait for my test equipment to arrives so I can start putting some of this theory to practice in my test lab (at home, in my basement!)

    SoN][c



    Quote Originally Posted by spaethco
    Couple things:

    - Spanning tree is evil. It was designed for a much more simple scenario of being able to inject a redundant learning bridge into a simple network topology. On average, with modern networks spanning tree will cause you at least one major outage per year when a link state event causes it to not be able to converge on a topology.

    - If you're using L3 switches in the racks, don't run HSRP on the 6500s. HSRP is designed for edge devices that can only use a fixed default gateway (ie servers & workstations). You should never be running HSRP between network devices capable of running an IGP. OSPF, EIGRP, heck even RIP would provide a better solution. There is no interaction between HSRP and any routing protocol, nor is there any awareness about upstream connectivity. You have a pair/group of router interfaces and they are configured to serve a VIP (the default gateway) and the active router is chosen based on its configured priority. It only fails over when the interfaces can no longer see each other to heartbeat.

    As for the VLANs, on IOS-based switches you can configure individual ports as routed interfaces. Or if you're using layer3 uplinks you can re-use the same set of vlans on every switch. The more of a layer3 hierarchy that you build the larger the design will be able to scale.

  6. #6
    Join Date
    Oct 2004
    Location
    Nevada
    Posts
    887
    '- Each 6509 is connected to the same two upstream ISP's via BGP (so that means 4 links in total)'
    Only if you are in a data center and connect to two different edge routers (or switches that are connected to two different core routers) There is little reason to have two uplinks to the same edge router, unless you are bonding them (e.g. with etherchannel).

    'The 6509's will have a single gigabit crossconnect between them so that they can talk to each other for HSRP, iBGP, et' 'They will be configured in two 802.3ad link aggregation groups (ie 2 x 1gbit links per LAG group). '
    Then you will need two GigEs between the routers.

    'Each dedicated server will get 5 usable IP addresses,' ' I was thinking (hoping) for simplicity that I could get away with having all my L3 switches and the 6509's all connected to the same subnet. Each 6509 and all the L3's would get their own IP address (x.x.x.1, x.x.x.2, x.x.x.3, etc).'
    If you are doing 1,000 servers there is no way to have them all on the same subnet. (You are not thinking of doing NAT, right?) If you do a /29 per server and 30 servers per rack, you are looking at a /24 per RACK.

    ' Which cisco switch would be good as my L3 access switch? It would have to support ~30 servers in the rack plus have at least 4x1gbit ports that I can config into two lag groups to uplink to the core.'
    Response 1 is that I would suggest that you reconsider your requirements for two GigE uplinks per rack to the core. Second response is to ask if you have considered two switches if you really want two gige uplinks. First would be an agg switch such as a 4908L3 or 4912G, then a switch for the racks such as a 3750. Then connect switches from the racks up to the agg switch.

    I would run everything on iBGP, not HSRP

    As pointed out above, simply assign the IPs to the ports on the L3 switch, you dont need VLANS.

    Finally, hire a network engineer to help you. You are talking about some huge $$s, and the upfront expenditure will be cost-effective.

  7. #7
    Join Date
    Apr 2004
    Posts
    33
    Dennis,

    Thank you very much for your response.

    I have a few clarifications:

    - I'll be in a carrier neutral facility and will connect to different edge routers within the building, just for added redundancy (doesnt cost me anything extra, they will aggregate the billing accross the two links so I still get volume discounts)

    - Makes sense what you're saying about having 2x1gbit cross connects between the 6509's. I just didnt anticipate having much traffic between them.

    -Regarding the IP addressing I stated, I was referring to the network between the L3 rack switches and the core 6509's. Since I'm running that as L3, I wasnt sure if I had to do a 'point to point' network with a /31 for the connection from each rack's L3 switch to the core? Or if I could use a larger subnet and have all the L3's and 6509's on it (simpler to setup and manage, but I wasnt sure about what implications it would have on network traffic). Does that make sense? It might not be as scalable.

    - Do you think 2x1gbit uplinks are not enough? I guess (as per Eric's suggestion) if I dont run HSRP and use OSPF on the network between the L3 switches and the core 6509's, I guess I could use eqal cost routing between both LAG groups (remember, i plan to have TWO 2x1gbit LAG groups, each going to a different core 6509). Effectivley, that would give me 4gbit uplink speed, correct?

    - I do plan to hire a Network Engineer, but I'm also in the process of learning this and what better way!

    thansk again for your assistance

    -SoN][c

    Quote Originally Posted by Dennis Nugent
    '- Each 6509 is connected to the same two upstream ISP's via BGP (so that means 4 links in total)'
    Only if you are in a data center and connect to two different edge routers (or switches that are connected to two different core routers) There is little reason to have two uplinks to the same edge router, unless you are bonding them (e.g. with etherchannel).

    'The 6509's will have a single gigabit crossconnect between them so that they can talk to each other for HSRP, iBGP, et' 'They will be configured in two 802.3ad link aggregation groups (ie 2 x 1gbit links per LAG group). '
    Then you will need two GigEs between the routers.

    'Each dedicated server will get 5 usable IP addresses,' ' I was thinking (hoping) for simplicity that I could get away with having all my L3 switches and the 6509's all connected to the same subnet. Each 6509 and all the L3's would get their own IP address (x.x.x.1, x.x.x.2, x.x.x.3, etc).'
    If you are doing 1,000 servers there is no way to have them all on the same subnet. (You are not thinking of doing NAT, right?) If you do a /29 per server and 30 servers per rack, you are looking at a /24 per RACK.

    ' Which cisco switch would be good as my L3 access switch? It would have to support ~30 servers in the rack plus have at least 4x1gbit ports that I can config into two lag groups to uplink to the core.'
    Response 1 is that I would suggest that you reconsider your requirements for two GigE uplinks per rack to the core. Second response is to ask if you have considered two switches if you really want two gige uplinks. First would be an agg switch such as a 4908L3 or 4912G, then a switch for the racks such as a 3750. Then connect switches from the racks up to the agg switch.

    I would run everything on iBGP, not HSRP

    As pointed out above, simply assign the IPs to the ports on the L3 switch, you dont need VLANS.

    Finally, hire a network engineer to help you. You are talking about some huge $$s, and the upfront expenditure will be cost-effective.

  8. #8
    Join Date
    Feb 2006
    Location
    Bristol, UK
    Posts
    280
    The easiest way to remove the access layer single point of failure is to have two switches per rack (assuming all your servers have two network interfaces) and plug each server into both switches (obviously this requires teaming or bonding to be configured within the server OS).

    I'd agree with 3750s as access switches. (4948s are the other option). None of Cisco's layer two switches will give you 4 x GigE uplinks and >= 30 access ports except for the WS-C2960G-48TC-L.
    Network EQ
    UK VPS
    , cPanel Hosting, Dedicated Servers and Hosted Exchange

  9. #9
    Join Date
    Nov 2005
    Location
    Minneapolis, MN
    Posts
    1,648
    Quote Originally Posted by SoN][c
    Thanks for your advice about spanning tree. If I shouldnt use it due to its instability, how would I get around the loop caused since the L3 switch is connected to each 6509, and the 6509's are cross connected??
    Spanning tree blocks ports to prevent layer2 loops. Adding layer3 hops segments out your broadcast domains, so even though it still physically looks like a ring when connected the broadcast domain between the 6500s would be different from the broadcast domain between the edge switches and the 6500s.
    Quote Originally Posted by SoN][c
    I was thinking (hoping) for simplicity that I could get away with having all my L3 switches and the 6509's all connected to the same subnet. ... How well would that scale?
    This shouldn't be a problem. In fact you don't need to waste your public IP address space for that connection, RFC1918 (ie, 10.x.x.x) address space would work just fine to "glue" the rack switches together with your 6500 distribution.
    Quote Originally Posted by SoN][c
    Do you think it would be better instead to run OSPF on the network (per above) between the rack switches and the core 6509's?
    Hell yes.
    Quote Originally Posted by SoN][c
    If one of the 6509's were to go down, how long would it take OSPF to converge and update the routing tables so that traffic would go to the other 6509? Also, if just one link to one of the ISP's were to go down, would the 6509 be smart enough to change its routing table <cut>?
    OSPF convergence is tunable by adjusting several metrics, but a default OSPF configuration should converge in a couple seconds. (literally 1-2 seconds, compared to 10+ seconds for HSRP) Tweaking even further can bring that number down; right now I'm using BiDirectional Forwarding Detection (BFD) on our metro core to provide link state notification to OSPF and observed convergence times are under 250ms. As for the forwarding to the 6509 with the valid uplink, that depends on how you're configured. Redistributing your BGP table into OSPF would make that work but it would produce a lot of excess baggage. You can work around a few options with distributing a default route via your 6500s, but those should be discussed with a network engineer.
    Quote Originally Posted by SoN][c
    Regarding the VLAN's and scalability. If I were to reuse the same VLAN numbers in each rack, I would have to UNTAG the egress traffic from the L3 uplink's to the 6509's, correct?
    Again VLAN tags are only good within a layer2 domain. As soon as you wedge a layer3 hop in there the layer2 domain on each side becomes completely independent.
    Quote Originally Posted by gavint
    The easiest way to remove the access layer single point of failure is to have two switches per rack (assuming all your servers have two network interfaces) and plug each server into both switches
    Easiest != Best. (especially in this case) If you really need redundancy for content you'd have a pair of servers, each plugged into different switches, with a fault tolerant load balancer splitting the traffic between them. For those even more serious, you'd have a DNS-based failover mechanism to swap between load balanced VIPs at two geographically separate DCs.
    Quote Originally Posted by gavint
    I'd agree with 3750s as access switches.
    Eh, the 3750 is really a poor value for the money. Unless you have a dire need to spend outrageously large sums of cash for a product with a bridge logo on it, I'd instead look at the Nortel 5510-48 or Force10 S50. Not only are those options cheaper than the 3750 by nearly half, but they also brutally pummel the 3750 every day of the week when it comes to forwarding performance. OSPF is a standards based routing protocol, and 802.3ad is vendor independent as well. There are no interoperability issues with mixing Nortel/Force10 edge switches with Cisco 6500 distribution hardware.
    Eric Spaeth
    Enterprise Network Engineer :: Hosting Hobbyist :: Master of Procrastination
    "The really cool thing about facts is they remain true regardless of who states them."

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •