Results 1 to 40 of 40
  1. #1

    question about dc's network aggregration

    Hi there! We're looking at facilities and found a design which we thought was odd. They have a row of 25 cabinets uplinking to a cisco 2960g L2 switch which then feeds into their core. It appears all their rows of about 20 cabinets are going to a single 2960g.

    Now correct me if I'm wrong but should each cabinet be flowing directly back into the core into the L3 switch like a catalyst 6500 or something else? They say it's just doing L2 only and from observing the lights flashing in synch on all the ports it would appear there's no vlans on there either.

    Shouldn't we be concerned about this?

  2. #2
    Join Date
    Aug 2007
    Location
    L.A., CA
    Posts
    3,706
    if its doing L2, there wouldn't be any VLANs obviously.
    They are using it as an aggregation point, and the VLANs are done upstream of it.

  3. #3

    ostsLa Keep

    So are you saying it's fine and we shouldn't be concerned?

  4. #4
    Join Date
    Aug 2007
    Location
    L.A., CA
    Posts
    3,706
    Ask them how they plan to segregate your traffic from others. If it is just an "open" network, then I would tell them I want a private VLAN or a direct port on an L3 device with the IPs routed directly to your port.

    Otherwise, it should be fine. I assume no one is really doing that much bandwidth usage based on the setup (few hundred megs per row or so)

  5. #5
    So without a private vlan on that switch our firewalls will be filtering out traffic that isn't destined for us. That creates additional, unecessary load on our gear doesn't it?

  6. #6
    Join Date
    Aug 2007
    Location
    L.A., CA
    Posts
    3,706
    There wont be other traffic coming to you for the most part (just broadcast/arps), but it will be possible for others to sniff your traffic (and for you to sniff theirs). Also, if there is a DDOS the ARP WHO-IS traffic will hit everyones port as well (if there aren't VLANs or L3 ports)
    So its insecure to not have a direct L3 port or a VLAN. Now, you should first ASK them how they are segregating the traffic, don't just assume they dont have VLANs.

  7. #7
    I appreciate your replies but you're confusing me. In my first post I said they told me it's only doing l2. You replied if it's only doing l2 there are no vlans.

    So doesn't that give me the answer?

  8. #8
    Join Date
    Aug 2007
    Location
    L.A., CA
    Posts
    3,706
    the VLANs can be done UPSTREAM of that 2960G (and they would have to be, as that is not a L3 switch).
    So there being a 2960G there does not mean something is wrong.

    Why not just ask them right now if they will be giving you a private VLAN or not?
    That will clear up all your questions and you can either move on with requesting a VLAN or L3 port or finding a new provider if the open L2 network will not work for you (it wouldn't for me) and they cant give you a VLAN or L3 port.... simple.

  9. #9
    Their contract says a private vlan. I appreciate you taking the time to answer but find what you're saying confusing.

    You say l2 doesn't do vlans but having a private vlan upstream works. That doesn't make sense to me, at least not the way you're saying it.

    I'm under the impression that l2 does have vlans but you need l3 to do inter vlan routing. Maybe that's what you're trying to say but just not being clear.

    So if the upstream switch is doing l3 inter vlan routing, the uplink l2 switch port to core is untagged, and they have my port tagged with my private vlan, then I'm all good? Except for it being a single point of failure, lol.

  10. #10
    Join Date
    Mar 2006
    Location
    Reston, VA
    Posts
    3,132
    Quote Originally Posted by cheezus View Post
    Their contract says a private vlan. I appreciate you taking the time to answer but find what you're saying confusing.

    You say l2 doesn't do vlans but having a private vlan upstream works. That doesn't make sense to me, at least not the way you're saying it.

    I'm under the impression that l2 does have vlans but you need l3 to do inter vlan routing. Maybe that's what you're trying to say but just not being clear.

    So if the upstream switch is doing l3 inter vlan routing, the uplink l2 switch port to core is untagged, and they have my port tagged with my private vlan, then I'm all good? Except for it being a single point of failure, lol.

    vlan = L2. They are just trunking the lines from their core to their aggregation point. I'd be more concerned with over-subscription than anything else...

  11. #11
    Join Date
    Aug 2008
    Location
    Seattle, WA
    Posts
    463
    Quote Originally Posted by cheezus View Post
    Hi there! We're looking at facilities and found a design which we thought was odd. They have a row of 25 cabinets uplinking to a cisco 2960g L2 switch which then feeds into their core. It appears all their rows of about 20 cabinets are going to a single 2960g.

    Now correct me if I'm wrong but should each cabinet be flowing directly back into the core into the L3 switch like a catalyst 6500 or something else? They say it's just doing L2 only and from observing the lights flashing in synch on all the ports it would appear there's no vlans on there either.

    Shouldn't we be concerned about this?
    It's not a design I would use. I prefer to do L3 at the rack-level so that you can use fast and reliable routing protocols to add redundancy and capacity. Instead, all 25 cabinets now have a common congestion point and a common single point of failure.

    The LED process on cisco switches only runs at certain intervals, especially if the switch is running at a high CPU load. You can't discern how they are configured from the way the lights are blinking. With not very much traffic, all the lights will blink nearly identically. On a desktop network where not much is happening, you might be able to see that some broadcasts cause some LEDs to blink but not others, but as soon as you add a bit of traffic, that becomes impossible.
    Seattle, WA - Gigabit Carrier & Network Services Provider
    Uptime | Atlas Networks

  12. #12
    Join Date
    Aug 2007
    Location
    L.A., CA
    Posts
    3,706
    And if all the cabinets were connected to 1 6509 for direct L3 (for example), it would STILL be a single point of failure / congestion point, so theres really not much difference.
    You can only get so much with just 1 uplink to your rack.

  13. #13
    Quote Originally Posted by CGotzmann View Post
    And if all the cabinets were connected to 1 6509 for direct L3 (for example), it would STILL be a single point of failure / congestion point, so theres really not much difference.
    You can only get so much with just 1 uplink to your rack.
    We are looking for 2 gig feeds, one for each firewall in our cluster. In this design both feeds go into that single switch. Not sure why you assumed we only requested a single feed.

  14. #14
    Join Date
    Aug 2008
    Location
    Seattle, WA
    Posts
    463
    Quote Originally Posted by CGotzmann View Post
    And if all the cabinets were connected to 1 6509 for direct L3 (for example), it would STILL be a single point of failure / congestion point, so theres really not much difference.
    You can only get so much with just 1 uplink to your rack.
    True, but...

    1) A 6509 can be configured with redundant SUPs and redundant PSUs, so I'd generally call it more reliable than a 2960.
    2) 2 SPOFs are twice as bad as one, and there are two SPOFs if there's a 2960 and a 6509.
    3) It's much easier to throw in a second 6509 if the links are layer-3 than if they're vlans - otherwise, you have to deal with either heartbeating the gateway IP around or using HSRP.
    Seattle, WA - Gigabit Carrier & Network Services Provider
    Uptime | Atlas Networks

  15. #15
    Join Date
    Mar 2006
    Location
    Reston, VA
    Posts
    3,132
    Quote Originally Posted by atlasnetworkseric View Post
    It's not a design I would use. I prefer to do L3 at the rack-level so that you can use fast and reliable routing protocols to add redundancy and capacity. Instead, all 25 cabinets now have a common congestion point and a common single point of failure.
    Its proven that L3 TOR is a complete waste of IP space and not effective.

    There is a complete NANOG presentation about this topic http://www.nanog.org/meetings/nanog4...zQ2&nm=nanog46

  16. #16
    Join Date
    Aug 2008
    Location
    Seattle, WA
    Posts
    463
    Quote Originally Posted by cheezus View Post
    We are looking for 2 gig feeds, one for each firewall in our cluster. In this design both feeds go into that single switch. Not sure why you assumed we only requested a single feed.
    It sounds like they're operating a low-budget network. Are you sure that's what you want?
    Seattle, WA - Gigabit Carrier & Network Services Provider
    Uptime | Atlas Networks

  17. #17
    Quote Originally Posted by Spudstr View Post
    vlan = L2. They are just trunking the lines from their core to their aggregation point. I'd be more concerned with over-subscription than anything else...
    Thanks Spudstr. I have to say I look forward to your replies in these forums as you're the most clear (to me anyway).

    Can you elaborate on over-subscription? Specifically what we can look for to make the call if this switch is over subscribed?

  18. #18
    Quote Originally Posted by atlasnetworkseric View Post
    It sounds like they're operating a low-budget network. Are you sure that's what you want?
    Not at all! The problem is, and why I'm here, is because they don't think it is a low budget network! So I'm here to get more input to validate my thoughts that it is a low budget approach.

  19. #19
    Join Date
    Mar 2006
    Location
    Reston, VA
    Posts
    3,132
    Quote Originally Posted by cheezus View Post
    Thanks Spudstr. I have to say I look forward to your replies in these forums as you're the most clear (to me anyway).

    Can you elaborate on over-subscription? Specifically what we can look for to make the call if this switch is over subscribed?

    I don't know the network design but usually those 2960's might have 1-2G "trunks" to their core. I don't know if each rack is 100m or 1g each? Where is the traffic going inner cabinets? Or all going to the outside world?

    Thanks for the comments, I might not come across as the nicest person on the forums but I am to the point and speak the truth. Sorry if some people don't appreciate that but I would rather not sugar coat things and make them appear a different way.

  20. #20
    Join Date
    Aug 2008
    Location
    Seattle, WA
    Posts
    463
    Quote Originally Posted by Spudstr View Post
    Its proven that L3 TOR is a complete waste of IP space and not effective.

    There is a complete NANOG presentation about this topic http://www.nanog.org/meetings/nanog4...zQ2&nm=nanog46
    I've read that presentation before. There's no proof that it's not effective in it. The usage of IP space is minimal (/31's for PtP links), and hardly what I would call 'a complete waste of space' given the other 'justifications' for IP space in common use today.

    And with IPv6, that argument evaporates.

    In any case, the OP's described network suggests a budget facility, and I would have serious concerns about putting equipment there. One DOS to any customer in any of the 25 cabinets and the already-heavily-used uplink will fill and take everyone down. Sounds lame to me.
    Seattle, WA - Gigabit Carrier & Network Services Provider
    Uptime | Atlas Networks

  21. #21
    Join Date
    Mar 2006
    Location
    Reston, VA
    Posts
    3,132
    Quote Originally Posted by atlasnetworkseric View Post
    I've read that presentation before. There's no proof that it's not effective in it. The usage of IP space is minimal (/31's for PtP links), and hardly what I would call 'a complete waste of space' given the other 'justifications' for IP space in common use today.

    And with IPv6, that argument evaporates.

    In any case, the OP's described network suggests a budget facility, and I would have serious concerns about putting equipment there. One DOS to any customer in any of the 25 cabinets and the already-heavily-used uplink will fill and take everyone down. Sounds lame to me.

    And if said provider only has gigE circuits, one said DOS that would fill up that uplink port would also saturate said providers circuit(s) to their upstreams.

    So whats your point?

  22. #22
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,571
    A 2960 used to aggregate 25 racks is not something I'd call high end, in fact it's a down right hack job if true IMO. One or two L2 device(s) at the top of each rack, each going straight into an L3 agg. layer would be something far more reasonable.
    Fast Serv Networks, LLC | AS29889 | Fully Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  23. #23
    Join Date
    Feb 2011
    Posts
    669
    Quote Originally Posted by CGotzmann View Post
    There wont be other traffic coming to you for the most part (just broadcast/arps), but it will be possible for others to sniff your traffic (and for you to sniff theirs). Also, if there is a DDOS the ARP WHO-IS traffic will hit everyones port as well (if there aren't VLANs or L3 ports)
    So its insecure to not have a direct L3 port or a VLAN. Now, you should first ASK them how they are segregating the traffic, don't just assume they dont have VLANs.
    Its a switch not a hub so you will only be able to get packets destined for your mac address and broadcasts (destined for all macs) the only thing you could sniff of someone elses is the broadcasts. (without mac emulation and that will cause problems and should be noticed.)

  24. #24
    The cabinets have their own switches for inter-rack communications (customer provided). Most customers have the DC feeds coming into their firewalls. Then their rack switches behind the firewalls.

    Uplinks from cabinets are 100 or 1000 and in our case we're asking for 2 x 1000. We have bursts up around 160 but please let's not get sidetracked on that.

    The facility has multiple 10gig feeds coming into it. The switch uplinks are 2 x 1 gig feeds into the core.

    Our opinion is this a budget setup but the facility seems to think otherwise.

  25. #25
    Join Date
    Aug 2008
    Location
    Seattle, WA
    Posts
    463
    Quote Originally Posted by Spudstr View Post
    And if said provider only has gigE circuits, one said DOS that would fill up that uplink port would also saturate said providers circuit(s) to their upstreams.

    So whats your point?
    Apparently, that's not the case.

    Anyways, the point is that hanging 25 cabinets off a L2 switch that then hangs off the core is a budget way to get GigE to the rack. Does it work? Yes. Is it the best way to do it? Hell no!

    Quote Originally Posted by cheezus View Post
    Our opinion is this a budget setup but the facility seems to think otherwise.
    That tells you what you need to know about this facility then. If you don't even operate in the same definition of the word 'budget', they're not a good fit, and you'll probably see other instances of this later in the relationship.
    Seattle, WA - Gigabit Carrier & Network Services Provider
    Uptime | Atlas Networks

  26. #26
    Join Date
    Mar 2006
    Location
    Reston, VA
    Posts
    3,132
    Quote Originally Posted by atlasnetworkseric View Post
    Apparently, that's not the case.

    Anyways, the point is that hanging 25 cabinets off a L2 switch that then hangs off the core is a budget way to get GigE to the rack. Does it work? Yes. Is it the best way to do it? Hell no!


    That tells you what you need to know about this facility then. If you don't even operate in the same definition of the word 'budget', they're not a good fit, and you'll probably see other instances of this later in the relationship.
    And who said/what happens if this provider was to run them all into a 6500 running a 6148 card? that 48 port rj46 gigE card that is 8:1 over subscribed? i.e sharing 1G for 8 1G ports?

    Doesn't matter how you slice it there is shadyness all around and what appears to be one thing might not actually hold true.

    No network is going to plug a customer directly into a core, the whole point is to have segregation and put customers on aggregation/distribution switches.

    Router ports are expensive, switch ports are not.

  27. #27
    So spudstr, are you of the opinion, given the information I've provided, that it is a; budget hack? acceptable setup? standard way of doing things?

  28. #28
    Join Date
    Mar 2006
    Location
    Reston, VA
    Posts
    3,132
    Quote Originally Posted by cheezus View Post
    So spudstr, are you of the opinion, given the information I've provided, that it is a; budget hack? acceptable setup? standard way of doing things?
    to be honest its a standard practice for aggregation, but then it depends on a few things. How much "outbound" bandwidth do all 20 cabinets have? Is that 2960 just for those 20 cabinets?

    Some people view it as the cheap way of doing something which it is.. cheaper So something isn't terminated into a switch like a 2960 but goes into a 6500 ok great, what did you gain by doing this? still a single point of failure the only difference is now you have a great bandwidth pipe leaving the entire network than a point that can be congestion.

    To be honest we terminate 100M cabinets into a 3550 switch with 2x1g to a 6500.. Why? its cost effective way to terminate those customers and we are not making a cabling mess on our 6500, instead we have 6-12" cables going from the patch panel right to the switch under it thats clean and organized. Now customers who have HSRP with us we run right into the 6500 because you have to.

    In the short everyone is going to run into a aggregation device of some point, weather its a small 48 port switch or a 8-12 blade switch. Core gear is expensive and should be used as the core, not distribution or aggregation.

  29. #29
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,571
    I'm sorry but I wouldn't even consider a 2960 proper choice for a datacenter with it's ridiculously small shared buffer. The fact home depot sells them should be warning enough.
    Fast Serv Networks, LLC | AS29889 | Fully Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  30. #30
    Join Date
    Aug 2008
    Location
    Seattle, WA
    Posts
    463
    Quote Originally Posted by Spudstr View Post
    to be honest its a standard practice for aggregation, but then it depends on a few things. How much "outbound" bandwidth do all 20 cabinets have? Is that 2960 just for those 20 cabinets?

    Some people view it as the cheap way of doing something which it is.. cheaper So something isn't terminated into a switch like a 2960 but goes into a 6500 ok great, what did you gain by doing this? still a single point of failure the only difference is now you have a great bandwidth pipe leaving the entire network than a point that can be congestion.

    To be honest we terminate 100M cabinets into a 3550 switch with 2x1g to a 6500.. Why? its cost effective way to terminate those customers and we are not making a cabling mess on our 6500, instead we have 6-12" cables going from the patch panel right to the switch under it thats clean and organized. Now customers who have HSRP with us we run right into the 6500 because you have to.

    In the short everyone is going to run into a aggregation device of some point, weather its a small 48 port switch or a 8-12 blade switch. Core gear is expensive and should be used as the core, not distribution or aggregation.
    An attacker can fit enough PPS into a DDOS to make that 3550 keel over in about 200-300 Mbps. A 6509 with a 720-3B SUP will just see a cute little bump in CPU usage, but the distribution layer switch and all of the customers on it will be down, all for a very small DDOS.

    Core gear is not expensive. WAN Edge (ASR, CRS) gear is expensive. That $25,000 6509 starts to look really cheap next to an ASR 9000.
    Seattle, WA - Gigabit Carrier & Network Services Provider
    Uptime | Atlas Networks

  31. #31
    Join Date
    Mar 2006
    Location
    Reston, VA
    Posts
    3,132
    Quote Originally Posted by atlasnetworkseric View Post
    An attacker can fit enough PPS into a DDOS to make that 3550 keel over in about 200-300 Mbps. A 6509 with a 720-3B SUP will just see a cute little bump in CPU usage, but the distribution layer switch and all of the customers on it will be down, all for a very small DDOS.

    Core gear is not expensive. WAN Edge (ASR, CRS) gear is expensive. That $25,000 6509 starts to look really cheap next to an ASR 9000.


    We've seen 3550's hit 800mbps with high PPS counts with zero issues, as long as your not using the 3550XL you are fine. I have seen to many people think their 6500's are brick houses to fall pray to ddos's against them and watch high rate of PPS nail their acl and grid lock their sup.

    If you have some sort of better core asr/MX/M/CSR and using 6500 for customer facing thats great, its what it should be doing, but if someones using the 6500 for a core then often they will use aggregation switches.

  32. #32
    Join Date
    Oct 2004
    Location
    Houston, Tx
    Posts
    307
    Quote Originally Posted by Spudstr View Post
    To be honest we terminate 100M cabinets into a 3550 switch with 2x1g to a 6500.. Why? its cost effective way to terminate those customers and we are not making a cabling mess on our 6500, instead we have 6-12" cables going from the patch panel right to the switch under it thats clean and organized. Now customers who have HSRP with us we run right into the 6500 because you have to.
    Huh? Why do you need direct connections to the 6500?

  33. #33
    Join Date
    Mar 2006
    Location
    Reston, VA
    Posts
    3,132
    Quote Originally Posted by DPG View Post
    Huh? Why do you need direct connections to the 6500?
    Do you know how HSRP/VRRP works? Its monitored state protocol. Sure you can put the HSRP interface on a vlan but you have a monitor tracking a physical interface, without this tracking it wouldn't know when to failover. Doing hsrp on a vlan is more work than required and a pain in the rear.

    This is why you have to put people directly on the device that is performing the routing and controls the HSRP group. You can't do HSRP any other way, unless you are working with spanning tree and just want to extend a vlan and let spanning tree do its magic.. which.. i highly advise against. HSRP ports don't work on aggregation switches that are only L2.

  34. #34
    Join Date
    Mar 2008
    Location
    Los Angeles, CA
    Posts
    555
    We only use 2960's here at the edge level but for distribution level (row of racks) we typically go with 4948 or 4948-10ge.

    The 2960 only having 2 gigs of uplink to the core for 25 racks seems pretty low if you ask me. Too much over subscription for my tastes. I know some of our larger rows that have 15-20 racks have 4 gigs of uplink capacity to the core and even those are higher than I would like (60-75% capacity).

  35. #35
    Join Date
    Aug 2007
    Location
    L.A., CA
    Posts
    3,706
    It depends on the average type of client base though. Some datacenters don't have clients that use any real amounts of bandwidth.

  36. #36
    Join Date
    Oct 2004
    Location
    Houston, Tx
    Posts
    307
    Quote Originally Posted by Spudstr View Post
    Do you know how HSRP/VRRP works? Its monitored state protocol. Sure you can put the HSRP interface on a vlan but you have a monitor tracking a physical interface, without this tracking it wouldn't know when to failover. Doing hsrp on a vlan is more work than required and a pain in the rear.

    This is why you have to put people directly on the device that is performing the routing and controls the HSRP group. You can't do HSRP any other way, unless you are working with spanning tree and just want to extend a vlan and let spanning tree do its magic.. which.. i highly advise against. HSRP ports don't work on aggregation switches that are only L2.

    I am well aware how first hop redundancy protocols work. Please tell us why you shouldn't use HSRP on vlan interfaces.

  37. #37
    Join Date
    Mar 2006
    Location
    Reston, VA
    Posts
    3,132
    Quote Originally Posted by DPG View Post
    I am well aware how first hop redundancy protocols work. Please tell us why you shouldn't use HSRP on vlan interfaces.

    What are you going to "monitor" the standby group with? the link going to the aggregation switch? That doesn't fly your customers link can go down on the switch up from your 6500 and your 6500 will have no idea that it happened.

    So what type of magic are you pulling off to monitor a port on another device? I am interested to know.

  38. #38
    Join Date
    Oct 2004
    Location
    Houston, Tx
    Posts
    307
    Quote Originally Posted by Spudstr View Post
    What are you going to "monitor" the standby group with? the link going to the aggregation switch? That doesn't fly your customers link can go down on the switch up from your 6500 and your 6500 will have no idea that it happened.

    So what type of magic are you pulling off to monitor a port on another device? I am interested to know.

    The magic is in the fact that the customer would have two uplinks into your network. The 6500's only need to be concerned with each other's reach-ability.

    http://imageshack.us/photo/my-images/863/hsrp.png/

  39. #39
    Join Date
    Mar 2006
    Location
    Reston, VA
    Posts
    3,132
    Quote Originally Posted by DPG View Post
    The magic is in the fact that the customer would have two uplinks into your network. The 6500's only need to be concerned with each other's reach-ability.

    http://imageshack.us/photo/my-images/863/hsrp.png/
    yeah, thats not really the practical way to do that and over complicates it because now your introducing other problems like spanning tree depending on how you've designed it.

  40. #40
    Join Date
    Oct 2004
    Location
    Houston, Tx
    Posts
    307
    Quote Originally Posted by Spudstr View Post
    yeah, thats not really the practical way to do that and over complicates it because now your introducing other problems like spanning tree depending on how you've designed it.

    Exactly. It all comes down to the design. It isn't all the complicated. You would just make your primary router the spanning tree root for each vlan that has HSRP.

    Of course this is coming from a Cisco fanboy. I would be interested to see how other vendors recommend the design of campus networks.

Similar Threads

  1. A Network Question
    By bomozah in forum Colocation and Data Centers
    Replies: 13
    Last Post: 12-09-2009, 07:41 PM
  2. network question....
    By Jeremy in forum Colocation and Data Centers
    Replies: 3
    Last Post: 04-01-2009, 09:10 PM
  3. Network question
    By koepp in forum Colocation and Data Centers
    Replies: 7
    Last Post: 11-21-2007, 05:07 AM
  4. Network Question
    By MikeWalczak in forum Computers and Peripherals
    Replies: 11
    Last Post: 08-23-2006, 04:37 PM
  5. network question
    By Eiv in forum Dedicated Server
    Replies: 10
    Last Post: 06-19-2003, 06:02 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •