Web Hosting Talk : Web Hosting Main Forums : Colocation and Data Centers : Multihome Across 2x6509's

[jdvachal]

I am looking for some suggestions on multihoming 2+ uplinks across 2 6509's.

Each 6509 will take only partial routes from each upstream - so the 3bxl will not be used.

My question is about whether or not the 6509's need to be cross connected via some internal protocol. Assume that each access switch connects to both of the 6509's.

Thanks!

[Techee]

Get the 3bxls, they are cheap right now.

Yes, you need to cross-connect the two 6509s, and run iBGP between them. Some topologies also suggest that you run OSPF between them.

BTW, the cross-connect between the two 6509s has to be large enough to handle the TOTAL traffic if one of the carriers disappears. Not an issue now (with two uplinks) but I have seen companies that have expanded, added a couple of GigE and a 10Gig uplink, yet still had only a single GigE between the 6509s!

[IDediServer Kevin]

I would suggest the use of a port channel as well thou obviously look at the pro's or con's regarding if one would be helpful for your topology / configuration

[IRCCo Jeff]

The 6509 is a core switch. Do you not have edge routers?

[jdvachal]

Thanks to all of you thus far!

I definitely understand the requirement for iBGP but I am not certain whether we need to implement OSPF of not.

As for the core vs edge device - we plan to use the 6509 as an edge device. We have yet decided on which switch to use at our core/aggregation layer. Opinions on using the 6509 at the edge?

[Amanah-Sales]

I've seen 6509/3bxl combo used on the edge in a lot of smaller scale deployments, ranging from multi 1G, to multi 10G, with around ~50 bgp neighbors without issue(mix of transit/peering). Im sure they're drawbacks, but its quite affordable.

Good luck in your deployment!

[FastServ]

Originally Posted by IRCCo Jeff

The 6509 is a core switch. Do you not have edge routers?

In a collapsed core approach so there is no separate core/edge. The 6509 is an excellent choice given this type of arrangement, especially on a smaller scale.

Originally Posted by Techee

Yes, you need to cross-connect the two 6509s, and run iBGP between them. Some topologies also suggest that you run OSPF between them.

If you plan on using HSRP/VRRP/STP/ect. on your downstream switches you'll probably want an L2 trunk between them as well.

[Spudstr]

Originally Posted by Techee

Get the 3bxls, they are cheap right now.

Yes, you need to cross-connect the two 6509s, and run iBGP between them. Some topologies also suggest that you run OSPF between them.

BTW, the cross-connect between the two 6509s has to be large enough to handle the TOTAL traffic if one of the carriers disappears. Not an issue now (with two uplinks) but I have seen companies that have expanded, added a couple of GigE and a 10Gig uplink, yet still had only a single GigE between the 6509s!

Realistically the channel between the two devices is ment for igp traffic. If you have a core die, your channel/trunk is't going to do you very much.

Cisco has a great igp protocol called EIGRP which frankly if your fully a cisco shop is the best internal routing protocol to use. BGP works but it can be cumbersom. OSPF is one of the more preferred cross vender/platforms for IGP. But EIGRP wins hands down in a cisco shop.

HSRP on both devices for your vlans, multi home each server directly to each 6500 for best results on HSRP.

Keep in mind to keep life "easier" on you run connections to both carriers to both devices or at least pick 1 carrier to run to both devices, if your inbound preferred routes drop on Carrier A. Carrier B will have to take over and your inbound traffic, i.e all your traffic is going to be hosed until your BGP re-converges on the internet. This is eliminated if you have preferred inbound traffic over a single network that has network diversity to your equipment. You can run both connections to a single device of the network of choice or to different cores on your upstream which will give you better network resilience.

Make sense?

[Spudstr]

Originally Posted by FastServ

In a collapsed core approach so there is no separate core/edge. The 6509 is an excellent choice given this type of arrangement, especially on a smaller scale.



If you plan on using HSRP/VRRP/STP/ect. on your downstream switches you'll probably want an L2 trunk between them as well.

... HSRP is a link state protocol iirc. If you have a upstream network device go pop and the link is still active, your gateway wont failover to router "b" due to link state.

Its always best practice to plug EACH device DIRECTLY into each device running HSRP for the link-state/failover. Or else you might end up with a lot of traffic going from one end of y our network across your internal trunk then back across to the other route to distribute traffic/routes due to the vlans.. and frankly that will make a mess of things and extra latency inside your network that is not needed. i.e longer pathway.

[genlee]

Originally Posted by Spudstr

... HSRP is a link state protocol iirc. If you have a upstream network device go pop and the link is still active, your gateway wont failover to router "b" due to link state.

Its always best practice to plug EACH device DIRECTLY into each device running HSRP for the link-state/failover. Or else you might end up with a lot of traffic going from one end of y our network across your internal trunk then back across to the other route to distribute traffic/routes due to the vlans.. and frankly that will make a mess of things and extra latency inside your network that is not needed. i.e longer pathway.

You can use track objects to monitor route or peer availability for HSRP.

As for the original question, iBGP will handle the routes between 6500's so no IGP is needed(unless you are peering loopbacks) until you add additional switches over layer 3 links. Be careful not to become a transit between your upstream providers. Use an outbound filter ^$ to allow only local prefixes advertised. Your providers should filter your prefixes but I wouldn't take the chance.

[Jay Suds]

One of the big PITAs is having dual 6500s in a collapsed core/edge with multi-homed, full routing tables is that each of your 6509s will have different routing tables, due to the way BGP path selection works, especially if you use iBGP to redistribute eBGP learned routes to your other 6500. This can be problematic for network trouble shooting purposes - customers routing through one switch will take one route, other customers going through the other switch may take another route.

[bizness]

Originally Posted by Jay Suds

One of the big PITAs is having dual 6500s in a collapsed core/edge with multi-homed, full routing tables is that each of your 6509s will have different routing tables, due to the way BGP path selection works, especially if you use iBGP to redistribute eBGP learned routes to your other 6500. This can be problematic for network trouble shooting purposes - customers routing through one switch will take one route, other customers going through the other switch may take another route.

and this is where fun begins...

there are always route reflectors... or star topology iBGP....

[genlee]

Originally Posted by Jay Suds

One of the big PITAs is having dual 6500s in a collapsed core/edge with multi-homed, full routing tables is that each of your 6509s will have different routing tables, due to the way BGP path selection works, especially if you use iBGP to redistribute eBGP learned routes to your other 6500. This can be problematic for network trouble shooting purposes - customers routing through one switch will take one route, other customers going through the other switch may take another route.

The routing tables would be the same unless you are modifying weight/local pref or using MED with your provider. I have yet to see any internet prefixes come down to iBGP vs EBGP for path selection.

[bizness]

Originally Posted by genlee

The routing tables would be the same unless you are modifying weight/local pref. I have yet to see any internet prefixes come down to iBGP vs EBGP for path selection.

agreed, unless you are changing local prefs, weights they should be the same... the difference can come when you have 3 routers with only 2 of them connecting to the eBGP peers and the 3rd being in the middle with iBGP


ISP1------R1------R2--------R3-------ISP2


in the above scenario i can see how ODD things can happen if you only have iBGP between R1 to R2 and R2 to R3 and NO iBGP from R1 to R3...

[genlee]

Originally Posted by bizness

agreed, unless you are changing local prefs, weights they should be the same... the difference can come when you have 3 routers with only 2 of them connecting to the eBGP peers and the 3rd being in the middle with iBGP


ISP1------R1------R2--------R3-------ISP2


in the above scenario i can see how ODD things can happen if you only have iBGP between R1 to R2 and R2 to R3 and NO iBGP from R1 to R3...

Weight is a local attribute that works similar to local pref. Local pref is carried throughout the AS while weight is local to the router. By adjusting weight, you will modify the local routing table since Cisco checks weight first for path selection.

In your scenario, R1 or R3 would not receive each others routes, only R2 would have the correct routing table. That is a broken design anyway and don't see why it would be implemented.