Web Hosting Talk : Web Hosting Main Forums : Colocation and Data Centers : BGP default route - how to failover when provider peers down?

[eger]

I'm setting up BGP so I can be multi-homed to a second provider. My switch doesn't have enough memory to receive a full table so I will do equal cost balancing with default routes.

My primary reason for adding a second provider is to help with unplanned downtime when a provider (or their peers) do maintenance. However, my understanding is that with a default route originated to me, even if the providers peers are down, they will still send me a default route and I would still try to send them traffic. This would still make me essentially down (or having at least 50% packet loss) while one of the 2 providers are having issues.

So my question is, what mechanism out there would let me suppress receiving a default route or ignore it when that providers peers are down or having an issue? How can I work around the issue of receiving a default route for a router that might not actually have a working internet connection (or may be having issues)?

[Jay Suds]

If your BGP session is down with a peer, they will not be able to advertise any routes, default or otherwise, to you. You can easily admin down a peer, however, if they are having network troubles and your BGP session is still up. On IOS, it's like ...

to disable:
conf t
router bgp xxxxx
neighbor x.x.x.x shutdown
end
wr mem

to enable:
conf t
router bgp xxxxx
no neighbor x.x.x.x shutdown
end
wr mem

[appliedops]

Originally Posted by eger

My primary reason for adding a second provider is to help with unplanned downtime when a provider (or their peers) do maintenance. However, my understanding is that with a default route originated to me, even if the providers peers are down, they will still send me a default route and I would still try to send them traffic.

That is 100% correct, you will be affected. You need full routes to handle peer outages.

So my question is, what mechanism out there would let me suppress receiving a default route or ignore it when that providers peers are down or having an issue?

Your provider would need to support a conditional advertisement, but I'd say thats unlikely to really provide what you need.

[rhythmic]

Originally Posted by eger

So my question is, what mechanism out there would let me suppress receiving a default route or ignore it when that providers peers are down or having an issue? How can I work around the issue of receiving a default route for a router that might not actually have a working internet connection (or may be having issues)?

It depends on your equipment. There are a lot of different options you have available to you. The most common is to create a ping target that is hit through each connection. If the ping target is down, the router marks the port administratively down proactively, continuing to ping that target until it is working again.

I personally wouldn't recommend this for multihoming, though. If your providers are legit, each of them should have full routing tables from their carriers. This means if they lose any given peer, you might see degraded performance, but you'll still have full accessibility to the Internet. If each of your providers only has one provider themselves, you've got bigger issues!

By the way, keep in mind that ECMP is only going to affect your outbound traffic. You'll have to do creative things with prepending and subnet splitting to balance your outbound traffic.

Do you mind sharing what gear you're using for this?

[FastServ]

What you need is either SLA tracking to up/down poorly performing peers. But this is usually not available some lower end devices. The other option would be to write some scripts to do the SLA tracking on the LAN segment and use something like rancid to automate neighbor changes (like Jay Suds example) on the router when problems are detected.

[eger]

Originally Posted by rhythmic

Do you mind sharing what gear you're using for this?

I am using a Foundry FESX448 to try and do this. I'm an extremely small time shop so I can't afford a real router at the moment. I did finally find some examples which show sla tracking. I also found some examples for doing conditional default-originate with a route-map. But it looks like this needs to be done at the provider end and I doubt many would want to configure and maintain something like that.

The main thing that worries me is that I have had downtime due to "fiber cuts" (not quite sure what the actual problem is yet as still waiting on a RFO) where my network was down but the network port was. I am adding a second provider to take over in the event one is down. But I worry that even though the provider may not pass traffic, the peer session will still be UP and I'll just be sending 50% of the traffic to /dev/null until someone complains or I notice

By the way, thanks for the comments everyone! Really wish I could just do a full view.

[Jay Suds]

During that outage, were you able to ping your providers default gateway? Where exactly was the fiber cut? Between you and them, or between them and the world?

[eger]

I wasn't able to test as I was not on site. I'm guessing it was between them and the world. I only know that my port with them never went down from logging (I log port up/downs). I suppose I should start monitoring my gateways too!

[Jay Suds]

It's pretty sad that a single fiber cut apparently took your provider entirely offline. No redundant transport / transit is not acceptable in this day.

[rhythmic]

Originally Posted by eger

I wasn't able to test as I was not on site. I'm guessing it was between them and the world. I only know that my port with them never went down from logging (I log port up/downs). I suppose I should start monitoring my gateways too!

As long as the BGP session goes down, it doesn't matter if the port is up. The session will timeout in whatever threshold you set (3-5 minutes is typically the default), at which point that default gateway will disappear. I have had plenty of carrier issues where the port was up but the network was down, particularly with providers that backhaul you to another data center for routing. I can't think of a time where I've ever had a provider with the session up but all traffic on their network down, and I'd be rightly pissed it happened.

[Spudstr]

Originally Posted by Jay Suds

It's pretty sad that a single fiber cut apparently took your provider entirely offline. No redundant transport / transit is not acceptable in this day.

To add to this I have seen people who use media converters which give a false positive of "links" being up and hence causing a bgp session to stay "up" until it default times out.... lower bgp timeouts can help with this but I've seen it way to often due to low end media converters.

[appliedops]

Originally Posted by rhythmic

I can't think of a time where I've ever had a provider with the session up but all traffic on their network down, and I'd be rightly pissed it happened.

I've experienced this a few times, it was pretty irritating.
This happens especially with carriers who do multihop/etc, where the direct attach device is sending default+loopbacks, and continues to send default even though the upstream router is down/broken. If you have full routes its less of a big deal, but it can be pretty annoying.

[condoraam]

If your interface down, your BGP will down and your router will auto use your second route in your routing table.

If your upstream whole network down, you can use SLA tracking to send ping request to a target IP. If the connection break, switch to second link.

If your upstream down only a part of network, You cannot control if you are not recvice a full BGP table. If you have full route table. Your router will find second AS-path to your target network.