HE FMT2 Core Router

[dmst]

So far today I've seen at least 3 outages at Hurricane Electric's FMT2 facility. The latest occurred just a few minutes ago. Their looking glass shows BGP session durations in the sub three minute range. We've suffered outages there 5 times this week. Things really seem to have gone downhill at FMT2 this year.

Each time this happens I open a ticket to their support to get confirmation so I can document this ongoing problem. Occasionally I get someone clueful who gives me some tidbit or another about what is going on. The best I can piece together is that their router core1.fmt2.he.net has been the subject of repeated DDoS attacks and these attacks create a resetting BGP failure and spike the CPU on the router. One support engineer confided that they are waiting for a security update from Brocade to better manage this problem.

I know a number of you host servers at HE FMT2 in some capacity. Have you detected similar issues on your network?

I'm really sick of all these alerts and issues and I'm sure they are growing sick of me documenting my case.

[ItsChrisG]

Gonna continue happening as a lot of Chinese DDOS users are hosting at HE Fremont now.

[dmst]

So you're saying that HE's network is the *source* of the DDoS attacks?

That sucks.

The hosting provider or providers are violating the ToS.

http://www.he.net/aup.html

Or are these client machines otherwise compromised?

[ItsChrisG]

The machines hosted on HEs network are the targets. The attacks are coming from Chinese ISP's (and else where as well) to servers hosted on HE's network by resellers of dedicated servers.

[dmst]

I'm more sympathetic to the notion that HE is hosting some targets of DDoS attacks. To date HE have indicated that their router itself was the target of the attacks. I suspected it was someone hosting there. Any specific sites you know of that are being subjected to DDoS?

Thanks

[ItsChrisG]

No, they are Chinese websites and/or services.
Gameservers, game cheats, etc etc. They all ddos each other over and over.

[Visbits]

We've black holed many /8s that are in use by china ISP.

A$$holes.

[dot_txt]

We've black holed many /8s that are in use by china ISP.

A$$holes.

+1, we have a number of Chinese subnets blackholed at our edge routers too..

[swiftnoc]

Yes HE.net FMT2 router is under attack a lot lately by substantial DDOS attacks. The latest attack (several hours ago) was ~10 gigabit/s and 12 Mpps
But fact is, a decent Juniper or Cisco router would have swallowed that without too much of an issue. True, without filtering devices in place, the segment behind it would be affected, but it would not affect the whole datacenter. Brocade/Foundry routers have specific weaknesses to DDOS attacks that are currently being exploited. Brocade finally seem to have promised some security update (this i understand from HE.net), but that is in my opinion several years too late - when we last tested Brocade/Foundry (for routing purposes) we already noticed their extreme vulnerability to specific DDOS attacks.

Offtopic
The good news is, that the XMR / RX8 / RX16 series convert to decent coffee tables, just as the classic Cisco 12000 series

[ddosguru]

So you're saying HE is losing entire nodes to 10 Gbps DDoS attacks?

[swiftnoc]

So you're saying HE is losing entire nodes to 10 Gbps DDoS attacks?

Not exactly - but they do get a hell of a lot of Packet-loss across the whole DC. There are some specific DDOS attacks that the RX8/RX16 and its brothers and sisters cannot handle. Personally i would not use them as routers, if i would need to use them, then they would be deployed as (core) switches.

[ddosguru]

I hear absolutely nothing but horrible reviews about Brocade around these parts...

[swiftnoc]

I hear absolutely nothing but horrible reviews about Brocade around these parts...

Yes - as core routers they are very vulnerable. Its pretty easy to remotely reset a BGP session of a Brocade. Many providers handle this by setting up BGP sessions over private IPs with their uplink providers - but its a vulnerability that should not be present in the first place.
These devices have a whole range of vulnerabilities to DDOS attacks.

[ai-danno]

I know a number of you host servers at HE FMT2 in some capacity. Have you detected similar issues on your network?

yes, we've seen this at the same time you have. Got a really friendly guy on the phone.
You know it's bad when they're either extremely friendly or completely frazzled, because it indicates they know something is happening.

[brendanm]

Hmm... This is a interesting thread, especially after we just bought MLXe's (XMR). Guess they are now going to be expensive core switches instead of border routers.

Can anyone PM me specific's on this? All the research I did before buying never discovered this. Why is juniper not effected by the same thing?

[jaymef]

Looks like HE is down again

[mrbmw99]

Down again. Was down for several minutes, now it's sporadic at best (losing about 15% of pings to anywhere beyond their core router in Fremont).

[FastServ]

I have seen this happen a couple times on their Ashburn and UK core as well -- mass BGP flappage blamed on DDOS. Maybe twice in the past year, so this issue isn't new.

[nonprofit-worker]

Yes HE.net FMT2 router is under attack a lot lately by substantial DDOS attacks. The latest attack (several hours ago) was ~10 gigabit/s and 12 Mpps
But fact is, a decent Juniper or Cisco router would have swallowed that without too much of an issue. True, without filtering devices in place, the segment behind it would be affected, but it would not affect the whole datacenter. Brocade/Foundry routers have specific weaknesses to DDOS attacks that are currently being exploited. Brocade finally seem to have promised some security update (this i understand from HE.net), but that is in my opinion several years too late - when we last tested Brocade/Foundry (for routing purposes) we already noticed their extreme vulnerability to specific DDOS attacks.

Offtopic
The good news is, that the XMR / RX8 / RX16 series convert to decent coffee tables, just as the classic Cisco 12000 series

10 gbit is nothing for HE. 12Mpps is also nothing. The routers they would use would handle over 1 billion packets/sec at least

What kind of router is FMT2 using?

[swiftnoc]

10 gbit is nothing for HE. 12Mpps is also nothing. The routers they would use would handle over 1 billion packets/sec at least

In theory they can, in practice they cannot. Depends also what type of attacks. There are many different types of DDOS attacks.
HE.net uses routers from Brocade.

[eming]

Offtopic
The good news is, that the XMR / RX8 / RX16 series convert to decent coffee tables, just as the classic Cisco 12000 series

ya, I've got two of those

[nonprofit-worker]

In theory they can, in practice they cannot. Depends also what type of attacks. There are many different types of DDOS attacks.
HE.net uses routers from Brocade.

Everyone had always suggested to not use foundry for core routing for as long as I can remember so that explains a lot. I have heard they make great switches.

I would never use a foundry for core routing indeed and for that matter Brocade/Foundry

[ShaunH]

Yes HE.net FMT2 router is under attack a lot lately by substantial DDOS attacks. The latest attack (several hours ago) was ~10 gigabit/s and 12 Mpps
But fact is, a decent Juniper or Cisco router would have swallowed that without too much of an issue. True, without filtering devices in place, the segment behind it would be affected, but it would not affect the whole datacenter. Brocade/Foundry routers have specific weaknesses to DDOS attacks that are currently being exploited. Brocade finally seem to have promised some security update (this i understand from HE.net), but that is in my opinion several years too late - when we last tested Brocade/Foundry (for routing purposes) we already noticed their extreme vulnerability to specific DDOS attacks.

Offtopic
The good news is, that the XMR / RX8 / RX16 series convert to decent coffee tables, just as the classic Cisco 12000 series

This is rather interesting. I wonder what Wiredtree's experience has been? Wiredtree Zac mentioned they recently upgraded to Brocade MLXe core routers.

I never realized Brocade had such an issue with Doss attacks.

[swiftnoc]

I never realized Brocade had such an issue with Doss attacks.

They are not vulnerable to every type of DDOS attacks, but they have a few specific vulnerabilities that can be exploited by the wrong kind of people

[kenny408]

Our connection keeps dropping at the FM2 location this morning 9AM PST. I called their NOC and they said they are under DDOS attacks.