14Hours + network outage experience with 100TB, could someone help me?
I was a happy user with 100TB, yeah, I was... but today, I was totally disappointed. Here is the story of the broken heart customer.
I have some machines on 100TB, they all work fine, except sometimes there are some minor network outage, annoying but still acceptable. Today, I was woke by a phone call at 7:00 am, I was told that our website, whole the service was down. I then check what happened to my service. And I eventually found a ticket in the support system of 100TB. Here they said:
DDOS: Automated System Alert - Null Route - 173.xxx
This ticket was automatically generated by the Softlayer Network Protection System. Due to the large amount of traffic targeted to your IP address 173.xxx, SoftLayer has automatically injected a null route for the IP address into our network. This null route automatically tells our carriers to drop all traffic destined to the IP address 173.xxx at the edge of their networks. The null route of 173.xxx will remain in place for 24 hours and traffic levels to this IP address will be retested.
Please let us know if you have any questions.
Dedicated Server Engineer
What !? They took off all my traffic for 24 hours just because of DDOS attack? Well, even there are DDOS attacks, so what? These things happen all the days. And I inspected the traffic of the server
I even can't tell is there a attack from the traffic diagram. It was 2:00 AM in our local time, the traffic was very low, I even don't know what kind of attack it was, I even have no chance to set the IPtables, then my site was killed. What kind of DDOS can take my site down for 24 hours, but our service provider did. What kind of protection it is?
If the server was a loading balancing server, that's not a big deal. But unfortunately, all infrastructures are in that machine, such as MySQL, nginx, message bus. I have no redundant machine for those infrastructure services. There are so many configurations was tunned, it's hard to deploy those services on another server in short period.
All I can do was to submit tickets.... but all the response I got was something like
"We're handling this, please stand by for update"
"Please note that we are awaiting update from the SL engineers, shall get back to you once we hear from them."
I have already received some blaming emails from users who know my email address. I have no courage to reply them and explain that we are sorry, but our server was killed by the hosting provider.
14+ hours has been pass, my machine was still isolated. All I can do is sit here, watch the ping result, refresh the page, refresh gmail to see ticket response, ping again, ping again and ping again.....
I'm not going to blame 100TB, their are really good at providing good bandwidth in such a good price.
Here, I would like to ask... does anybody encounter same problem? Are there any action I can take to solve the problem? Are there other hosting providers can also provide bandwidth in good price but also have good quality?
DDoS affect everybody, 100TB, 100TB clients, and you.
DDoS attacks can take your site down for months no problem! it can go from 10 Mbps to 40 Gbps (that is the biggest one registered I think), but they are not free, if you are getting DDoS its because somebody is paying for the attack (most of the times it cost $30 to $100 per hour)
Getting null routed is a common procedure among host, is just a block in the higher carrier, in this case your traffic gets null routed at the internet provider/carrier of 100TB so that traffic doesn't affect other 100TB clients in their network, you will get nulled until the attack ends.
Changing host doesn't really solve the problem because the attack is in your domain (most of the times) and not the IP. There are friendly DDoS host but they are expensive and have their limits also.
The only true solution against a Distributed Denial of Service DDoS attack is a Geo-distributed Hosting like for example google, having thousand of servers around the world and this is out of the question. Every other solutions like DDoS hardware or DDoS friendly host will help a lot but still have limits of bandwidth or hardware PPS.