Results 1 to 20 of 20
Thread: Linode London - Completely down
-
12-27-2015, 01:50 AM #1Web Hosting Master
- Join Date
- Nov 2000
- Location
- localhost
- Posts
- 3,771
Linode London - Completely down
Connectivity Issue
Investigating - We are aware of connectivity issues to Linodes in our London datacenter and are currently investigating.
Dec 27, 05:37 UTC
Disappointing it took nearly 25 minutes to update the status page.
-
12-27-2015, 02:12 AM #2Web Hosting Guru
- Join Date
- Mar 2004
- Location
- UK
- Posts
- 265
Offline for 1 hour and counting...
-
12-27-2015, 02:25 AM #3WHT Addict
- Join Date
- May 2014
- Location
- UK / USA (California)
- Posts
- 105
Same problem here. Linode became an absolute crap in the last 3 days. All their locations are down all day.
Useless service.
-
12-27-2015, 03:42 AM #4Web Hosting Master
- Join Date
- Apr 2000
- Location
- Brisbane, Australia
- Posts
- 2,602
my Linode Fremont, CA servers all fine.. touch wood
maybe they are using holiday period for some maintenance ?
-
12-27-2015, 04:42 AM #5Web Hosting Guru
- Join Date
- Mar 2014
- Posts
- 292
My Dallas Linodes were down twice in the last two days totaling around 25 minutes
Sent from my mobile device.
-
12-27-2015, 04:46 AM #6Web Hosting Master
- Join Date
- Nov 2000
- Location
- localhost
- Posts
- 3,771
@eva2000 its not planned maint
It's been down due to a DDOS for ~3.5hrs now. Traceroutes stop in Osaka Japan at the moment. It almost seems like they turn it back on every ~30 minutes and you see 95% packet loss for a few minutes, then they stop announcing again,as if they are simply waiting for it to fade out. First major 3hr+ outage in London Linode I recall in 5 years
-
12-27-2015, 04:48 AM #7Newbie
- Join Date
- Oct 2015
- Posts
- 23
Looks like another DDoS: http://status.linode.com/
This meant a 5:08am wakeup call for me
To be fair to Linode though it's a double-bank-holiday weekend, the night shift must have been a skeleton. These things happen and it takes time to get engineers physically on location and it appears they were all-hands-on-deck by 6am.
-
12-27-2015, 05:02 AM #8WHT Addict
- Join Date
- Mar 2012
- Posts
- 121
Appears to be a complete blackout in London. Every hour IPs come back online, and then they go offline again. It's as though they have been null routed to prevent packet loss to other customers in the Telecity facility.
-
12-27-2015, 05:06 AM #9Web Hosting Master
- Join Date
- Nov 2000
- Location
- localhost
- Posts
- 3,771
Unfortunately with Linode you never know, they are super secret in post-mortems about all failures, I doubt we'll understand more than "its was a big DDOS" which makes it really difficult to evaluate future risk.
Agreed, yes thankfully it is morning still and double bank holiday weekend, however if a DDOS takes out the whole of London network (which it practically has - practically completely out = partial outage) I worry about next week and the week after. At their size of network they should definitely have already reached out to people like ArborNetworks etc..
-
12-27-2015, 05:37 AM #10Web Hosting Master
- Join Date
- Apr 2000
- Location
- Brisbane, Australia
- Posts
- 2,602
ah DDOSes ! nasty indeed
maybe Linode 2016 will roll out DDOS protection
-
12-30-2015, 06:51 PM #11WHT Addict
- Join Date
- Mar 2012
- Posts
- 121
Linode are being hit hard again. London is down. Looks like uptime may slip into 98% this month... what a nightmare.
-
12-30-2015, 07:15 PM #12WHT Addict
- Join Date
- May 2014
- Location
- UK / USA (California)
- Posts
- 105
What should we expect when after another day of complete downtimes, they said yesterday "no other attacks occurred in the last hour so we consider the problem as resolved". This was on their status page.
Per their opinion a problem is solved with it stops on its own. From their announcement it was pretty obvious that they resolve the problems with the attacks by just waiting the attacks to stop. And voila the attacks are back today for sixth day.
-
12-31-2015, 09:39 AM #13WHT Addict
- Join Date
- Mar 2012
- Posts
- 121
Another day, another Linode outage. My phone now has hundreds of unread SMS outage noifications... this is getting tiresome.
-
01-01-2016, 06:04 PM #14Disabled
- Join Date
- Feb 2015
- Posts
- 100
Atlanta appears to be under attack now, this is pretty ridiculous.
-
01-01-2016, 06:44 PM #15Newbie
- Join Date
- Oct 2015
- Posts
- 23
Here's their first big public update:
In progress - An update from Linode about the recent DDoS attacks
Dec 31, 22:45 UTC
Scheduled - I’d like to share some updates about the recent DDoS attacks.
I am one of several network engineers at Linode who have been working around the clock on DDoS mitigation. While things are stable, I would like to take a moment to publicly address the large and frequent DDoS attacks that we have been receiving since Christmas Day.
It has become evident in the past two days that a bad actor is purchasing large amounts of botnet capacity in an attempt to significantly damage Linode’s business. The following is a partial list of attacks we have received in no particular order:
- Multiple volumetric attacks simultaneously directed toward all of our authoritative nameservers, causing DNS hosting outages
- Multiple volumetric attacks simultaneously directed toward all of our public-facing websites, causing Linode Manager outages
- Layer 7 (“400 bad request”) attacks toward our web and application servers, causing Linode Manager outages
- Large volumetric attacks toward our colocation provider’s upstream interconnection points, overwhelming the router control planes and causing significant congestion/packet loss
- Large volumetric attacks toward Linode network infrastructure, overwhelming the router control planes and causing significant congestion/packet loss
All of these attacks have occurred multiple times. Over the course of the last week, we have seen over 30 attacks of significant duration and impact. As we have found ways to mitigate these attacks, the vectors used inevitably change.
As of this afternoon, we have mostly hardened ourselves against the above attack vectors, but we expect more to come. We are working extremely closely with all of our technical partners, including our network equipment vendors and our colocation providers, to prevent future attacks.
Once these attacks stop, we plan to share a complete technical explanation about what has been happening. Additionally, we will be announcing the details of an ongoing project to significantly improve our internet connectivity and resiliency.
We would like to apologize for the lack of detail in some of our recent status-page updates. Please know that we are dedicating all resources from multiple departments to stopping these attacks. We acknowledge the amount of downtime we’ve been experiencing is completely unacceptable, and we appreciate the understanding and support we have received over the past several days. We will share more information as our investigation continues.
Alex Forster
Network Engineer at Linode
-
01-01-2016, 08:55 PM #16Junior Guru Wannabe
- Join Date
- Jan 2006
- Location
- Western New York
- Posts
- 89
Atlanta has been under attack for 8 hours plus now... sites I use there are completely down. Read a ton today and looks Linode has had some major attacks over the years. Is there something they can be doing but not?
Last edited by Frank Broughton; 01-01-2016 at 09:08 PM.
-
01-02-2016, 01:17 AM #17Disabled
- Join Date
- Dec 2015
- Posts
- 40
Too bad Linode does not provide DDoS protection, even as addon.
-
01-02-2016, 03:37 AM #18Junior Guru Wannabe
- Join Date
- Jan 2006
- Location
- Western New York
- Posts
- 89
-
01-03-2016, 02:13 AM #19Disabled
- Join Date
- Dec 2015
- Posts
- 40
-
02-10-2016, 05:33 AM #20New Member
- Join Date
- Sep 2015
- Posts
- 0
Same problem
Similar Threads
-
Networkx Completely Down
By DATARTIM in forum Providers and Network Outages and UpdatesReplies: 2Last Post: 09-12-2006, 09:14 PM -
Rackspace - London 2 - Down!
By AH-99P in forum Providers and Network Outages and UpdatesReplies: 4Last Post: 07-18-2006, 07:26 AM -
Server At PixelFX Completely Down!!
By chiehkai in forum Providers and Network Outages and UpdatesReplies: 8Last Post: 07-05-2006, 08:41 PM -
a2hosting.com - completely down?
By harriska2 in forum Providers and Network Outages and UpdatesReplies: 3Last Post: 06-01-2006, 12:11 PM -
Pair.com completely down?!
By fluffybunny888 in forum Providers and Network Outages and UpdatesReplies: 8Last Post: 04-01-2006, 10:15 PM