Please be aware that we will be carrying out emergency maintenance work on one of our London core routers this evening, starting at 9.30pm. This work should not be service affecting, and the maintenance window is expected to last 15 minutes.
Once this work has been completed, additional capacity from Level3 will be bought online tomorrow afternoon to increase our network throughput and latency at both Maidenhead and London data centres.
Please feel free to contact us should you have any further questions or comments.
As you may have been aware we suffered a loss of our Level3 connectivity after the maintenance of our router this evening.
We were saddened to hear that this downtime was caused by human error on Level3's part.
Some weeks ago Level3 removed our AS number from the RIPE database for reasons not yet known to us. Because tonight's maintenance took down our BGP session to Level3 for a very brief period, the changes in the RIPE database came in to effect, and our AS numbers and IP range was withdrawn from the BGP sessions on the Level3 routers.
We immediately contacted Level3 once our connection was lost, and they began investigation. After 1 hour of investigation they were no closer to resolution. We then realised that the RIPE entry was missing and informed Level3 of this error. They then confirmed this was an error on their part after a further 1 hour of checking and continuous engagement.
The RIPE database was then updated at midnight by Level3 and we waited for the BGP information, as well as the RIPE database to propagate to the internet.
We now have full connectivity restored at 12.30.
We will be following this matter up with Level3 senior management tomorrow morning, and we will expect to receive a detailed fault report which we can pass onto our customers.
As we are sure you will agree, this kind of 'error' is not acceptable and should not have happened if the proper checks were in place by Level3 from the start.
We sincerely apologise for any inconvenience this downtime may have caused you and your customers.
Rest assured we will be pursuing this matter further.