@elmister - I can confirm that at this time, we have isolated the issue to failed hardware.
@QuickWeb-Roel - There is redundancy in place, however, due to the way that this has failed the failover did not work as expected. Details about this should be released in the RFO.
Our network team is still working to resolve the issue at this time, and they are working in tandem with Cisco TAC.
I appreciate everyones patience during this outage, and as before as soon as I get more information I will update it here. On a side note, we will be sending a full detailed RFO to affected customers early next week.
Even though Kody has updated already, I thought I would address two points.
We do run redundant cisco nexus equipment but due to the nature of the failure it caused issues on the vpc peer-link (Cisco Nexus world) which in turn caused issues on links to the N5k switch fabric. It’s a very complex issue as you can imagine, we had all of our network team & Cisco working on this. Things appear to be back now. Obviously I'd like to apologise and we'll immediately as a team get together and see what else we can do here.
Next week we'll also release our status update code into our CP. Due to us having 9 locations, we'll want it so you can select + see all + pull maintenance info from all locations + from our internal teams + our SoftLayer locations etc. We'll make sure this is launched next week.
Well, we have to release that feature immediately so we will. If you could send your support concerns to Kody - who is in this thread (you can PM him) - he will look at them immediately, he is a major part of the support team we have. Again, I apologise.