Results 1 to 30 of 30
  1. #1

    Data Center's Canada DOWN

    Wouldn't it be nice to have an update?

  2. #2
    Join Date
    Nov 2006
    Location
    USA
    Posts
    1,265
    Would be nice if you actually took the time to post some relevant information.....

    Company Name ... would be an excellent start.

  3. #3
    The company name is Data Center's Canada...........

  4. #4
    Join Date
    Nov 2006
    Location
    USA
    Posts
    1,265
    Quote Originally Posted by Chasem View Post
    The company name is Data Center's Canada...........
    My apologies!

    I saw you other posts and assumed you were trying to bring up someone else based on your response to the Continuum outage thread.

  5. #5
    Sorry to be snappy... Frustration sets in fast when customers are calling you, and it's not your fault.

  6. #6
    I've been calling into their NOC for updates since about 7am ... our servers started to have problems about 3:45am then at 6am we went into a complete outage situation. Up until then we were troubleshooting our own gear ... i called into the NOC at about 7am and they said there was a peering problem with or between Rogers and Bell ... it's unclear the extent of the issue but it certainly is an issue for DCC (and us as well)

    I've called in several times and the last update i got was that they were trying to re-route ... I've noticed since that services worked from my home Rogers connection but no where else (that was about 9am) then about 10 min ago i couldn't connect from Rogers anymore but can from the US ... so obviously there are still routing issues.

    Any other customers of DCC have updates from their end they'd like to share?

    I have many peeved off clients calling me directly now ... so the more info the better.

    ETA would be nice.

  7. #7
    I just called for an update and the mailbox seems full now ... we've been up and down since just before 4am ... this is insane.

    Anyone have a recent status?

  8. #8
    Join Date
    Mar 2003
    Location
    WhereIts, Summer
    Posts
    34

    Also down.

    Same here, been up & down, but mostly down since 3.30 ET.

    Quote Originally Posted by OpenLeaf View Post
    I just called for an update and the mailbox seems full now ... we've been up and down since just before 4am ... this is insane.

    Anyone have a recent status?

  9. #9
    And the first order of business after the servers are up and running.... Going on 7 hours of downtime now.

  10. #10
    I'm at the data centre now - there are about 12 or more clients here I think ... DCC staff just provided an update to one of the guys saying Bell injected a whole bunch of bad routes ... No word on why it's not resolved though.

    After almost 18 years of managing BGP connections for ISPs I've never seen this ... This should be solvable quickly ... And we should be getting updates.

  11. #11
    Thanks for the update OpenLeaf. Maybe you can help them fix it? haha

    I just saw the Uptime Robot Alert, servers started having issues at 1:14AM

    Pretty mind boggling that its actually over 10 hours of service outtage

  12. #12
    Any word on the disaster...?

  13. #13
    The DCC people should have a customer communication plan in place for issues like this.

    The fact that I cannot reach anyone for an update (nor their website/email because their DNS is also hosted entirely internally) is bad news. Redundancy doesn't just mean multiple connections.

    I started having connectivity problems around 2am with full loss at 5am. According to my contract I am entitled to one day of service for each hour of downtime, so we are at roughly 11 days of service compensation so far.

    Can anyone out there confirm that this is going to be resolved today?

  14. #14
    I have a VoIP client claiming they are back up - but we need to verify still.

  15. #15
    Here is a very fresh press release (yesterday). Anyone want to speculate on whether this change went badly and is related:

    [missing]

    the forum won't let me post a link because i'm new .. google for the terms:

    "gtt data centers canada" and click the first link

  16. #16
    Now ... that IS fascinating ... my traceroutes from the US were showing hops with GT in the DNS name ... i immediately thought GroupTelecom or something like that ... but now that we (appear) to be back online i don't see those hops any more ...

    We are still checking - some customers seem up but this happened earlier today to so we're not counting the chickens quite yet ...

  17. #17
    We are STILL down... *sigh*

  18. #18
    Another interesting issue ... since this is a BGP issue - why has one of my access ports to the access switch at DCC bounced twice ... seems like someone is rebooting switches ... but what would access layer devices have to do with it ... i still think they are doing stuff ...

    we are not out of the wood ...

  19. #19
    If I was in Ontario I would be pulling all my equipment right now.

    Unfortunately I'm in Alberta...

  20. #20
    Join Date
    Mar 2003
    Location
    WhereIts, Summer
    Posts
    34

    Can help

    So your done with things ?

    I'm sure Openleaf, my tech, or someone local could assist with that process

    I know this sucks, but we can only hope things will be better whenever its finally back up.

    Quote Originally Posted by Chasem View Post
    If I was in Ontario I would be pulling all my equipment right now.

    Unfortunately I'm in Alberta...

  21. #21
    Quote Originally Posted by matador View Post
    So your done with things ?

    I'm sure Openleaf, my tech, or someone local could assist with that process

    I know this sucks, but we can only hope things will be better whenever its finally back up.
    Thanks for the support, I have a business partner going there tonight to start offloading data.

    I chose the absolute wrong place to host Telecom hardware. Phones being down for 11+ hours doesn't exactly work out.

    It was my mistake for not being geodiversified.

  22. #22
    We have 2 sites and a VPS for DR name services in the US ... today though may have cost us our largest client - but in the end we too need to beef up the other DC and get even more redundancy - at least our corp services and email was up and running so that we could work today ... very frustrating - but the last place we left over a year ago had a 26 hour outage (4 hour planned maintenance that went wrong) and no updates ... i think this will hopefully be a learning experience for EVERYONE that colos and the colo themselves ...

    Especially if this was due to a new upstream being turned online today (as per the press release) ... i'll hold judgement for when i see the incident report and the rebate.

  23. #23
    Up and running now... Hopefully the compensation balances out the customers I lost... heh

  24. #24
    interesting ... their website is not up ... everything seems fine but their website is still not accessible

  25. #25
    Join Date
    Jan 2006
    Location
    Toronto, Canada
    Posts
    249
    They are back up, I just got home from the facility. Was like a movie scene - cars everywhere, people everywhere... Glad it's back up. Looking forward to getting a formal RFO.

  26. #26
    RFO came out - i'm not too happy with it as it seems to try to indicate this was not a problem that fits the SLA criteria and therefore a financial reimbursement.

  27. #27
    Join Date
    Mar 2003
    Location
    WhereIts, Summer
    Posts
    34
    OpenLeaf,

    I never got the RFO, would be interested to see this.

  28. #28
    I think you have to open a ticket with support or noc to get the report. if you send a request they will send it over.

  29. #29
    Openleaf,

    Can you clarify what in the RFO makes the SLA not apply?

  30. #30
    There was a note at the end "No physical hardware or fiber failures occurred during this outage, the issue lay outside the network edges." ... it concerned me.

    Hoping to clarify today with them ... by now everyone else should have gotten the incident report if you were affected ... did you get it?

Similar Threads

  1. I want to buy vps that data center in canada
    By pc1986 in forum VPS Hosting
    Replies: 2
    Last Post: 02-09-2012, 06:23 AM
  2. Need Best Data Center in Canada
    By Vinayak_Sharma in forum Dedicated Server
    Replies: 17
    Last Post: 10-19-2011, 08:25 PM
  3. Data Center in Canada OK for USA Website?
    By Crim50 in forum Web Hosting
    Replies: 14
    Last Post: 07-08-2008, 07:55 PM
  4. East Coast Canada Data Center
    By Timelord_ in forum Dedicated Server
    Replies: 11
    Last Post: 03-22-2008, 05:27 PM
  5. Data Center in Calgary, AB? Canada!
    By SeanK in forum Dedicated Server
    Replies: 27
    Last Post: 06-24-2003, 03:33 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •