Page 1 of 2 12 LastLast
Results 1 to 40 of 68
  1. #1
    Join Date
    Apr 2008
    Posts
    439

    FDC Denver Outage

    Looks like FDC Denver location is down. Anyone experiencing the issue?

  2. #2
    Join Date
    Aug 2004
    Location
    SF, CA, USA
    Posts
    155
    I have some servers colo'd there and they shutdown because of thermal overload....they were reporting 114F+ on the ambient temp....

    I wonder if their CRAC shut down

  3. #3
    Join Date
    Apr 2008
    Posts
    439
    Did you get any response from techs? I sent two tickets, one to Denver 1 hour ago and one to Chicago. Chicago tech moved ticket to Denver, then Azam from Denver moved it to Chicago. However, I still have nothing in both tickets.

    Btw, Denver looking glass is UP...

  4. #4
    Join Date
    Aug 2004
    Location
    SF, CA, USA
    Posts
    155
    I have not, opened an emergency ticket in Denver 14 minutes ago asking what's going on.

  5. #5
    Join Date
    Jul 2006
    Location
    Australia
    Posts
    3,059
    I have 2 servers coloed at Denver and they are both fine.

  6. #6
    Join Date
    Aug 2004
    Location
    SF, CA, USA
    Posts
    155
    Quote Originally Posted by BeZazz View Post
    I have 2 servers coloed at Denver and they are both fine.
    Can you check the ambient temperatures for the two boxes you have colo'd? Via IPMI or the OS?

    Just want to compare notes.

  7. #7
    Join Date
    Apr 2008
    Posts
    439
    Still down, 1 hour.

    I am wondering why techs are not responding, they are there for sure and they saw my tickets.

  8. #8
    Join Date
    Aug 2004
    Location
    SF, CA, USA
    Posts
    155
    Just got a response to my emergency ticket:

    There is outage in denver datacenter, we are working on restoring service, that
    you for your patience.

  9. #9
    Join Date
    Jul 2006
    Location
    Australia
    Posts
    3,059
    Quote Originally Posted by rnavarro View Post
    Can you check the ambient temperatures for the two boxes you have colo'd? Via IPMI or the OS?

    Just want to compare notes.
    Here is the best I can do.

    Server 1

    [[email protected] ~]# sensors
    k8temp-pci-00c3
    Adapter: PCI adapter
    Core0 Temp: +61.0C
    Core1 Temp: +62.0C

    k8temp-pci-00cb
    Adapter: PCI adapter
    Core0 Temp: +54.0C
    Core1 Temp: +55.0C


    Server 2
    (Been running scripts flat out on it for hours)

    [[email protected] ~]# sensors
    k8temp-pci-00c3
    Adapter: PCI adapter
    Core0 Temp:
    +72C
    Core1 Temp:
    +72C

    k8temp-pci-00cb
    Adapter: PCI adapter
    Core0 Temp:
    +68C
    Core1 Temp:
    +71C

  10. #10
    Join Date
    Apr 2008
    Posts
    439
    rnavarro, can you please check where exactly your servers located? I mean rack ID, row ID or VLAN ID. Maybe we are located in the same segment and the segment is experiencing power/networking/cooling issue.

  11. #11
    Join Date
    Aug 2004
    Location
    SF, CA, USA
    Posts
    155
    Quote Originally Posted by Ilya74 View Post
    rnavarro, can you please check where exactly your servers located? I mean rack ID, row ID or VLAN ID. Maybe we are located in the same segment and the segment is experiencing power/networking/cooling issue.
    PM'd .

  12. #12
    Join Date
    Aug 2004
    Location
    SF, CA, USA
    Posts
    155
    @BeZazz - Thanks for that....those CPU temps look a little high for my blood...but I'm glad your gear is working

  13. #13
    Join Date
    Jul 2006
    Location
    Australia
    Posts
    3,059
    Quote Originally Posted by rnavarro View Post
    @BeZazz - Thanks for that....those CPU temps look a little high for my blood...but I'm glad your gear is working
    Yes they are a bit high but not bad enough that I would shut them down.


    thanks

  14. #14
    Join Date
    Aug 2004
    Location
    SF, CA, USA
    Posts
    155
    Hmmm it appears temperatures are still going up....my server is shutdown but the ambient is now reading:

    Ambient Temp: 120.20F/49.00C

    Edit

    08:52 PST - The networking for my IPMI just went down.... *sad panda*
    Last edited by rnavarro; 12-05-2013 at 12:53 PM.

  15. #15
    Join Date
    Jul 2006
    Location
    Australia
    Posts
    3,059
    I just lost connection to one server.

    EDIT: Back up.
    Last edited by BeZazz; 12-05-2013 at 12:59 PM.

  16. #16
    Join Date
    Apr 2008
    Posts
    439
    BeZazz, may I know where is that server located? I am trying to understand if there is issue in one segment or it is global Denver issue.

    Since Denver looking glass is still online and loads fast, it seems not all DC is experiencing the issue.

    Edit:
    It seems routers are also online, since all gateways are pinging.

  17. #17
    Join Date
    Jul 2006
    Location
    Australia
    Posts
    3,059
    Both are in
    Basement: B-200

  18. #18
    Join Date
    Apr 2008
    Posts
    439
    I meant rack ID or row ID. As far I know, "Basement: B-200" is just the name of the DC space.

  19. #19
    Join Date
    Jul 2006
    Location
    Australia
    Posts
    3,059
    The one that down ofr about 15 mins
    VLAN 140 - 10/100 Racks: Shelf 92
    8


    The other
    VLAN 140 - 10/100 Racks: Shelf 101
    35
    Last edited by BeZazz; 12-05-2013 at 01:14 PM.

  20. #20
    Join Date
    Apr 2008
    Posts
    439
    Thank you. We both located in the neighboring VLANs and racks. So the problem may be in there somewhere.

  21. #21
    Join Date
    Jul 2006
    Location
    Australia
    Posts
    3,059
    I have just lost connection to both servers I have there.

  22. #22
    Join Date
    Mar 2009
    Location
    Houston, TX
    Posts
    1,114
    Hopefully they fix this soon. We haven't had issues with them for ages in Denver.

  23. #23
    Join Date
    Apr 2008
    Posts
    439
    The tech just replied:
    Due to extreme weather conditions in the Denver area, we are are currently experiencing equipment failure. We hope to get it resolved asap.

  24. #24
    Join Date
    Aug 2004
    Location
    SF, CA, USA
    Posts
    155
    Quote Originally Posted by Ilya74 View Post
    The tech just replied:
    I just got the same response....canned message! haha

  25. #25
    Join Date
    Mar 2012
    Posts
    53
    What weather?
    http://www.wunderground.com/cgi-bin/...y=denver%2C+co

    My colo is down as well, almost 2 hours...

  26. #26
    Join Date
    Aug 2004
    Location
    SF, CA, USA
    Posts
    155
    Quote Originally Posted by spork985 View Post
    What weather?
    http://www.wunderground.com/cgi-bin/...y=denver%2C+co

    My colo is down as well, almost 2 hours...
    Haha I took a look at the weather map and asked the same question!

    I can't even reach my IPMI interfaces anymore, so I'm completely in the dark....

    I opened my ticket 3 hours ago...

  27. #27
    Join Date
    Apr 2008
    Posts
    439
    As far as I know it's cold enough for Denver.

    Compared to 2012: http://www.wunderground.com/history/...q_statename=NA
    And now: http://www.wunderground.com/history/...q_statename=NA

  28. #28
    Join Date
    Aug 2004
    Location
    SF, CA, USA
    Posts
    155
    Well it is pretty cold in Denver right now relative to the average....but that should make it easier to cool the servers! haha

    Just open the doors

  29. #29
    Join Date
    Apr 2008
    Posts
    439
    Perhaps the cooling system has failed (frozen? heh), which is why there was so hot. If your sensors were correct, then it was +50C in the DC and -24C outside. Looks bad.

  30. #30
    Join Date
    Aug 2004
    Location
    SF, CA, USA
    Posts
    155
    SWEEEET I think things are coming back online.

    My IP just came back, and IPMI is now reporting:

    Ambient Temp: 69.80F/21.00C

    Which is much more inline with expectations

  31. #31
    Join Date
    Apr 2008
    Posts
    439
    Yeah, glad you are online now. Our switches are still down

  32. #32
    Join Date
    Mar 2012
    Posts
    53
    Mine's back online as well. I didn't get any explanation as to the problem though.

  33. #33
    Any news ?

  34. #34
    3 mines are still down

  35. #35
    Join Date
    Apr 2008
    Posts
    439
    Our switches are still down too. No updates from support.

  36. #36
    Join Date
    Aug 2004
    Location
    SF, CA, USA
    Posts
    155
    I haven't heard anything back with respect to cause, but the temperatures seems to be holding steady at Ambient Temp: 82.40F/28.00C

    I've gone ahead and powered on my servers again and i'm running some consistency checks on the drives and file systems.

  37. #37
    Any idea what happens over there ?

  38. #38
    Join Date
    Sep 2005
    Location
    Albany, NY
    Posts
    3,795
    I believe this is a very isolated issue in the basement, I was told that AC failed there, no ETA yet. We only have 2 or 3 machines down, which is a small fraction of what we have there. Still not fun though. Just have to wait.

  39. #39
    Join Date
    Apr 2008
    Posts
    439
    They told
    We believe all connectivity has now been restored. Please reopen this ticket if
    you are still experiencing issues.

    Thanks
    However, we are still down. Anyone else?

  40. #40
    2 of 3 come back online, 1 still down
    the one in : Shelf 1

Page 1 of 2 12 LastLast

Similar Threads

  1. FDC - Denver is down
    By badboyx in forum Providers and Network Outages and Updates
    Replies: 0
    Last Post: 01-31-2013, 08:08 AM
  2. FDC Denver DOWN?
    By linuxperts in forum Providers and Network Outages and Updates
    Replies: 11
    Last Post: 05-19-2011, 02:58 PM
  3. FDC - denver is down
    By badboyx in forum Providers and Network Outages and Updates
    Replies: 22
    Last Post: 11-12-2010, 07:24 AM
  4. FDC Denver Down
    By brc_csf in forum Providers and Network Outages and Updates
    Replies: 22
    Last Post: 10-28-2010, 04:58 PM
  5. {ASK} FDC Chicago v/s FDC Denver ! Which one is better ?
    By fdc-downalways in forum Dedicated Server
    Replies: 8
    Last Post: 06-11-2010, 03:02 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •