Page 1 of 2 12 LastLast
Results 1 to 25 of 29

Thread: rackspace down?

  1. #1
    Join Date
    Mar 2004
    Location
    Los Angeles
    Posts
    622

    rackspace down?

    seems like rackspace.com down.
    any others who can access their servers?

  2. #2
    Join Date
    Jan 2005
    Location
    Scotland, UK
    Posts
    2,681
    Yeap we have multiple systems down too, started exactly 15 minutes ago.

  3. #3
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681
    i have 2 clients whom are down.

  4. #4
    Join Date
    Jan 2005
    Location
    Scotland, UK
    Posts
    2,681
    Only appears to be servers in the DFW datacenter which just happens to be where the vast majority of ours and most peoples servers are.

  5. #5
    Join Date
    Jun 2002
    Location
    Normal, IL
    Posts
    21
    Yep, all of our CloudSites are offline.

  6. #6
    Join Date
    Jun 2003
    Location
    Westminster, MD
    Posts
    60
    Anyone know what happened? Their phones are down as well!

  7. #7
    Join Date
    Jan 2008
    Posts
    643
    yes, rackspace is also down from my end (portugal)


    Does the email exchange is also down?

    Does anyone knows what is the url of the rackspace network status / noc site?

  8. #8
    Join Date
    Jan 2005
    Location
    Scotland, UK
    Posts
    2,681
    They updated twitter with, "We are having an issue that is affecting part of our DFW data center. No details yet. Will update as we get more information."

  9. #9
    Pretty poor they don't have an alternative status page that shows up on a google search for rackspace status.

    I only noticed because I use Rackspace's email service, which I didn't expect to go down, ever. Stupid of me to be that optimistic I guess.

  10. #10
    Join Date
    Jun 2002
    Location
    Phoenix, AZ
    Posts
    190
    Quote Originally Posted by sieb View Post
    Yep, all of our CloudSites are offline.
    holy crap, sieb is on WHT

  11. #11
    Word from inside is that it is another power issue, of which they've had 2-3 in the past month.

  12. #12
    they do have a status page, status.rackspacecloud.com

  13. #13
    Join Date
    Aug 2005
    Posts
    3,587
    ah damn, that's why apture.com is down too. That's just terrible, RackSpace is having more and more problems lately.

  14. #14
    Join Date
    Jan 2008
    Posts
    643
    Quote Originally Posted by jyarnold View Post
    they do have a status page, status.rackspacecloud.com
    that site is only for the mosso/cloud network.

    i guess rackspace does not have a similar site for they core-business!

    very bad...

  15. #15
    Join Date
    Jan 2008
    Posts
    643
    back online for me

  16. #16
    They do for rackspace but funny enough it's down too.

  17. #17
    here too! Power related again. They've been doing a bunch of UPS and Generator testing the last number of weeks so perhaps it is related.

  18. #18
    Join Date
    Aug 2005
    Posts
    3,587

  19. #19
    Join Date
    May 2005
    Location
    New York
    Posts
    737
    just to follow up on this incident
    http://www.datacenterknowledge.com/a...of-25-million/

    Interesting lol - wouldn't want to be writing those sort of cheques myself

  20. #20
    Join Date
    Jan 2005
    Location
    Scotland, UK
    Posts
    2,681
    They are out again, doesn't seem to effect as many of our systems as the last one did but it's certainly another wide spread issue, including their own website.

  21. #21
    Join Date
    Jan 2005
    Location
    Scotland, UK
    Posts
    2,681
    Another power issue,

    # uptime
    11:25:22 up 4 min

  22. #22
    Join Date
    Mar 2009
    Location
    $HOME
    Posts
    85
    Rackspace's explanation is roughly:
    • 13:51: utility breaker trip. Critical load on UPS. Gen cluster starting.
    • 14:35: synchronization failure in gen cluster. Critical load back on UPS.
    • 15:15: UPSes exhausted, load off.

    They don't explicitly say that the UPSes operated from generator for a time, but imply it; and 84 minutes would be very long for battery run-time. Assuming the generators did parallel and power UPSes from 13:51 to 14:35, that would be 40 minutes run-time, which is believable assuming 2N UPSes.

    But: if the generators were running in parallel, how did they lose synchronization? It's not as if a generator needs to "try" and stay synchronized. The current on the emergency bus, and in the windings of each generator, forces the generator to stay in sync. I have read that you could shut off fuel supply to a generator, and if the parallel cluster is large enough, the energy on the emergency bus and in the windings will change it from being a generator into being an electric motor: the rest of the cluster will drag it along, making it turn in sync with the rest of the generators, even while it is turned-off. So I don't understand how a set of parallel generators would lose sync.

    I did read one account that the generators lost sync with the UPSes. Would that imply that the generators were configured to parallel off the UPSes' critical bus? Maybe that would explain it, if the UPS output Hz. was drifting beyond generator tolerance (but why?... was there no bypass input present when running from generator? If bypass input was also from gen, then the UPS should have sync'd to the generator cluster).
    Last edited by speedcolo; 07-13-2009 at 06:02 PM.

  23. #23
    Join Date
    Mar 2009
    Location
    $HOME
    Posts
    85
    Rackspace wrote:

    “What we saw yesterday was a situation where the generators started fighting with one another on the bus,” Rackspace said of the generator challenges. “The generators were unable to get properly synchronized. Eventually, they failed in a cascading manner and we lost all of the generators. Each generator failed on a loss of excitation – an inability to maintain the magnetic field. But it was really the inability to get synchronized that created that fault.”
    This implies that the generators were not quite paralleled, but yet were still powering load for half an hour. I don't understand how that could work. Once the paralleling bus closes, if they weren't in sync, then they would yank each other into sync... violently if necessary.

  24. #24
    Join Date
    Dec 2001
    Location
    Houston Texas
    Posts
    4,420
    "Each generator failed on a loss of excitation – an inability to maintain the magnetic field"

    They probably just need to dump a bottle of viagra in the diesel tanks in case they need them to be able to keep it up next time. Loss of excitation stinks.

    but on a serious note

    Does anyone know how often they test the gen sets and if they do a full load transfer or if they just start them up and exercise them with no load?

  25. #25
    Join Date
    Feb 2007
    Location
    USA, UK, Singapore
    Posts
    3,481
    Most generally, this is done every 15-30 days by doing a full load transfer in most of the datacenters. At very least, I am aware of Handy Networks doing this regularly.

Page 1 of 2 12 LastLast

Similar Threads

  1. Rackspace.co.uk
    By pushedtomb in forum Web Hosting
    Replies: 3
    Last Post: 05-01-2006, 06:57 PM
  2. Rackspace in the UK
    By tazd9t9 in forum Dedicated Server
    Replies: 2
    Last Post: 06-24-2004, 03:35 PM
  3. Rackspace
    By box0p in forum Dedicated Server
    Replies: 16
    Last Post: 11-01-2003, 04:59 PM
  4. Rackspace
    By StartAnISP in forum Dedicated Server
    Replies: 9
    Last Post: 10-08-2003, 09:56 AM
  5. besides rackspace ?
    By always-neo in forum Dedicated Server
    Replies: 19
    Last Post: 09-30-2003, 08:14 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •