Results 1 to 25 of 29
Thread: rackspace down?
-
06-29-2009, 04:28 PM #1Web Hosting Master
- Join Date
- Mar 2004
- Location
- Los Angeles
- Posts
- 622
rackspace down?
seems like rackspace.com down.
any others who can access their servers?
-
06-29-2009, 04:29 PM #2Engineer
- Join Date
- Jan 2005
- Location
- Scotland, UK
- Posts
- 2,681
Yeap we have multiple systems down too, started exactly 15 minutes ago.
-
06-29-2009, 04:29 PM #3Problem Solver
- Join Date
- Mar 2003
- Location
- California USA
- Posts
- 13,681
i have 2 clients whom are down.
-
06-29-2009, 04:34 PM #4Engineer
- Join Date
- Jan 2005
- Location
- Scotland, UK
- Posts
- 2,681
Only appears to be servers in the DFW datacenter which just happens to be where the vast majority of ours and most peoples servers are.
-
06-29-2009, 04:37 PM #5Newbie
- Join Date
- Jun 2002
- Location
- Normal, IL
- Posts
- 21
Yep, all of our CloudSites are offline.
-
06-29-2009, 04:37 PM #6Junior Guru Wannabe
- Join Date
- Jun 2003
- Location
- Westminster, MD
- Posts
- 60
Anyone know what happened? Their phones are down as well!
-
06-29-2009, 04:43 PM #7Web Hosting Master
- Join Date
- Jan 2008
- Posts
- 643
yes, rackspace is also down from my end (portugal)
Does the email exchange is also down?
Does anyone knows what is the url of the rackspace network status / noc site?
-
06-29-2009, 04:44 PM #8Engineer
- Join Date
- Jan 2005
- Location
- Scotland, UK
- Posts
- 2,681
They updated twitter with, "We are having an issue that is affecting part of our DFW data center. No details yet. Will update as we get more information."
-
06-29-2009, 04:49 PM #9Newbie
- Join Date
- Jul 2007
- Posts
- 5
Pretty poor they don't have an alternative status page that shows up on a google search for rackspace status.
I only noticed because I use Rackspace's email service, which I didn't expect to go down, ever. Stupid of me to be that optimistic I guess.
-
06-29-2009, 04:52 PM #10Junior Guru
- Join Date
- Jun 2002
- Location
- Phoenix, AZ
- Posts
- 190
-
06-29-2009, 04:55 PM #11New Member
- Join Date
- Jun 2009
- Posts
- 0
Word from inside is that it is another power issue, of which they've had 2-3 in the past month.
-
06-29-2009, 04:55 PM #12New Member
- Join Date
- Jun 2009
- Posts
- 0
they do have a status page, status.rackspacecloud.com
-
06-29-2009, 04:59 PM #13Web Hosting Master
- Join Date
- Aug 2005
- Posts
- 3,587
ah damn, that's why apture.com is down too. That's just terrible, RackSpace is having more and more problems lately.
-
06-29-2009, 05:02 PM #14Web Hosting Master
- Join Date
- Jan 2008
- Posts
- 643
-
06-29-2009, 05:03 PM #15Web Hosting Master
- Join Date
- Jan 2008
- Posts
- 643
back online for me
-
06-29-2009, 05:05 PM #16New Member
- Join Date
- Jun 2009
- Posts
- 0
They do for rackspace but funny enough it's down too.
-
06-29-2009, 05:09 PM #17New Member
- Join Date
- Jun 2009
- Posts
- 0
here too! Power related again. They've been doing a bunch of UPS and Generator testing the last number of weeks so perhaps it is related.
-
06-29-2009, 05:11 PM #18Web Hosting Master
- Join Date
- Aug 2005
- Posts
- 3,587
-
07-02-2009, 09:10 PM #19Design and Hosting Solutions
- Join Date
- May 2005
- Location
- New York
- Posts
- 737
just to follow up on this incident
http://www.datacenterknowledge.com/a...of-25-million/
Interesting lol - wouldn't want to be writing those sort of cheques myself
-
07-07-2009, 12:10 PM #20Engineer
- Join Date
- Jan 2005
- Location
- Scotland, UK
- Posts
- 2,681
They are out again, doesn't seem to effect as many of our systems as the last one did but it's certainly another wide spread issue, including their own website.
-
07-07-2009, 12:33 PM #21Engineer
- Join Date
- Jan 2005
- Location
- Scotland, UK
- Posts
- 2,681
Another power issue,
# uptime
11:25:22 up 4 min
-
07-13-2009, 05:54 PM #22Junior Guru Wannabe
- Join Date
- Mar 2009
- Location
- $HOME
- Posts
- 85
Rackspace's explanation is roughly:
• 13:51: utility breaker trip. Critical load on UPS. Gen cluster starting.
• 14:35: synchronization failure in gen cluster. Critical load back on UPS.
• 15:15: UPSes exhausted, load off.
They don't explicitly say that the UPSes operated from generator for a time, but imply it; and 84 minutes would be very long for battery run-time. Assuming the generators did parallel and power UPSes from 13:51 to 14:35, that would be 40 minutes run-time, which is believable assuming 2N UPSes.
But: if the generators were running in parallel, how did they lose synchronization? It's not as if a generator needs to "try" and stay synchronized. The current on the emergency bus, and in the windings of each generator, forces the generator to stay in sync. I have read that you could shut off fuel supply to a generator, and if the parallel cluster is large enough, the energy on the emergency bus and in the windings will change it from being a generator into being an electric motor: the rest of the cluster will drag it along, making it turn in sync with the rest of the generators, even while it is turned-off. So I don't understand how a set of parallel generators would lose sync.
I did read one account that the generators lost sync with the UPSes. Would that imply that the generators were configured to parallel off the UPSes' critical bus? Maybe that would explain it, if the UPS output Hz. was drifting beyond generator tolerance (but why?... was there no bypass input present when running from generator? If bypass input was also from gen, then the UPS should have sync'd to the generator cluster).Last edited by speedcolo; 07-13-2009 at 06:02 PM.
-
07-13-2009, 06:08 PM #23Junior Guru Wannabe
- Join Date
- Mar 2009
- Location
- $HOME
- Posts
- 85
Rackspace wrote:
“What we saw yesterday was a situation where the generators started fighting with one another on the bus,” Rackspace said of the generator challenges. “The generators were unable to get properly synchronized. Eventually, they failed in a cascading manner and we lost all of the generators. Each generator failed on a loss of excitation – an inability to maintain the magnetic field. But it was really the inability to get synchronized that created that fault.”
-
07-13-2009, 11:13 PM #24Web Hosting Master
- Join Date
- Dec 2001
- Location
- Houston Texas
- Posts
- 4,420
"Each generator failed on a loss of excitation – an inability to maintain the magnetic field"
They probably just need to dump a bottle of viagra in the diesel tanks in case they need them to be able to keep it up next time. Loss of excitation stinks.
but on a serious note
Does anyone know how often they test the gen sets and if they do a full load transfer or if they just start them up and exercise them with no load?
-
07-14-2009, 01:45 AM #25Web Hosting Master
- Join Date
- Feb 2007
- Location
- USA, UK, Singapore
- Posts
- 3,481
Most generally, this is done every 15-30 days by doing a full load transfer in most of the datacenters. At very least, I am aware of Handy Networks doing this regularly.
Similar Threads
-
Rackspace.co.uk
By pushedtomb in forum Web HostingReplies: 3Last Post: 05-01-2006, 06:57 PM -
Rackspace in the UK
By tazd9t9 in forum Dedicated ServerReplies: 2Last Post: 06-24-2004, 03:35 PM -
Rackspace
By box0p in forum Dedicated ServerReplies: 16Last Post: 11-01-2003, 04:59 PM -
Rackspace
By StartAnISP in forum Dedicated ServerReplies: 9Last Post: 10-08-2003, 09:56 AM -
besides rackspace ?
By always-neo in forum Dedicated ServerReplies: 19Last Post: 09-30-2003, 08:14 PM