Web Hosting Talk : Web Hosting Main Forums : Industry Announcements : Providers and Network Outages and Updates : Rackspace Cloud Server Outage

[Joshua]

One of Rackspace's Cloud Host Servers has failed. This is the second major outage on this server since June, when it was down for 8 hours.

Dear Customer,

Today December 23rd 2010, at approximately 11:02:35 UTC, your host server became unresponsive and after extensive troubleshooting and attempts to bring it back online, we determined that it experienced a catastrophic failure. All customer data on that host has been lost. If you have backups, we strongly urge you to create a new server from that backup, and we will share the old server's IP address over to the new one.

We sincerely apologize for any inconvenience this may have caused you. If you have any further questions, please contact a member of our support team by visiting us in live-chat or by calling 1.877.934.0407 or INTL +1.210.581.0407. Thank you for your support.

Best Regards,
The Rackspace Cloud

Their SLA policy is as follows, though 5% refund for every hour of downtime beyond the first is basically nothing.

Cloud Server Hosts
We guaranty the functioning of all cloud server hosts including compute, storage, and hypervisor. If a cloud server host fails, we guaranty that restoration or repair will be complete within one hour of problem identification.

Glad to have a failover backup hosted at BurstNet for times like this.

[media-hosts_com]

In the email you quoted, they refer to the "host server" in singular... This is what's sparking the questions in my mind.

Correct me if I'm wrong... Isn't the whole idea of a "Cloud" to ensure that if one piece of hardware experiences "catastrophic" failure, the cloud can stay online? Basically eliminating the single point of failure?

If there's a single point of failure, then it's not really a cloud... Unless there was a giant power surge or something that fried a whole rack of servers..

[dotHostel]

Originally Posted by media-hosts_com

In the email you quoted, they refer to the "host server" in singular... This is what's sparking the questions in my mind.

Correct me if I'm wrong... Isn't the whole idea of a "Cloud" to ensure that if one piece of hardware experiences "catastrophic" failure, the cloud can stay online? Basically eliminating the single point of failure?

If there's a single point of failure, then it's not really a cloud... Unless there was a giant power surge or something that fried a whole rack of servers..

I couldn't agree more.

If it is a cloud there is no oportunity to "catastrophic failures" -- except natural catastrophes of course, nor things like "All customer data on that host has been lost" as there isn't "that host". It seems a serious issue with their SAN (or a Sata Hdd ) and no resilient/robust architecture adopted building their "cloud".

[Joshua]

In this case, it seems the "cloud" is simply a standard VPS host server.

[media-hosts_com]

They also mention their cloud consists of "compute, storage, and hypervisor" nodes...

So loosing all of the customer data would mean the storage node was wiped. But this should be a redundant storage array.

Then they could just re-create your account on a different hypervisor node from the same filesystem. But in no way should the data be non-recoverable requiring an off-site backup restoration.

Unless the storage array is causing the failure...

[dotHostel]

Originally Posted by media-hosts_com

But in no way should the data be non-recoverable requiring an off-site backup restoration.

Unfortunately it is possible when using proprietary NAS/SAN solutions. I was a Softlayer customer when they lost all customer data stored on their (sophisticated and expensive) NAS: unrecoverable. But it was just a backup storage device ...

[dotHostel]

http://ir.rackspace.com/phoenix.zhtm...761&highlight=

SAN ANTONIO, Jul 19, 2010

Rackspace(R) Hosting (NYSE:RAX) today announced the launch of OpenStack(TM), an open-source cloud platform designed to foster the emergence of technology standards and cloud interoperability. Rackspace, the leading specialist in the hosting and cloud computing industry, is donating the code that powers its Cloud Files and Cloud Servers public-cloud offerings to the OpenStack project. The project will also incorporate technology that powers the NASA Nebula Cloud Platform. Rackspace and NASA plan to actively collaborate on joint technology development and leverage the efforts of open-source software developers worldwide.

...

OpenStack will feature several cloud infrastructure components including a fully distributed object store based on Rackspace Cloud Files, available today at OpenStack.org. The next component planned for release is a scalable compute-provisioning engine based on the NASA Nebula cloud technology and Rackspace Cloud Servers technology. It is expected to be available later this year. Using these components, organizations would be able to turn physical hardware into scalable and extensible cloud environments using the same code currently in production serving tens of thousands of customers and large government projects.

...

Rackspace and NASA have committed to use OpenStack to power their cloud platforms, and Rackspace will dedicate open-source developers and resources to support adoption of OpenStack among enterprises and service providers. An OpenStack Design Summit hosted by Rackspace was held July 13-16 in Austin, where more than 100 technical advisors, developers and founding members joined to validate the code and ratify the project roadmap. More than 25 companies were represented at the Design Summit including AMD, Autonomic Resources, Citrix, Cloud.com, Cloudkick, Cloudscaling, CloudSwitch, Dell, enStratus, FathomDB, Intel, iomart Group, Limelight, Nicira, NTT DATA, Opscode, PEER 1, Puppet Labs, RightScale, Riptano, Scalr, SoftLayer, Sonian, Spiceworks, Zenoss and Zuora.

[Rens]

Host server - data lost - cloud. Can't be combined.

[JohnnyUtah]

This is actually unreal considering that they are one of the most expensive provider out there...

[ObjectZone]

Way too often do I see "cloud" thrown around in place of a VPS account. Tell customers what it really is. Not saying VPS is bad, just that you shouldn't give your customers an impression that their hosting account is indestructible. I've heard numerous supposedly clued in IT professionals tell me that they don't need backups because they're using a cloud service -- after a few questions about their setup, I discover they're just on a glorified VPS account and have to give them a shocking wake-up call.

--Chris

[dotHostel]

Taking in account the press-release Rackspace is running a "true" cloud. It seems an issue with the storage system (hardware or the "Cloud Files" file system).

[bizwizkid]

What annoys me is that they notify us by email and ignore the latest communication vehicle ... you know, text messaging!! grrrr

I was just told they will tell us later in the afternoon what happened - so they must've been attacked.

What irritates me to no end is when we set this site up they didn't even automatically do a backup image nor make it clear we should do this. I literally bumped into this feature (thank God) as I would be face-down right now with a potential loss of devastating proportions (major database for a non-profit).

I've never been a fan of RS and will be shopping this service out. I can't stand buying into 'cloud' hype when in essense I bought server space where these rocket scientists probably put my db and website on the same server. Oh, to boot we didn't backup my .svn site (if it matters) and I'm betting we lost tons there too. Hopefully we can restore that from our pc's but, just another pain.

[layer0]

Originally Posted by dotHostel

Taking in account the press-release Rackspace is running a "true" cloud. It seems an issue with the storage system (hardware or the "Cloud Files" file system).

Cloud Files is used for their CDN product I believe.

Their cloud is in fact nothing more than a glorified VPS last I checked.

The host server in this case had its own storage array fail I am guessing.

[Mxhub]

Lot of disappointment with big names these few days.. first is skype's 24 hours downtime.. now, rackspace cloud couldn't promise 100% uptime or even have a proper backup procedure to keep data safe.

-joseph

[bizwizkid]

I gathered from my chats with tech 'support' that there were external forces (they referred to legal issues) that helped them crash. Who knows. All I know is thank goodness I had a backup and no help to them for our having discovered that on our own.

I'll be looking to pay a bit more and move from these waters. They're so caught up on their 'fanatical support' they actually believe it. Too much Kool-Aid is my take.