Sharkspace is having a bad day. Their last forum update was about an hour and a half ago. I've only been with them for a couple of months, with really good service, but this is a bit jarring. Here's the thread for their forum. Any thoughts on their response?

This all started about 8 p.m. or so PST. Site's are still down at 6 p.m. PST.


Affecting Server - S9C4 | Priority - Critical
The server S9C4 will be taken offline shortly and a manual FSCK will be completed to fix a file system error. We will keep you updated in this post.

Update:
The FSCK is not showing errors, however the file system still goes read only. We are working on the journal now. Note the server may be up most of this time.

Update:
The server will be going offline shortly for another FSCK. We hope that this will resolve the issue. ETA 20-40 minutes.

Update:
The FSCK did not find any errors. We blieve it may be failed hardware at this point. We are now checking that.

Update:
While we continue to work on the corruption issue. We are going to be starting a r1soft restore to a backup server. If this server is not fixed by the time the restore completes we will replace it with the spare.

Update:
We have moved the 4 raid 10 harddrives to a spare server. The spare is now online and websites are loading. This should resolve any hardware issues. We will continue to monitor.

Update:
Some IPs are not working on the new server. We are fixing them now.

Update:
Unfortantely the read only file system came back with the new hardware. We are now restoring from backup.
Date - 03/30/2010 08:20
Last Updated - 03/30/2010 16:45

Well at least they have a backup -- that's good.

The FSCK is not showing errors, however the file system still goes read only. We are working on the journal now. Note the server may be up most of this time.

o.0

Albeit the hardware should have been checked early on during the initial file system issues I think they're handling it fine. Unfortunately when hardware failures happen (especially when data is concerned) it's best to tread on caution and fully evaluate the situation / work on rectifying it while maintaining the integrity of the data - which I believe they've done an admiral job of.

The last update was a bit ago (according to you) however it's likely just the RAID rebuilding. The services are online right?

Actually, the sites remain down. They did come back up for a time during the middle part of the day but are offline currently.

Like you, I thought they were handling this well. But, with sites still down I'm concerned.

does anyone have any update? I am dying over here.. I have clients calling me on my personal cell phone. The last update is old... no word yet as to when its coming back up. help us.

Actually, the sites remain down. They did come back up for a time during the middle part of the day but are offline currently.

Like you, I thought they were handling this well. But, with sites still down I'm concerned.

Ah, I would be concerned too - I was under the impression that the last update meant the machine was online however rebuilding the array and such. They did mention IP issues - are you sure all sites are down or are you possibly one of the unlucky individuals who are having issues?

Just an update. We finally got FSCK to fix some errors after disabling the journal. Now FSCK is running again with the journal and we hope everything will be online shortly. We have the backup if this fails.

does anyone have any update? I am dying over here.. I have clients calling me on my personal cell phone. The last update is old... no word yet as to when its coming back up. help us.

They have a phone number listed on their main page. Have you tried calling them?

thanks for the update. It sounds like all sites on the server are down.

Just to note we are restoring the r1soft backup to a spare duplicate server and still working on fixing the original why the backup runs.

do you know how long it will take to a get a back up server online? are we looking at 1-2 hours or longer ?

do you know how long it will take to a get a back up server online? are we looking at 1-2 hours or longer ?

Unfortunately we don't have an ETA. I would hope the r1soft restore finishes within the next 3 hours. There was a delay at the start, because r1soft didn't have the network drivers needed.

http://www.sharkspace.com/termsofservice.html
(99.9% - 99.5% uptime) -- 10% Refund

If it bothers you that much you can claim back on SLA.

At least they had backups.

Unfortunately we don't have an ETA. I would hope the r1soft restore finishes within the next 3 hours. There was a delay at the start, because r1soft didn't have the network drivers needed.

Ok that is good.... anymore past 3 hours and I am not sure I can make it. I just finished a tub of sprinkles, and now I am heading for the cookie dough in the fridge.

"The FSCK has now failed. "Unexpected inconsistency." The r1soft bare metal restore is still running and once complete we hope everything will be online." what happens if the bare metal restore brings back the same errors? are we doomed? is their an older back up around?

Ok that is good.... anymore past 3 hours and I am not sure I can make it. I just finished a tub of sprinkles, and now I am heading for the cookie dough in the fridge.

DUDE! COOKIE DOUGH! = Yumm!

"The FSCK has now failed. "Unexpected inconsistency." The r1soft bare metal restore is still running and once complete we hope everything will be online." what happens if the bare metal restore brings back the same errors? are we doomed? is their an older back up around?

We have 7 days of backups to work from.

"The FSCK has now failed. "Unexpected inconsistency." The r1soft bare metal restore is still running and once complete we hope everything will be online." what happens if the bare metal restore brings back the same errors? are we doomed? is their an older back up around?



We have 7 days of backups to work from.


I was going to say originally R1Soft will also carry over the corruption obviously as it's block level backup so the start of the issues any points it did would be bad. Also it coming back clean was a force check done? Some times those pesky systems come back clean unless you force it through.


File system corruption sucks though I feel for anyone with some. We had a machine with some we ended up using a different superblock in our case then moving the server to new hardware a week later. Those sorts of issues typically end in multi hour outages no matter what :(

The file system corruption is not getting any better (getting worse) so we are going to OS reload the original server in case the r1soft restore fails we cant start a new restore instantly.

I wont be available for update for a few hours. Other staff are monitoring the restore and will post an update in announcement section if something goes wrong.


I am really sorry to everyone on this server. I just want you to know we are doing everything possible. File system corruption is a nightmare.

The file system corruption is not getting any better (getting worse) so we are going to OS reload the original server in case the r1soft restore fails we cant start a new restore instantly.

I wont be available for update for a few hours. Other staff are monitoring the restore and will post an update in announcement section if something goes wrong.


I am really sorry to everyone on this server. I just want you to know we are doing everything possible. File system corruption is a nightmare.

Dimmy, why did you do this to me? Please Dimmy, I'm afraid.


You still have my files right!?

We still have the r1soft backups. As long as there is no corruption in them we will be fine.

I've no idea if I'm just one of the unlucky ones, but from the rest of the thread, it would seem that there's a significant widespread problem.

I'm still relatively new to Sharkspace, although I'm preparing to migrate several other sites over. This is my first experience with how they handle such an issue, but frankly I'm not knowledgeable enough to evaluate the response. (I do understand that outages happen to even the best host.)

Obviously, they're working hard; it's a good company. But, I would appreciate any thoughts on whether or not this is being handled in the most appropriate manner.

I hope there is a backup of the backup with the way it is going!!! I am getting nervous over here....

The restore is currently 70% completed.

The restore is currently 70% completed.

Good to hear its 70% completed, How many servers was lost? Also what caused this?

Good to hear its 70% completed, How many servers was lost? Also what caused this?

This only affected 1 server. It was file system corruption that popped out of no where. We replaced all hardware and are restoring to a spare.

Dimmy, why did you do this to me? Please Dimmy, I'm afraid.

You still have my files right!?Nobody has "done" anything "to you." File system issues can [and most certainly will] happen. Realistically if the provider loses your data and their backup is corrupted (they would have to be really unlucky) and you don't have your own copy of your sites/files/databases you really have nobody to blame but yourself. You should *always* maintain an up to date copy of your sites if they are important to you.

I hope there is a backup of the backup with the way it is going!!! I am getting nervous over here....I've been through file system issues and I can tell you that they're probably more nervous about than you are... Think about it - you have a few sites offline where as they have at least a few hundred that they are supposed to keep online... You may be sweating bullets but I'm sure somebody else is probably sweating cannonballs.

Good luck to all involved.

I hope there is a backup of the backup with the way it is going!!! I am getting nervous over here....

They said they have backups of the backups of the backups. If you want to worry, you can worry about how long it will take, but I wouldn't worry too much about them getting it back eventually.

We have 7 days of backups to work from.

We are still down on all sites. Over 10 hours now I believe of downtime.....

I have a resellers account with them, used mainly for myself, friends and a few clients.

Clients are furious about their sites being down as I cant give them an answer as when their sites will be online again.

Sites and email are down for almost a day, first intermittently now permanently for a long tine.

This really sucks.

The r1soft restore is going as fast as we can make it go. If no issues arise it wont be too much longer.

The r1soft restore is going as fast as we can make it go. If no issues arise it wont be too much longer.

Could you please specify what "wont be too much longer" means.

I have just lost a customer because I cannot provide a time frame to when this issue will be resolved.

More than a day with intermittent down-time is not acceptable.

Thank you.

The restore has finished and we are working on getting it to boot.

It stalls at the grub section of boot. And simply shows "grub" with a flashing cursor. If anyone has any suggestions please let me know.

We otherwise are trying to restore the /boot section again.

At this point it appears the backups are also corrupt. We are now going to re-setup the accounts and then if needed we should be able to restore your files and databases. You may need to re-add your addon/sub-domains.

We are extremely sorry, but this file system corruption was unpreventable and there was no way we could have know it was in the backups.

Restoring

/etc/passwd
/etc/shadow
/etc/group
/etc/wwwacct.conf
/etc/quota.conf
/etc/domainalias
/etc/remotedomains
/etc/reservedipreasons
/etc/reservedips
/etc/secondarymx
/etc/localdomains
/etc/userdomains
/etc/valiases
/etc/vfilters
/etc/vmail
/etc/tru*
/etc/ips
/etc/domainips
/etc/services
/etc/sysconfig/pure-ftpd
/etc/pure-ftpd.conf
/etc/pure-ftpd
/etc/proftpd
/etc/proftpd.*
/var/cpanel
/var/spool/cron
/usr/share/ssl
/usr/local/cpanel
/usr/local/apache/conf
/usr/local/frontpage
/root/.my.cnf
/etc/my.cnf
/var/named
/etc/named.conf
/etc/rndc.conf
/var/lib/mysql
/home/

Should take care of everything, you'll need to rebuild quotas though.

Thanks Jedito. Will try that also.

It stalls at the grub section of boot. And simply shows "grub" with a flashing cursor. If anyone has any suggestions please let me know.

We otherwise are trying to restore the /boot section again.

At this point it appears the backups are also corrupt. We are now going to re-setup the accounts and then if needed we should be able to restore your files and databases. You may need to re-add your addon/sub-domains.

We are extremely sorry, but this file system corruption was unpreventable and there was no way we could have know it was in the backups.

Now I am starting to panic (see above) and another client is also threatening to leave.

Can anyone suggest a new reseller web host please?

The restore just will not work at this point. The r1soft backup is corrupt and will not restore anything.

We are setting up main accounts on a new server and you will need to change name servers and restore from personal backup.

We are very sorry for the data loss. Each client will be credited several months once things settle down.

But from this point you should be receiving a new welcome email. Change your name servers then upload your personal backup.

Once again I am sorry beyond believe.

Hardware failures happen daily, nothing you can do. Backups also fail daily, even when there's amazing providers out there like Sharkspace. It's a risk everyone takes which is why every provider always suggests taking your own backups. Don't rely on a single method of backup, ever.

I think they did the right thing, they are giving you updates, giving you credits and trying their best to get your sites up.

Hardware failures happen daily, nothing you can do. Backups also fail daily, even when there's amazing providers out there like Sharkspace. It's a risk everyone takes which is why every provider always suggests taking your own backups. Don't rely on a single method of backup, ever.

I think they did the right thing, they are giving you updates, giving you credits and trying their best to get your sites up.

Thank you nerdie. We also will continue to work on the backups, but at this point I would say you need to restore from personal backup.

@sharkspace

Are you going to upload the backups to each account?


I dont have a personal backup :(

Good luck to Sharkspace, data loss is pretty much as bad as it gets. Every provider goes through it eventually.

The server and backups are corrupt. At this point we are sending new welcome emails to clients. Please change your dns to the ones listed in the welcome email and restore your accounts from your personal backup.


We are extremely sorry and will provide a several month credit once things settle down to all affected clients.


There is still a remote chance we can recover the server from r1soft backup. It just wont be for awhile due to the 12+ hour restore time. But I do not want to leave anyone waiting around for that. So please follow the instructions above.



We have determined to give all affected clients a 1 yr account credit. Hopefully this will help. It will be applied as soon as things settle down.

@sharkspace

Are you going to upload the backups to each account?


I dont have a personal backup :(

Neither do I, not for my client's accounts anyway and I am 99% sure they do not have either.

I am screwed.

I am almost certain we will be able to recover data, but do not rely on that. Please try to get everything up from personal backup.

I am almost certain we will be able to recover data, but do not rely on that. Please try to get everything up from personal backup.

:pray::pray:

:pray::pray:

Not to jump the gun, but it looks like databases will be available in a few hours.

Is R1 providing any assistance on this? (i.e. working with sharkspace to get the backups?)

Is R1 providing any assistance on this? (i.e. working with sharkspace to get the backups?)

Yes, they have been helping since yesterday.

Sharkspace, could you clarify? I'm currently recreating accounts but my personal backup is about a week old, and I'd prefer the latest backup you have — even if it takes a little longer.

How best should I proceed?

Not to jump the gun, but it looks like databases will be available in a few hours.

This will be a great help since allmost all my sites are database driven :D

Sharkspace, could you clarify? I'm currently recreating accounts but my personal backup is about a week old, and I'd prefer the latest backup you have — even if it takes a little longer.

How best should I proceed?

Restore from your personal backups and if we can get ours to work then we can always overwrite them.

Not to jump the gun, but it looks like databases will be available in a few hours.

Please update us when you can.

Thank you.

I would not recommend sharkspace to anyone after seeing the current fiasco that just happened to my reseller hosting.

All clients sites have dropped off the web, all unavailable. No notification what so ever from sharkspace. Furthermore I have now found the notification from them on their forums stating they have lost all the data and are unable to restore the backups. Apparently they are sending new welcome emails and we must start from scratch.

Dont believe me ? Read it for yourself under network issues on the sharkspace forums.

ALL DATA GONE
BACKUPS NON RECOVERABLE
SITES OFFLINE NOW FOR 8+ HOURS

Live Support ? Offline predictably. This is catastrophic hosting failure. The way its being dealt with is a complete shocker and the fact that it could happen at all is ridiculous.

I would not recommend sharkspace to anyone after seeing the current fiasco that just happened to my reseller hosting.

All clients sites have dropped off the web, all unavailable. No notification what so ever from sharkspace. Furthermore I have now found the notification from them on their forums stating they have lost all the data and are unable to restore the backups. Apparently they are sending new welcome emails and we must start from scratch.

Dont believe me ? Read it for yourself under network issues on the sharkspace forums.

ALL DATA GONE
BACKUPS NON RECOVERABLE
SITES OFFLINE NOW FOR 8+ HOURS

Live Support ? Offline predictably. This is catastrophic hosting failure. The way its being dealt with is a complete shocker and the fact that it could happen at all is ridiculous.

Are you sure? They advertise that they are using R1Soft backups.. :confused:

That said, all the best in recovering from the situation :agree:

I need an update on this. You have not sent out any "welcome emails" for my hosting. All my sites are still offline. Changing dns is a huge issue with my co.za domain names and takes 24 hours for them to process. This is a huge disaster. No notification that anything is down. No notification on how to fix anything. Quite frankly its disgusting. I am regretting the day I swapped to sharkspace.

Actually feeling physically Ill over this. I am due to go away for a week tomorrow morning and now I am going to have to cancel flights so I can spend the next few days trying to piece together everything.

Greetings,
I am very sorry to hear that you lost all of your customer data, but there is a SharkSpace rep on WHT who is reporting that they are restoring data from an R1Soft Solution so I am not sure where you are seeing that they may have lost the data. As someone who has been in that situation twice when I was a reseller. I know it is very hard to rebuild from scratch.

I need to also be the bad guy here and remind people that they should ALWAYS maintain their own server backups, even if the host claims to keep them.

My data is mostly backed up. I however have clients with e-commerce where a weekly backup simply is not good enough.

The R1Soft restore failed according to them. They are resending welcome emails (which apparently they couldnt be bothered to do with me) Even with backups in hand all my sites are down, no one is sending out mails advising there is/was a problem nothing.

I believe the OP may be referring to
https://www.sharkspace.com/sharkcenter/networkissues.php?view=nid81

It shows that the backups are corrupt, and they will be sending new welcome emails with 1 year worth of credit to customers.

I need an update on this. You have not sent out any "welcome emails" for my hosting. All my sites are still offline. Changing dns is a huge issue with my co.za domain names and takes 24 hours for them to process. This is a huge disaster. No notification that anything is down. No notification on how to fix anything. Quite frankly its disgusting. I am regretting the day I swapped to sharkspace.

Actually feeling physically Ill over this. I am due to go away for a week tomorrow morning and now I am going to have to cancel flights so I can spend the next few days trying to piece together everything.

Same problem here my fellow SA friend, 95% of my sites have .co.za names.

I just hope that updating my personal name servers will do the trick. That's aside from creating all the accounts.

We are actually looking at 24 hours for the update to be processed and then another 12 hours for DNS to propagate etc.

:angry::angry::angry:

Greetings,
As the case with most heavy business sites, a backup from 5 minutes ago can be outdated before the backup is finished so I know your pain. If I ever lost my WHMCS database I would loose it (Hence why I mirror it on a staging box in case). Not to mention loosing a week of e-mails would hurt.

I truly hope that the R1Soft did not fully fail, when they re-create your account I would advise you to log into the R1Soft control panel and see if you can self restore.

I am not even sure the 1 year service credit can make up for the hit to your companies reputation.

I believe the OP may be referring to
https://www.sharkspace.com/sharkcenter/networkissues.php?view=nid81

It shows that the backups are corrupt, and they will be sending new welcome emails with 1 year worth of credit to customers.

Hello,
I did see this but something does not make sense, R1Soft is a remote backup tool (of course the server can reside next to it) but I do not see how both can be corrupt unless some serious s$%# happened.

The file system was lost as explained already. It corrupted all the data and backups. You will need to restore from your personal backups to the new server.

We still are working on restoring the server with all data, but I cant promise anything due to the extreme corruption.

Changing dns is a huge issue with my co.za domain names and takes 24 hours for them to process.


Same problem here my fellow SA friend, 95% of my sites have .co.za names.

I just hope that updating my personal name servers will do the trick. That's aside from creating all the accounts.

We are actually looking at 24 hours for the update to be processed and then another 12 hours for DNS to propagate etc.

Keep in mind the difference between changing name servers and changing the data on them. When you get a new IP address you will need to update the A record but unless your current DNS server was on the crashed server you shouldn't need to change to a different DNS server, which is what would take the processing time for a co.za. To the extent any of the affected customers control their own DNS records, one thing you can do while you wait for a new IP address is to lower the TTL as low as you can. Depending on the expiration of any existing cached records around the world, that may help reduce the propagation time.

Great news the mysql databases are available. Open a support ticket with the databases you need and we will put them under your account for you to restore.

It is sad to hear such tragedy happening but this happens all the time and even to the best hosting providers. SharkSpace seem to be handling the situation quite well but I can understand the client's frustrations as well.

@SharkSpace: Do you have any other forms of backup besides R1Soft? Eg. Rsync or Duplicity?

Great news the mysql databases are available. Open a support ticket with the databases you need and we will put them under your account for you to restore.

:pray: now just the files

You should beable to leave the nameservers for your .co.za domains alone and only have the update take place locally. Your domains should propogate on there own. I wouldn't cancel any flights until you are sure you are going to HAVE TO BE HOME/AT YOUR OFFICE to make your sites become available to the internet.

I know it's off topic but did anyone read the SharkSpace privacy policy when they signed up. http://www.sharkspace.com/privacypolicy.html part 6. It states that they "SharkSpace reserves the right to sell or transfer your information (including, but not limited to, name, address information, and other information you provided to SharkSpace) to a third party that (i) concentrates its business in Web hosting or communication products or services"
so I kept reading and they state if you do not agree to the sale or transfer of your information you are agreeing to cancel your services.

Just an interesting note..

You should beable to leave the nameservers for your .co.za domains alone and only have the update take place locally. Your domains should propogate on there own. I wouldn't cancel any flights until you are sure you are going to HAVE TO BE HOME/AT YOUR OFFICE to make your sites become available to the internet.

I know it's off topic but did anyone read the SharkSpace privacy policy when they signed up. http://www.sharkspace.com/privacypolicy.html part 6. It states that they ""
so I kept reading and they state if you do not agree to the sale or transfer of your information you are agreeing to cancel your services.

Just an interesting note..


We would never sell any client information to a third party company. That is just a generic privacy policy.

Any update on the files from R1?

Any update on the files from R1?

The restore keeps failing due to the file system corruption. At this time its looks as though the files will need to be restored from your personal backup.

Is there any hope file recovery at all as of right now ? 5:23 EST time I am talking about hope here.. just hope. give me something to hang on for anything at all

R1soft is still working on it. We will know for sure by tomorrow if any files can be saved. The main thing so far is we saved the databases which contain the most critical data.

What is contained in the databases? what about webpage data? what exactly are you saying here by the databases are saved?

What is contained in the databases?
The mysql databases if you run forums, blogs, etc...

what about webpage data?
They will know for sure by tomorrow if any files can be saved

what exactly are you saying here by the databases are saved?
Meaning customers got the databases they had on their accounts

What is contained in the databases? what about webpage data? what exactly are you saying here by the databases are saved?

Nearly every script uses a database (mysql mostly). The mysql database contains all the data for the script. You should be able to download and reupload the script files from the script developer and it will work as it did before.

are additional staff members in place to assist with restores? what about live chat can't we get that turned on to help us?

are additional staff members in place to assist with restores? what about live chat can't we get that turned on to help us?

Generally speaking tickets are far more efficient especially during something like this.

are additional staff members in place to assist with restores? what about live chat can't we get that turned on to help us?

We have an extra staff member in the helpdesk doing the database restores. Turn around is very quick.

What is contained in the databases? what about webpage data? what exactly are you saying here by the databases are saved?

If you are using static html pages that don't change on a regular basis, someone must have created them and uploaded them to the server in the first place. Hopefully you can trace back to where that came from, and upload it again. If you are running a shopping cart, forum or other type of site, that changes minute by minute the databases is where that information is stored, so that is the most critical thing to get back as current as possible, but the database is only useful after you get back the web pages that interface with it.

I'm one of the lucky, unlucky ones. I've a relatively new reseller account with them and I'd only moved one production site — and a few test sites that I can lose — over to test the waters. Once word went out the database backups were available, I put in a ticket and the database for the live site was restored in less than 10 minutes.

I hope that offers at least some encouragement.

Now, I have to decide if I want to hang in there with my Arctic Reseller, because the water was a bit colder than I expected. That's probably not fair to Sharkspace, which really seems to be doing all possible, but it's hard not to lose confidence.

Unlucky.

All hosting companies go through this from time to time.

Hope you get it all sorted soon.

I called today and had a phone support ticket opened. The person I spoke with acted completely oblivious to the reported issues and didn't seem all that worried that our site was down.

We were notified that we would receive free hosting for one year via a support ticket a few hours later. I will see if I can get this transferred over to my own account since my personal sites aren't "that bad" if lost, and I can spend that time reevaluating if SharkSpace is ready for a business site. I realize that all companies have problems, but it just baffled me that such poor internal communication took place here. Furthermore, the fact that we were sent a welcome email with zero explanation until I called and had to wait on hold was down right unacceptable. Problems are acceptable, and how you handle them teaches you a lot about a company. I think that making zero attempt to reach out to us as customers until we waited on the phone proved how serious our account is taken by sharkspace.

At this time, with how support acted completely oblivious to such a major issue, I cannot recommend to my employer that we move our site. Fortunately we only had one development site and one live site with them.

Edit- I just got this support ticket update
Hello,

The r1soft restore has failed on all 7 backups. At this point the files must be restored from your own backups. We do have databases available. If you need db backups, update this ticket with the databases you need.

=========================================
Regards,
Alex

My experience was the same with phone support. I called trying to get a quick clarification of something mentioned on this forum, but the person on the call seemed to have no knowledge of an outage. They opened a ticket, which I obviously could have done on my own.

Even if they couldn't provide an answer, I would've felt better if they'd at least been aware a problem existed, which tells me the phone support is probably limited to the "Hey, I lost my password" situations.

Hello,

The r1soft restore has failed on all 7 backups. At this point the files must be restored from your own backups. We do have databases available. If you need db backups, update this ticket with the databases you need.

=========================================
Regards,
Alex
I wonder what the definition of "failed" is in this particular case. I wonder if they are simply still having the new server go read-only or if it literally has not succeeded at restoring to the standby server.

How come backups are corrupted ? Your server backed "the corrupted data" in the r1soft backup server :confused: ?

How come backups are corrupted ? Your server backed "the corrupted data" in the r1soft backup server :confused: ?

Don't really know 100% how there servers are setup, but I do know a server will backup with errors in it.

That's why all web host tell there clients to make sure they do regular backup.

You could do weekly backups of your main site or even monthly if there no images added, ect added and build a simple php script to email your MySQL DB nightly or hourly.

Now regard what happen, I know the feeling shark space (wish them well) is going though regarding the errors he is seeing and I can say there nothing you can do to recover from that error as we had 1 server do the same thing a few years ago.

We had 4 tech onsite trying everything me and 3 DC techs, plus a few on the phone.

Our Backups recovered fine, but regard OS recover we had to replace the whole server as it was 100% caused by hardware failure that caused the HD to corrupt & shows grub section.

Upgrade the grub made things worse...

Also for the shark space affected clients there is no warning to these events, one minutes your site running the next all hell break lose that's just how servers work sometimes.

it's like you sitting in front of your home computer and for no reason the Blue Screen show... error, except in a servers it could me death :crap:

hopefully that make some sense...

My experience was the same with phone support. I called trying to get a quick clarification of something mentioned on this forum, but the person on the call seemed to have no knowledge of an outage. They opened a ticket, which I obviously could have done on my own.

Even if they couldn't provide an answer, I would've felt better if they'd at least been aware a problem existed, which tells me the phone support is probably limited to the "Hey, I lost my password" situations.

I would like to hear the reply from SharkSpace in this regard.

This is R1Soft at its finest :)

@sharkspace

Are you going to upload the backups to each account?


I dont have a personal backup :(

Guys why do you use the forum for the issues which can be solved via help desk?

We got the server restored from an r1soft baremetal restore and it booted up past the fsck (where it would fail completely on the other attempts.)

We are now working on getting the networking up. I am in the server via KVM and it looks ok so far, but it may go read only so this is only an informational post at this time. I will update soon with more information.

Great news everyone! The old server is online at 66.219.29.34/cpanel or 66.219.29.34/whm

We are getting the secondary IPs up now. We had to disable cpanel backup for the moment while the raid finishes rebuilding. Once that is completed we can transfer any accounts over you need and forward the dns. This is all relying on that the system does not go read only again.

All data is back up! The server restored from r1soft bare metal restore from the previous days backup.

Please merge this with thread http://www.webhostingtalk.com/showthread.php?p=6727995

I will keep everyone updated there.

The secondary IPs are up.

Great news everyone! The old server is online at 66.219.29.34/cpanel or 66.219.29.34/whm

As in all the old data?

If so, holy s*** that's great news to the peeps with no personal backups. :)

All old data is present.

All old data is present.

I'd say that's good news.

EDIT: Actually, great news, or stellar news or even OMGWTFBBQ really?

Everything is going to have an happy end after all. If i look at this as 'non' client things we're handled very good.

They done what they could and informed everyone fairly well. Before this one failure they had only good reviews, i think this little failure could happen everywhere with every hoster.

YES YES YES DATA IS BACK :D


Lesson Learned:
Make your own backups something I did not do

We have determined to give all affected clients a 1 yr account credit. Hopefully this will help. It will be applied as soon as things settle down.That's a generous offer. For my case, that will be good enough. Awaiting Sharkspace's notification.

WOW, stellar on the park of Sharkspace I must say.

They handled this great. The downtime sucked but all of there clients have every little bit of data they had before and they are offering the affected clients a free year of service. I commend them and will recommend them to anyone who is looking for shared hosting I come across. TTT for SharkSpace.

And congratulations to the users who were discouraged. You must stay with SharkSpace. I have been in the industry for almost 6 years now and this is absolutely what I love to see a hosting company do. If you switch providers you are one of those people that no matter what you do to help them they will never be happy. Great job SharkSpace!

Lesson Learned:
Make your own backups something I did not do

Everybody should take some lessons away from this. If you are a customer of Sharkspace or any other hosting company, you can't watch something like this, and not think about the status of your own backups. Even if you were making regular backups, you should be thinking about how often you make what kind of backups, where you store them, how long you keep them, and how sure are you that you can restore your backup.

For Sharkspace and all the rest of us in the hosting business, we should all know that something like this could happen to any of us. We all need to give a new look to our contingency plans. For customers of Sharkspace you should probably assume that what didn't kill them, will make them stronger, and you can be sure they are, or soon will be (after they get a little sleep), going over every detail of this incident to determine what they could do better, from now on. They don't ever want to go through this again, and if they ever do, they want the recovery to go smoother, so you can be sure that any changes they can reasonably make to make a similar indecent less likely or less severe, will be made.

We have determined to give all affected clients a 1 yr account credit. Hopefully this will help. It will be applied as soon as things settle down.

That's a generous offer. For my case, that will be good enough. Awaiting Sharkspace's notification.

You'll note that the offer of compensation has quietly been removed from that notice :(

Also, the fact that I was charged for my next months hosting after the offer was removed would suggest that no compensation will be forthcoming.

You'll note that the offer of compensation has quietly been removed from that notice :(

Also, the fact that I was charged for my next months hosting after the offer was removed would suggest that no compensation will be forthcoming.

No it's still there, see
http://www.webhostingtalk.com/showpost.php?p=6726184&postcount=44

Notice that it also says:
It will be applied as soon as things settle down. I'm guessing that things haven't totally "settled down" yet.

Yes, I know it is still in the post on *this* forum. I mean it has been removed from the post on sharkspace's network announcements page (after the post here had been made).

I think they jumped the gun on compensation here. Surely they were willing to provide 1 year when they thought they were losing people's data but now that they have everyone up and it was just some downtime that will be forgotten next week/month it's hard to follow through with 1 year of free service. Though likely they will have to since they announcement it publicly on WHT and they cannot go back and edit the post.

Luckily the only customers that will call them on it are the customers who have already been privy to the premature announcement that they made.

Well I wouldn't care about the free 1 year of service. I would just want my data back and that's it.

Their clients did get their data back. So all should be good now.

As I've mentioned before. I'd only moved one site to Sharkspace to test the waters before moving others. And you can often tell the most about a company during something like this.

The quick offer of a year's free hosting, to put it mildly, was impressive. I'd have to go back and check, but I believe that came even before it looked as if all the data was lost, but I may be wrong.

Regardless of the timing, the offer just screamed of a company willing to go above and beyond. If they really did pull that offer after making the promise, well, that certainly shapes my perception.

For me, I don't care about the free hosting. I'm happy to pay for reliability, because I've got a handful of clients who depend on me to make the right decision on where to host their sites. I've now moved a few sites over to one of the managed VPS providers praised so often here.

It's more expensive. This host could also experience a "bad luck" loss of data. But, you've got to play the odds, and so far I'm impressed with the service.

Promising something to upset customers, and then pulling it back when it only means days of downtime versus complete data loss … Well, everyone has to judge for themselves.

To claim the SLA refund contact the sales department. The 1yr one was for full data loss, since we recovered completely within a reasonable amount of time (about the same as a rysnc/cpbackup restore) we will be tripling our SLA and offering 3 months free hosting.

To claim the SLA refund contact the sales department. The 1yr one was for full data loss, since we recovered completely within a reasonable amount of time (about the same as a rysnc/cpbackup restore) we will be tripling our SLA and offering 3 months free hosting.

Back pedaling on an a credit offer isn't horribly classy - I'd imagine only a hand full would claim the credit and in addition to that it would have been in good faith to keep the offer up due to the amount of time as well as uncertainty your clients went through during the whole ordeal. Just my 2 cents :)

As stated we are still tripling our SLA and giving 3 months free. If you look over this thread it was originally several months (3 months planned). That is still far more than most providers will offer for an unpreventable issue.

We have determined to give all affected clients a 1 yr account credit. Hopefully this will help. It will be applied as soon as things settle down.

You guys made a mistake trying to spin this and you need to take your lumps and follow through with what you said here. Obviously it was an attempt to diffuse the situation and I understand that because I have been in your situation before but to simply renig on your offer since things are ok now is very immature.

You guys made a mistake trying to spin this and you need to take your lumps and follow through with what you said here. Obviously it was an attempt to diffuse the situation and I understand that because I have been in your situation before but to simply renig on your offer since things are ok now is very immature.

If you read through this thread that was only made in sequence with the data loss. Once we restored all data hours later, we went back to our original several months free offer. If you realized how much money we put into this issue to resolve it you would not be making such claims. We have 4 x $4000+ servers tied up in this restore. Two of which will be unavailable for other use indefinitely. In sysadmin payments we spent thousands extra all to get everything restored. We could have stopped working at the original backup failure and said clients would have been fine with the 1 yr credit, but we didn't. We invested thousands more (more than we would have paid out in SLA claims to the 50 clients on this server) to get everything restored and working.

1. I read through the whole thread as I was following it during the issue.

2. (2) servers need to be burned to the ground because they were used during a restoration process? Please explain why they cannot be used "indefinitely".

3. Your backup system was obviously untested, now you want credit for doing something that according to your webpage you should have been able to do anyways? You spent thousands because your backup system failed to restore.

Anyways, you overreacted and made an offer to provide all affected clients with 1 year of credit without any stipulations and now you are rescinding the offer. that's how I see it, just voicing my opinion.

sounds like these guys did everything possible to get this system back and appeared to provide updates on progress - you cant ask much more than that - these things happen

A years free hosting would be nice, but I think a 3 month deal and knowing that these guys are able to pull out all the stops to get you going again in the event of a failure... Priceless

1. I read through the whole thread as I was following it during the issue.

2. (2) servers need to be burned to the ground because they were used during a restoration process? Please explain why they cannot be used "indefinitely".

3. Your backup system was obviously untested, now you want credit for doing something that according to your webpage you should have been able to do anyways? You spent thousands because your backup system failed to restore.

2) We used four servers to make this restore work. Currently two are only being used to store the old data.

3) Our backup system was tested. We have done more than 25 test baremetal restores. All finished within 6 hours. This was a completely different issue since file system corruption was also in the backups and required a ton of work to fix.

So wait since 2 servers are being used to store old data you can NEVER EVER use them for anything again?

It is common to use a couple of servers in a situation like this and does in no way mean that you cannot use the servers for something else next week, I mean come on you know that's just wrong.

So wait since 2 servers are being used to store old data you can NEVER EVER use them for anything again?

It is common to use a couple of servers in a situation like this and does in no way mean that you cannot use the servers for something else next week, I mean come on you know that's just wrong.

That is not what I said. The servers will be tied up for at least 6 months. We need to keep them online in case clients find any missed files.

All that matters here is we did everything we could possibly could do and fixed the problem and restored all data. If that's not enough, we are giving 3x our stated SLA.

I agree that you did do what you are supposed to do and you did a good job.

I am just giving you some grief so that you remember this next time and dont jump the gun.


All in all you guys did put in the time and the effort required to make this happen and your customers have likely already forgotten about the whole ordeal.

I agree that you did do what you are supposed to do and you did a good job.


Thank you. Hopefully there will be no next time. Now all our efforts are to setup a totally new solution to prevent any downtime from one of these events.

That is not what I said. The servers will be tied up for at least 6 months. We need to keep them online in case clients find any missed files.

For the $4000 these servers cost, can you not transfer the files from these servers to a NAS or online storage company to handle this?

That way you could provide clients with access to this data without needing to tie up these servers.

To claim the SLA refund contact the sales department. The 1yr one was for full data loss, since we recovered completely within a reasonable amount of time (about the same as a rysnc/cpbackup restore) we will be tripling our SLA and offering 3 months free hosting.

Comming back on an announcement (kind of a promise) doesn't look very professional. When u made the posting u said it was a year compensation, u didn't say it was gonna be 3 months if the data was back (u should have done this).

Now it just looks like u made the announcement to fast and u're comming back on a given word.

Comming back on an announcement (kind of a promise) doesn't look very professional. When u made the posting u said it was a year compensation, u didn't say it was gonna be 3 months if the data was back (u should have done this).

Now it just looks like u made the announcement to fast and u're comming back on a given word.

I agree, I am a client of theirs (reseller) and as I lost a couple of clients due to this fiasco a years free hosting would have been nice to soften the blow.

They made the offer to avoid their clients to jump ship and now want to change the offer. At no time did I understand that the years credit was for total data loss.

Yes they did recover all data which is what they are suppose to do in a situation like this but the fact of the matter is they made an offer out of their own free will and now want to go back on it.

If this is so then I will be using the 3 months credit to dust off my life preserver (find a new host) and jump ship anyway.

I would guess it's a fair assumption as they had several failed restores. i.e. They assumed there was a total data loss, hence the original offer of one year free. At least that's how I read into it. Basically "we lost all of your data, we're sorry, here is a free year on the house. However they were able to restore the data, hence a minor inconvenience instead of total data loss. Again, that's how I was reading into it, perhaps I overlooked something?

Their clients got their data back + 3 months free hosting.

Technically, Sharkspace only had to give their clients 1 month of free hosting, but they gave them 3 months free hosting instead. So they actually got an extra 2 months of hosting for free.

Can't complain really.

I am sorry for the trouble I caused with my initial post. It was too early and I should have waited until the findings/process was completed.

In addition to the 3 months free we have approved a free upgrade to the next package for the life of the account. Contact the sales department and they can give you a free upgrade to the next package.


This hopefully will allow you to be able to give your clients free upgrades and help improve the entire situation.

For future reference, clients should only be privied to basic updates.

i.e. we're working on it... we're working with r1soft to restore the data... we ran into troubles with the restore, still working with r1... etc... Giving people too much information causes panic and troubles you don't need. Not to mention the free year issue because of the assumption of a total loss. Client's only see the bad, rarely the good. Keeping that in mind, updates should be short and simple.

Strange things are happening.

I logged into my reseller account today only to find all my accounts missing except the main account.

Other accounts created after the previous fiasco are also missing and sites that were updated recently now suddenly does not reflect these updates.

I did contact support and they are "working on it", anyone having the same problem?

Thanks.

It likely is just they need to be re-assigned under your main account or you are logging in on the old server (there is a notice posted on the WHM and Cpanel pages). There are no new issues. Support will fix this for you directly in your ticket.

It likely is just they need to be re-assigned under your main account or you are logging in on the old server (there is a notice posted on the WHM and Cpanel pages). There are no new issues. Support will fix this for you directly in your ticket.

Everything was working fine a few days ago, so what happened now?

Really depends on what you changed (name servers, etc.). Please respond directly in your ticket since this is not a support medium.

Really depends on what you changed (name servers, etc.). Please respond directly in your ticket since this is not a support medium.

I am busy awaiting an answer from support, I was only asking here if someone is experiencing the same problem.

Issue was fixed. Was just a minor fix where the sub-accounts just needed to be re-assigned to the reseller and now they all show under the reseller's whm. The accounts were always present on the server and online.


Hopefully this thread can now be closed since the original issue was resolved now almost 2 weeks ago.

This is where I think I would take a host from you guys.
Will ask you in a PM then.