Web Hosting Talk : Web Hosting Main Forums : Dedicated Server : Serverbeach / Peer1 Formats My Production RAID 1 Hard Disk and Reload OS

[photonplus]

Hello all,

I'm not your average noob when it comes to servers and UNIX / Linux. I've over 2 dozens websites hosted for my personal, business, hobby and non profit Temple usage.

I'm in shock right now, I've been Serverbeach / Peer1 USA and UK customer since last 15 or 16 months. The server runs few small sites and mostly used for deploying scripts for other my projects.

Y'days SB tech supported ended up formatting my Raid 1 1TB Disk. Last known good backup was taken 4-5 days ago. I've lost some important Python, PHP and shell scripts and sql data which was not in the backups.

History - my server went down at 02-08-2010 14:29:27 so I opened TT and an hour later they told me:

Our technicians report that the power supply in the server had died. They replaced the power supply and the server is now online and responds to SSH.

After login into server I need to verify disk integrity as server died so I asked them

I'd like to run fsck to check a file systems. But, when I click on rapid rescue > rescue option in CP, nothing happens.

Again tech replied:

Unfortunately at this time the Rapid Rescue feature is not working properly. If you wish to have the server placed into rescue you will need to submit a ticket and we can manually place the server into the rescue environment.

I am sending this ticket to our data center technicians to have them place the server into rescue for you.

This is *where* my nightmare started, instead of starting server in the rescue mode, tech formatted my hard disk and 2 hours later told me:

Unfortunately I have bad news. Our datacenter tech reports that when they booted the server into the Rescue environment, the system began to format the drive as if the server was to be reloaded. We're still investigating why this occurred, but while the tech stopped the format, the system will not boot from the drive.

22 hours later they reloaded my server and now I've to restore whatever I have. The changes I made to scripts were critical including data. But due due to power supply problem my remote backup server did not kicked in time to fetch changes made to all scripts and their tech ended up deleting / formatting everything for now. I've asked them to give me access to old disk and so far no one replied to me and left me in shock. This is the last info I got from Facil Yohannes:

The reason for the server getting a reload has been identified and our Developers have already corrected the problems, so this will not occur again. I sincerely apologize for this issue and will be contacting Client Relations to assist in further communications with you. I will be happy to give you a call again tomorrow morning, so I can provide updates on the technical side of the situation for you.

My questions to community -

• How come they deployed such the rescue system which formats clients working setup and destroyes all data?
  
• They advertise on their site that 'rescue mode' is self help tool for geeks, but it did not worked as advertised. There are other hosts who will changes $100 for same setup but I'm paying twice the amount of money due to Peer1 name and this is what I get in return.
• I don't believe that the rescue mode will automatically reload os (both after separates option in cp and can only be done by their support). I've five servers cluster hosted at Softlayer and never had any issue with the rescue mode ever to fix disk problems or OCFS2 cluster file systems or any other server problems. How come they deployed such system if it wasn't ready in production?
  
• The worst part is I do have backup (80-85%) but I lost the all changes I made by me and my other developer as backup cron cannot get into server during first power supply problem. What are my options now?

Sorry for the long post. I'm posting this so that others know that Peer1 (they call them self Enterprise hosting provider and hosting for geek my a$$) does not care to hire good sys admins and developers to test their *beeping* systems. They are now not replying my final query and not provided me access to old hard drive to see if I can use some tool to recover data:

Can I access my old hard disk? What are my chances getting missing data and scripts?

I now started to believe that Peer1 / SB is run by some bunch of morons.

[UNIXy]

This is indeed horrible. DC techs do need to be composed and careful when handling production servers. Perhaps the tech chose the wrong menu option. Anyway, I wish you good luck rebuilding the box and hope this incident will be behind you soon.

Regards
Joe / UNIXY

[photonplus]

I will mostly move to another provider Avoid serverbeach service Like the Plague. Any suggestion for good service provider?

[CKGroup]

Originally Posted by photonplus

I will mostly move to another provider Avoid serverbeach service Like the Plague. Any suggestion for good service provider?

Location? Budget? Managed/Unmanaged?

[photonplus]

US or EU will do around USD 200 - 2GB, 1TB RAID1, Intel P4/ Celeron, Linux. I hope I will at least get refund for such poor service as most of my sites are non profit nature.

[irmet]

Hi
sorry to hear the story
its time to move
you can use gigenet
great company :-)

[XTremo]

Originally Posted by photonplus

US or EU will do around USD 200 - 2GB, 1TB RAID1, Intel P4/ Celeron, Linux. I hope I will at least get refund for such poor service as most of my sites are non profit nature.

Shoot a PM to Manny at HostDime....he's got the HostDime user account here! He'll see you right!

FutureHosting have some good deals at the moment too.

[photonplus]

Thanks for suggestion. I will check it out. I've placed a cancellation request with SB and requested refund. I really hope if they have any shame left they will at least refund my money. BTW, I dialed toll free support number 000-800-100-3491 and it says "Invalid number".

[HKPlane]

I will leave as fast as I can. Random format customer's HD is evil.

[net]

Looking at the message from the tech, it seems a problem with their system not caused by the technician.

[XLHost]

Originally Posted by UNIXy

This is indeed horrible. DC techs do need to be composed and careful when handling production servers. Perhaps the tech chose the wrong menu option. Anyway, I wish you good luck rebuilding the box and hope this incident will be behind you soon.

Regards
Joe / UNIXY

It sounds to me like the server PXE booted into a re-format rather than PXE booting into 'rescue mode' which isn't really a tech's fault unless he accidentally selected the wrong option when configuring the PXE session (which shouldn't be possible in the first place).

A lot of people really enjoy all of these auto-reformat and auto-rescue systems, but as we automate everything in this industry these sorts of things will happen. It adds time/cost to the process but when/if we ever deploy our auto-reformat system we will be taking images/backups before the re-install runs.

I would also like to mention without starting a war here but RAID-1 (especially software RAID) is not a replacement for a solid multi-tiered backup strategy.

[photonplus]

Originally Posted by XLHost

It sounds to me like the server PXE booted into a re-format rather than PXE booting into 'rescue mode' which isn't really a tech's fault unless he accidentally selected the wrong option when configuring the PXE session (which shouldn't be possible in the first place).

A lot of people really enjoy all of these auto-reformat and auto-rescue systems, but as we automate everything in this industry these sorts of things will happen. It adds time/cost to the process but when/if we ever deploy our auto-reformat system we will be taking images/backups before the re-install runs.

I would also like to mention without starting a war here but RAID-1 (especially software RAID) is not a replacement for a solid multi-tiered backup strategy.

I do have / has offsite backup server but as I said eariler backup server did not get to pickup changes made during last 24-36 hrs due to power supply failure. So changed made 2 days ago all are lost. They are not giving me access to old disks at all.

Also I do not believe about a bug in system. They formatted it and now blaming on automated system. They must be using same system for months. May deploying 5 to 100 system a day.

[photonplus]

Can you attach 2 raid cards in the same server and boot from 1st raid to recover data from 2nd raid card? Another option they are offering is to attach 4 port raid card, but I'm not sure how am I suppose to use 4 port raid card to get data from old disks. Can you configure two raid 1 volumes from four port raid card?

[hhw]

Originally Posted by photonplus

Can you configure two raid 1 volumes from four port raid card?

Yes, you absolutely can.

[ServerBeach]

Photonplus,

On behalf of ServerBeach, I want to publicly apologize for our failure. I just emailed you and we will explain over the phone exactly what happened. In short, it was a combination of both human error as well as a legacy erroneous script which combined to create the lost data. (Both of these areas have been addressed and eliminated, by the way).

We will also issue you an additional credit and coooperate fully with your data recovery efforts in order to ease your move to another hosting solution, whether that be with us or with another provider.

At ServerBeach, we take customer service seriously. I hate seeng customers leave us - and it really hurts when I know it's something we could have prevented.

This experience has revealed several areas where we need to make some changes. The first step, in my mind at least, is to own up to our failures publicly to you and others and then get busy putting the permanent fix in place. You have my commitment that we will follow through on these tasks quickly.

We look forward to speaking with you soon.

Respectfully,
Robert Miggins
Big Kahuna - ServerBeach