hosted by liquidweb


Go Back   Web Hosting Talk : Web Hosting Main Forums : Dedicated Server : Serverbeach / Peer1 Formats My Production RAID 1 Hard Disk and Reload OS
Reply

Forum Jump

Serverbeach / Peer1 Formats My Production RAID 1 Hard Disk and Reload OS

Reply Post New Thread In Dedicated Server Subscription
 
Send news tip View All Posts Thread Tools Search this Thread Display Modes
  #1  
Old 08-03-2010, 02:07 AM
photonplus photonplus is offline
Newbie
 
Join Date: Aug 2010
Posts: 8

Serverbeach / Peer1 Formats My Production RAID 1 Hard Disk and Reload OS


Hello all,

I'm not your average noob when it comes to servers and UNIX / Linux. I've over 2 dozens websites hosted for my personal, business, hobby and non profit Temple usage.

I'm in shock right now, I've been Serverbeach / Peer1 USA and UK customer since last 15 or 16 months. The server runs few small sites and mostly used for deploying scripts for other my projects.

Y'days SB tech supported ended up formatting my Raid 1 1TB Disk. Last known good backup was taken 4-5 days ago. I've lost some important Python, PHP and shell scripts and sql data which was not in the backups.

History - my server went down at 02-08-2010 14:29:27 so I opened TT and an hour later they told me:
Quote:
Our technicians report that the power supply in the server had died. They replaced the power supply and the server is now online and responds to SSH.
After login into server I need to verify disk integrity as server died so I asked them
Quote:
I'd like to run fsck to check a file systems. But, when I click on rapid rescue > rescue option in CP, nothing happens.
Again tech replied:
Quote:
Unfortunately at this time the Rapid Rescue feature is not working properly. If you wish to have the server placed into rescue you will need to submit a ticket and we can manually place the server into the rescue environment.

I am sending this ticket to our data center technicians to have them place the server into rescue for you.
This is *where* my nightmare started, instead of starting server in the rescue mode, tech formatted my hard disk and 2 hours later told me:
Quote:
Unfortunately I have bad news. Our datacenter tech reports that when they booted the server into the Rescue environment, the system began to format the drive as if the server was to be reloaded. We're still investigating why this occurred, but while the tech stopped the format, the system will not boot from the drive.
22 hours later they reloaded my server and now I've to restore whatever I have. The changes I made to scripts were critical including data. But due due to power supply problem my remote backup server did not kicked in time to fetch changes made to all scripts and their tech ended up deleting / formatting everything for now. I've asked them to give me access to old disk and so far no one replied to me and left me in shock. This is the last info I got from Facil Yohannes:
Quote:
The reason for the server getting a reload has been identified and our Developers have already corrected the problems, so this will not occur again. I sincerely apologize for this issue and will be contacting Client Relations to assist in further communications with you. I will be happy to give you a call again tomorrow morning, so I can provide updates on the technical side of the situation for you.
My questions to community -
  • How come they deployed such the rescue system which formats clients working setup and destroyes all data?
  • They advertise on their site that 'rescue mode' is self help tool for geeks, but it did not worked as advertised. There are other hosts who will changes $100 for same setup but I'm paying twice the amount of money due to Peer1 name and this is what I get in return.
  • I don't believe that the rescue mode will automatically reload os (both after separates option in cp and can only be done by their support). I've five servers cluster hosted at Softlayer and never had any issue with the rescue mode ever to fix disk problems or OCFS2 cluster file systems or any other server problems. How come they deployed such system if it wasn't ready in production?
  • The worst part is I do have backup (80-85%) but I lost the all changes I made by me and my other developer as backup cron cannot get into server during first power supply problem. What are my options now?

Sorry for the long post. I'm posting this so that others know that Peer1 (they call them self Enterprise hosting provider and hosting for geek my a$$) does not care to hire good sys admins and developers to test their *beeping* systems. They are now not replying my final query and not provided me access to old hard drive to see if I can use some tool to recover data:
Quote:
Can I access my old hard disk? What are my chances getting missing data and scripts?
I now started to believe that Peer1 / SB is run by some bunch of morons.



Sponsored Links
  #2  
Old 08-03-2010, 02:15 AM
UNIXy UNIXy is offline
Warp Speed!
 
Join Date: Feb 2008
Location: Houston, Texas, USA
Posts: 2,837
This is indeed horrible. DC techs do need to be composed and careful when handling production servers. Perhaps the tech chose the wrong menu option. Anyway, I wish you good luck rebuilding the box and hope this incident will be behind you soon.

Regards
Joe / UNIXY

__________________
|- UNIXY :: Fully Managed Servers and Clusters Since 2006
|- DC POP :: Houston, Los Angeles, Atlanta, & Rotterdam NL
|- Managed Magento Varnish Servers w/ ESI. < 250ms Page Load / TTFB
L- We LOVE helping our clients!

  #3  
Old 08-03-2010, 04:17 AM
photonplus photonplus is offline
Newbie
 
Join Date: Aug 2010
Posts: 8
I will mostly move to another provider Avoid serverbeach service Like the Plague. Any suggestion for good service provider?

Sponsored Links
  #4  
Old 08-03-2010, 04:23 AM
CKGroup CKGroup is offline
Disabled
 
Join Date: Apr 2009
Location: England, UK
Posts: 643
Quote:
Originally Posted by photonplus View Post
I will mostly move to another provider Avoid serverbeach service Like the Plague. Any suggestion for good service provider?
Location? Budget? Managed/Unmanaged?

  #5  
Old 08-03-2010, 04:32 AM
photonplus photonplus is offline
Newbie
 
Join Date: Aug 2010
Posts: 8
US or EU will do around USD 200 - 2GB, 1TB RAID1, Intel P4/ Celeron, Linux. I hope I will at least get refund for such poor service as most of my sites are non profit nature.

  #6  
Old 08-03-2010, 04:41 AM
irmet irmet is offline
WHT Addict
 
Join Date: Sep 2009
Posts: 114
Hi
sorry to hear the story
its time to move
you can use gigenet
great company :-)

  #7  
Old 08-03-2010, 04:54 AM
XTremo XTremo is offline
ShillBuster
 
Join Date: Jun 2003
Location: Spain
Posts: 3,982
Quote:
Originally Posted by photonplus View Post
US or EU will do around USD 200 - 2GB, 1TB RAID1, Intel P4/ Celeron, Linux. I hope I will at least get refund for such poor service as most of my sites are non profit nature.
Shoot a PM to Manny at HostDime....he's got the HostDime user account here! He'll see you right!

FutureHosting have some good deals at the moment too.

  #8  
Old 08-03-2010, 04:57 AM
photonplus photonplus is offline
Newbie
 
Join Date: Aug 2010
Posts: 8
Thanks for suggestion. I will check it out. I've placed a cancellation request with SB and requested refund. I really hope if they have any shame left they will at least refund my money. BTW, I dialed toll free support number 000-800-100-3491 and it says "Invalid number".

  #9  
Old 08-03-2010, 06:58 AM
HKPlane HKPlane is offline
Junior Guru
 
Join Date: Jul 2004
Location: Hong Kong
Posts: 195
I will leave as fast as I can. Random format customer's HD is evil.

  #10  
Old 08-03-2010, 09:12 AM
net net is offline
Community Liaison
 
Join Date: Mar 2003
Posts: 11,128
Looking at the message from the tech, it seems a problem with their system not caused by the technician.

__________________
.
JoneSolutions.Com + SSS = Your Number One Choice On The Net - since 2001

It's Fully Managed and Secured. Ask us at sales @ jonesolutions.com .

  #11  
Old 08-03-2010, 09:19 AM
XLHost XLHost is offline
Web Hosting Evangelist
 
Join Date: Nov 2003
Posts: 534
Quote:
Originally Posted by UNIXy View Post
This is indeed horrible. DC techs do need to be composed and careful when handling production servers. Perhaps the tech chose the wrong menu option. Anyway, I wish you good luck rebuilding the box and hope this incident will be behind you soon.

Regards
Joe / UNIXY
It sounds to me like the server PXE booted into a re-format rather than PXE booting into 'rescue mode' which isn't really a tech's fault unless he accidentally selected the wrong option when configuring the PXE session (which shouldn't be possible in the first place).

A lot of people really enjoy all of these auto-reformat and auto-rescue systems, but as we automate everything in this industry these sorts of things will happen. It adds time/cost to the process but when/if we ever deploy our auto-reformat system we will be taking images/backups before the re-install runs.

I would also like to mention without starting a war here but RAID-1 (especially software RAID) is not a replacement for a solid multi-tiered backup strategy.

__________________
XLHost.com
Dedicated Servers, Virtual Private Servers, and more since 1995.
drew @ xlhost.com


Last edited by XLHost; 08-03-2010 at 09:22 AM.
  #12  
Old 08-03-2010, 10:57 AM
photonplus photonplus is offline
Newbie
 
Join Date: Aug 2010
Posts: 8
Quote:
Originally Posted by XLHost View Post
It sounds to me like the server PXE booted into a re-format rather than PXE booting into 'rescue mode' which isn't really a tech's fault unless he accidentally selected the wrong option when configuring the PXE session (which shouldn't be possible in the first place).

A lot of people really enjoy all of these auto-reformat and auto-rescue systems, but as we automate everything in this industry these sorts of things will happen. It adds time/cost to the process but when/if we ever deploy our auto-reformat system we will be taking images/backups before the re-install runs.

I would also like to mention without starting a war here but RAID-1 (especially software RAID) is not a replacement for a solid multi-tiered backup strategy.
I do have / has offsite backup server but as I said eariler backup server did not get to pickup changes made during last 24-36 hrs due to power supply failure. So changed made 2 days ago all are lost. They are not giving me access to old disks at all.

Also I do not believe about a bug in system. They formatted it and now blaming on automated system. They must be using same system for months. May deploying 5 to 100 system a day.

  #13  
Old 08-03-2010, 11:51 AM
photonplus photonplus is offline
Newbie
 
Join Date: Aug 2010
Posts: 8
Can you attach 2 raid cards in the same server and boot from 1st raid to recover data from 2nd raid card? Another option they are offering is to attach 4 port raid card, but I'm not sure how am I suppose to use 4 port raid card to get data from old disks. Can you configure two raid 1 volumes from four port raid card?

  #14  
Old 08-03-2010, 12:11 PM
hhw hhw is offline
Web Hosting Master
 
Join Date: Oct 2002
Location: Vancouver, B.C.
Posts: 2,314
Quote:
Originally Posted by photonplus View Post
Can you configure two raid 1 volumes from four port raid card?
Yes, you absolutely can.

__________________
Han Hwei Woo, ASTUTE HOSTING AS54527 *Advanced and customized solutions for the savvy customer!*
Dedicated Hosting and CDN out of Vancouver, Seattle, LA, Toronto, NYC, Miami, and London
We include CDN, anycast DNS, onboard KVMoIP, firewall, local and global load-balancing, and privatenet with all servers.
sales@astutehosting.com

  #15  
Old 08-03-2010, 05:17 PM
Peer1 Peer1 is offline
New Member
 
Join Date: Jul 2003
Location: San Antonio, TX
Posts: 3
Photonplus,

On behalf of ServerBeach, I want to publicly apologize for our failure. I just emailed you and we will explain over the phone exactly what happened. In short, it was a combination of both human error as well as a legacy erroneous script which combined to create the lost data. (Both of these areas have been addressed and eliminated, by the way).

We will also issue you an additional credit and coooperate fully with your data recovery efforts in order to ease your move to another hosting solution, whether that be with us or with another provider.

At ServerBeach, we take customer service seriously. I hate seeng customers leave us - and it really hurts when I know it's something we could have prevented.

This experience has revealed several areas where we need to make some changes. The first step, in my mind at least, is to own up to our failures publicly to you and others and then get busy putting the permanent fix in place. You have my commitment that we will follow through on these tasks quickly.

We look forward to speaking with you soon.

Respectfully,
Robert Miggins
Big Kahuna - ServerBeach

Reply

Similar Threads
Thread Thread Starter Forum Replies Last Post
Peer1 VS ServerBeach? hosteur Dedicated Server 2 02-01-2010 07:04 PM
400GB Hard Disk Drives in RAID 0, RAID 5 and RAID 10 Arrays: Performance Analysis UFBSolutions Hosting Security and Technology 0 03-07-2007 03:19 PM
Is Serverbeach pure Peer1 BW now? helios3 Dedicated Server 2 08-04-2006 09:54 AM
Raid Drive and 2nd Hard Disk saj Dedicated Server 7 05-02-2006 01:20 AM
ServerBeach acquired by Peer1? RikRok Web Hosting Lounge 2 10-12-2004 12:20 PM

Related posts from TheWhir.com
Title Type Date Posted
New Zealand Court Rules Warrants Used in 2012 Dotcom Mansion Raid Legal Web Hosting News 2014-02-19 12:06:55
PEER 1 Hosting Extends VMware Public Cloud to US Web Hosting News 2013-09-24 12:04:51
PEER 1 Hosting Provides Cloud, Managed Services to Dell Customers in Canada Web Hosting News 2013-07-22 10:54:01
Interactive Map of the Internet App Shows How Networks Connect in 3D Blog 2013-03-06 12:06:20
Microsoft Acquires Cloud-Integrated Storage Provider StorSimple Web Hosting News 2012-10-16 17:10:45


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes
Postbit Selector

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump
Login:
Log in with your username and password
Username:
Password:



Forgot Password?
Advertisement:
Web Hosting News:



 

X

Welcome to WebHostingTalk.com

Create your username to jump into the discussion!

WebHostingTalk.com is the largest, most influentual web hosting community on the Internet. Join us by filling in the form below.


(4 digit year)

Already a member?