Results 1 to 15 of 15
  1. #1

    Serverbeach / Peer1 Formats My Production RAID 1 Hard Disk and Reload OS

    Hello all,

    I'm not your average noob when it comes to servers and UNIX / Linux. I've over 2 dozens websites hosted for my personal, business, hobby and non profit Temple usage.

    I'm in shock right now, I've been Serverbeach / Peer1 USA and UK customer since last 15 or 16 months. The server runs few small sites and mostly used for deploying scripts for other my projects.

    Y'days SB tech supported ended up formatting my Raid 1 1TB Disk. Last known good backup was taken 4-5 days ago. I've lost some important Python, PHP and shell scripts and sql data which was not in the backups.

    History - my server went down at 02-08-2010 14:29:27 so I opened TT and an hour later they told me:
    Our technicians report that the power supply in the server had died. They replaced the power supply and the server is now online and responds to SSH.
    After login into server I need to verify disk integrity as server died so I asked them
    I'd like to run fsck to check a file systems. But, when I click on rapid rescue > rescue option in CP, nothing happens.
    Again tech replied:
    Unfortunately at this time the Rapid Rescue feature is not working properly. If you wish to have the server placed into rescue you will need to submit a ticket and we can manually place the server into the rescue environment.

    I am sending this ticket to our data center technicians to have them place the server into rescue for you.
    This is *where* my nightmare started, instead of starting server in the rescue mode, tech formatted my hard disk and 2 hours later told me:
    Unfortunately I have bad news. Our datacenter tech reports that when they booted the server into the Rescue environment, the system began to format the drive as if the server was to be reloaded. We're still investigating why this occurred, but while the tech stopped the format, the system will not boot from the drive.
    22 hours later they reloaded my server and now I've to restore whatever I have. The changes I made to scripts were critical including data. But due due to power supply problem my remote backup server did not kicked in time to fetch changes made to all scripts and their tech ended up deleting / formatting everything for now. I've asked them to give me access to old disk and so far no one replied to me and left me in shock. This is the last info I got from Facil Yohannes:
    The reason for the server getting a reload has been identified and our Developers have already corrected the problems, so this will not occur again. I sincerely apologize for this issue and will be contacting Client Relations to assist in further communications with you. I will be happy to give you a call again tomorrow morning, so I can provide updates on the technical side of the situation for you.
    My questions to community -
    • How come they deployed such the rescue system which formats clients working setup and destroyes all data?
    • They advertise on their site that 'rescue mode' is self help tool for geeks, but it did not worked as advertised. There are other hosts who will changes $100 for same setup but I'm paying twice the amount of money due to Peer1 name and this is what I get in return.
    • I don't believe that the rescue mode will automatically reload os (both after separates option in cp and can only be done by their support). I've five servers cluster hosted at Softlayer and never had any issue with the rescue mode ever to fix disk problems or OCFS2 cluster file systems or any other server problems. How come they deployed such system if it wasn't ready in production?
    • The worst part is I do have backup (80-85%) but I lost the all changes I made by me and my other developer as backup cron cannot get into server during first power supply problem. What are my options now?


    Sorry for the long post. I'm posting this so that others know that Peer1 (they call them self Enterprise hosting provider and hosting for geek my a$$) does not care to hire good sys admins and developers to test their *beeping* systems. They are now not replying my final query and not provided me access to old hard drive to see if I can use some tool to recover data:
    Can I access my old hard disk? What are my chances getting missing data and scripts?
    I now started to believe that Peer1 / SB is run by some bunch of morons.

  2. #2
    Join Date
    Feb 2008
    Location
    Houston, Texas, USA
    Posts
    3,262
    This is indeed horrible. DC techs do need to be composed and careful when handling production servers. Perhaps the tech chose the wrong menu option. Anyway, I wish you good luck rebuilding the box and hope this incident will be behind you soon.

    Regards
    Joe / UNIXY
    UNIXy - Fully Managed Servers and Clusters - Established in 2006
    Server Management - Unlimited Servers. Unlimited Requests. One Plan!
    cPanel Varnish Plugin -- Seamless SSL Caching (Let's Encrypt, AutoSSL, etc)
    Slow Site or Server? Unable to handle traffic? Same day performance fix: joe@unixy

  3. #3
    I will mostly move to another provider Avoid serverbeach service Like the Plague. Any suggestion for good service provider?

  4. #4
    Join Date
    Apr 2009
    Location
    England, UK
    Posts
    645
    Quote Originally Posted by photonplus View Post
    I will mostly move to another provider Avoid serverbeach service Like the Plague. Any suggestion for good service provider?
    Location? Budget? Managed/Unmanaged?

  5. #5
    US or EU will do around USD 200 - 2GB, 1TB RAID1, Intel P4/ Celeron, Linux. I hope I will at least get refund for such poor service as most of my sites are non profit nature.

  6. #6
    Join Date
    Sep 2009
    Posts
    114
    Hi
    sorry to hear the story
    its time to move
    you can use gigenet
    great company :-)

  7. #7
    Join Date
    Jun 2003
    Location
    Spain
    Posts
    4,251
    Quote Originally Posted by photonplus View Post
    US or EU will do around USD 200 - 2GB, 1TB RAID1, Intel P4/ Celeron, Linux. I hope I will at least get refund for such poor service as most of my sites are non profit nature.
    Shoot a PM to Manny at HostDime....he's got the HostDime user account here! He'll see you right!

    FutureHosting have some good deals at the moment too.

  8. #8
    Thanks for suggestion. I will check it out. I've placed a cancellation request with SB and requested refund. I really hope if they have any shame left they will at least refund my money. BTW, I dialed toll free support number 000-800-100-3491 and it says "Invalid number".

  9. #9
    Join Date
    Jul 2004
    Location
    Hong Kong
    Posts
    195
    I will leave as fast as I can. Random format customer's HD is evil.

  10. #10
    Join Date
    Mar 2003
    Location
    /root
    Posts
    23,990
    Looking at the message from the tech, it seems a problem with their system not caused by the technician.

    Specially 4 U
    Reseller Hosting: Boost Your Websites | Fully Managed KVM VPS: 3.20 - 5.00 Ghz, Pure Dedicated Power
    JoneSolutions.Com is on the net 24/7 providing stable and reliable web hosting solutions, server management and services since 2001
    Debian|Ubuntu|cPanel|DirectAdmin|Enhance|Webuzo|Acronis|Estela|BitNinja|Nginx

  11. #11
    Join Date
    Nov 2003
    Posts
    538
    Quote Originally Posted by UNIXy View Post
    This is indeed horrible. DC techs do need to be composed and careful when handling production servers. Perhaps the tech chose the wrong menu option. Anyway, I wish you good luck rebuilding the box and hope this incident will be behind you soon.

    Regards
    Joe / UNIXY
    It sounds to me like the server PXE booted into a re-format rather than PXE booting into 'rescue mode' which isn't really a tech's fault unless he accidentally selected the wrong option when configuring the PXE session (which shouldn't be possible in the first place).

    A lot of people really enjoy all of these auto-reformat and auto-rescue systems, but as we automate everything in this industry these sorts of things will happen. It adds time/cost to the process but when/if we ever deploy our auto-reformat system we will be taking images/backups before the re-install runs.

    I would also like to mention without starting a war here but RAID-1 (especially software RAID) is not a replacement for a solid multi-tiered backup strategy.
    Last edited by XLHost; 08-03-2010 at 09:22 AM.
    XLHost.com
    Dedicated Servers, Virtual Private Servers, and more since 1995.
    drew @ xlhost.com

  12. #12
    Quote Originally Posted by XLHost View Post
    It sounds to me like the server PXE booted into a re-format rather than PXE booting into 'rescue mode' which isn't really a tech's fault unless he accidentally selected the wrong option when configuring the PXE session (which shouldn't be possible in the first place).

    A lot of people really enjoy all of these auto-reformat and auto-rescue systems, but as we automate everything in this industry these sorts of things will happen. It adds time/cost to the process but when/if we ever deploy our auto-reformat system we will be taking images/backups before the re-install runs.

    I would also like to mention without starting a war here but RAID-1 (especially software RAID) is not a replacement for a solid multi-tiered backup strategy.
    I do have / has offsite backup server but as I said eariler backup server did not get to pickup changes made during last 24-36 hrs due to power supply failure. So changed made 2 days ago all are lost. They are not giving me access to old disks at all.

    Also I do not believe about a bug in system. They formatted it and now blaming on automated system. They must be using same system for months. May deploying 5 to 100 system a day.

  13. #13
    Can you attach 2 raid cards in the same server and boot from 1st raid to recover data from 2nd raid card? Another option they are offering is to attach 4 port raid card, but I'm not sure how am I suppose to use 4 port raid card to get data from old disks. Can you configure two raid 1 volumes from four port raid card?

  14. #14
    Join Date
    Oct 2002
    Location
    Vancouver, B.C.
    Posts
    2,699
    Quote Originally Posted by photonplus View Post
    Can you configure two raid 1 volumes from four port raid card?
    Yes, you absolutely can.
    ASTUTE INTERNET: Advanced, customized, and scalable solutions with AS54527 Premium Performance and Canadian Optimized Network (Level3, Shaw, CogecoPeer1, GTT/Tinet),
    AS63213 Cost Effective High Performance Network (Cogent, HE, GTT/Tinet)
    Dedicated Hosting, Colo, Bandwidth, and Fiber out of Vancouver, Seattle, LA, Toronto, NYC, and Miami

  15. #15
    Join Date
    Jul 2003
    Location
    San Antonio, TX
    Posts
    3
    Photonplus,

    On behalf of ServerBeach, I want to publicly apologize for our failure. I just emailed you and we will explain over the phone exactly what happened. In short, it was a combination of both human error as well as a legacy erroneous script which combined to create the lost data. (Both of these areas have been addressed and eliminated, by the way).

    We will also issue you an additional credit and coooperate fully with your data recovery efforts in order to ease your move to another hosting solution, whether that be with us or with another provider.

    At ServerBeach, we take customer service seriously. I hate seeng customers leave us - and it really hurts when I know it's something we could have prevented.

    This experience has revealed several areas where we need to make some changes. The first step, in my mind at least, is to own up to our failures publicly to you and others and then get busy putting the permanent fix in place. You have my commitment that we will follow through on these tasks quickly.

    We look forward to speaking with you soon.

    Respectfully,
    Robert Miggins
    Big Kahuna - ServerBeach

Similar Threads

  1. Peer1 VS ServerBeach?
    By hosteur in forum Dedicated Server
    Replies: 2
    Last Post: 02-01-2010, 07:04 PM
  2. 400GB Hard Disk Drives in RAID 0, RAID 5 and RAID 10 Arrays: Performance Analysis
    By donniesd in forum Hosting Security and Technology
    Replies: 0
    Last Post: 03-07-2007, 03:19 PM
  3. Is Serverbeach pure Peer1 BW now?
    By helios3 in forum Dedicated Server
    Replies: 2
    Last Post: 08-04-2006, 09:54 AM
  4. Raid Drive and 2nd Hard Disk
    By saj in forum Dedicated Server
    Replies: 7
    Last Post: 05-02-2006, 01:20 AM
  5. ServerBeach acquired by Peer1?
    By RikRok in forum Web Hosting Lounge
    Replies: 2
    Last Post: 10-12-2004, 12:20 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •