Page 1 of 2 12 LastLast
Results 1 to 25 of 38
  1. #1

    * ServerMatrix Support Data Loss

    Hi All,

    I had a server at SM for a few personal sites as they offer some really great special offers, a few days ago my server had a main HD failure and I asked for a reload onto a new HD and to install my existing main drive as a slave drive so I could attempt to retrieve the data (the drive was not totally dead). However firstly it took nearly 12 hours to do a restore (this I can live with as its only personal), what I can't live with is the fact that they accidently wiped the bad drive to a clean cpanel install as well. When I raised this point to them they did not respond for just over 24 hours then just closed the ticket !

    This is the worst hosting company for support I have ever had the misfortune to use, I do not recommend that anyone uses them who wishes to ensure data integrity or prompt support.

    If anyone from SM reads this and wishes to contact me to explain this it is in regards to ticket 247794PLNT_which you closed 7:13 am your time today with no comment on the fact you wiped all my data by mistake.

    Ticket Number: 247794PLNT_
    Ticket Type: Accounting_
    Status: CLOSED_
    Last Updated: 05/10/2004 09:17:10_

    ------------------------------------------
    (mdeater-05/08/04-20:34):
    Thank you for choosing Server Matrix, your server is up and running, and your old harddrive has been placed in your box, It it hdc.
    --------------------------------------
    (c13639fcol-05/09/04-07:13):
    What have you done with the data ? It appears the second drive has been formated or is a different drive. Please restore my data. See below for analysis of the two drives

    Filesystem Size Used Avail Use% Mounted on

    /dev/hdc3 72G 2.3G 66G 4% /

    /dev/hda3 72G 2.3G 66G 4% /old

    hdc is now an exact mirror of hda, this drive had about 35-40gb of data on it._
      0 Not allowed!

  2. #2
    The reason the ticket was not updated is when they finished the install it was sent to billing. The Accounting dept is closed on weekends.

    A rep is contacting you within the next few minutes.
    Jeff Reinis
    (469) 607-3005 - jreinis@cogentco.com
    www.cogentco.com
    Datacenter Colocation - Tier-1 IP Transit - Bandwidth Solutions
      0 Not allowed!

  3. #3
    Join Date
    Jun 2003
    Location
    Ottawa, Canada
    Posts
    124
    Not to take anything away from the mess up SM did, but you should have had your own backups. Never trust anyone with your only copy of your data. Things like this happens sometimes and will probably always happens every now and then. Only way to prevent it is to have your own backups. This isn't a SM only problem, i recall seeing problems like this at ev1 as well and probably every provider out there.

    If they can't find your data then it sucks to have to redo everything but at least you learned to cover your but from now on.

    Sorry for being a bit brash, yes SM messed up, but so did you.
      0 Not allowed!

  4. #4
    I just talked with someone on the phone who will investigate. However, my point is not that the HD failed, as I mentioned, it was a personal site that I can reupload, what I do object to and think is poor is.

    1/. A tech was asked to install cpanel on a new drive and put the second drive as slave, un mounted. They proceeded to put a cpanel install on both drives fresh.
    2/. When I asked SM what happened there was no responce, sure the ticket may have gone to billing, but why did billing not push to technical to investigate when they saw my responce rather than closing.

    All my production systems are fully backed up both to secondary drives and an off machine backup server

    Alan
      0 Not allowed!

  5. #5
    SM/ThePlanet does actually have quite a few screwups (more lately than before), I have had serveral recently, and I have been using their Total Control Servers.
    crucialparadigm - Affordable, Reliable, Professional :
    Web Hosting
    24/7 Support • Web Hosting • Reseller Hosting • Cloud/VPS Plans • Dedicated Servers •
      0 Not allowed!

  6. #6
    Join Date
    Nov 2002
    Location
    WebHostingTalk
    Posts
    8,901
    Originally posted by crucialparad
    SM/ThePlanet does actually have quite a few screwups (more lately than before), I have had serveral recently, and I have been using their Total Control Servers.
    Funny... in a managed.com thread, you said you had problems with them too. Have you found a provider that doesn't screw up your stuff?

    Can I help you find the common denominator?

    Sirius
    I support the Human Rights Campaign!
    Moving to the Tampa, Florida area? Check out life in the suburbs in Trinity, Florida.
      0 Not allowed!

  7. #7
    Join Date
    Nov 2003
    Posts
    1,093
    Bare in mind, they're adding at least 1000 new servers a month and therefore as they add more servers the likelihood of screwups only gets larger...
      0 Not allowed!

  8. #8
    sirius:

    Do you need explanation of the problems?

    Manged.com
    --------------
    - unreliable network
    - can cancel anyones server!!! (now I don't think there is much explanation for that)
    - don't listen to support tickets

    ThePlanet
    -----------
    - has been quite good until recently and I still like their service, but they DID have a few screwups
    - Firstly I ordered a Total Control Server over $400/month, the server kept having problems and was freezing and having intermittent network problems. It turns out they gave me the wrong IP addresses, so the name servers that I had placed on those IPs had to be removed, and caused me 2 days worth of down time for some sites. I got a refund for this so I am quite happy about it.
    - I asked them to enable HT on the server and update with the appropriate kernel. They did this, later I had problems and the server so I tried rebooting, it had problems starting up. Turns out they messed up the Grub config file. This was no major problem (execept for 1 hour downtime!). This was yet another screwup.

    Like I stated in my message SM/TC does have screwups, I didn't say I am not satisfied with them. They provided me decent refunds in all instances and I am quite happy with them actually.

    And managed.com on the other hand should not be used for mission critical applications for obvious reasons.
    crucialparadigm - Affordable, Reliable, Professional :
    Web Hosting
    24/7 Support • Web Hosting • Reseller Hosting • Cloud/VPS Plans • Dedicated Servers •
      0 Not allowed!

  9. #9
    Join Date
    Feb 2004
    Posts
    208
    I encountered a failed drive also with managed.com. The support staff actually ask me if I want to attach the failed drive as slave so I can salvage my data. I don't intend to try because I afraid the corrupted data might do more harm. So in less than 2 hours I have a new drive with the default O/S in the same server.

    I learned that I better backup my data. I don't feel good that I have the server for just over 1 month and the drive failed, but glad that the support is not bad after all.
      0 Not allowed!

  10. #10
    Join Date
    Sep 2001
    Location
    Seattle, WA
    Posts
    3,085
    Originally posted by sirius

    Can I help you find the common denominator?

    Sirius


    That seems to be the case with a lot of chronic complainers.
    Jim Reardon - jim/amusive.com
      0 Not allowed!

  11. #11
    Join Date
    Feb 2004
    Location
    Your Screen
    Posts
    3,999
    sc_dean, can you post a copy of the complete transaction / entire ticket?

    When I went through the same thing w/ SM (actually I had a rooted box) I:
    (a) called them on the phone and speak to the tech doing the work, and specifically requested the OS reload be done on a new drive;
    (b) open a separate ticket authorizing the set-up + monthly charges for the old drive to be remounted on the chassis (SM required this)

    Obviously we only have about 5% of the story here, we have no idea what the actual interaction was. And I do not wish to suggest that SM did not make an error -- screw-ups happen. However. Their standard OS reload practice is to wipe the drive and reload the OS on that same drive. I would not be at all surprised if the tech assigned the reload simply progressed "as usual," they do dozens of reloads every day.

    What really puzzles me is why they would wipe and reload on a "bad" drive. A bad drive should be completely swapped out. Did you request it as a reload? Or was your ticket put through as a hardware failure?

    Just, odd.

    Bailey
    Let's Connect on Twitter! @thatsmsgeek2u || Fighting mediocrity one thread at a time.
      0 Not allowed!

  12. #12
    Join Date
    Apr 2003
    Location
    Washington, DC
    Posts
    595
    Originally posted by amusive.com


    That seems to be the case with a lot of chronic complainers.
    That's funny.
    Benoît Brookens III
    President - Dataracks, Inc.
    Dataracks.net | Ultra-Reliable Hosting.™
      0 Not allowed!

  13. #13
    There is only one other post in the restore request (you are correct there was a ticket before that workign out it was an HD error) That was

    Details:
    This is a request to reload the Operating System on my server. The request details are as follows:

    Hardware Name : <removed> (Server)
    Hardware Other :

    Location info : Rack : AE48
    Row : AE
    Unknown : Unknown

    Operating System: Red Hat Enterprise Linux, Version 3 (recommended)
    Comments : Hi,

    Please reload RHE w/cpanel onto a NEW 80gb HD and put the second HD in as a slave (you will not be able to mount). I beleive there is no charge for either the new drive or the reload as this restore is due to HW failure. I would appreciate this being worked on ASAP as the server has now been down for over 6 hours. I will open a ticket to have the damaged drive removed within 3 days once I have tried to remove data from it. Ref number of existing ticket is 247601

    Alan
      0 Not allowed!

  14. #14
    Join Date
    Feb 2004
    Location
    Your Screen
    Posts
    3,999
    This is a request to reload the Operating System on my server.
    Well there's the cause of the problem. If you were looking for replacement of faulty hardware, requesting an OS reload was the wrong work the have requested.

    Replacement of faulty hardware, and an OS reload, are two totally, completely, different things.

    I realize that you explained it farther below, however, requesting the wrong service at the outset very likely caused the issue. (Tech tunnel vision... I'm not saying it was right, because obviously the tech didn't read... and he should have)

    You requested an OS reload, and that's exactly what he did.

    Bailey
    Let's Connect on Twitter! @thatsmsgeek2u || Fighting mediocrity one thread at a time.
      0 Not allowed!

  15. #15
    Join Date
    Apr 2004
    Posts
    1,834
    Originally posted by sirius
    Can I help you find the common denominator?
    The common denominator is this. Large bargain dedicated hosts play a numbers game. They have standard offerings and 97% of their customers do not have problems. Hope and pray you are not one of the unfortunate 3%. They are not equiped to handle problems efficiently.

    Here, let's do the math. I had a cronic hardware problem at server matrix with 3 seperate and unidentifable failures. This outage cost me a downtime of 10 days total per month. One of these outages took 6 days to resolve. During this 6 days my server was down they added 6/7 X 1000/4 servers which is 214 new customers before my server was back online.

    Now, if you ran the company would you prefer to have 1 happy existing customer or 214 new customers?
      0 Not allowed!

  16. #16
    Actually I rang them in regards to the first ticket and they said to open a restore ticket and ask for the second drive to be put in as slave, what I did was following the advice of the tech on the phone.
      0 Not allowed!

  17. #17
    Join Date
    Feb 2004
    Location
    Your Screen
    Posts
    3,999
    ...which by then was probably too late.

    Sorry to hear of the trouble, but, lesson learned I guess? Again I am not saying SM is in the right here, obviously the tech did not read the details... but I certainly can understand how an OS Reload came about off a ticket entitled an OS Reload request.

    BTW what of the data on your backup drive? Was your data not saved there as it should have been, or...?

    Bailey
    Let's Connect on Twitter! @thatsmsgeek2u || Fighting mediocrity one thread at a time.
      0 Not allowed!

  18. #18
    Join Date
    Mar 2003
    Location
    Charlotte, NC
    Posts
    2,761
    I've heard of this happening at EV1, as well. I am not a proponent of using extra harddrives to do backups on, as the data isn't safe if a box is compromised, therefore I always have sent backups offsite. If you're concerned about bandwidth, use RSYNC or R-Diff to save yourself some transfer. Even here in the office, we do backups between computers, and onto CD-RWs as well.

    -Josh
      0 Not allowed!

  19. #19
    Join Date
    Aug 2002
    Location
    Trouble will find me!
    Posts
    1,470
    Originally posted by sc_dean
    Actually I rang them in regards to the first ticket and they said to open a restore ticket and ask for the second drive to be put in as slave, what I did was following the advice of the tech on the phone.
    Originally posted by bithost(NET)
    ...which by then was probably too late.
    How was that too late? This was the first ticket in regards to an "OS restore" for which he followed SM/TP procedures as he was told.

    I would call it incompetant techs and bad management...

    I hope you get it worked out ...
      0 Not allowed!

  20. #20
    Bailey,

    I do not understand, in the trouble shooting ticket where we worked out the problem they asked me to open a second restore ticket to have it restored. This was purely a mistake by SM. I am not complaining that the HD failed OR that I didn't have a backup, if they had followed the instructions and put the drive in and I was unable to retrieve the data I would not have said a word. However since my post here they have been very helpful and here is the resolution

    Resolution : -

    1/. SM had their 'Hardware Manager' look at my system, the system was offline for c. 4/5 hours, when it came back on they had 'found' my data and hdc was mounted as /data with all my data on.

    ------------------------------------------
    (lcarter-05/10/04-15:46):
    Your server is back online. The original drive is mounted as /data. Please take a look and let us knwo when the damages drive needs to be removed.

    2/. SM refunded me for the restore even though they were not meant to have charged me for it as it was under the Hardware SLA, I had already confirmed this with the tech on phone and reiterated it in the restore ticket. I was not even aware I had been charged.

    ------------------------------------------
    (bhooker-05/10/04-16:10):
    I have refunded $75.00 to your credit card.

    I am hoping that ServerMatrix will offer something under the SLA or as a goodwill gesture for the hassle, here is the problems SM needs to sort and things they don't from my experiance

    PROBLEM 1 (the worst): Techs and Billing people do not read the tickets in detail, they skim read and miss important instructions
    PROBLEM 2 : There should be a tag or similar in the restore ticket to specify why a restore was needed as restores due to Hardware failures should not be charged under their Hardware SLA
    GOOD 1 : They where able to find my HD, my guess is they actually put two new drives in.
    GOOD 2 : Once they did realise the problem they where helpful and called me from USA to Spain three times (though perhaps when calling international they should check timezones, the last call was at 1am for me )

    Hopefully they will offer me something to keep my personal/test server there so next time I can post how they did right
      0 Not allowed!

  21. #21
    Join Date
    Feb 2004
    Location
    Your Screen
    Posts
    3,999
    sc_dean, the problem is, based on the ticket data you have provided here, you never asked for a restore. You asked for an OS reload. These are completley and entirely different things -- unrelated, in fact.

    s.n.a.zz.y, just FYI, with ServerMatrix reload tickets are opened in a special section of Orbit and are assigned to a separate team of techs. The reason I was guessing it was likely 'too late' is because the first thing a Reload Tech does, is pop the drive in the imager and wipe it. Get it started ASAP, that makes the work finished quicker, as reload customers are always screaming to have their boxes up ASAP. It's all 'hurry, hurry' when it comes to reloads (because they're offline), and the reload crew does try to keep that in mind. (Or so they have assured me! LOL)

    sc_dean, I would personally guess the confusion stems from the wording in the ticket provided above. I figure you 'n I are not connecting in terms of nuances, so would it help to provide my interpretation of your 'further details' as provided in this same thread -- how I would read it as a tech (understanding of course that we are still missing huge chunks of ticket history)? I don't want to do this to point fingers at all!!!! I just am trying to show how there are different ways to read the instructions in the ticket, hopefully to help avoid situations like this in the future.

    Details: This is a request to reload the Operating System on my server.
    Please reload RHE w/cpanel onto a NEW 80gb HD and put the second HD in as a slave (you will not be able to mount). I beleive there is no charge for either the new drive or the reload as this restore is due to HW failure.
    • One, why is anything being reloaded on a new drive? If you need a reload, the only thing that can be reloaded is the original (bad) drive.
    • "...put the second HD in as a slave..." OK, this is a question of which drive is which. Normally users refer to the first drive as the one being already installed in the server, so the second hard drive is the new one.
    • "I beleive there is no charge for either the new drive or the reload..." This sentence appears to be refering to two drives: one new clean drive, and an OS-reloaded drive that is failing...
    • "...as this restore is due to HW failure." We don't know anything about a restore, this is the first mention of it, so that must be something the customer's working on.


    The ticket is very specific to requesting a reload. However the clean drive would not require any sort of OS reload, it should have a clean OS already on it as a boot drive, and it should go in bay #1. The original drive should not be touched -- no reload, no restore -- and just go in bay #2. So why does the customer make any mention of an OS reload at all?!?! This is why I am confused and why I would not be at all surprised if the failing drive had wound up wiped and reloaded. Sorry I wasn't more detailed in explaining, before.

    sc_dean, I am sorry, I don't wish to point fingers, I realize that you've had a very long couple of days. A drive failure -- or compromise too -- makes for some nasty long hours of server work, and I can definitely empathize. However the words restore and reload are not interchangeable. One brings data back. The other destroys it.



    In the end, were the rolls reversed, I would probably feel very lucky walking away with everything so intact, based on the scenario presented... The SM Hardware Techs are very knowledgeable and have fine tools on hand, I am very glad to hear that they were able to recover/find your data (whatever the case may be). I do believe that they really do want to help customers and give them the best outcome possible, based on my time spent on the phone with a number of their techs; they were as nice and helpful as the day is long -- as professional as I have worked with at any data center.

    If SM was able to sort out that you just wanted the old drive untouched in bay 2, and a clean RHES/cPanel boot drive in bay 1, based on what we've read here ---major kudos to SM. They made it happen and that's fantastic.

    With 35-40 GB of data, good heavens, I'd also invest in a backup drive they're not "just" handy for compromises (BTW, only root compromises can result in possibly losing all back-up data, but regardless, even then, the full back-up isn't always lost). Data centers will be happy to give you a fresh drive when the old one fails, but their TOS's are pretty clear on "not responsible for lost data." Backups are essential. With a backup drive on board & being used, they could have just swapped out your boot drive and have been done with it.

    Anyway sc_dean, no hard feelings and I promise I am not trying to hassle you at all I just hope this helps explain how the interpretation of the data provided might have been what resulted in the confusion.

    Bailey
    Let's Connect on Twitter! @thatsmsgeek2u || Fighting mediocrity one thread at a time.
      0 Not allowed!

  22. #22
    I am always forgetting, although American is based on English, we do not actually speak the same language !

    Let me clarify the points you raised

    "Please reload RHE w/cpanel onto a NEW 80gb HD"
    This specifically askes for an OS to be loaded onto a new drive, it is being 're'loaded to imply that the same operating system and panel are to be used as the previous install. To install or to load are the same thing.

    "...put the second HD in as a slave..."
    I have just refered to a new drive being loaded with software, there are now two drives in the machine, 'the second drive' quite obviously refers to the 'other' drive than the one I just refered to. To put it as slave means to set the jumper to salve and attach it to the IDE chain. This is more efficient that talking about an old and new drive, how do you know which is the old and new once they are pulled ?

    "I beleive there is no charge for either the new drive or the reload..." & "...as this restore is due to HW failure."
    This refers to the fact there should be no monthly charge for having two drives in the system and that there should not be the usualy $75 OS install fee as the OS install is due to a HW failure.

    Quote " We don't know anything about a restore, this is the first mention of it, so that must be something the customer's working on."
    It clearly references the old ticket to explain EXACTLY what is going on if the tech had any questions. In the SM panel it is called an OS restore, so it is fairly obvious what that refers to at any rate.

    My words words where correctly used based on the context I used them in. I could have been more elequent with my wording, but at goen midnight adter 6 hours troubleshooting I should not have to explain my problem more than I did, especially after a 5 min telephone conversation with a technican

    I do not understand why you are trying to find some error with what I did. I requested something via phone in detail, from that phone call I was asked to raise a ticket which I did. Neither the phone or ticket instructions where followed, and if the tech was being as padantic as you where in the reply the ticket would have made 0 sence and he should have queried what I wanted (even though I have just explained it in detail on the phone).
      0 Not allowed!

  23. #23
    Join Date
    Aug 2002
    Location
    Trouble will find me!
    Posts
    1,470
    Originally posted by bithost(NET)
    sc_dean, the problem is, based on the ticket data you have provided here, you never asked for a restore. You asked for an OS reload. These are completley and entirely different things -- unrelated, in fact.

    s.n.a.zz.y, just FYI, with ServerMatrix reload tickets are opened in a special section of Orbit and are assigned to a separate team of techs. The reason I was guessing it was likely 'too late' is because the first thing a Reload Tech does, is pop the drive in the imager and wipe it. Get it started ASAP, that makes the work finished quicker, as reload customers are always screaming to have their boxes up ASAP. It's all 'hurry, hurry' when it comes to reloads (because they're offline), and the reload crew does try to keep that in mind. (Or so they have assured me! LOL)

    sc_dean, I would personally guess the confusion stems from the wording in the ticket provided above. I figure you 'n I are not connecting in terms of nuances, so would it help to provide my interpretation of your 'further details' as provided in this same thread -- how I would read it as a tech (understanding of course that we are still missing huge chunks of ticket history)? I don't want to do this to point fingers at all!!!! I just am trying to show how there are different ways to read the instructions in the ticket, hopefully to help avoid situations like this in the future.


    • One, why is anything being reloaded on a new drive? If you need a reload, the only thing that can be reloaded is the original (bad) drive.
    • "...put the second HD in as a slave..." OK, this is a question of which drive is which. Normally users refer to the first drive as the one being already installed in the server, so the second hard drive is the new one.
    • "I beleive there is no charge for either the new drive or the reload..." This sentence appears to be refering to two drives: one new clean drive, and an OS-reloaded drive that is failing...
    • "...as this restore is due to HW failure." We don't know anything about a restore, this is the first mention of it, so that must be something the customer's working on.


    The ticket is very specific to requesting a reload. However the clean drive would not require any sort of OS reload, it should have a clean OS already on it as a boot drive, and it should go in bay #1. The original drive should not be touched -- no reload, no restore -- and just go in bay #2. So why does the customer make any mention of an OS reload at all?!?! This is why I am confused and why I would not be at all surprised if the failing drive had wound up wiped and reloaded. Sorry I wasn't more detailed in explaining, before.

    sc_dean, I am sorry, I don't wish to point fingers, I realize that you've had a very long couple of days. A drive failure -- or compromise too -- makes for some nasty long hours of server work, and I can definitely empathize. However the words restore and reload are not interchangeable. One brings data back. The other destroys it.



    In the end, were the rolls reversed, I would probably feel very lucky walking away with everything so intact, based on the scenario presented... The SM Hardware Techs are very knowledgeable and have fine tools on hand, I am very glad to hear that they were able to recover/find your data (whatever the case may be). I do believe that they really do want to help customers and give them the best outcome possible, based on my time spent on the phone with a number of their techs; they were as nice and helpful as the day is long -- as professional as I have worked with at any data center.

    If SM was able to sort out that you just wanted the old drive untouched in bay 2, and a clean RHES/cPanel boot drive in bay 1, based on what we've read here ---major kudos to SM. They made it happen and that's fantastic.

    With 35-40 GB of data, good heavens, I'd also invest in a backup drive they're not "just" handy for compromises (BTW, only root compromises can result in possibly losing all back-up data, but regardless, even then, the full back-up isn't always lost). Data centers will be happy to give you a fresh drive when the old one fails, but their TOS's are pretty clear on "not responsible for lost data." Backups are essential. With a backup drive on board & being used, they could have just swapped out your boot drive and have been done with it.

    Anyway sc_dean, no hard feelings and I promise I am not trying to hassle you at all I just hope this helps explain how the interpretation of the data provided might have been what resulted in the confusion.

    Bailey
    You do not need to talk in laymen terms for someone to understand what you want, it is clearly evident to everyone what was request apart from you it seems...

    Besides it seems you speak and understand different english to us?
    ^^ IM WITH STUPID!! ^^

    "The only way to overcome fear, is to challenge it head on"
    "The quickest way to get over a woman, is to get under another"
      0 Not allowed!

  24. #24
    Join Date
    Nov 2002
    Location
    Kansas City, MO
    Posts
    46
    So where did you move your hosting to?
    Thank You,

    Chad McCan
      0 Not allowed!

  25. #25
    They have been temporarily moved to my private cage where i have my main business operations, I would like to take a second chance with SM for this server, but am waiting to see what kind of responce I get from them. I think it is fair to expect some compensation from them, even if it is a few months free.
      0 Not allowed!

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •