Results 1 to 25 of 38
Thread: ServerMatrix Support Data Loss
-
05-10-2004, 10:47 AM #1Disabled
- Join Date
- May 2002
- Posts
- 17
ServerMatrix Support Data Loss
Hi All,
I had a server at SM for a few personal sites as they offer some really great special offers, a few days ago my server had a main HD failure and I asked for a reload onto a new HD and to install my existing main drive as a slave drive so I could attempt to retrieve the data (the drive was not totally dead). However firstly it took nearly 12 hours to do a restore (this I can live with as its only personal), what I can't live with is the fact that they accidently wiped the bad drive to a clean cpanel install as well. When I raised this point to them they did not respond for just over 24 hours then just closed the ticket !
This is the worst hosting company for support I have ever had the misfortune to use, I do not recommend that anyone uses them who wishes to ensure data integrity or prompt support.
If anyone from SM reads this and wishes to contact me to explain this it is in regards to ticket 247794PLNT_which you closed 7:13 am your time today with no comment on the fact you wiped all my data by mistake.
Ticket Number: 247794PLNT_
Ticket Type: Accounting_
Status: CLOSED_
Last Updated: 05/10/2004 09:17:10_
------------------------------------------
(mdeater-05/08/04-20:34):
Thank you for choosing Server Matrix, your server is up and running, and your old harddrive has been placed in your box, It it hdc.
--------------------------------------
(c13639fcol-05/09/04-07:13):
What have you done with the data ? It appears the second drive has been formated or is a different drive. Please restore my data. See below for analysis of the two drives
Filesystem Size Used Avail Use% Mounted on
/dev/hdc3 72G 2.3G 66G 4% /
/dev/hda3 72G 2.3G 66G 4% /old
hdc is now an exact mirror of hda, this drive had about 35-40gb of data on it._0
-
05-10-2004, 10:53 AM #2Web Hosting Guru
- Join Date
- Oct 2002
- Posts
- 288
The reason the ticket was not updated is when they finished the install it was sent to billing. The Accounting dept is closed on weekends.
A rep is contacting you within the next few minutes.Jeff Reinis
(469) 607-3005 - jreinis@cogentco.com
www.cogentco.com
Datacenter Colocation - Tier-1 IP Transit - Bandwidth Solutions0
-
05-10-2004, 10:56 AM #3WHT Addict
- Join Date
- Jun 2003
- Location
- Ottawa, Canada
- Posts
- 124
Not to take anything away from the mess up SM did, but you should have had your own backups. Never trust anyone with your only copy of your data. Things like this happens sometimes and will probably always happens every now and then. Only way to prevent it is to have your own backups. This isn't a SM only problem, i recall seeing problems like this at ev1 as well and probably every provider out there.
If they can't find your data then it sucks to have to redo everything but at least you learned to cover your but from now on.
Sorry for being a bit brash, yes SM messed up, but so did you.0
-
05-10-2004, 11:05 AM #4Disabled
- Join Date
- May 2002
- Posts
- 17
I just talked with someone on the phone who will investigate. However, my point is not that the HD failed, as I mentioned, it was a personal site that I can reupload, what I do object to and think is poor is.
1/. A tech was asked to install cpanel on a new drive and put the second drive as slave, un mounted. They proceeded to put a cpanel install on both drives fresh.
2/. When I asked SM what happened there was no responce, sure the ticket may have gone to billing, but why did billing not push to technical to investigate when they saw my responce rather than closing.
All my production systems are fully backed up both to secondary drives and an off machine backup server
Alan0
-
05-10-2004, 12:05 PM #5Web Hosting Master
- Join Date
- Feb 2004
- Posts
- 2,197
SM/ThePlanet does actually have quite a few screwups (more lately than before), I have had serveral recently, and I have been using their Total Control Servers.
crucialparadigm - Affordable, Reliable, Professional :
Web Hosting
• 24/7 Support • Web Hosting • Reseller Hosting • Cloud/VPS Plans • Dedicated Servers •0
-
05-10-2004, 12:23 PM #6Retired Moderator
- Join Date
- Nov 2002
- Location
- WebHostingTalk
- Posts
- 8,901
Originally posted by crucialparad
SM/ThePlanet does actually have quite a few screwups (more lately than before), I have had serveral recently, and I have been using their Total Control Servers.
Can I help you find the common denominator?
SiriusI support the Human Rights Campaign!
Moving to the Tampa, Florida area? Check out life in the suburbs in Trinity, Florida.0
-
05-10-2004, 12:29 PM #7Web Hosting Master
- Join Date
- Nov 2003
- Posts
- 1,093
Bare in mind, they're adding at least 1000 new servers a month and therefore as they add more servers the likelihood of screwups only gets larger...
0
-
05-10-2004, 12:37 PM #8Web Hosting Master
- Join Date
- Feb 2004
- Posts
- 2,197
sirius:
Do you need explanation of the problems?
Manged.com
--------------
- unreliable network
- can cancel anyones server!!! (now I don't think there is much explanation for that)
- don't listen to support tickets
ThePlanet
-----------
- has been quite good until recently and I still like their service, but they DID have a few screwups
- Firstly I ordered a Total Control Server over $400/month, the server kept having problems and was freezing and having intermittent network problems. It turns out they gave me the wrong IP addresses, so the name servers that I had placed on those IPs had to be removed, and caused me 2 days worth of down time for some sites. I got a refund for this so I am quite happy about it.
- I asked them to enable HT on the server and update with the appropriate kernel. They did this, later I had problems and the server so I tried rebooting, it had problems starting up. Turns out they messed up the Grub config file. This was no major problem (execept for 1 hour downtime!). This was yet another screwup.
Like I stated in my message SM/TC does have screwups, I didn't say I am not satisfied with them. They provided me decent refunds in all instances and I am quite happy with them actually.
And managed.com on the other hand should not be used for mission critical applications for obvious reasons.crucialparadigm - Affordable, Reliable, Professional :
Web Hosting
• 24/7 Support • Web Hosting • Reseller Hosting • Cloud/VPS Plans • Dedicated Servers •0
-
05-10-2004, 12:43 PM #9Junior Guru
- Join Date
- Feb 2004
- Posts
- 208
I encountered a failed drive also with managed.com. The support staff actually ask me if I want to attach the failed drive as slave so I can salvage my data. I don't intend to try because I afraid the corrupted data might do more harm. So in less than 2 hours I have a new drive with the default O/S in the same server.
I learned that I better backup my data. I don't feel good that I have the server for just over 1 month and the drive failed, but glad that the support is not bad after all.0
-
05-10-2004, 01:57 PM #10Web Hosting Master
- Join Date
- Sep 2001
- Location
- Seattle, WA
- Posts
- 3,085
Originally posted by sirius
Can I help you find the common denominator?
Sirius
That seems to be the case with a lot of chronic complainers.Jim Reardon - jim/amusive.com0
-
05-10-2004, 02:10 PM #11Too smart for her own good.
- Join Date
- Feb 2004
- Location
- Your Screen
- Posts
- 3,999
sc_dean, can you post a copy of the complete transaction / entire ticket?
When I went through the same thing w/ SM (actually I had a rooted box) I:
(a) called them on the phone and speak to the tech doing the work, and specifically requested the OS reload be done on a new drive;
(b) open a separate ticket authorizing the set-up + monthly charges for the old drive to be remounted on the chassis (SM required this)
Obviously we only have about 5% of the story here, we have no idea what the actual interaction was. And I do not wish to suggest that SM did not make an error -- screw-ups happen. However. Their standard OS reload practice is to wipe the drive and reload the OS on that same drive. I would not be at all surprised if the tech assigned the reload simply progressed "as usual," they do dozens of reloads every day.
What really puzzles me is why they would wipe and reload on a "bad" drive. A bad drive should be completely swapped out. Did you request it as a reload? Or was your ticket put through as a hardware failure?
Just, odd.
BaileyLet's Connect on Twitter! @thatsmsgeek2u || Fighting mediocrity one thread at a time.0
-
05-10-2004, 02:11 PM #12Web Hosting Master
- Join Date
- Apr 2003
- Location
- Washington, DC
- Posts
- 595
Originally posted by amusive.com
That seems to be the case with a lot of chronic complainers.0
-
05-10-2004, 02:23 PM #13Disabled
- Join Date
- May 2002
- Posts
- 17
There is only one other post in the restore request (you are correct there was a ticket before that workign out it was an HD error) That was
Details:
This is a request to reload the Operating System on my server. The request details are as follows:
Hardware Name : <removed> (Server)
Hardware Other :
Location info : Rack : AE48
Row : AE
Unknown : Unknown
Operating System: Red Hat Enterprise Linux, Version 3 (recommended)
Comments : Hi,
Please reload RHE w/cpanel onto a NEW 80gb HD and put the second HD in as a slave (you will not be able to mount). I beleive there is no charge for either the new drive or the reload as this restore is due to HW failure. I would appreciate this being worked on ASAP as the server has now been down for over 6 hours. I will open a ticket to have the damaged drive removed within 3 days once I have tried to remove data from it. Ref number of existing ticket is 247601
Alan0
-
05-10-2004, 02:32 PM #14Too smart for her own good.
- Join Date
- Feb 2004
- Location
- Your Screen
- Posts
- 3,999
This is a request to reload the Operating System on my server.
Replacement of faulty hardware, and an OS reload, are two totally, completely, different things.
I realize that you explained it farther below, however, requesting the wrong service at the outset very likely caused the issue. (Tech tunnel vision... I'm not saying it was right, because obviously the tech didn't read... and he should have)
You requested an OS reload, and that's exactly what he did.
BaileyLet's Connect on Twitter! @thatsmsgeek2u || Fighting mediocrity one thread at a time.0
-
05-10-2004, 02:37 PM #15Web Hosting Master
- Join Date
- Apr 2004
- Posts
- 1,834
Originally posted by sirius
Can I help you find the common denominator?
Here, let's do the math. I had a cronic hardware problem at server matrix with 3 seperate and unidentifable failures. This outage cost me a downtime of 10 days total per month. One of these outages took 6 days to resolve. During this 6 days my server was down they added 6/7 X 1000/4 servers which is 214 new customers before my server was back online.
Now, if you ran the company would you prefer to have 1 happy existing customer or 214 new customers?0
-
05-10-2004, 03:26 PM #16Disabled
- Join Date
- May 2002
- Posts
- 17
Actually I rang them in regards to the first ticket and they said to open a restore ticket and ask for the second drive to be put in as slave, what I did was following the advice of the tech on the phone.
0
-
05-10-2004, 05:53 PM #17Too smart for her own good.
- Join Date
- Feb 2004
- Location
- Your Screen
- Posts
- 3,999
...which by then was probably too late.
Sorry to hear of the trouble, but, lesson learned I guess? Again I am not saying SM is in the right here, obviously the tech did not read the details... but I certainly can understand how an OS Reload came about off a ticket entitled an OS Reload request.
BTW what of the data on your backup drive? Was your data not saved there as it should have been, or...?
BaileyLet's Connect on Twitter! @thatsmsgeek2u || Fighting mediocrity one thread at a time.0
-
05-10-2004, 07:35 PM #18Taking a break from hosting
- Join Date
- Mar 2003
- Location
- Charlotte, NC
- Posts
- 2,761
I've heard of this happening at EV1, as well. I am not a proponent of using extra harddrives to do backups on, as the data isn't safe if a box is compromised, therefore I always have sent backups offsite. If you're concerned about bandwidth, use RSYNC or R-Diff to save yourself some transfer. Even here in the office, we do backups between computers, and onto CD-RWs as well.
-Josh0
-
05-10-2004, 08:27 PM #19Web Hosting Master
- Join Date
- Aug 2002
- Location
- Trouble will find me!
- Posts
- 1,470
Originally posted by sc_dean
Actually I rang them in regards to the first ticket and they said to open a restore ticket and ask for the second drive to be put in as slave, what I did was following the advice of the tech on the phone.Originally posted by bithost(NET)
...which by then was probably too late.
I would call it incompetant techs and bad management...
I hope you get it worked out ...0
-
05-11-2004, 05:15 AM #20Disabled
- Join Date
- May 2002
- Posts
- 17
Bailey,
I do not understand, in the trouble shooting ticket where we worked out the problem they asked me to open a second restore ticket to have it restored. This was purely a mistake by SM. I am not complaining that the HD failed OR that I didn't have a backup, if they had followed the instructions and put the drive in and I was unable to retrieve the data I would not have said a word. However since my post here they have been very helpful and here is the resolution
Resolution : -
1/. SM had their 'Hardware Manager' look at my system, the system was offline for c. 4/5 hours, when it came back on they had 'found' my data and hdc was mounted as /data with all my data on.
------------------------------------------
(lcarter-05/10/04-15:46):
Your server is back online. The original drive is mounted as /data. Please take a look and let us knwo when the damages drive needs to be removed.
2/. SM refunded me for the restore even though they were not meant to have charged me for it as it was under the Hardware SLA, I had already confirmed this with the tech on phone and reiterated it in the restore ticket. I was not even aware I had been charged.
------------------------------------------
(bhooker-05/10/04-16:10):
I have refunded $75.00 to your credit card.
I am hoping that ServerMatrix will offer something under the SLA or as a goodwill gesture for the hassle, here is the problems SM needs to sort and things they don't from my experiance
PROBLEM 1 (the worst): Techs and Billing people do not read the tickets in detail, they skim read and miss important instructions
PROBLEM 2 : There should be a tag or similar in the restore ticket to specify why a restore was needed as restores due to Hardware failures should not be charged under their Hardware SLA
GOOD 1 : They where able to find my HD, my guess is they actually put two new drives in.
GOOD 2 : Once they did realise the problem they where helpful and called me from USA to Spain three times (though perhaps when calling international they should check timezones, the last call was at 1am for me )
Hopefully they will offer me something to keep my personal/test server there so next time I can post how they did right0
-
05-11-2004, 11:47 AM #21Too smart for her own good.
- Join Date
- Feb 2004
- Location
- Your Screen
- Posts
- 3,999
sc_dean, the problem is, based on the ticket data you have provided here, you never asked for a restore. You asked for an OS reload. These are completley and entirely different things -- unrelated, in fact.
s.n.a.zz.y, just FYI, with ServerMatrix reload tickets are opened in a special section of Orbit and are assigned to a separate team of techs. The reason I was guessing it was likely 'too late' is because the first thing a Reload Tech does, is pop the drive in the imager and wipe it. Get it started ASAP, that makes the work finished quicker, as reload customers are always screaming to have their boxes up ASAP. It's all 'hurry, hurry' when it comes to reloads (because they're offline), and the reload crew does try to keep that in mind. (Or so they have assured me! LOL)
sc_dean, I would personally guess the confusion stems from the wording in the ticket provided above. I figure you 'n I are not connecting in terms of nuances, so would it help to provide my interpretation of your 'further details' as provided in this same thread -- how I would read it as a tech (understanding of course that we are still missing huge chunks of ticket history)? I don't want to do this to point fingers at all!!!! I just am trying to show how there are different ways to read the instructions in the ticket, hopefully to help avoid situations like this in the future.
Details: This is a request to reload the Operating System on my server.
Please reload RHE w/cpanel onto a NEW 80gb HD and put the second HD in as a slave (you will not be able to mount). I beleive there is no charge for either the new drive or the reload as this restore is due to HW failure.- One, why is anything being reloaded on a new drive? If you need a reload, the only thing that can be reloaded is the original (bad) drive.
- "...put the second HD in as a slave..." OK, this is a question of which drive is which. Normally users refer to the first drive as the one being already installed in the server, so the second hard drive is the new one.
- "I beleive there is no charge for either the new drive or the reload..." This sentence appears to be refering to two drives: one new clean drive, and an OS-reloaded drive that is failing...
- "...as this restore is due to HW failure." We don't know anything about a restore, this is the first mention of it, so that must be something the customer's working on.
The ticket is very specific to requesting a reload. However the clean drive would not require any sort of OS reload, it should have a clean OS already on it as a boot drive, and it should go in bay #1. The original drive should not be touched -- no reload, no restore -- and just go in bay #2. So why does the customer make any mention of an OS reload at all?!?! This is why I am confused and why I would not be at all surprised if the failing drive had wound up wiped and reloaded. Sorry I wasn't more detailed in explaining, before.
sc_dean, I am sorry, I don't wish to point fingers, I realize that you've had a very long couple of days. A drive failure -- or compromise too -- makes for some nasty long hours of server work, and I can definitely empathize. However the words restore and reload are not interchangeable. One brings data back. The other destroys it.
In the end, were the rolls reversed, I would probably feel very lucky walking away with everything so intact, based on the scenario presented... The SM Hardware Techs are very knowledgeable and have fine tools on hand, I am very glad to hear that they were able to recover/find your data (whatever the case may be). I do believe that they really do want to help customers and give them the best outcome possible, based on my time spent on the phone with a number of their techs; they were as nice and helpful as the day is long -- as professional as I have worked with at any data center.
If SM was able to sort out that you just wanted the old drive untouched in bay 2, and a clean RHES/cPanel boot drive in bay 1, based on what we've read here ---major kudos to SM. They made it happen and that's fantastic.
With 35-40 GB of data, good heavens, I'd also invest in a backup drive they're not "just" handy for compromises (BTW, only root compromises can result in possibly losing all back-up data, but regardless, even then, the full back-up isn't always lost). Data centers will be happy to give you a fresh drive when the old one fails, but their TOS's are pretty clear on "not responsible for lost data." Backups are essential. With a backup drive on board & being used, they could have just swapped out your boot drive and have been done with it.
Anyway sc_dean, no hard feelings and I promise I am not trying to hassle you at all I just hope this helps explain how the interpretation of the data provided might have been what resulted in the confusion.
BaileyLet's Connect on Twitter! @thatsmsgeek2u || Fighting mediocrity one thread at a time.0
-
05-11-2004, 12:14 PM #22Disabled
- Join Date
- May 2002
- Posts
- 17
I am always forgetting, although American is based on English, we do not actually speak the same language !
Let me clarify the points you raised
"Please reload RHE w/cpanel onto a NEW 80gb HD"
This specifically askes for an OS to be loaded onto a new drive, it is being 're'loaded to imply that the same operating system and panel are to be used as the previous install. To install or to load are the same thing.
"...put the second HD in as a slave..."
I have just refered to a new drive being loaded with software, there are now two drives in the machine, 'the second drive' quite obviously refers to the 'other' drive than the one I just refered to. To put it as slave means to set the jumper to salve and attach it to the IDE chain. This is more efficient that talking about an old and new drive, how do you know which is the old and new once they are pulled ?
"I beleive there is no charge for either the new drive or the reload..." & "...as this restore is due to HW failure."
This refers to the fact there should be no monthly charge for having two drives in the system and that there should not be the usualy $75 OS install fee as the OS install is due to a HW failure.
Quote " We don't know anything about a restore, this is the first mention of it, so that must be something the customer's working on."
It clearly references the old ticket to explain EXACTLY what is going on if the tech had any questions. In the SM panel it is called an OS restore, so it is fairly obvious what that refers to at any rate.
My words words where correctly used based on the context I used them in. I could have been more elequent with my wording, but at goen midnight adter 6 hours troubleshooting I should not have to explain my problem more than I did, especially after a 5 min telephone conversation with a technican
I do not understand why you are trying to find some error with what I did. I requested something via phone in detail, from that phone call I was asked to raise a ticket which I did. Neither the phone or ticket instructions where followed, and if the tech was being as padantic as you where in the reply the ticket would have made 0 sence and he should have queried what I wanted (even though I have just explained it in detail on the phone).0
-
05-11-2004, 12:38 PM #23Web Hosting Master
- Join Date
- Aug 2002
- Location
- Trouble will find me!
- Posts
- 1,470
Originally posted by bithost(NET)
sc_dean, the problem is, based on the ticket data you have provided here, you never asked for a restore. You asked for an OS reload. These are completley and entirely different things -- unrelated, in fact.
s.n.a.zz.y, just FYI, with ServerMatrix reload tickets are opened in a special section of Orbit and are assigned to a separate team of techs. The reason I was guessing it was likely 'too late' is because the first thing a Reload Tech does, is pop the drive in the imager and wipe it. Get it started ASAP, that makes the work finished quicker, as reload customers are always screaming to have their boxes up ASAP. It's all 'hurry, hurry' when it comes to reloads (because they're offline), and the reload crew does try to keep that in mind. (Or so they have assured me! LOL)
sc_dean, I would personally guess the confusion stems from the wording in the ticket provided above. I figure you 'n I are not connecting in terms of nuances, so would it help to provide my interpretation of your 'further details' as provided in this same thread -- how I would read it as a tech (understanding of course that we are still missing huge chunks of ticket history)? I don't want to do this to point fingers at all!!!! I just am trying to show how there are different ways to read the instructions in the ticket, hopefully to help avoid situations like this in the future.
- One, why is anything being reloaded on a new drive? If you need a reload, the only thing that can be reloaded is the original (bad) drive.
- "...put the second HD in as a slave..." OK, this is a question of which drive is which. Normally users refer to the first drive as the one being already installed in the server, so the second hard drive is the new one.
- "I beleive there is no charge for either the new drive or the reload..." This sentence appears to be refering to two drives: one new clean drive, and an OS-reloaded drive that is failing...
- "...as this restore is due to HW failure." We don't know anything about a restore, this is the first mention of it, so that must be something the customer's working on.
The ticket is very specific to requesting a reload. However the clean drive would not require any sort of OS reload, it should have a clean OS already on it as a boot drive, and it should go in bay #1. The original drive should not be touched -- no reload, no restore -- and just go in bay #2. So why does the customer make any mention of an OS reload at all?!?! This is why I am confused and why I would not be at all surprised if the failing drive had wound up wiped and reloaded. Sorry I wasn't more detailed in explaining, before.
sc_dean, I am sorry, I don't wish to point fingers, I realize that you've had a very long couple of days. A drive failure -- or compromise too -- makes for some nasty long hours of server work, and I can definitely empathize. However the words restore and reload are not interchangeable. One brings data back. The other destroys it.
In the end, were the rolls reversed, I would probably feel very lucky walking away with everything so intact, based on the scenario presented... The SM Hardware Techs are very knowledgeable and have fine tools on hand, I am very glad to hear that they were able to recover/find your data (whatever the case may be). I do believe that they really do want to help customers and give them the best outcome possible, based on my time spent on the phone with a number of their techs; they were as nice and helpful as the day is long -- as professional as I have worked with at any data center.
If SM was able to sort out that you just wanted the old drive untouched in bay 2, and a clean RHES/cPanel boot drive in bay 1, based on what we've read here ---major kudos to SM. They made it happen and that's fantastic.
With 35-40 GB of data, good heavens, I'd also invest in a backup drive they're not "just" handy for compromises (BTW, only root compromises can result in possibly losing all back-up data, but regardless, even then, the full back-up isn't always lost). Data centers will be happy to give you a fresh drive when the old one fails, but their TOS's are pretty clear on "not responsible for lost data." Backups are essential. With a backup drive on board & being used, they could have just swapped out your boot drive and have been done with it.
Anyway sc_dean, no hard feelings and I promise I am not trying to hassle you at all I just hope this helps explain how the interpretation of the data provided might have been what resulted in the confusion.
Bailey
Besides it seems you speak and understand different english to us?^^ IM WITH STUPID!! ^^
"The only way to overcome fear, is to challenge it head on"
"The quickest way to get over a woman, is to get under another"0
-
05-11-2004, 01:00 PM #24Junior Guru Wannabe
- Join Date
- Nov 2002
- Location
- Kansas City, MO
- Posts
- 46
So where did you move your hosting to?
Thank You,
Chad McCan0
-
05-11-2004, 01:11 PM #25Disabled
- Join Date
- May 2002
- Posts
- 17
They have been temporarily moved to my private cage where i have my main business operations, I would like to take a second chance with SM for this server, but am waiting to see what kind of responce I get from them. I think it is fair to expect some compensation from them, even if it is a few months free.
0