Results 1 to 25 of 35
-
12-11-2010, 09:00 AM #1Junior Guru Wannabe
- Join Date
- Jan 2005
- Location
- Kozani - Greece
- Posts
- 59
Lost everything from dedicated server
Hello everybody.
I have a dedicated server in A very reputable company.
I have server administration in a team that supports me and i have a backup server in another company.
So every nigh my server takes backup localy and then moves them to the remote backup server's company.
On Wednestday a hacker got into my system and done the following :
1) deleted all accounts public_html directories
2) deleted all mysql files
3) deleted all local backups
4) logged in remote backup company and deleted backups from there, too.
As you can understand my position is awful right now, as i need to apologise to my clients for this big loss. I have some static html files on disk but no joomla/wordpres/forum sites can be recovered.
Questions :
1) Do companies take backup of their clients dedicated servers ?
2) Do backup server companies need to take clients backup data, backups ? even if this is once a month ?
3) Who shall accuse for this situtation ?
Thankfully i have a local backup of my server's accounts from January 2010, and i can restore some older accounts but i've lost about 30 sites and lots of updates from portal sites.
What would you do in my position ? From where shall i start ?
-
12-11-2010, 09:33 AM #2Greece
- Join Date
- Jan 2004
- Location
- Greece
- Posts
- 2,211
1) If it's managed dedicated server and say that they do backups yes. If they say nothing about backups then don't expect them to make backups.
2) Most of the backup companies I know just create an account and allow you to upload backups. If they don't say that they keep backups of the backups then don't expect them to offer it.
I hope your customers have their own backups so you can restore their sites.
-
12-11-2010, 09:42 AM #3Web Hosting Guru
- Join Date
- Dec 2004
- Location
- Spain
- Posts
- 255
I think that you are really in a difficult position to claim to the hosting company. Because from your post, I do understand that your hosting company was really doing backups even offline. However it has been the intrusion that has deleted everything.
The only thing you can do is ask if the breach was your fault of your hosting company fault. If they where in charge of securing your server, then you can ask them for liabilities. If they where not and it was your responsibility, I'm afraid that you are at a dead end here.
Generally speaking, It is smart to use an offline backup that rsyncs from the other server. Not a push from your server to an external backup. A pull backup methodology does not require you to save the backup server password locally. Let's face it, the case of a HD failure is way more rare than a hacking security breach that deletes files on server, so it is good to be protected for that second case.
QComfortHost.NET. Top quality hosting. And a personal touch.
** web hosting ** reseller hosting ** VPS ** Managed Servers **
-
12-11-2010, 09:43 AM #4Web Hosting Guru
- Join Date
- Aug 2008
- Posts
- 275
Aah that's painful. But hopefully your customers also made backups of their websites. If not, you might consider compensating those by giving free hosting for x time.
I don't know the success rate of data recovery software / companies, but that might also be an option for you.██ SolidSRV Internet Solutions | OVH France HQ
██ Unmetered Bandwidth with every server
██ sales@solidsrv.com | +31 233-020-200
-
12-11-2010, 09:47 AM #5Web Hosting Guru
- Join Date
- Dec 2004
- Location
- Spain
- Posts
- 255
greatseeder, that's a great point. if the backup or your drive was on an ext3 partition you may be able to recover the files if nothing else has saved on top of those. Ask your host, maybe you hit luck...
ComfortHost.NET. Top quality hosting. And a personal touch.
** web hosting ** reseller hosting ** VPS ** Managed Servers **
-
12-11-2010, 09:47 AM #6Junior Guru Wannabe
- Join Date
- Jan 2005
- Location
- Kozani - Greece
- Posts
- 59
Thanks for your answer cretaforce.
Unfortunately for me the dedicated server doesn't hold backups.
Also the backup company didn't keep backups, and it is a reputable company.
Most of my clients don't even know cpanel usage so nobody holds backups.
I think that i am really at the thought of closing up my freelancing in web after that.
I did the following.
I asked my dedicated server company to create a new hard disk, and send me the disk for data recovery (because i work on a data recovery company). They refused to send me the disk because it's against their policy.
So i asked them to connect the old disk on usb so i can perform remote data recovery. I managed to find some files .gz and .tar that are recovered without proper naming. So i need to test all these files for corruption right now. Most of them are corrupted though and cannot be decompressed. It would be a life saver if i could find some of them working but this seems not feasible.
Any other ideas you may propose after this catastrophy ?
-
12-11-2010, 12:24 PM #7Junior Guru Wannabe
- Join Date
- Jan 2005
- Location
- Kozani - Greece
- Posts
- 59
So is there anyone else using a backup server or it's best practice to pull things to a Local NAS ? The problem is that our server was about 30GB and it's not easily transferable through an small adsl line.
-
12-11-2010, 01:20 PM #8Web Hosting Guru
- Join Date
- Feb 2005
- Location
- localhost
- Posts
- 5,473
That's nasty - was this a cPanel server? how did the hacker get access to you backup account?
Respectfully,
Mr. Terrence
-
12-11-2010, 01:26 PM #9Problem Solver
- Join Date
- Mar 2003
- Location
- California USA
- Posts
- 13,681
What worries me is how did they get into the system in the first place?
There has been several root escalation exploits in the past few months. If they were not patched someone could just walk into your server through a vulnerable php script such as a outdated joomla install.
Is your administration team proactive? Some are not and you must ask for updates to be applied.
Companies like bqinternet will backup your account to another account if you ask for it.Last edited by Steven; 12-11-2010 at 01:32 PM.
Steven Ciaburri | Industry's Best Server Management - Rack911.com
Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance
-
12-11-2010, 03:24 PM #10
I agree with that. It's also important that the remote backup user only have read access to your files. If the remote backup provider got hacked, and the remote server had write access on your servers, then they could also delete all the backups as well as all your files locally, and you would be in exactly the same position you are now. So no matter what you do, you need to make sure that neither server has permissions to delete files off the other server.
IOFLOOD.com -- We Love Servers
Phoenix, AZ Dedicated Servers in under an hour
★ Ryzen 9: 7950x3D ★ Dual E5-2680v4 Xeon ★
Contact Us: sales@ioflood.com ★
-
12-11-2010, 04:54 PM #11WHT Addict
- Join Date
- Nov 2010
- Location
- Casablanca
- Posts
- 153
This case shows that you have not chosen a good company for your backup.
Normally, backup servers are very protected against hackers with no HTTP or SSH access (just FTP).
If it was professional backup services, these backup servers should be backuped on servers not connected to the internet in the same DC.
If I was in your situation, I'll blame on the backup company and change them as soon as possible.
-
12-11-2010, 04:57 PM #12Greece
- Join Date
- Jan 2004
- Location
- Greece
- Posts
- 2,211
The problem is that possible for transferring the backups to the remote server ssh keys used which allow the attacker to connect from the webserver to the backup server.
-
12-12-2010, 04:45 AM #13Junior Guru Wannabe
- Join Date
- Jan 2005
- Location
- Kozani - Greece
- Posts
- 59
Cretaforce you are probably right, because the backup company told me that i deleted the files. But i believe that it is mandatory for the backup company to make backups of the client. I didn't read their terms in the past, but i thought i pay them to keep my backups safe... I really don't know who to blame now for this catastrophy.
-
12-12-2010, 05:13 AM #14Junior Guru Wannabe
- Join Date
- Dec 2009
- Posts
- 37
Just a bad luck. The customers if their information is important to them should have backups too. Just a lesson to learn.
-
12-12-2010, 09:34 AM #15Greece
- Join Date
- Jan 2004
- Location
- Greece
- Posts
- 2,211
mmingos as I told you if the backup provider didn't say that they keep backups of your data then don't expect them to offer it. So I wouldn't blame the backup provider.
-
12-12-2010, 01:46 PM #16Problem Solver
- Join Date
- Mar 2003
- Location
- California USA
- Posts
- 13,681
I am suspecting a missed software update on the server....
Steven Ciaburri | Industry's Best Server Management - Rack911.com
Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance
-
12-14-2010, 07:42 PM #17Web Hosting Evangelist
- Join Date
- Sep 2002
- Location
- Canada
- Posts
- 456
I feel for you man. This must be really tough. I can suggest few things:
a) Evaluate how much the data is worth, and whether your clients will go after you for the lost data.
b) If data is worth recovering then:
1. You want to recover, the best way is to get data recovery done on the server. It will be costly to do a professional block level recovery. IF this is a delete only, most data recovery providers can get you the data back.
Try to convince your hosting provider to allow a data recovery agent to come onsite to pickup the disk. Offer to pay them the cost of the disk, double or triple what it cost in the market.
2. Recovery agent will probably recommend making copy of the disk later for investigation. Find out how you were hacked and make a plan to prevent that in the future.
You will not be able to blame this on anyone. Your hosting company is not responsible for your data. Your management team is also not responsible for your data, and even if it was security issue, you can't blame it on them unless they have a guarantee you are not to be hacked!!! They are as much responsible as the OS developers, and you can't blame them either.
Your backup provider is only responsible for the space where your backup resides, and most likely limited to their TOS and SLA.
In other words, no one will take responsibility for your data (or your customers). Liability insurance does not cover data loss, and even E&O insurance would be unpredictable in cases like these.
So focus on recovering the data on your disks first and foremost.Reliability • Performance • Integrity
-
12-14-2010, 09:23 PM #18
So no one is safe.
You should make offsite backups with at least 2-3 providers and use the "pull" method. You must choose reliable offsite backup providers who you can trust though... I would like to know which offsite backup provider(s) you were using? It could happen to other users using the same company...
Edit: PS I'm not saying it was the backup providers fault. But whether it was their fault or not, it would good for you to let other users know which provider(s) you were using for offsite backups. The provider could give us more info too. Same with the Dedicated Server company you're using.Last edited by HostXNow; 12-14-2010 at 09:27 PM.
HostXNow - Shared Web Hosting | Semi Dedicated Hosting | Enterprise Reseller Hosting | VPS Hosting
-
12-14-2010, 09:27 PM #19Web Hosting Evangelist
- Join Date
- May 2010
- Location
- 10.0.0.17
- Posts
- 480
Shouldn't the important thing be dedicating the time and effort into trying to recover your client's data, rather than trying to find someone to lay blame on? All too often people are quick to blame their host for a situation that they may not have control over.
-
12-14-2010, 09:31 PM #20HostXNow - Shared Web Hosting | Semi Dedicated Hosting | Enterprise Reseller Hosting | VPS Hosting
-
12-14-2010, 10:20 PM #21Web Hosting Evangelist
- Join Date
- May 2010
- Location
- 10.0.0.17
- Posts
- 480
-
12-15-2010, 12:44 AM #22Web Hosting Master
- Join Date
- Jul 2005
- Posts
- 1,062
Oh that is just a sad thing to happen. Well, there is nothing much you can do at this point.
You must tell your customers to make backups of their own at all times.More than decade with webhosting!
-
12-15-2010, 01:24 AM #23Newbie
- Join Date
- Oct 2010
- Posts
- 16
wow. nightmare. here is what i suggest.
Technically you got most of the stuff covered with the generous community in the preceding posts. Therefore
a) proceed to recover data as much as possible.
b) figure out future prevention and your service terms.
c) estimate your costs for the above
and
d) Compensate your customers with offerings which help easing the pain for their loss, or even better if it makes your customers look at your service as an exceptional one as you stood by your trusting clients.
e) Add a better security and service tag as a veteran in cyber security
as you have seen the dragon in the eyes and survived.(the best)
-
12-15-2010, 01:47 AM #24Retired Moderator
- Join Date
- May 2004
- Location
- Toronto, Canada
- Posts
- 5,105
I really do feel sorry for you but I think you are spending too much time looking for someone ELSE that did something wrong here. Don't be looking for someone to blame, we know who to blame, the hacker! The ONLY thing that could have prevented this is having your system hardened from intrusion. Did you use the same password for the system and the backup as well?
The backup company is not under any obligation to backup your backups. They should be providing reliable storage but if through a fault on your part the backups are wiped out, you have really shouldn't be even thinking about what someone else did wrong.
Stay on the track of data recovery. Ask to BUY the hard drive and explain the situation or ask them to send to a recovery center of their choosing if they won't sell it to you.CloudNexus Technology Services
Managed Services
-
12-15-2010, 04:33 AM #25Junior Guru Wannabe
- Join Date
- Jan 2005
- Location
- Kozani - Greece
- Posts
- 59
Backup Servers are no good when hacked !
Thanks for your support Guys.
1) Thank God i had a local backup (a bit old though) of my server so i saved most clients.
2) I need to remake only 2 websites out of 120 i host, so i hired some proffessionals to help me in this difficult situation.
3) All clients that had problem, were given free 2 year hosting4
4) Lost 2-3 clients out of that problem and after 10 years it's the first time a client moves to another company.
5) Most of my clients are satisfied of our effords because we tried hard to be up again within 2-3 days, and this happened during weekend. So we had little mail loss and the sites were up again in 3 days. Event occured on Thursday and we were up on Monday evening.
Let this be a lesson to everybody :
1) Dedicated Server companies will not be responsible for data loss from hacking
2) Server Management teams will also not be responsible for data loss from hacking
3) Backup Server companies will not be responsible for data loss through rm -rf command. My company told me that if the ticket was submited before 24 hours from the event, they may have done something. So i found out that i issued a ticket and it took them about 5 hours to get answer from the technical stuff on the backup server, as they were not doing backup servers themselves. So actually nobody could do anything about my files.
Clue : You are alone in there. Backup Servers are only good for other types of failures. They cannot protect you because as far as i asked, no backup server company backs up data.
By the way, the hacker deleted any records of the logs.
Similar Threads
-
Lost SSH on dedicated server
By Formas in forum Hosting Security and TechnologyReplies: 6Last Post: 05-01-2008, 07:14 AM -
Lost and Confused in finding a reliable Dedicated Fully Managed Host
By zapatab in forum Managed Hosting and ServicesReplies: 37Last Post: 11-15-2007, 01:52 AM -
Which dedicated server provider has the best free or lost cost managed solution?
By Shoey in forum Dedicated ServerReplies: 1Last Post: 03-19-2004, 08:47 AM -
CPanel Dedicated Server market a lost cause?
By bteeter in forum Dedicated ServerReplies: 7Last Post: 07-01-2003, 03:21 PM -
@Webhost has finally lost its mind:: $69 dedicated servers, 300gb bw and free CP!
By Asher S in forum Dedicated Hosting OffersReplies: 22Last Post: 11-15-2002, 10:11 PM