Page 1 of 2 12 LastLast
Results 1 to 25 of 35
  1. #1
    Join Date
    Jan 2005
    Location
    Kozani - Greece
    Posts
    59

    Lost everything from dedicated server

    Hello everybody.
    I have a dedicated server in A very reputable company.
    I have server administration in a team that supports me and i have a backup server in another company.

    So every nigh my server takes backup localy and then moves them to the remote backup server's company.

    On Wednestday a hacker got into my system and done the following :
    1) deleted all accounts public_html directories
    2) deleted all mysql files
    3) deleted all local backups
    4) logged in remote backup company and deleted backups from there, too.

    As you can understand my position is awful right now, as i need to apologise to my clients for this big loss. I have some static html files on disk but no joomla/wordpres/forum sites can be recovered.

    Questions :
    1) Do companies take backup of their clients dedicated servers ?
    2) Do backup server companies need to take clients backup data, backups ? even if this is once a month ?
    3) Who shall accuse for this situtation ?

    Thankfully i have a local backup of my server's accounts from January 2010, and i can restore some older accounts but i've lost about 30 sites and lots of updates from portal sites.

    What would you do in my position ? From where shall i start ?

  2. #2
    Join Date
    Jan 2004
    Location
    Greece
    Posts
    2,211
    1) If it's managed dedicated server and say that they do backups yes. If they say nothing about backups then don't expect them to make backups.

    2) Most of the backup companies I know just create an account and allow you to upload backups. If they don't say that they keep backups of the backups then don't expect them to offer it.

    I hope your customers have their own backups so you can restore their sites.

  3. #3
    Join Date
    Dec 2004
    Location
    Spain
    Posts
    255
    I think that you are really in a difficult position to claim to the hosting company. Because from your post, I do understand that your hosting company was really doing backups even offline. However it has been the intrusion that has deleted everything.

    The only thing you can do is ask if the breach was your fault of your hosting company fault. If they where in charge of securing your server, then you can ask them for liabilities. If they where not and it was your responsibility, I'm afraid that you are at a dead end here.

    Generally speaking, It is smart to use an offline backup that rsyncs from the other server. Not a push from your server to an external backup. A pull backup methodology does not require you to save the backup server password locally. Let's face it, the case of a HD failure is way more rare than a hacking security breach that deletes files on server, so it is good to be protected for that second case.

    Q
    ComfortHost.NET. Top quality hosting. And a personal touch.
    ** web hosting ** reseller hosting ** VPS ** Managed Servers **

  4. #4
    Aah that's painful. But hopefully your customers also made backups of their websites. If not, you might consider compensating those by giving free hosting for x time.
    I don't know the success rate of data recovery software / companies, but that might also be an option for you.
    SolidSRV Internet Solutions | OVH France HQ
    Unmetered Bandwidth with every server
    sales@solidsrv.com | +31 233-020-200

  5. #5
    Join Date
    Dec 2004
    Location
    Spain
    Posts
    255
    greatseeder, that's a great point. if the backup or your drive was on an ext3 partition you may be able to recover the files if nothing else has saved on top of those. Ask your host, maybe you hit luck...
    ComfortHost.NET. Top quality hosting. And a personal touch.
    ** web hosting ** reseller hosting ** VPS ** Managed Servers **

  6. #6
    Join Date
    Jan 2005
    Location
    Kozani - Greece
    Posts
    59
    Thanks for your answer cretaforce.
    Unfortunately for me the dedicated server doesn't hold backups.
    Also the backup company didn't keep backups, and it is a reputable company.
    Most of my clients don't even know cpanel usage so nobody holds backups.
    I think that i am really at the thought of closing up my freelancing in web after that.

    I did the following.
    I asked my dedicated server company to create a new hard disk, and send me the disk for data recovery (because i work on a data recovery company). They refused to send me the disk because it's against their policy.
    So i asked them to connect the old disk on usb so i can perform remote data recovery. I managed to find some files .gz and .tar that are recovered without proper naming. So i need to test all these files for corruption right now. Most of them are corrupted though and cannot be decompressed. It would be a life saver if i could find some of them working but this seems not feasible.

    Any other ideas you may propose after this catastrophy ?

  7. #7
    Join Date
    Jan 2005
    Location
    Kozani - Greece
    Posts
    59
    So is there anyone else using a backup server or it's best practice to pull things to a Local NAS ? The problem is that our server was about 30GB and it's not easily transferable through an small adsl line.

  8. #8
    Join Date
    Feb 2005
    Location
    localhost
    Posts
    5,473
    That's nasty - was this a cPanel server? how did the hacker get access to you backup account?
    Respectfully,
    Mr. Terrence

  9. #9
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681
    What worries me is how did they get into the system in the first place?

    There has been several root escalation exploits in the past few months. If they were not patched someone could just walk into your server through a vulnerable php script such as a outdated joomla install.

    Is your administration team proactive? Some are not and you must ask for updates to be applied.

    Companies like bqinternet will backup your account to another account if you ask for it.
    Last edited by Steven; 12-11-2010 at 01:32 PM.
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  10. #10
    Quote Originally Posted by Quel View Post
    I think that you are really in a difficult position to claim to the hosting company. Because from your post, I do understand that your hosting company was really doing backups even offline. However it has been the intrusion that has deleted everything.

    The only thing you can do is ask if the breach was your fault of your hosting company fault. If they where in charge of securing your server, then you can ask them for liabilities. If they where not and it was your responsibility, I'm afraid that you are at a dead end here.

    Generally speaking, It is smart to use an offline backup that rsyncs from the other server. Not a push from your server to an external backup. A pull backup methodology does not require you to save the backup server password locally. Let's face it, the case of a HD failure is way more rare than a hacking security breach that deletes files on server, so it is good to be protected for that second case.

    Q
    I agree with that. It's also important that the remote backup user only have read access to your files. If the remote backup provider got hacked, and the remote server had write access on your servers, then they could also delete all the backups as well as all your files locally, and you would be in exactly the same position you are now. So no matter what you do, you need to make sure that neither server has permissions to delete files off the other server.
    IOFLOOD.com -- We Love Servers
    Phoenix, AZ Dedicated Servers in under an hour
    ★ Ryzen 9: 7950x3D ★ Dual E5-2680v4 Xeon ★
    Contact Us: sales@ioflood.com

  11. #11
    Join Date
    Nov 2010
    Location
    Casablanca
    Posts
    153
    This case shows that you have not chosen a good company for your backup.
    Normally, backup servers are very protected against hackers with no HTTP or SSH access (just FTP).
    If it was professional backup services, these backup servers should be backuped on servers not connected to the internet in the same DC.
    If I was in your situation, I'll blame on the backup company and change them as soon as possible.

  12. #12
    Join Date
    Jan 2004
    Location
    Greece
    Posts
    2,211
    The problem is that possible for transferring the backups to the remote server ssh keys used which allow the attacker to connect from the webserver to the backup server.

  13. #13
    Join Date
    Jan 2005
    Location
    Kozani - Greece
    Posts
    59
    Cretaforce you are probably right, because the backup company told me that i deleted the files. But i believe that it is mandatory for the backup company to make backups of the client. I didn't read their terms in the past, but i thought i pay them to keep my backups safe... I really don't know who to blame now for this catastrophy.

  14. #14
    Join Date
    Dec 2009
    Posts
    37
    Just a bad luck. The customers if their information is important to them should have backups too. Just a lesson to learn.

  15. #15
    Join Date
    Jan 2004
    Location
    Greece
    Posts
    2,211
    mmingos as I told you if the backup provider didn't say that they keep backups of your data then don't expect them to offer it. So I wouldn't blame the backup provider.

  16. #16
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681
    I am suspecting a missed software update on the server....
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  17. #17
    Join Date
    Sep 2002
    Location
    Canada
    Posts
    456
    I feel for you man. This must be really tough. I can suggest few things:

    a) Evaluate how much the data is worth, and whether your clients will go after you for the lost data.

    b) If data is worth recovering then:

    1. You want to recover, the best way is to get data recovery done on the server. It will be costly to do a professional block level recovery. IF this is a delete only, most data recovery providers can get you the data back.

    Try to convince your hosting provider to allow a data recovery agent to come onsite to pickup the disk. Offer to pay them the cost of the disk, double or triple what it cost in the market.


    2. Recovery agent will probably recommend making copy of the disk later for investigation. Find out how you were hacked and make a plan to prevent that in the future.

    You will not be able to blame this on anyone. Your hosting company is not responsible for your data. Your management team is also not responsible for your data, and even if it was security issue, you can't blame it on them unless they have a guarantee you are not to be hacked!!! They are as much responsible as the OS developers, and you can't blame them either.

    Your backup provider is only responsible for the space where your backup resides, and most likely limited to their TOS and SLA.

    In other words, no one will take responsibility for your data (or your customers). Liability insurance does not cover data loss, and even E&O insurance would be unpredictable in cases like these.

    So focus on recovering the data on your disks first and foremost.
    Reliability • Performance • Integrity

  18. #18
    So no one is safe.

    You should make offsite backups with at least 2-3 providers and use the "pull" method. You must choose reliable offsite backup providers who you can trust though... I would like to know which offsite backup provider(s) you were using? It could happen to other users using the same company...

    Edit: PS I'm not saying it was the backup providers fault. But whether it was their fault or not, it would good for you to let other users know which provider(s) you were using for offsite backups. The provider could give us more info too. Same with the Dedicated Server company you're using.
    Last edited by HostXNow; 12-14-2010 at 09:27 PM.
    HostXNow - Shared Web Hosting | Semi Dedicated Hosting | Enterprise Reseller Hosting | VPS Hosting

  19. #19
    Join Date
    May 2010
    Location
    10.0.0.17
    Posts
    480
    Shouldn't the important thing be dedicating the time and effort into trying to recover your client's data, rather than trying to find someone to lay blame on? All too often people are quick to blame their host for a situation that they may not have control over.

  20. #20
    Quote Originally Posted by Aldryic C'boas View Post
    Shouldn't the important thing be dedicating the time and effort into trying to recover your client's data, rather than trying to find someone to lay blame on? All too often people are quick to blame their host for a situation that they may not have control over.
    It's not so much just finding someone to blame, it's more about finding out how it happened, so you can try to prevent the same thing from happening again.
    HostXNow - Shared Web Hosting | Semi Dedicated Hosting | Enterprise Reseller Hosting | VPS Hosting

  21. #21
    Join Date
    May 2010
    Location
    10.0.0.17
    Posts
    480
    Quote Originally Posted by HostXNow View Post
    It's not so much just finding someone to blame, it's more about finding out how it happened, so you can try to prevent the same thing from happening again.
    This is very true, and part of the point I was trying to get across. Apologies if I worded that poorly.

  22. #22
    Oh that is just a sad thing to happen. Well, there is nothing much you can do at this point.

    You must tell your customers to make backups of their own at all times.
    More than decade with webhosting!

  23. #23
    wow. nightmare. here is what i suggest.
    Technically you got most of the stuff covered with the generous community in the preceding posts. Therefore
    a) proceed to recover data as much as possible.
    b) figure out future prevention and your service terms.
    c) estimate your costs for the above
    and
    d) Compensate your customers with offerings which help easing the pain for their loss, or even better if it makes your customers look at your service as an exceptional one as you stood by your trusting clients.
    e) Add a better security and service tag as a veteran in cyber security
    as you have seen the dragon in the eyes and survived.(the best)

  24. #24
    Join Date
    May 2004
    Location
    Toronto, Canada
    Posts
    5,105
    Quote Originally Posted by mmingos View Post
    Cretaforce you are probably right, because the backup company told me that i deleted the files. But i believe that it is mandatory for the backup company to make backups of the client. I didn't read their terms in the past, but i thought i pay them to keep my backups safe... I really don't know who to blame now for this catastrophy.
    I really do feel sorry for you but I think you are spending too much time looking for someone ELSE that did something wrong here. Don't be looking for someone to blame, we know who to blame, the hacker! The ONLY thing that could have prevented this is having your system hardened from intrusion. Did you use the same password for the system and the backup as well?

    The backup company is not under any obligation to backup your backups. They should be providing reliable storage but if through a fault on your part the backups are wiped out, you have really shouldn't be even thinking about what someone else did wrong.

    Stay on the track of data recovery. Ask to BUY the hard drive and explain the situation or ask them to send to a recovery center of their choosing if they won't sell it to you.
    CloudNexus Technology Services
    Managed Services

  25. #25
    Join Date
    Jan 2005
    Location
    Kozani - Greece
    Posts
    59

    Backup Servers are no good when hacked !

    Thanks for your support Guys.
    1) Thank God i had a local backup (a bit old though) of my server so i saved most clients.
    2) I need to remake only 2 websites out of 120 i host, so i hired some proffessionals to help me in this difficult situation.
    3) All clients that had problem, were given free 2 year hosting4
    4) Lost 2-3 clients out of that problem and after 10 years it's the first time a client moves to another company.
    5) Most of my clients are satisfied of our effords because we tried hard to be up again within 2-3 days, and this happened during weekend. So we had little mail loss and the sites were up again in 3 days. Event occured on Thursday and we were up on Monday evening.

    Let this be a lesson to everybody :
    1) Dedicated Server companies will not be responsible for data loss from hacking
    2) Server Management teams will also not be responsible for data loss from hacking
    3) Backup Server companies will not be responsible for data loss through rm -rf command. My company told me that if the ticket was submited before 24 hours from the event, they may have done something. So i found out that i issued a ticket and it took them about 5 hours to get answer from the technical stuff on the backup server, as they were not doing backup servers themselves. So actually nobody could do anything about my files.

    Clue : You are alone in there. Backup Servers are only good for other types of failures. They cannot protect you because as far as i asked, no backup server company backs up data.

    By the way, the hacker deleted any records of the logs.

Page 1 of 2 12 LastLast

Similar Threads

  1. Lost SSH on dedicated server
    By Formas in forum Hosting Security and Technology
    Replies: 6
    Last Post: 05-01-2008, 07:14 AM
  2. Lost and Confused in finding a reliable Dedicated Fully Managed Host
    By zapatab in forum Managed Hosting and Services
    Replies: 37
    Last Post: 11-15-2007, 01:52 AM
  3. Replies: 1
    Last Post: 03-19-2004, 08:47 AM
  4. CPanel Dedicated Server market a lost cause?
    By bteeter in forum Dedicated Server
    Replies: 7
    Last Post: 07-01-2003, 03:21 PM
  5. Replies: 22
    Last Post: 11-15-2002, 10:11 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •