Page 1 of 2 12 LastLast
Results 1 to 40 of 48
  1. #1
    Join Date
    Apr 2006
    Posts
    69

    Someone did rm -rf / on my server

    I was logged in under the root ...and someone in my office did run this command..

    rm -rf /

    Now.i can't ftp to the server nor enter the shell panel..it's a cpanel server..

    Please guide what needs to be done.. i have asked to reboot the server and don't know what should be done here..

    I have 40-50 sites on this server and none seems to be loading right now..

    Any help is appreciated..

  2. #2
    Join Date
    Mar 2003
    Location
    Canada
    Posts
    8,910
    1) Do not reboot.

    2) You do realize what that command does, right?

    3) Does the command ls work?
    Patrick William | RACK911 Labs | Software Security Auditing
    400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com

    www.HostingSecList.com - Security notices for the hosting community.

  3. #3
    Join Date
    Jun 2002
    Location
    Waco, TX
    Posts
    5,292
    rm -rf /, just wiped out your entire server, time to pull out the backups!

  4. #4
    Join Date
    Apr 2006
    Posts
    69
    The server has been rebooted..However we had a backup server..but will that bring everything..??
    I mean the mysql and the email data..

  5. #5
    Join Date
    Jun 2002
    Location
    Waco, TX
    Posts
    5,292
    Quote Originally Posted by wtim View Post
    The server has been rebooted..However we had a backup server..but will that bring everything..??
    I mean the mysql and the email data..
    It depends, do your backups contain the mysql and email partitions/files?

  6. #6
    Join Date
    Apr 2006
    Posts
    69
    I
    Quote Originally Posted by (Stephen) View Post
    It depends, do your backups contain the mysql and email partitions/files?
    I don't know..we had a second drive configured on that server for weekly backups...does that backup mysql as well??

  7. #7
    Join Date
    Mar 2003
    Location
    Canada
    Posts
    8,910
    Quote Originally Posted by wtim View Post
    does that backup mysql as well??
    You should hope so, along with hoping the rm -rf / didn't erase the backup drive if it was mounted too.

  8. #8
    Join Date
    Apr 2006
    Posts
    69
    Quote Originally Posted by Pat H View Post
    You should hope so, along with hoping the rm -rf / didn't erase the backup drive if it was mounted too.
    Yes..it was mounted..but I don't know as I am not being able to login to the shell prompt..

    I can feel I am having a bad day..my entire work of 2-3 years has gone in vain if there isn't anything much to be done..

    I have a 3rd party backup as well..but I only have www files over there..I am just scared of the mysql databases..because there really isn't any way to backthem up..

  9. #9
    Join Date
    Mar 2003
    Location
    Canada
    Posts
    8,910
    Quote Originally Posted by wtim View Post
    Yes..it was mounted..but I don't know as I am not being able to login to the shell prompt..

    I can feel I am having a bad day..my entire work of 2-3 years has gone in vain if there isn't anything much to be done..

    I have a 3rd party backup as well..but I only have www files over there..I am just scared of the mysql databases..because there really isn't any way to backthem up..


    All you can do now is ask your data center to check the backup hard drive to see if the content is intact.

    If the content is no longer available, you could have the drive sent to a data recovery company, that is assuming your data center will release the hard drive to you.

  10. #10
    Join Date
    Apr 2006
    Posts
    69
    Quote Originally Posted by Pat H View Post


    All you can do now is ask your data center to check the backup hard drive to see if the content is intact.

    If the content is no longer available, you could have the drive sent to a data recovery company, that is assuming your data center will release the hard drive to you.
    Our server is on layeredtech ..and they are a self managed company..I heard from the technician that they don't do restore stuff..however they can reload the OS and asking me to restore from the secondary drive...( considering if the data is still there )..

  11. #11
    Join Date
    Apr 2006
    Posts
    69
    Can someone tell me where does the .sql file resides in a CPanel server for every account created..

    let's say..i have abc.com on a cpanel server..and have 3 db's ..can you tell where can i find the sql files for them in a server??

  12. #12
    Join Date
    Apr 2006
    Posts
    69
    Just recieved a message from the datacenter that they can't enter into any run levels even from the single user mode..

    The only option they suggest is to reload the primary drive OS and see if the data still exists in secondary drive and could be restored..

    What do you guys think..

    a) Will the data exists on the secondary drive.
    b) Reloading the OS on the primary drive effect the secondary drive.


    I am just thinking..if the secondary drive data also got deleted..then there is no option i believe..??

  13. #13
    Join Date
    Mar 2003
    Location
    Canada
    Posts
    8,910
    Quote Originally Posted by wtim View Post
    a) Will the data exists on the secondary drive.
    b) Reloading the OS on the primary drive effect the secondary drive.
    If the secondary hard drive was mounted and the rm -rf / command was left to execute, then there's a good chance the data has been erased. Hopefully by some fluke, it's still there...

    Reloading the OS on the primary hard drive should have no effect on the secondary drive. The biggest concern would be a data center technician not paying attention and accidentally formatting the drive, or installing the OS on it... make sure they are well aware to not touch the secondary drive.

  14. #14
    Join Date
    Apr 2006
    Posts
    69
    Quote Originally Posted by Pat H View Post
    If the secondary hard drive was mounted and the rm -rf / command was left to execute, then there's a good chance the data has been erased. Hopefully by some fluke, it's still there...

    Reloading the OS on the primary hard drive should have no effect on the secondary drive. The biggest concern would be a data center technician not paying attention and accidentally formatting the drive, or installing the OS on it... make sure they are well aware to not touch the secondary drive.
    They said..they would reloading the OS on a seperate drive..keeping the two primary and secondary drive untouched...

    Let's keep the fingers crossed..and thanks PatH for the responses..it;s you guys who keep the noobs alive by these response..really much appreicated..

  15. #15
    Join Date
    Aug 2006
    Location
    Canada
    Posts
    756
    Who would do such thing in your office? I'll need to be careful too after hearing this, somebody might come over and just do it for the fun of it.
    Otto Yiu
    Rsync Palace ● Providing offsite backups since 2007.
    Backomatic ● Hassle-free Automated cPanel/WHM, DirectAdmin, FTP, and MySQL backups.

  16. #16
    Join Date
    Oct 2005
    Posts
    435
    Quote Originally Posted by -OY- View Post
    Who would do such thing in your office? I'll need to be careful too after hearing this, somebody might come over and just do it for the fun of it.
    yeah I was wondering the same....

    I guess you must have stepped away and left your computer unlocked? If you wiped out a fortune 500 company's data like that, chances are you might go to jail. So whoever did this to you, you need to try and find them and have them face the penalty. If you know who it might be, report them to your manager (assuming by office you meant your day job). I would even consider a lawsuit against the offender. That's BS!
    Last edited by subzer0; 03-05-2008 at 01:38 AM.

  17. #17
    Join Date
    Jan 2003
    Location
    U.S.A.
    Posts
    3,911
    I'm so sorry to hear that someone would do something like this. I will be a little more cautious when logged in as root.

  18. #18
    Join Date
    Jan 2004
    Location
    Greece
    Posts
    2,123
    I haven't test what happens when rm -fr / deletes the /bin/rm

    Does deleting of files still continue or it stops? I think it continues because the command loaded on the memory.

  19. #19
    Join Date
    Jul 2004
    Posts
    63
    would be handy to be able to setup a password when removing files from certain directories, suppose this is where not using root directly comes in.

    If you've run "rm -rf /" than i would say it would remove all data in your mounts. Your only option is to have the hard disk sent to you or a 3rd party company for data recovery, it may even be possible to pay layeredtech do do such a thing but i'm thinking this is your only option.
    Cast-Control
    --------------
    Shoutcast Control Panel - Standalone - Billing Features - GeoIP Features - Stream Authentication - MSN Control - Cluster - Reselling - Video Streaming
    Managed VPS Hosting

  20. #20
    Join Date
    Apr 2006
    Posts
    69
    Quote Originally Posted by Extreme43 View Post

    If you've run "rm -rf /" than i would say it would remove all data in your mounts.
    This is what scaring me the most..I have all the data files backed up..but not the database..on a 3rd party server...

    I can easily restore all the php., html files from the 3rd disk..but it won't function until the whole DB is not there...

  21. #21
    Join Date
    Jul 2004
    Posts
    63
    How old is the backup and what directories are you backing up? If you have the files backed up than the mysql files should still be intact (depending on your backup scheme).

    Checkout /var/lib/mysql/ on your backup, if you have backed this up you will most likely find the files within this directory. If not, do a search for *.MYD and *.MYI (MySQL)files and hopefully it will come up with something.

    I would say the last resort would be to contact your customers, offer them a refund for the full or partial period with a free future month or two thrown in if they wish to continue your services - ask them if they have made there own backups, you will find often that customers wont trust hosting services and make there own regular backups.

    Than you would be on your steps to recovering your business, if your customers want something - throw it in free. After all, you were responsible for there websites and you have failed (no intentional offense). If it were me i would probably make the person responsible for running that command take on the bulk of the work (if they are your employee that is).
    Cast-Control
    --------------
    Shoutcast Control Panel - Standalone - Billing Features - GeoIP Features - Stream Authentication - MSN Control - Cluster - Reselling - Video Streaming
    Managed VPS Hosting

  22. #22
    Join Date
    Apr 2006
    Posts
    69
    Quote Originally Posted by Extreme43 View Post
    How old is the backup and what directories are you backing up? If you have the files backed up than the mysql files should still be intact (depending on your backup scheme).

    Checkout /var/lib/mysql/ on your backup, if you have backed this up you will most likely find the files within this directory. If not, do a search for *.MYD and *.MYI (MySQL)files and hopefully it will come up with something.

    I would say the last resort would be to contact your customers, offer them a refund for the full or partial period with a free future month or two thrown in if they wish to continue your services - ask them if they have made there own backups, you will find often that customers wont trust hosting services and make there own regular backups.

    Than you would be on your steps to recovering your business, if your customers want something - throw it in free. After all, you were responsible for there websites and you have failed (no intentional offense). If it were me i would probably make the person responsible for running that command take on the bulk of the work (if they are your employee that is).

    Yeah..I am all at this..I have already informed the clients and offer them a solution..they are happy with it..

    Unfortunately..I just backed up the home partition and nothing else.....which was the problem..
    We haven't recieved any notice from the datacenter about the second drive..which is what I am waiting for at the moment...If the second drive data is also deleted..then the only solution left for me is to start fresh with copying just the files from the 3rd server..
    Now.. Is there any way to transfer the files from one FTP server to another server.using a ftp software.i.e 2 remote servers..as it would be impossible for me to copy 1 files each from the linux server..or download them on a local machine and then re-upload on the main machine..

  23. #23
    Join Date
    Jul 2004
    Posts
    63
    ok, glad to see your customers are understanding with the situation. I don't see why it would be impossible to copy the files, you could tar the home directory - move it to the webserver and wget it into your other server. Than simply untar the files.

    You could use a site-to-site transfer (FXP i think it is) but it would take forever for the fact there are would be so many individual files.

    I still think you should look into a data recovery service, it is not as difficult as it sounds. To break it down, when you delete a file on the disk the sector/block is simply set to "Allow data to overwrite" and your data is NOT destroyed. I know this applies to NTFS but am unsure about others - wouldn't see a reason not too.

    Furthermore your customers will be impressed to see this.
    Cast-Control
    --------------
    Shoutcast Control Panel - Standalone - Billing Features - GeoIP Features - Stream Authentication - MSN Control - Cluster - Reselling - Video Streaming
    Managed VPS Hosting

  24. #24
    Join Date
    Apr 2006
    Posts
    69
    Quote Originally Posted by Extreme43 View Post
    ok, glad to see your customers are understanding with the situation. I don't see why it would be impossible to copy the files, you could tar the home directory - move it to the webserver and wget it into your other server. Than simply untar the files.

    You could use a site-to-site transfer (FXP i think it is) but it would take forever for the fact there are would be so many individual files.

    I still think you should look into a data recovery service, it is not as difficult as it sounds. To break it down, when you delete a file on the disk the sector/block is simply set to "Allow data to overwrite" and your data is NOT destroyed. I know this applies to NTFS but am unsure about others - wouldn't see a reason not too.

    Furthermore your customers will be impressed to see this.
    Yeah...I would request the LT guys to check with data recovery on the primary drive which will be kept on a pending que..but I am not sure how much it might cost ....and since they are a self managed company..I doubt they would escalate the process of data recovery on the primary drive..

    Yeah..I would try the TAR option ..just wanted to check if there is an easier process..However..since the server is a CPANEL server..does the mail folder in the /home/x123/mail/ would still contain all the mails??

  25. #25
    Join Date
    May 2006
    Location
    EU & USA
    Posts
    3,684
    OMG, i hope you find out who did this, these are no jokes. Can't imagine anyone would do this to any server. Lesson learned : Off-site backups no mounts.

  26. #26
    Join Date
    Jan 2005
    Location
    Scotland, UK
    Posts
    2,549
    That is the importance of offsite backups, if your drive was mounted chances are it was wiped too.

    "rm -rf /" itself will not do anything, it will just return an error so I assume you mean "rm -rf /*", when it was done you should still be able to browse as once it gets to /bin most of the utils will be gone but you can still browse around with the built in shell commands such as "echo *"
    Server Management - AdminGeekZ.com
    Infrastructure Management, Web Application Performance, mySQL DBA. System Automation.
    WordPress/Magento Performance, Apache to Nginx Conversion, Varnish Implimentation, DDoS Protection, Custom Nginx Modules
    Check our wordpress varnish plugin. Contact us for quote: [email protected]

  27. #27
    Join Date
    Apr 2006
    Posts
    69
    Quote Originally Posted by Scott.Mc View Post
    That is the importance of offsite backups, if your drive was mounted chances are it was wiped too.
    Yes..offsite backups of every server is must..which we had..the only thing i feel sad about is the mysql wasn't backed up..and until/unless the datacenter confirms..i am still holding on with the second drive..which was mounted yes..I will confirm here if the files were deleted...

    Quote Originally Posted by Scott.Mc View Post
    "rm -rf /" itself will not do anything, it will just return an error so I assume you mean "rm -rf /*", when it was done you should still be able to browse as once it gets to /bin most of the utils will be gone but you can still browse around with the built in shell commands such as "echo *"
    "rm -rf /" was the command since the employee was trying to delete a directory of a user account but accidentally hit enter after the slash and didn't notice..


    However..since the server is a CPANEL server..does the mail folder in the /home/x123/mail/ would still contain all the mails and if we create new accounts in the server..would copy/paste the mail folder on the new drive would bring back all the mails??

  28. #28
    Join Date
    Mar 2007
    Location
    UK
    Posts
    852
    So was the command found running and Cntrl + C, or was it found out after the server just went down?

    I really think the rm command should ask you for the root password again when you run "rm -rf /*"

    Amount of people/servers I have seen thats something along them lines has happened.
    ZXPlay
    Premium Virtual Private Servers | Dedicated Media Streaming Servers
    Dedicated Resources | EU Based
    www.zxplay.co.uk

  29. #29
    Join Date
    Jun 2006
    Location
    Amex & Amex
    Posts
    1,276
    Is there anyway to block the rm -rf / command?

  30. #30
    Join Date
    Feb 2003
    Location
    Europe
    Posts
    55
    Quote Originally Posted by wtim View Post
    Yes..it was mounted..but I don't know as I am not being able to login to the shell prompt..

    I can feel I am having a bad day..my entire work of 2-3 years has gone in vain if there isn't anything much to be done..

    I have a 3rd party backup as well..but I only have www files over there..I am just scared of the mysql databases..because there really isn't any way to backthem up..
    I feel sorry for you, every good admin learned his or her lessons.
    However, there are good ways to backup most things, including mysql.

    There is no excuse for that, but you can learn some lessons.

    mysqldump is included in your setup for backing up mysql
    backups should always be copied to another place, so if the server fails completely you can restore it again.

    rm -fr / is serious, but recoverable if you dont mind paying an expert or you are experienced in hard disk recovery.

    Best of luck,

    Neil

  31. #31
    Join Date
    Nov 2001
    Location
    Philadelphia, Pa
    Posts
    949
    Does deleting of files still continue or it stops? I think it continues because the command loaded on the memory.
    It will continue. Try it once. Install a vps (vmware, xen, your choice of software), and do it in a vm. It's actually entertaining (when you're not doing it on a real server with real data!).
    Last edited by derek.bodner; 03-05-2008 at 04:18 PM.

  32. #32
    Join Date
    Jun 2006
    Location
    USA
    Posts
    58
    Quote Originally Posted by Starsky View Post
    yeah I was wondering the same....

    I guess you must have stepped away and left your computer unlocked? If you wiped out a fortune 500 company's data like that, chances are you might go to jail. So whoever did this to you, you need to try and find them and have them face the penalty. If you know who it might be, report them to your manager (assuming by office you meant your day job). I would even consider a lawsuit against the offender. That's BS!
    I disagree. It's large security risk to leave your computer unlocked for anything that takes you out of sight of it. The person who gets blamed for deleting company files is going to be the one who left their computer unlocked. You'd need verifiable proof that it was someone else, since all the logging evidence would point to your username.

    This was a terrible lapse in judgement, especially since you were logged in as root.

    Where I work, leaving one's computer unlocked is an invitation for anyone who notices it to screw with it. It's only a nuisance, but it's enough so that everyone locks their screen now.

    I'm sorry you had to go through that in order to learn your lesson though. The person who did that to you was particularly cruel.

  33. #33
    Join Date
    Aug 2004
    Location
    London
    Posts
    883
    Quote Originally Posted by wtim View Post
    I was logged in under the root ...and someone in my office did run this command..

    rm -rf /
    If that were someone in my office they'd be escorted off the premises immediately and then a police complain filed for criminal damages.

    What is the data worth to you?

    less than $750 - take the loss don't pay recovery firm, this will be cheaper than their fees.
    $750-2000 - Seriously consider paying for data recovery
    $2000+ - Pretty much a no brainer. Go with the data recovery.
    ...loading

  34. #34
    Join Date
    Oct 2001
    Posts
    1,315
    Hi There,

    I'm sorry to hear about what happened

    1) the rm command will not stop even after the rm binary is deleted because a copy of the executable is made when the program runs

    2) an rm -r -f / is very difficult to recover from if you're running ext3. EXT3 wipes out the inode block #s and so its virtually impossible to reconstruct very large files. If the datacenter will ship you the drive, put it in your computer, purchase a copy of WINHEX and search for key phrases in important files. If you're looking for MySQL data, search for key MySQL phrases or use the MySQL docs to search for the hex codes that can identify a MySQL file. If you're lucky, large amounts of data from MySQL files will be continuous on the drive

    The best thing to come out of this will be learning some lessons:
    1) Backups on a 3rd harddrive mounted as /backup are NOT sufficient even when combined with RAID. Off-site/off-server backups are MUST

    2) Be sure to backup not only /home but also use MySQL dump to backup MySQL files and vital files in /var/cpanel and /etc

    3) Write a wrapper to prevent rm -r -f / from running under ANY circumstances! But this gets messy if any os updates try to update the rm binary.
    Avi Brender
    Reliable Web Hosting by Elite Hosts, Inc
    CPANEL Reseller Hosting - Fantastico - Rvskins - ClientExec

  35. #35
    I agree offsite backup is a must, always backup your entire disk (not just the /home directory) less the package that can be reinstall from rpm and/or deb repository. Lot of configure and data file are located else where. Also use rdiff-backup, you get incremental backup and uses less bandwidth.
    Sitek XEN Hosting - Reliable VPS Hosting

    CRM OnDemand - Fully Managed / Customized vTiger Hosting

  36. #36
    Join Date
    Oct 2005
    Posts
    435
    Quote Originally Posted by ZX-Ashley View Post
    I really think the rm command should ask you for the root password again when you run "rm -rf /*"
    I agree. or at least "are you sure". There should be more safety precautions built into the command when you execute it with those parameters.

  37. #37
    by default, it does.

    But the problem is you put in the option of -f, -f mean that you say yes to everything without prompt. Since you are root, no permission denied there. So basically, all is GONE.
    Sitek XEN Hosting - Reliable VPS Hosting

    CRM OnDemand - Fully Managed / Customized vTiger Hosting

  38. #38
    Join Date
    Oct 2005
    Posts
    435
    Quote Originally Posted by PrezKennedy View Post
    I disagree. It's large security risk to leave your computer unlocked for anything that takes you out of sight of it. The person who gets blamed for deleting company files is going to be the one who left their computer unlocked. You'd need verifiable proof that it was someone else, since all the logging evidence would point to your username.
    Well duh, the point I was making was if it was an act of sabotage and you knew exactly who did it. Anyway, in the case of OP, the rm appears to have been carried out accidentally by one of his employees.

  39. #39
    Join Date
    Oct 2005
    Posts
    435
    Quote Originally Posted by jamesmoey View Post
    by default, it does.

    But the problem is you put in the option of -f, -f mean that you say yes to everything without prompt. Since you are root, no permission denied there. So basically, all is GONE.
    Then rm should be programmed to ignore -f when path = "/"

  40. #40
    Join Date
    Apr 2006
    Posts
    69
    Just an update..:

    The second drive backup got deleted too! Probably because it was mounted too!

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •