Results 1 to 12 of 12
  1. #1

    /dev/md0 100% server down

    Good day,

    I have my server down due to /dev/md0 100% full. Could anyone please offer advise on how to make space?

    Any comments much appreciated.

  2. #2
    Join Date
    Dec 2011
    Location
    Germany
    Posts
    1,153
    Depends on what you have on this partition. A good option would be to delete logs and backups.
    Inbound Marketing & real SEO for web hosting providers
    ✎ Get in touch with me: co<at>infinitnet.de

  3. #3
    Hi infinitnet,

    My host recently installed backups for me and I'm seeing 9G.

    Could you please advise how I would check if backups was in my dev/md0 drive?

    Thank you.

  4. #4
    Join Date
    Nov 2002
    Location
    Portland, Oregon
    Posts
    2,948
    If you've been experiencing technical issues on the box (things like reboots, system errors, failures, etc.) there's a good chance the partition is stuffed with crash logs as @infinitnet explained.

    If SSH isn't running and you haven't got some sort of console-level access -- IPMI/KVMoIP or the like, your DC support folks may have to do this for you.
    | John Edel Jetfire Networks L.L.C. Trusted Hosting Solutions
    | Consistent, Reliable, Stable OpenVZ & KVM Virtual Private Servers
    | SpamWall AV & Full SMTP Filtering
    Now an SSLStore Titanium Partner!

  5. #5
    You could install ncdu to check on the disk usage. Kindly note that since the disk space is already full free some space and have a detailed check using this tool.

    If you are using fedora you could install directly from the repository. For any other refer URL : http://www.hosting.com/support/linux...-linux-server/

  6. #6
    Hi jfnllc

    No emergency support from my ISP on weekends

    I have gained a tiny amount of space and just got ssh running.

    Not enough space to get MYSql running so all my clients sites are still down.

    Any idea where my crash logs folder would be please.

    Also I have a feeling my backups are on the same partition /backup

    I'm seeing 9G in there. If this is the case then deleting my weekly backups as a temporary fix would be ok?

  7. #7
    Join Date
    Nov 2002
    Location
    Portland, Oregon
    Posts
    2,948
    Quote Originally Posted by bub666 View Post
    Hi jfnllc

    No emergency support from my ISP on weekends

    I have gained a tiny amount of space and just got ssh running.

    Not enough space to get MYSql running so all my clients sites are still down.

    Any idea where my crash logs folder would be please.

    Also I have a feeling my backups are on the same partition /backup

    I'm seeing 9G in there. If this is the case then deleting my weekly backups as a temporary fix would be ok?

    Yikes, if your backup directory isn't on a separate partition this can absolutely cause the problem, especially if you've got weekly backups configured in WHM with the "monthly" option enabled too (top of the month to you!) then you may actually have 2 instances of each account backup.

    Can you send me a 'df -h' output please?

    Also, /var/log/ is where you'll find most of the important system logs.

    PHP Code:
     [~]# du -sh /var/log/
    245M    /var/log
    | John Edel Jetfire Networks L.L.C. Trusted Hosting Solutions
    | Consistent, Reliable, Stable OpenVZ & KVM Virtual Private Servers
    | SpamWall AV & Full SMTP Filtering
    Now an SSLStore Titanium Partner!

  8. #8
    Join Date
    Apr 2007
    Location
    US, UK, Europe, ME
    Posts
    258
    Quote Originally Posted by bub666 View Post
    Hi jfnllc

    No emergency support from my ISP on weekends

    I have gained a tiny amount of space and just got ssh running.

    Not enough space to get MYSql running so all my clients sites are still down.

    Any idea where my crash logs folder would be please.

    Also I have a feeling my backups are on the same partition /backup

    I'm seeing 9G in there. If this is the case then deleting my weekly backups as a temporary fix would be ok?
    Can you show us the output of "df -h" please? also "cat /proc/mdstat" could be useful as well.

    Go to your / "cd /" and run the following command:

    du -h --max-depth=1

    If it's your /backup directory which is causing this then yes try to delete some old or unneeded backups

  9. #9
    Thank you for your help.

    I have made 5% free by the following:

    rm -rf /usr/local/cpanel/src/3rdparty/gpl/*.gz
    rm -rf /usr/local/apache.backup_archive

    cd /usr/local/apache/logs

    rm -rf suexec_log
    touch suexec_log
    chown root:nobody suexec_log
    chmod 600 suexec_log

    And also deleting several backups from /weekly

    As requested:

    # df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/md0 19G 17G 971M 95% /
    /dev/md2 437G 17G 399G 4% /home
    tmpfs 7.9G 0 7.9G 0% /dev/shm



    # cat /proc/mdstat
    Personalities : [raid1]
    md1 : active raid1 sdb2[1] sda2[0]
    3911744 blocks [2/2] [UU]

    md2 : active raid1 sdb3[1] sda3[0]
    464929024 blocks [2/2] [UU]

    md0 : active raid1 sdb1[1] sda1[0]
    19542976 blocks [2/2] [UU]

    unused devices: <none>


    # du -h --max-depth=1
    20K ./.ssh
    56K ./.HttpRequest
    1.7M ./.spamassassin
    8.7M ./tmp
    12K ./cpanel3-skel
    8.0K ./public_html
    532K ./.MirrorSearch
    44K ./.gnupg
    36K ./.cpobjcache
    1012K ./rkhunter-1.4.0
    28K ./.cpanm
    67M ./.cpan
    4.0K ./perl5
    2.0M ./csf
    82M ./.cpanel
    4.0K ./public_ftp
    170M .

  10. #10
    Join Date
    Nov 2002
    Location
    Portland, Oregon
    Posts
    2,948
    Can you post the output of the following command (redacting any sensitive information like usernames/domain names) please? Let's dig in & find those little buggers.



    PHP Code:
    # du -h / | grep ^[2-9][0-9][0-9][0-9.]*M 
    | John Edel Jetfire Networks L.L.C. Trusted Hosting Solutions
    | Consistent, Reliable, Stable OpenVZ & KVM Virtual Private Servers
    | SpamWall AV & Full SMTP Filtering
    Now an SSLStore Titanium Partner!

  11. #11
    Thank you jfnllc


    du -h / | grep ^[2-9][0-9][0-9][0-9.]*M

    568M /backup/cpbackup/weekly/xxxx.0
    290M /backup/cpbackup/weekly/dirs
    232M /backup/cpbackup/monthly/dirs
    du: cannot read directory `/proc/9049': No such file or directory
    272M /home/xxxx/mail/xxxx.co.uk/sales/new
    272M /home/xxxx/mail/xxxx.co.uk/sales
    475M /home/xxxx/mail/xxxx.co.uk
    484M /home/xxxx/mail
    251M /home/xxxx
    240M /home/xxxx
    321M /home/xxxx
    200M /home/xxxx
    279M /home/xxxx/public_html/administrator
    334M /home/xxxx/public_html
    419M /home/xxxx
    205M /home/xxxx/mail
    372M /home/xxxx
    220M /home/xxxx
    225M /home/xxxx
    210M /home/xxxx
    375M /home/xxxx/public_html
    231M /home/xxxx/mail/xxx.co.uk
    465M /home/hxxxx/mail/new
    696M /home/xxxx/mail
    279M /home/xxxx/public_html/administrator
    345M /home/xxxx/public_html
    523M /home/xxxx
    316M /home/xxxx/public_html
    279M /home/xxxx/mail/new
    336M /home/xxxx/mail
    219M /home/backup-10.26.2011_19-30-25_drive
    221M /home/xxxx
    525M /home/xxxx/public_html/administrator
    561M /home/xxxx/public_html
    685M /home/xxxx
    538M /home/xxxx
    259M /home/xxxx/mail/new
    270M /home/xxxx/mail
    365M /home/xxxx
    208M /home/xxxx/public_html
    217M /home/xxxx
    457M /home/xxxx
    294M /home/cpeasyapache/src/php-5.3.21
    407M /home/cpeasyapache/src
    422M /home/cpeasyapache
    303M /home/xxxx
    368M /lib/modules
    394M /lib
    369M /usr/local/cpanel/base
    374M /usr/local/cpanel/tmp/rpm_downloads
    374M /usr/local/cpanel/tmp
    286M /usr/local/cpanel/whostmgr/bin
    314M /usr/local/cpanel/whostmgr
    213M /usr/local/cpanel/modules-install/clamavconnector-Linux-x86_64
    213M /usr/local/cpanel/modules-install
    254M /usr/local/cpanel/src/3rdparty/gpl
    342M /usr/local/cpanel/src/3rdparty
    351M /usr/local/cpanel/src
    220M /usr/local/cpanel/logs
    363M /usr/local/cpanel/bin
    375M /usr/local/apache/logs/archive
    463M /usr/local/apache/logs
    535M /usr/local/apache
    871M /usr/share
    349M /usr/lib
    339M /usr/lib64
    272M /var/netenberg/archives/fantastico_de_luxe
    272M /var/netenberg/archives
    272M /var/netenberg
    201M /var/cpanel/bandwidth
    414M /var/cpanel
    642M /var/lib/mysql
    826M /var/lib

  12. #12
    Join Date
    Mar 2005
    Location
    Ten1/0/2
    Posts
    2,509
    Quote Originally Posted by bub666 View Post
    # df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/md0 19G 17G 971M 95% /
    /dev/md2 437G 17G 399G 4% /home
    tmpfs 7.9G 0 7.9G 0% /dev/shm



    # cat /proc/mdstat
    Personalities : [raid1]
    md1 : active raid1 sdb2[1] sda2[0]
    3911744 blocks [2/2] [UU]

    md2 : active raid1 sdb3[1] sda3[0]
    464929024 blocks [2/2] [UU]

    md0 : active raid1 sdb1[1] sda1[0]
    19542976 blocks [2/2] [UU]

    unused devices: <none>
    While your / partion at 20G Meets the CPanel Recommendation as the Minimum size, It reallydoes not cut it these days.

    http://docs.cpanel.net/twiki/bin/vie.../ConfiguringOs

    Really, if you look at the Advanced Guide, and add everyhting up, it comes to

    / 40G
    /var 16G
    /usr 16G
    /tmp 2G

    Now, this adds up to 74G - which is nearly 4x what you have and no wonder you are having problems.

    http://docs.cpanel.net/twiki/bin/vie...dvancedOptions


    The short answer here is that the default install just does not cut it and you really should consider doing a re-build of this server with a sensible LVM setup. I have posted several times about this - so search my posts and you will find a lot of info.

    I really wish that CPanel would update thier recommendations and actually change the 20G minimum to at least a 50G minimum for /

    Honestly, LVM has been around for a long time now and I really am failing to see why By default CPanel will install if it is NOT set up - would solve a heap of problems like this. Then again, I think I understand why - it is because CPanel is trying to make anyone who can open a web browser think they are a Systems Administrator, when they are not.


    If you also search, you will see that someone posts exactly the same thing as you have just done about once a week.
    Last edited by RRWH; 11-03-2013 at 07:29 PM.
    CPanel Shared and Reseller Hosting, OpenVZ VPS Hosting. West Coast (LA) Servers and Nodes
    Running Linux since 1.0.8 Kernel!
    Providing Internet Services since 1995 and Hosting Since 2004

Similar Threads

  1. /dev/sda3 full 100%
    By NSlapper in forum Hosting Security and Technology
    Replies: 9
    Last Post: 09-16-2013, 05:11 AM
  2. Disk /dev/md0 doesn't contain a valid partition table
    By justneedtoask in forum Dedicated Server
    Replies: 5
    Last Post: 02-25-2011, 02:17 PM
  3. Disk /dev/sda5 (/) 100%
    By NilinHost in forum Dedicated Server
    Replies: 7
    Last Post: 01-19-2010, 12:55 PM
  4. Disk (dev) 100 % What to do?
    By abusaifedden in forum Hosting Security and Technology
    Replies: 2
    Last Post: 01-17-2007, 01:39 PM
  5. /dev/hda /dev/hdb /dev/hdc /dev/hdd
    By pmak0 in forum Hosting Security and Technology
    Replies: 5
    Last Post: 01-16-2004, 04:13 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •