Results 1 to 19 of 19
  1. #1

    arg!!! high server load!?

    I think this is a problem thats come up recently (although I could be completely wrong), but high server load plagues my server constantly for no apparent reason. Im running a small vbul board with 30-80 visitors online at once. It's worst on the hour which leads me to believe its cron related, but i can find no such evidence.

    Around 12 oclock:
    Code:
    Server Load Averages  	35.73  34.51  30.38
    I cant even run commands in ssh
    Code:
    -bash: /usr/bin/whoami: Inappropriate ioctl for device
    -bash: fork: Resource temporarily unavailable
    -bash-3.1$
    Heres the crontab
    Code:
    55 4 * * * /scripts/upcp
    0 1 * * * /scripts/cpbackup
    */15 * * * * /usr/local/cpanel/whostmgr/bin/dnsqueue > /dev/null 2>&1
    2,58 * * * * /usr/local/bandmin/bandmin
    0 0 * * * /usr/local/bandmin/ipaddrmap
    14 23 * * * /usr/local/cpanel/whostmgr/docroot/cgi/cpaddons_report.pl --notify
    0 6 * * * /scripts/exim_tidydb > /dev/null 2>&1
    */5 * * * * /usr/local/cpanel/bin/dcpumon >/dev/null 2>&1
    and top
    Code:
    top - 14:21:32 up  1:57,  2 users,  load average: 14.78, 23.99, 21.66
    Tasks:  76 total,   7 running,  69 sleeping,   0 stopped,   0 zombie
    Cpu(s): 42.3% us, 10.2% sy,  0.0% ni, 46.9% id,  0.5% wa,  0.0% hi,  0.0% si
    Mem:    691200k total,   419104k used,   272096k free,        0k buffers
    Swap:        0k total,        0k used,        0k free,        0k cached
    
      PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                   
    11734 mysql     15   0  140m  30m 3452 S   72  4.6  69:34.81 mysqld                                                     
    29980 named     21   0  106m 3572 1876 S    0  0.5   0:00.16 named                                                      
    11995 root      16   0 27748  24m 1996 S    0  3.7   0:02.59 spamd                                                      
    12041 root      16   0 27748  23m  624 S    0  3.5   0:00.02 spamd                                                      
    12045 root      16   0 27748  23m  524 S    0  3.5   0:00.01 spamd                                                      
     3288 neomagic  17   0 20156 9.8m 4544 R   12  1.5   0:00.41 php                                                        
    13410 root      16   0 19792 9072  976 S    0  1.3   0:00.26 cpsrvd-ssl                                                 
     3358 neomagic  18   0 17040 6112 3832 R    5  0.9   0:00.17 php                                                        
     3384 neomagic  19   0 16756 5936 3772 R    5  0.9   0:00.16 php                                                        
    11430 neomagic  18   0 16204 4900 3456 S    0  0.7   0:00.10 php                                                        
    19872 neomagic  18   0 16204 4900 3456 S    0  0.7   0:00.08 php                                                        
    15625 neomagic  18   0 16200 4900 3456 S    0  0.7   0:00.08 php                                                        
     3394 neomagic  18   0 15612 3320 2432 R    2  0.5   0:00.08 php                                                        
    13545 root      18   0 15104 7900  320 S    0  1.1   0:00.00 cpdavd                                                     
    13575 root      34  19 13656  10m 1880 S    0  1.6   0:04.87 cpanellogd                                                 
    32539 nobody    15   0 12720 4648 1884 S    0  0.7   0:00.93 httpd                                                      
     9988 nobody    15   0 12720 4572 1840 S    1  0.7   0:01.66 httpd                                                      
    15362 nobody    16   0 12720 4572 1840 S    0  0.7   0:00.47 httpd                                                      
    18411 nobody    15   0 12712 4584 1856 S    0  0.7   0:00.57 httpd                                                      
    18417 nobody    15   0 12712 4588 1856 S    0  0.7   0:00.71 httpd                                                      
     7554 nobody    15   0 12712 4584 1856 S    0  0.7   0:00.43 httpd                                                      
     3501 nobody    16   0 12704 4564 1840 S    0  0.7   0:00.59 httpd                                                      
    16157 nobody    16   0 12584 4548 1840 S    0  0.7   0:00.66 httpd                                                      
    20097 nobody    15   0 12584 4528 1832 S    0  0.7   0:00.20 httpd                                                      
    17723 nobody    16   0 12568 4524 1836 S    0  0.7   0:00.63 httpd                                                      
     7454 nobody    16   0 12568 4516 1824 S    0  0.7   0:00.02 httpd                                                      
     7558 nobody    15   0 12568 4540 1840 S    0  0.7   0:00.36 httpd                                                      
    19726 nobody    15   0 12568 4440 1772 S    0  0.6   0:00.01 httpd

    It usually comes down to having to restart the server in order to make it usable. Sometimes I even have issues with apache randomly receiving a KILL signal.

    If there is anything else I can provide to track down the cuplrit please tell me.

    Thanks for your help

  2. #2
    Join Date
    Aug 2006
    Location
    Milton Keynes, UK
    Posts
    52
    Check what queries are being run in MySQL at the time as you have a high MySQL CPU usage. See if any queries are hanging around for along time. I had a problem with one of the arcade forum scripts a while ago (think it was IBP arcade for vBulletin) where the script was going through all the scores to find the highest score everytime somebody went to the homepage. There were over 100,000 scores in the database and the query took 99% of the CPU for long periods.

  3. #3
    I've never done any optimization on mysql. How do I check the queries?

  4. #4
    I'd hazard a guess that your logging is too verbose and that disk writes are queuing up.

    Cheers

  5. #5
    Join Date
    Aug 2006
    Location
    Milton Keynes, UK
    Posts
    52
    Log into WHM as root and then take a look through phpmyadmin. There's a link for looking at current queries. There are also some scripts for tracking inefficient queries but you'll need to do a search for them.

  6. #6
    This MySQL server has been running for 0 days, 1 hours, 13 minutes and 30 seconds. It started up on Mar 31, 2008 at 03:20 PM.

    Query statistics: Since its startup, 145,381 queries have been sent to the server.

    Im not really sure whats optimal or what i should be looking for. here are some screens

    http://www.neomagician.com/help

  7. #7
    Join Date
    Dec 2006
    Posts
    477
    at the time you experience the high load, connect to mysql and run:

    show processlist;

    that will show you what queries are running at that moment in time. Also turn on your slow query log to find out which queries are regularly slow.

  8. #8
    Join Date
    Aug 2005
    Location
    Egypt
    Posts
    110
    hello use this script to see where is the issue

    nano /root/load.sh
    #!/bin/bash


    LOAD=$(awk '{print int($1)}' /proc/loadavg)
    HOSTNAME=$(hostname)
    IP=$(hostname -i)
    EMAIL=( [email protected] [email protected] )

    if [ $LOAD -ge 5 ]
    then
    {
    printf "Hostname : $HOSTNAME\n"
    printf "IP : $IP\n"
    printf "Current Load : $LOAD"
    printf "\n\n==================\nMySQL Process List\n==================\n\n"
    mysqladmin proc
    printf "\n\n==================\nMySQL Extended Status\n==================\n\n"
    mysqladmin extended status
    printf "\n\n===================\nServer Process List\n===================\n\n"
    ps -ef
    printf "\n\n==================\nNobody Processes\n==================\n\n"
    ps --User nobody
    printf "\n\n==================\nTOP Snapshot\n==================\n\n"
    top -n1 -b
    printf "\n\n==================\nMemory Usage\n==================\n\n"
    free -m
    printf "\n\n==================\nDisk Usage\n==================\n\n"
    df -h
    printf "\n\n==================\n Apache Connections \n==================\n\n"
    netstat -alpn | grep :80 | awk '{print $5}' | cut -d: -f 1 | sort | uniq -c | sort -n
    } | mail -s "$HOSTNAME Overloaded !!!" $EMAIL[@]
    fi

    change [email protected] to ur mail
    and in crontab this cron * * * * * /root/load.sh

    Regards
    GNU/Linux system Engineer
    Contact Me: 00201003338749

  9. #9
    Join Date
    Dec 2002
    Location
    chica go go
    Posts
    11,858
    Guys, a high server load is not going to result in this guy being given "Inappropriate ioctl for device" errors.

    run this, send output:

    dmesg

  10. #10
    karem, didnt seem to do anything

    ub3r, no output.
    Code:
    [[email protected] ~]# dmesg
    [[email protected] ~]#

    apache has stopped at least 3 times today with no warning and nothing in the logs.
    if i restart it with 'service httpd start' shortly after the load will climb to at least 25-30 and remain for 30min to an hour.

  11. #11
    Join Date
    Dec 2002
    Location
    chica go go
    Posts
    11,858
    Are you on a vps?

  12. #12
    yes
    does that make a difference?

  13. #13
    also during high load there are an enormous amount of php processes running as well as mysql being the top cpu process

  14. #14
    Can you run:

    iostat 3 5

    while under load and post output?

    Cheers

  15. #15
    Code:
    [[email protected] ~]$ iostat 3 5
    Linux 2.6.9-023stab046.2-enterprise (neomagician.com) 	04/02/08
    
    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
              13.49    0.01    3.99    1.54    0.00   80.97
    
    Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
    
    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
              79.53    0.00   17.82    0.59    0.00    2.06
    
    Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
    
    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
              73.73    0.00   23.66    0.15    0.00    2.47
    
    Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
    
    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
              73.21    0.00   22.86    0.14    0.00    3.79
    
    Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
    
    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
              64.28    0.00   17.28    0.74    0.00   17.70
    
    Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
    
    [[email protected] ~]$
    load average: 17.45, 15.45, 14.75
    Last edited by mord3th; 04-02-2008 at 05:38 PM.

  16. #16
    As mentioned before you can try to reduce your logging verbosity. Those spamd processes for instance are probably writing a bunch of stuff to disk.

    You might be able to reduce the verbosity enough to help the situation but ultimately, unless you are on a VPS at a high-end provider that takes measures to reduce I/O contention, you're kind of SOL. Consider migrating to a dedicated box.

    Cheers

    Cheers

  17. #17
    where do i disable or change it?

  18. #18
    Join Date
    Mar 2008
    Posts
    72
    As your server isnt swapping and also not using full memory , You must optimize MySQL and httpd , Check Tutorials at dedicated server section.

  19. #19
    optimized mysql and everything is much more stable now. Problem seems to be fixed.

    Thanks all who helped very much appreciated

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •