Results 1 to 17 of 17
  1. #1
    Join Date
    May 2004
    Location
    Scotland
    Posts
    26

    Unhappy High Server Load Woes

    Since moving to a dedicated server recently I've been experiencing very high server loads, and I have been unable to pinpoint either whats causing it, or what I can do to reduce it.

    I am hosting only two sites, and one of them has only just started and has next to no traffic. The other runs an Invision Board forum, a Coppermine Gallery and has anything up to 140 users online at any given time.

    Sometimes, the server load stays at a reasonable level, 0.5-0.9 , but most of the time jumps up to loads ranging from anywhere between 3.0 to 12.0 and above and stays like that for ages for no apparant reason. I'm certain that even 140 users on-line at once should not be creating server loads this high on a 2.4ghz Celeron with 512mb RAM, running RHE 3.0.

    I've tried several things to try and reduce the load, including trying to configure the my.cnf file and installing Turck mmcache, but neither seems to have made any difference.

    I was planning on hosting more sites on the server, but i'm reluctant to do so until i can figure out how to solve this high server load issue.

    Can anyone suggest a course of action i could take to get to the bottom of this ? What is the most likely cause ? This is my first dedicated server, so i haven't had too much experience dealing with anything like this.

    Any help would be greatly appreciated. Thanks in advance.

  2. #2
    Join Date
    Mar 2003
    Location
    Charlotte, NC
    Posts
    2,760
    Can you post the output of "top" over SSH?

    -Josh

  3. #3
    I think it all comes back down to the celery you are using to host the sites. Although I have no experience with Celeron servers, celery's to begin with, are low-end desktop processors.

    You mentioned that you tried optimizing your my.cnf file and installing MMCache without results. That could seriously mean that your server is underpowered for the task, regardless of optimizations.
    YourCheapHost.com - Low cost multi domain hosting solutions. [Legal adult content friendly]
    Reliable web site hosting is our motto. We have Alertra stats to back that up.
    Proven provider of high quality shared and reseller accounts since 2002.

  4. #4
    Join Date
    Mar 2004
    Location
    Poland, Lublin
    Posts
    71
    take more ram

    show `top`...

    config mysql&httpd (for more connection...)

  5. #5
    Join Date
    May 2004
    Location
    Scotland
    Posts
    26
    Here's an example output from 'top' :

    15:08:44 up 115 days, 43 min, 1 user, load average: 5.09, 2.56, 2.02
    129 processes: 126 sleeping, 2 running, 0 zombie, 1 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 17.8% 0.0% 6.2% 0.9% 0.3% 74.5% 0.0%
    Mem: 505396k av, 499992k used, 5404k free, 0k shrd, 1744k buff
    383176k actv, 46752k in_d, 6824k in_c
    Swap: 1052248k av, 358516k used, 693732k free 159100k cached

    PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
    27661 mysql 15 0 48716 30M 1184 D 10.5 6.1 0:38 0 mysqld
    28295 nobody 15 0 3808 3228 2220 D 1.5 0.6 0:00 0 httpd
    28233 nobody 15 0 4892 4348 2764 S 1.3 0.8 0:00 0 httpd
    28335 mysql 15 0 48716 30M 1184 D 1.3 6.1 0:01 0 mysqld
    28334 mysql 15 0 48716 30M 1184 D 0.7 6.1 0:01 0 mysqld
    7317 nobody 15 0 8808 8080 4760 S 0.5 1.5 4:57 0 httpd
    27391 nobody 15 0 7992 7392 3932 S 0.5 1.4 0:03 0 httpd
    27790 mysql 15 0 48716 30M 1184 S 0.5 6.1 0:35 0 mysqld
    28238 nobody 15 0 3464 2884 1756 S 0.5 0.5 0:00 0 httpd
    7318 nobody 15 0 9084 8376 4628 S 0.3 1.6 4:56 0 httpd
    27511 nobody 15 0 6468 5872 3700 D 0.3 1.1 0:03 0 httpd
    27662 mysql 15 0 48716 30M 1184 S 0.3 6.1 0:41 0 mysqld
    27866 nobody 15 0 6160 5560 3240 S 0.3 1.1 0:02 0 httpd
    28277 nobody 15 0 3928 3348 2312 S 0.3 0.6 0:00 0 httpd
    28294 nobody 15 0 2104 1188 836 D 0.3 0.2 0:00 0 httpd
    28323 mysql 15 0 48716 30M 1184 S 0.3 6.1 0:01 0 mysqld
    28332 mysql 15 0 48716 30M 1184 D 0.3 6.1 0:00 0 mysqld
    5 root 15 0 0 0 0 SW 0.1 0.0 8:42 0 kswapd
    23274 nobody 15 0 7924 7244 3932 S 0.1 1.4 0:14 0 httpd
    27537 mysql 15 0 48716 30M 1184 S 0.1 6.1 0:33 0 mysqld
    27791 mysql 15 0 48716 30M 1184 D 0.1 6.1 0:21 0 mysqld
    28160 nobody 15 0 5208 4648 2804 S 0.1 0.9 0:00 0 httpd
    28232 root 15 0 1120 1120 712 R 0.1 0.2 0:00 0 top

  6. #6
    thats a lot of sql usage... what kinda sites are you hosting? could be the traffic to those sites generating your cpu load
    Joe
    www.DollarWebHosting.Biz
    |:| Shared |:| Reseller |:| Dedicated |:|

  7. #7
    Join Date
    May 2004
    Location
    Scotland
    Posts
    26
    I wouldn't say that i'm really receiving very much traffic, I only have one active site - the other only receives a couple of hundred hits a day. On average there are between 60 and 100 people on-line at any given time on my main site, which is a news site with an Invision Power Board forum and a Coppermine image gallery. I would have thought that a 2.5ghz Celeron could handle more traffic than that. The server load doesn't really seem to correspond with the number of users on-line either.

  8. #8
    Join Date
    May 2004
    Location
    Scotland
    Posts
    26
    Just received this warning email from the server aswell:

    While processing the log files for user xxxxxx, the cpu has been
    maxed out for more then a 6 hour period. The current load/uptime line on the server at the time of this email is 16:27:03 up 115 days, 2:01, 1 user, load average: 1.07, 3.24, 3.00. You should check the server to see why the load is so high and take steps to lower the load.


    Could this be an indication that the log file processing is in some way responsible for the high server load ?

  9. #9
    Join Date
    Aug 2003
    Location
    Florida
    Posts
    181
    Um, look at his iowait guys:

    CPU states: cpu user nice system irq softirq iowait idle
    total 17.8% 0.0% 6.2% 0.9% 0.3% 74.5% 0.0%

    74.5%

    That's the culprit. This is fairly common with RHE, I've seen it discussed often here - search will certainly turn something up.
    Tyler
    www.AdminZoom.com
    "Server Administration Done Right"

    Server setup, hardening, migrations and more

  10. #10
    Originally posted by Dhal
    Just received this warning email from the server aswell:

    While processing the log files for user xxxxxx, the cpu has been
    maxed out for more then a 6 hour period. The current load/uptime line on the server at the time of this email is 16:27:03 up 115 days, 2:01, 1 user, load average: 1.07, 3.24, 3.00. You should check the server to see why the load is so high and take steps to lower the load.


    Could this be an indication that the log file processing is in some way responsible for the high server load ?
    i had this error occuring before... rebuild apache and see if that fixes it, it did for me...
    Joe
    www.DollarWebHosting.Biz
    |:| Shared |:| Reseller |:| Dedicated |:|

  11. #11
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,294
    Originally posted by TySoft
    Um, look at his iowait guys:

    CPU states: cpu user nice system irq softirq iowait idle
    total 17.8% 0.0% 6.2% 0.9% 0.3% 74.5% 0.0%

    74.5%

    That's the culprit. This is fairly common with RHE, I've seen it discussed often here - search will certainly turn something up.

    try this out:


    add this to /etc/rc.local

    echo 100 > /proc/sys/vm/inactive_clean_percent
    echo 2 10 20 > /proc/sys/vm/pagecache

    make your /etc/my.cnf look like this:

    [mysqld]
    skip-locking
    key_buffer=16M
    join_buffer=1M
    record_buffer=1M
    sort_buffer=2M
    table_cache=1028
    max_connections=250
    thread_cache_size=286
    max_allowed_packet=5M
    connect_timeout=15
    query_cache_limit=1M
    query_cache_size=32M
    query_cache_type=1
    thread_concurrency=2
    wait_timeout=15


    in you httpd.conf change these settings to look like these:

    MaxRequestsPerChild 50
    KeepAlive Off


    Reboot the server.
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  12. #12
    Join Date
    May 2004
    Location
    Scotland
    Posts
    26
    Thanks for all your help and comments, I think we've finally found the cause of the problem - one of the scripts on the site seems to have been creating the high server loads, once that script was replaced the load dropped down to between 0.1-0.5 even with 130+ people on-line.

    It just goes to show how a single buggy script can bring the whole server to a crawl.

    Thanks again.

  13. #13
    Join Date
    Oct 2003
    Posts
    293
    may i know how do you manage to find the buggy script?
    it's a whole server and there are so many scripts running, i can't seem to able to find a single script that affects server load.

  14. #14
    Join Date
    May 2004
    Location
    Scotland
    Posts
    26
    It was actually more of a coincidence than anything, I was watching 'top' from the shell, and happened to load one of the pages that is generated by the script in question. The load increased immediately, which drew my attention to it.

    To test this, i disabled the script to see what would happen, and low and behold the load dropped exponentially and stayed low for as long as the script was inactive. The minute i reactivated it, the load increased again.

    Only be replacing this script with an alternative did the high server load problems disappear.

  15. #15
    Join Date
    Jan 2003
    Location
    singapore
    Posts
    291
    thanks for sharing your experience Dhal

    i encounter the same problem these days, my server load would run up to 10 within 24 hours after every reboot.

    it was wired...my top doesn't show much information, CPU is almost idle, it's just the load is high...

    until i disabled one site due to our AUP don't allow any pirated software, after that the server back to normal, and it's been fine for 2 days...i will keep watching...

  16. #16
    Join Date
    Nov 2003
    Posts
    63
    I have a VDS that's lightly used, meaning not many sites and not much activity. Server Load was always below 0.5, every time I checked it. Now, Server Load stays low for the first hour after rebooting, then starts to climb. It normally stays around 1 or 2, but has been above 12.

    Nothing changed between the "good times" and the high loads. Nothing was added. The only scripts I'm running are Invision Power Board, v1.31, but it was installed when Server Load was light.

    I've watched the TOP display for a while, it shows spamd and/or exim using 99.9% cpu when the problems occur. Does spamd need this percentage of cpu because the system is running slow, or does spamd slow the system by using this percentage of cpu?

    Is there a guide to troubleshooting high Server Loads?
    Jack.

  17. #17
    Join Date
    Oct 2002
    Location
    Canada, QC, Montreal
    Posts
    302
    May I know what "script" it was? Part of Coppermine? IPB? Or something custom coded?

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •