Results 1 to 9 of 9
  1. #1
    Join Date
    Jun 2004
    Location
    Bay Area
    Posts
    1,320

    Tracking down load issue

    Hi,

    I have a CentOS box here that is causing some strange load. Brand new X3230 (4*2.66Ghz), 2GB ram and 2*500GB SATA2 in raid-1 on a LSI MegaIDE controller.

    Basically the server is continuesly at a load of 1. However I dont see why. No CPU load, and no IO as far as I can see. Does anybody have a tip where to look for the problem?

    top - 05:01:56 up 1:53, 2 users, load average: 1.00, 1.02, 1.20
    Tasks: 81 total, 1 running, 80 sleeping, 0 stopped, 0 zombie
    Cpu0 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
    Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
    Cpu2 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
    Cpu3 : 0.3%us, 0.0%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
    Mem: 2072132k total, 591072k used, 1481060k free, 65652k buffers
    Swap: 2048276k total, 0k used, 2048276k free, 461532k cached
    # iostat
    Linux 2.6.18-53.el5 (xxxx.xandrios.net) 05/14/2008

    avg-cpu: %user %nice %system %iowait %steal %idle
    2.65 0.56 5.56 0.97 0.00 90.26

    Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
    sda 6.44 99.67 255.89 686581 1762792
    # hdparm -tT /dev/sda

    /dev/sda:
    Timing cached reads: 16092 MB in 2.00 seconds = 8057.39 MB/sec
    Timing buffered disk reads: 88 MB in 3.04 seconds = 28.92 MB/sec
    With the WHT benchmark test the machine scores about 520 points, seems not bad either.

    Only thing I could think of is that it has something to do with the raidcard. The data transfer rate is quite low. Could it be that the card is rebuilding the array? Can I check that without rebooting?
    Last edited by Xandrios; 05-13-2008 at 09:12 PM.
    Powered by Level3, GBLX and AT&T

  2. #2
    Join Date
    Dec 2004
    Location
    New York, NY
    Posts
    10,541
    You should be able to check the state of the array (no reboot required) by downloading the CLI and/or storage manager from your RAID card manufacturer.

  3. #3
    Join Date
    May 2006
    Posts
    130
    Load average refers to how many processes are using or waiting for the CPU, hence it is possible a single process can cause a load of 1 without using much CPU. I'm not sure under what circumstances this might occur but it is possible.

    Don't most RAID controllers require that you use the BIOS to rebuild a a RAID set?

  4. #4
    Join Date
    Jun 2004
    Location
    Bay Area
    Posts
    1,320
    Quote Originally Posted by layer0 View Post
    You should be able to check the state of the array (no reboot required) by downloading the CLI and/or storage manager from your RAID card manufacturer.
    Thanks. I guess I should contact support to figure out which card it is exactly since dmesg does not seem to give a model no.

    Load average refers to how many processes are using or waiting for the CPU, hence it is possible a single process can cause a load of 1 without using much CPU. I'm not sure under what circumstances this might occur but it is possible.

    Don't most RAID controllers require that you use the BIOS to rebuild a a RAID set?
    Well, all the 4 cores are 100% idle. So the problem kinda must be somewhere else. All running processes are sleeping as well, with one exception: The raidcard driver at status 'D'. (Should be expected).
    Powered by Level3, GBLX and AT&T

  5. #5
    Join Date
    Jun 2004
    Location
    Bay Area
    Posts
    1,320
    Turns out it is an onboard raidcontroller provided by the mainboard (Intel SR1530). They checked the HDD utilization and the drives were almost completely idle...so its not like there is a rebuild in progress or anything like that.

    Besides IO, what could cause such load? It is *exactly* 1.0:

    13:08:50 up 10:49, 2 users, load average: 1.00, 1.00, 1.00
    Powered by Level3, GBLX and AT&T

  6. #6
    Join Date
    Feb 2008
    Location
    California
    Posts
    315
    While running top, if you hit 'i' to filter out all of the idle processes, are you left with any processes in the list that are constantly in the "run" state?
    Take 2 Hosting, Inc. - Hosting Done Right
    Fully automated setup - new servers in as little as 10 minutes
    http://www.take2hosting.com/

  7. #7
    Join Date
    Aug 2007
    Location
    Minneapolis
    Posts
    109
    Quote Originally Posted by Xandrios View Post
    Turns out it is an onboard raidcontroller provided by the mainboard (Intel SR1530). They checked the HDD utilization and the drives were almost completely idle...so its not like there is a rebuild in progress or anything like that.

    Besides IO, what could cause such load? It is *exactly* 1.0:

    13:08:50 up 10:49, 2 users, load average: 1.00, 1.00, 1.00
    On my quadcore system, my load average is 4.00 all the time. Why? I run SETI@Home. Someone might be running one of these distributed computing programs so check for that. Wouldn't hurt to post the full process list.

  8. #8
    Join Date
    Apr 2005
    Location
    Oz
    Posts
    3,497
    * Moved to Tech/Security Issues.


    Alex

  9. #9
    Join Date
    Jun 2004
    Location
    Bay Area
    Posts
    1,320
    Quote Originally Posted by msherman View Post
    While running top, if you hit 'i' to filter out all of the idle processes, are you left with any processes in the list that are constantly in the "run" state?
    top - 06:57:39 up 1 day, 4:38, 1 user, load average: 1.00, 1.00, 1.00
    Tasks: 138 total, 1 running, 137 sleeping, 0 stopped, 0 zombie
    Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
    Mem: 2072132k total, 1188276k used, 883856k free, 122448k buffers
    Swap: 2048276k total, 56k used, 2048220k free, 925808k cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    423 root 15 0 0 0 0 D 0 0.0 0:00.00 megaide
    14063 root 15 0 2172 1028 792 R 0 0.0 0:00.02 top
    Seems that the raid driver does have something to dow with it. But what exactly..
    Powered by Level3, GBLX and AT&T

Related Posts from theWHIR.com

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •