Web Hosting Talk : Web Hosting Main Forums : Hosting Security and Technology : Tracking down load issue

[Xandrios]

Hi,

I have a CentOS box here that is causing some strange load. Brand new X3230 (4*2.66Ghz), 2GB ram and 2*500GB SATA2 in raid-1 on a LSI MegaIDE controller.

Basically the server is continuesly at a load of 1. However I dont see why. No CPU load, and no IO as far as I can see. Does anybody have a tip where to look for the problem?

top - 05:01:56 up 1:53, 2 users, load average: 1.00, 1.02, 1.20
Tasks: 81 total, 1 running, 80 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 0.3%us, 0.0%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2072132k total, 591072k used, 1481060k free, 65652k buffers
Swap: 2048276k total, 0k used, 2048276k free, 461532k cached

# iostat
Linux 2.6.18-53.el5 (xxxx.xandrios.net) 05/14/2008

avg-cpu: %user %nice %system %iowait %steal %idle
2.65 0.56 5.56 0.97 0.00 90.26

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 6.44 99.67 255.89 686581 1762792

# hdparm -tT /dev/sda

/dev/sda:
Timing cached reads: 16092 MB in 2.00 seconds = 8057.39 MB/sec
Timing buffered disk reads: 88 MB in 3.04 seconds = 28.92 MB/sec

With the WHT benchmark test the machine scores about 520 points, seems not bad either.

Only thing I could think of is that it has something to do with the raidcard. The data transfer rate is quite low. Could it be that the card is rebuilding the array? Can I check that without rebooting?

[layer0]

You should be able to check the state of the array (no reboot required) by downloading the CLI and/or storage manager from your RAID card manufacturer.

[Philio]

Load average refers to how many processes are using or waiting for the CPU, hence it is possible a single process can cause a load of 1 without using much CPU. I'm not sure under what circumstances this might occur but it is possible.

Don't most RAID controllers require that you use the BIOS to rebuild a a RAID set?

[Xandrios]

Originally Posted by layer0

You should be able to check the state of the array (no reboot required) by downloading the CLI and/or storage manager from your RAID card manufacturer.

Thanks. I guess I should contact support to figure out which card it is exactly since dmesg does not seem to give a model no.

Load average refers to how many processes are using or waiting for the CPU, hence it is possible a single process can cause a load of 1 without using much CPU. I'm not sure under what circumstances this might occur but it is possible.

Don't most RAID controllers require that you use the BIOS to rebuild a a RAID set?

Well, all the 4 cores are 100% idle. So the problem kinda must be somewhere else. All running processes are sleeping as well, with one exception: The raidcard driver at status 'D'. (Should be expected).

[Xandrios]

Turns out it is an onboard raidcontroller provided by the mainboard (Intel SR1530). They checked the HDD utilization and the drives were almost completely idle...so its not like there is a rebuild in progress or anything like that.

Besides IO, what could cause such load? It is *exactly* 1.0:

13:08:50 up 10:49, 2 users, load average: 1.00, 1.00, 1.00

[msherman]

While running top, if you hit 'i' to filter out all of the idle processes, are you left with any processes in the list that are constantly in the "run" state?

[ximi]

Originally Posted by Xandrios

Turns out it is an onboard raidcontroller provided by the mainboard (Intel SR1530). They checked the HDD utilization and the drives were almost completely idle...so its not like there is a rebuild in progress or anything like that.

Besides IO, what could cause such load? It is *exactly* 1.0:

13:08:50 up 10:49, 2 users, load average: 1.00, 1.00, 1.00

On my quadcore system, my load average is 4.00 all the time. Why? I run SETI@Home. Someone might be running one of these distributed computing programs so check for that. Wouldn't hurt to post the full process list.

[Alex]

* Moved to Tech/Security Issues.


Alex

[Xandrios]

Originally Posted by msherman

While running top, if you hit 'i' to filter out all of the idle processes, are you left with any processes in the list that are constantly in the "run" state?

top - 06:57:39 up 1 day, 4:38, 1 user, load average: 1.00, 1.00, 1.00
Tasks: 138 total, 1 running, 137 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2072132k total, 1188276k used, 883856k free, 122448k buffers
Swap: 2048276k total, 56k used, 2048220k free, 925808k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
423 root 15 0 0 0 0 D 0 0.0 0:00.00 megaide
14063 root 15 0 2172 1028 792 R 0 0.0 0:00.02 top

Seems that the raid driver does have something to dow with it. But what exactly..