Hi all,

A quick question for a server.

The forum runs on a single PIII 1Ghz CPU with 512 MB ram. eide hd.

The memory and disk swap "percent capacity" are less than 50%. But the server load average is higher than 50, which makes the forum unaccessible.

The forum is something like ikonboard, use cgi and mysql database. around 300-500 people online in 10 minutes. Daily transfer is about 8GB.

Anyone know the high server load average is normal? Is there a way to optimize it?

Thanks.

not sure that sounds high

[admin@plesk admin]$ uptime
12:38am up 18 days, 10:12, 1 user, load average: 0.00, 0.00, 0.00
[admin@plesk admin]$


thats my server and it has like 10 customers on it that pull about 150 gigs month and my loads are nothing and higest are like 0.40

so i dont think somthing is right on yours.

maybe hardware issue or somthing who knows

It seems ikonboard is getting banned from a few hosts due to being resource intensive. Maybe it's time to look for a php/mysql blend of board?

************,

If it's a pure downlaod site, I think the server load average should not be high. But it's a forum which need to manipulate the db all the time.

JustinK, I tried vB before and the server load average was higher than 30 which also made the forum not accessible.

Is there any server setting that can help optimize it? Or is there a way to tell whether the server is being attacked?

Thanks.

ebird : downloads dont make your load averages higher, scripts however do.

so if you are getting lots of forum views etc... then it will be higher than just downloading regular.zip or .jpg's etc..

Two questions:
You have enough free memory, do you know why you are using swap space at all?
What process consumes the CPU cycles? Can you provide us the output of the "top" commando?

Does this only happen when the board is busy?

Are you using apache?

IF so, what are these settings in your httpd conf file?

Timeout
KeepAlive
MaxKeepAliveRequests
KeepAliveTimeout
MinSpareServers
MaxSpareServers

StartServers
MaxClients
MaxRequestsPerChild

Also, if you have mod_status loaded, what does it's stats show?

Originally posted by Walter
Two questions:
You have enough free memory, do you know why you are using swap space at all?
What process consumes the CPU cycles? Can you provide us the output of the "top" commando?


Hi Walter,

The load average is lower than 1 now. Below is the result. I have not idea why when there's free memory, the server is using swap space. I believe a lot of "forum.cgi", "forum_show.cgi" "thread_show.cgi" are consuming cpu cycles. As you can see, even when the load average is low, there were still <defunct>s.

I am trying to upgrade the server to 1GB memory, but I am wondering whether it helps at all.

Please help.

Thanks.

3:49pm up 7:14, 1 user, load average: 0.81, 0.77, 0.59
84 processes: 77 sleeping, 3 running, 4 zombie, 0 stopped
CPU states: 74.4% user, 18.3% system, 0.0% nice, 7.1% idle
Mem: 513308K av, 367288K used, 146020K free, 72K shrd, 99656K buff
Swap: 265032K av, 111644K used, 153388K free 112292K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
19553 nobody 9 0 4592 4592 1484 S 5.7 0.8 0:00 threaded_show.c
19567 nobody 10 0 4592 4588 1484 S 5.7 0.8 0:00 topic_show.cgi
19565 nobody 9 0 0 0 0 Z 4.9 0.0 0:00 threaded_show.c <defunct>
19555 nobody 9 0 0 0 0 Z 4.3 0.0 0:00 forum_show.cgi <defunct>
19564 nobody 9 0 0 0 0 Z 3.9 0.0 0:00 topic_show.cgi <defunct>
19571 nobody 16 0 4092 4092 1312 R 3.9 0.7 0:00 threaded_show.c
19542 nobody 9 0 0 0 0 Z 3.3 0.0 0:00 threaded_show.c <defunct>
15055 nobody 8 0 6268 5656 3916 S 0.5 1.1 0:01 httpd
16489 nobody 9 0 4672 4056 3684 S 0.1 0.7 0:00 httpd
17545 nobody 8 0 4648 4032 3684 S 0.1 0.7 0:00 httpd
19129 nobody 9 0 4596 3960 3688 S 0.1 0.7 0:00 httpd
19304 erotic 10 0 1060 1060 832 R 0.1 0.2 0:00 top
19522 nobody 9 0 4556 4556 1484 S 0.1 0.8 0:00 topic_show.cgi
19570 mysql 9 0 16788 8576 1040 S 0.1 1.6 0:00 mysqld
1 root 8 0 120 76 52 S 0.0 0.0 0:06 init
2 root 9 0 0 0 0 SW 0.0 0.0 0:00 keventd
3 root 9 0 0 0 0 SW 0.0 0.0 0:00 kapm-idled
4 root 19 19 0 0 0 SWN 0.0 0.0 0:00 ksoftirqd_CPU0
5 root 9 0 0 0 0 SW 0.0 0.0 0:02 kswapd
6 root 9 0 0 0 0 SW 0.0 0.0 0:00 kreclaimd
7 root 9 0 0 0 0 SW 0.0 0.0 0:00 bdflush
8 root 9 0 0 0 0 SW 0.0 0.0 0:00 kupdated
9 root -1 -20 0 0 0 SW< 0.0 0.0 0:00 mdrecoveryd
13 root 9 0 0 0 0 SW 0.0 0.0 0:00 kjournald
136 root 9 0 0 0 0 SW 0.0 0.0 0:00 kjournald
137 root 9 0 0 0 0 SW 0.0 0.0 0:00 kjournald
138 root 9 0 0 0 0 SW 0.0 0.0 0:00 kjournald
139 root 9 0 0 0 0 SW 0.0 0.0 0:00 kjournald
140 root 9 0 0 0 0 SW 0.0 0.0 0:00 kjournald
574 root 9 0 208 148 108 S 0.0 0.0 0:00 syslogd
579 root 9 0 652 4 0 S 0.0 0.0 0:00 klogd
730 nobody 9 0 424 164 112 S 0.0 0.0 0:00 proftpd
748 root 9 0 516 416 332 S 0.0 0.0 0:00 sshd
781 root 9 0 200 4 0 S 0.0 0.0 0:00 xinetd
799 root 9 0 324 208 176 S 0.0 0.0 0:00 antirelayd
817 root 9 0 1172 896 504 S 0.0 0.1 0:00 chkservd
845 root 18 19 2312 1272 796 S N 0.0 0.2 0:00 cpanellogd
851 root 9 0 1092 4 0 S 0.0 0.0 0:00 cpaneld
852 root 9 0 672 4 0 S 0.0 0.0 0:00 webmaild
866 root 9 0 264 4 0 S 0.0 0.0 0:00 stunnel
868 root 9 0 264 4 0 S 0.0 0.0 0:00 stunnel
870 nobody 9 0 264 4 0 S 0.0 0.0 0:00 stunnel

Hi bitserve,

I have tried vBulletin, an all-cgi forum script, and ikonboard like script. The forum ran well for several days in the beginning, and then the server load average became high. Right now, there are about 100 people online in 10 mintues, the load average is lower than 1. But in the busy time, (5pm - 10am pacific time), the server load average were higher than 50 when there were about 300 people online in 10 minutes.

I am wondering whether it's possible that the server is under kind of attacking within the time frame.

I don't know how to get the httpd conf file. Would you please tell me how can I get the settings? Please also tell me how to get the mod_status stats?

Thanks.




Originally posted by bitserve
Does this only happen when the board is busy?

Are you using apache?

IF so, what are these settings in your httpd conf file?

Timeout
KeepAlive
MaxKeepAliveRequests
KeepAliveTimeout
MinSpareServers
MaxSpareServers

StartServers
MaxClients
MaxRequestsPerChild

Also, if you have mod_status loaded, what does it's stats show?

What confused me most is that when the server load average was higher than 50, there were still around 50% free memory left.

Why did this happen? :confused:

Here is the new "top" result. As you can see, the server load average is about 20, but there are still about 150MB free memory.

Would someone tell me whether this is normal? If not, what kind of setting should be modified? Again, how can I get the settings and how do I change it?

Thanks.



6:35pm up 10:00, 1 user, load average: 13.45, 21.15, 21.88
213 processes: 181 sleeping, 24 running, 8 zombie, 0 stopped
CPU states: 78.0% user, 21.9% system, 0.0% nice, 0.0% idle
Mem: 513308K av, 351944K used, 161364K free, 80K shrd, 15308K buff
Swap: 265032K av, 139860K used, 125172K free 48060K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
646 mysql 20 0 17860 9852 968 R 4.3 1.9 0:00 mysqld
652 mysql 20 0 17860 9852 968 R 4.3 1.9 0:00 mysqld
693 mysql 20 0 17860 9852 968 R 4.3 1.9 0:00 mysqld
605 mysql 19 0 17860 9852 968 R 4.1 1.9 0:00 mysqld
687 mysql 19 0 17860 9852 968 R 4.1 1.9 0:00 mysqld
608 mysql 18 0 17860 9852 968 R 3.9 1.9 0:00 mysqld
716 mysql 20 0 17860 9852 968 R 3.9 1.9 0:00 mysqld
733 nobody 17 0 4324 4324 1452 S 3.8 0.8 0:00 board_show.cgi
723 nobody 18 0 4248 4248 1356 R 3.6 0.8 0:00 threaded_show.c
731 nobody 19 0 4372 4372 1452 S 3.6 0.8 0:00 threaded_show.c
732 nobody 17 0 4316 4316 1452 S 3.2 0.8 0:00 board_show.cgi
740 nobody 19 0 3496 3496 1300 R 3.2 0.6 0:00 user_list.cgi
727 nobody 18 0 4320 4320 1452 S 3.0 0.8 0:00 board_show.cgi
738 nobody 18 0 3236 3236 1300 R 3.0 0.6 0:00 threaded_show.c
739 nobody 19 0 3412 3412 1300 R 3.0 0.6 0:00 threaded_show.c
742 nobody 20 0 3516 3516 1300 R 3.0 0.6 0:00 forum_show.cgi
724 nobody 15 0 4320 4320 1452 S 2.8 0.8 0:00 board_show.cgi
725 nobody 14 0 4148 4148 1452 S 2.8 0.8 0:00 forum_show.cgi
737 nobody 20 0 3524 3524 1300 R 2.8 0.6 0:00 threaded_show.c
726 nobody 16 0 4320 4320 1452 S 2.6 0.8 0:00 board_show.cgi
728 nobody 15 0 4324 4324 1452 S 2.6 0.8 0:00 board_show.cgi
743 nobody 20 0 3216 3216 1300 R 2.6 0.6 0:00 threaded_show.c
664 nobody 11 0 4672 4672 1484 S 2.4 0.9 0:00 topic_show.cgi
741 nobody 19 0 2584 2584 1180 R 2.4 0.5 0:00 topic_show.cgi
750 nobody 20 0 2328 2328 1180 R 1.5 0.4 0:00 threaded_show.c
660 erotic 13 0 1176 1176 832 R 1.3 0.2 0:00 top
709 nobody 13 0 4572 4572 1484 S 1.3 0.8 0:00 topic_show.cgi
749 nobody 19 0 2312 2312 1180 R 1.3 0.4 0:00 board_show.cgi
751 nobody 20 0 2216 2216 1172 R 1.3 0.4 0:00 threaded_show.c
744 nobody 20 0 1948 1948 1172 R 1.1 0.3 0:00 threaded_show.c
752 nobody 19 0 2236 2236 1172 R 1.1 0.4 0:00 threaded_show.c
714 nobody 11 0 4640 4640 1484 S 0.9 0.9 0:00 threaded_show.c
655 nobody 9 0 4552 4552 1484 S 0.7 0.8 0:00 threaded_show.c
707 nobody 11 0 4536 4536 1484 S 0.7 0.8 0:00 threaded_show.c
613 nobody 8 0 4584 3540 3224 S 0.5 0.6 0:00 httpd
676 nobody 10 0 4596 4596 1484 S 0.5 0.8 0:00 threaded_show.c
21852 nobody 9 0 4048 2740 2296 S 0.3 0.5 0:00 httpd
595 nobody 9 0 4532 3460 3200 S 0.3 0.6 0:00 httpd
679 nobody 10 0 4620 4620 1484 S 0.3 0.9 0:00 threaded_show.c
711 mysql 10 0 17860 9852 968 S 0.3 1.9 0:00 mysqld
717 mysql 10 0 17860 9852 968 S 0.3 1.9 0:00 mysqld
720 mysql 10 0 17860 9852 968 S 0.3 1.9 0:00 mysqld

Simple really.

Your system is basically running 3 processes:

htttpd
perl
mysqld

With shared memory you can run hundreds of each and not use much more memory than one instance of each.

All of your load is perl and mysql talking, major load there.

What can you do about it? Well, I'm not a mysql tweaking guru but I'm sure you can tweak it. As for perl, again not sure what the tweaks are available but I'm sure there are a few.

Looks to me, it's time for load balancing if the tweaks aren't forthcoming. Or at least putting your DB on one machine, and your web server on another and connecting them on a 100 mbit "back end" network (dual nics in each, have the db and the web server talk to each other on a private network with a crossover patch cable).

Here's something that "might" help, but it's a little tricky.

Ramdrive. Make a ram drive that's large enough to hold the mysql data and once every so often dump it all to the drive (if the system fails though you'll lose all new posts since last dump). I've done that sort of thing before but I paid the price occasionally in lost data.

Seperateing DB from Web server right now is probably your best bet. Or, actually I'd move to a php/mysql solution, php doesn't require shelling out to perl. It's possible you could also benefit from mod_perl but I'm not a big guru of mod_perl so I'm not sure.

Originally posted by ebird
Here is the new "top" result. As you can see, the server load average is about 20, but there are still about 150MB free memory.

Would someone tell me whether this is normal? If not, what kind of setting should be modified? Again, how can I get the settings and how do I change it?

Thanks.



6:35pm up 10:00, 1 user, load average: 13.45, 21.15, 21.88
213 processes: 181 sleeping, 24 running, 8 zombie, 0 stopped
CPU states: 78.0% user, 21.9% system, 0.0% nice, 0.0% idle
Mem: 513308K av, 351944K used, 161364K free, 80K shrd, 15308K buff
Swap: 265032K av, 139860K used, 125172K free 48060K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
646 mysql 20 0 17860 9852 968 R 4.3 1.9 0:00 mysqld
652 mysql 20 0 17860 9852 968 R 4.3 1.9 0:00 mysqld
693 mysql 20 0 17860 9852 968 R 4.3 1.9 0:00 mysqld
605 mysql 19 0 17860 9852 968 R 4.1 1.9 0:00 mysqld
687 mysql 19 0 17860 9852 968 R 4.1 1.9 0:00 mysqld
608 mysql 18 0 17860 9852 968 R 3.9 1.9 0:00 mysqld
716 mysql 20 0 17860 9852 968 R 3.9 1.9 0:00 mysqld
733 nobody 17 0 4324 4324 1452 S 3.8 0.8 0:00 board_show.cgi
723 nobody 18 0 4248 4248 1356 R 3.6 0.8 0:00 threaded_show.c
731 nobody 19 0 4372 4372 1452 S 3.6 0.8 0:00 threaded_show.c
732 nobody 17 0 4316 4316 1452 S 3.2 0.8 0:00 board_show.cgi
740 nobody 19 0 3496 3496 1300 R 3.2 0.6 0:00 user_list.cgi
727 nobody 18 0 4320 4320 1452 S 3.0 0.8 0:00 board_show.cgi
738 nobody 18 0 3236 3236 1300 R 3.0 0.6 0:00 threaded_show.c
739 nobody 19 0 3412 3412 1300 R 3.0 0.6 0:00 threaded_show.c
742 nobody 20 0 3516 3516 1300 R 3.0 0.6 0:00 forum_show.cgi
724 nobody 15 0 4320 4320 1452 S 2.8 0.8 0:00 board_show.cgi
725 nobody 14 0 4148 4148 1452 S 2.8 0.8 0:00 forum_show.cgi
737 nobody 20 0 3524 3524 1300 R 2.8 0.6 0:00 threaded_show.c
726 nobody 16 0 4320 4320 1452 S 2.6 0.8 0:00 board_show.cgi
728 nobody 15 0 4324 4324 1452 S 2.6 0.8 0:00 board_show.cgi
743 nobody 20 0 3216 3216 1300 R 2.6 0.6 0:00 threaded_show.c
664 nobody 11 0 4672 4672 1484 S 2.4 0.9 0:00 topic_show.cgi
741 nobody 19 0 2584 2584 1180 R 2.4 0.5 0:00 topic_show.cgi
750 nobody 20 0 2328 2328 1180 R 1.5 0.4 0:00 threaded_show.c
660 erotic 13 0 1176 1176 832 R 1.3 0.2 0:00 top
709 nobody 13 0 4572 4572 1484 S 1.3 0.8 0:00 topic_show.cgi
749 nobody 19 0 2312 2312 1180 R 1.3 0.4 0:00 board_show.cgi
751 nobody 20 0 2216 2216 1172 R 1.3 0.4 0:00 threaded_show.c
744 nobody 20 0 1948 1948 1172 R 1.1 0.3 0:00 threaded_show.c
752 nobody 19 0 2236 2236 1172 R 1.1 0.4 0:00 threaded_show.c
714 nobody 11 0 4640 4640 1484 S 0.9 0.9 0:00 threaded_show.c
655 nobody 9 0 4552 4552 1484 S 0.7 0.8 0:00 threaded_show.c
707 nobody 11 0 4536 4536 1484 S 0.7 0.8 0:00 threaded_show.c
613 nobody 8 0 4584 3540 3224 S 0.5 0.6 0:00 httpd
676 nobody 10 0 4596 4596 1484 S 0.5 0.8 0:00 threaded_show.c
21852 nobody 9 0 4048 2740 2296 S 0.3 0.5 0:00 httpd
595 nobody 9 0 4532 3460 3200 S 0.3 0.6 0:00 httpd
679 nobody 10 0 4620 4620 1484 S 0.3 0.9 0:00 threaded_show.c
711 mysql 10 0 17860 9852 968 S 0.3 1.9 0:00 mysqld
717 mysql 10 0 17860 9852 968 S 0.3 1.9 0:00 mysqld
720 mysql 10 0 17860 9852 968 S 0.3 1.9 0:00 mysqld

ikonboard seems to be hogging a lot of the resources there - all those running processes ending with .cgi look to be ikonboard components - I recommend you remove ikonboard.

Matt

Thanks for all the reply.

Dixiesys, I got a new machine with 2hd. I put the mysql on the 2nd HD. although it's not as good as get a server for db only, it should be better, right?

matt, that's not ikonboard. That's a ikonboard 3.01 like forum script developed by a Chinese developer. It uses perl and mysql together and it was out several months before ib 3 did the same thing. The forum is the only thing on that machine. (:0

As I said, I have tried several forum scripts, they all ran well for a few days, and then the load average became very high.

Any other suggestions?

Originally posted by ebird
Thanks for all the reply.

Dixiesys, I got a new machine with 2hd. I put the mysql on the 2nd HD. although it's not as good as get a server for db only, it should be better, right?

matt, that's not ikonboard. That's a ikonboard 3.01 like forum script developed by a Chinese developer. It uses perl and mysql together and it was out several months before ib 3 did the same thing. The forum is the only thing on that machine. (:0

As I said, I have tried several forum scripts, they all ran well for a few days, and then the load average became very high.

Any other suggestions?

Use a php / mysql based forum :)

Also, try tweaking Apache by using RLimitCPU and RLimitMEM.

Regards
Matt

I think that you'd definitely be okay if you tweaked apache and mysql.

If your cpu isn't maxed but your load average is high, it's because your machine is waiting to run those processes for some reason besides free cpu cycles. I'm guessing it's not able to spawn enough mysql or apache children and so it has to wait for ones to free up.

But you're cutting it close on your processor too. You could definitely use a board that doesn't hog so many resources.

Your httpd configuration file is probably called httpd.conf. Do a locate on it.