Web Hosting Talk : Web Hosting Main Forums : Hosting Security and Technology : Server is acting weird tonight

[C.P]

For some reason my server is acting very weird with high server load and it's 12 midnight. I never have this problem at this time.

00:30:16 up 29 min, 1 user, load average: 6.38, 6.56, 5.97
137 processes: 134 sleeping, 1 running, 2 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 16.9% 0.0% 5.8% 0.0% 0.4% 76.7% 0.0%
Mem: 1546620k av, 930748k used, 615872k free, 0k shrd, 41100k buff
777180k active, 114316k inactive
Swap: 2097136k av, 0k used, 2097136k free 253816k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
1580 nobody 15 0 19684 12M 11972 S 14.1 0.8 0:26 0 /usr/local/apache/bin/httpd -DSSL
2559 root 16 0 4144 1192 3812 R 3.0 0.0 0:36 0 top
1571 nobody 15 0 20724 13M 11972 S 1.3 0.8 0:22 0 /usr/local/apache/bin/httpd -DSSL
1850 mysql 15 0 161M 35M 3756 D 1.3 2.3 0:02 0 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --us
1570 nobody 16 0 19724 12M 11972 S 0.8 0.8 0:16 0 /usr/local/apache/bin/httpd -DSSL
2386 mysql 15 0 161M 35M 3756 S 0.8 2.3 0:03 0 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --us
1579 nobody 15 0 20560 13M 11972 S 0.4 0.8 0:27 0 /usr/local/apache/bin/httpd -DSSL
1583 nobody 15 0 20788 13M 11972 S 0.4 0.8 0:27 0 /usr/local/apache/bin/httpd -DSSL
1587 nobody 15 0 20708 13M 11972 S 0.4 0.8 0:25 0 /usr/local/apache/bin/httpd -DSSL
2503 nobody 15 0 20728 13M 11972 S 0.4 0.8 0:17 0 /usr/local/apache/bin/httpd -DSSL
2511 root 16 0 7028 2160 6388 S 0.4 0.1 0:00 0 sshd: root@pts/0
2560 nobody 15 0 20516 12M 11972 S 0.4 0.8 0:19 0 /usr/local/apache/bin/httpd -DSSL
2625 mailnull 16 0 113M 109M 6120 S 0.4 7.2 0:14 0 /usr/bin/perl -I/usr/mailscanner/lib /usr/mailscanner/bin/
1 root 16 0 1504 504 1352 S 0.0 0.0 0:01 0 init [3]
2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
3 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
4 root 5 -10 0 0 0 SW< 0.0 0.0 0:00 0 events/0
5 root 15 -10 0 0 0 SW< 0.0 0.0 0:00 0 khelper
6 root 14 -10 0 0 0 SW< 0.0 0.0 0:00 0 kacpid
21 root 5 -10 0 0 0 SW< 0.0 0.0 0:00 0 kblockd/0
32 root 20 0 0 0 0 SW 0.0 0.0 0:00 0 pdflush
33 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 pdflush
35 root 10 -10 0 0 0 SW< 0.0 0.0 0:00 0 aio/0
34 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 kswapd0
36 root 10 -10 0 0 0 SW< 0.0 0.0 0:00 0 xfslogd/0
37 root 5 -10 0 0 0 SW< 0.0 0.0 0:00 0 xfsdatad/0
38 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 xfsbufd
131 root 6 -10 0 0 0 SW< 0.0 0.0 0:00 0 ata/0
134 root 16 0 0 0 0 SW 0.0 0.0 0:00 0 kseriod
165 root 15 0 0 0 0 DW 0.0 0.0 0:00 0 kjournald
467 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 khubd
944 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
1344 root 15 0 1568 572 1400 S 0.0 0.0 0:00 0 syslogd -m 0
1348 root 15 0 1512 464 1348 S 0.0 0.0 0:00 0 klogd -x
1394 root 16 0 3640 1564 3420 S 0.0 0.1 0:00 0 /usr/sbin/sshd


Any ideas why?

[C.P]

It's midnight and there's barely anyone around and I'm getting hit pretty bad for some reason.

00:33:34 up 33 min, 1 user, load average: 21.58, 9.98, 7.16
150 processes: 115 sleeping, 34 running, 1 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 94.8% 0.0% 3.2% 0.3% 1.6% 0.0% 0.0%
Mem: 1546620k av, 1155812k used, 390808k free, 0k shrd, 41828k buff
988952k active, 125508k inactive
Swap: 2097136k av, 0k used, 2097136k free 284844k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
1575 nobody 17 0 23888 16M 11972 R 4.1 1.0 0:29 0 /usr/local/apache/bin/httpd -DSSL
1576 nobody 17 0 23780 16M 11972 R 4.1 1.0 0:33 0 /usr/local/apache/bin/httpd -DSSL
1571 nobody 17 0 23708 15M 11972 R 3.8 1.0 0:27 0 /usr/local/apache/bin/httpd -DSSL
1579 nobody 17 0 23680 16M 11972 R 3.8 1.0 0:30 0 /usr/local/apache/bin/httpd -DSSL
1581 nobody 17 0 23776 16M 11972 R 3.8 1.0 0:30 0 /usr/local/apache/bin/httpd -DSSL
1582 nobody 17 0 23752 16M 11972 R 3.8 1.0 0:39 0 /usr/local/apache/bin/httpd -DSSL
1583 nobody 17 0 23636 16M 11972 R 3.8 1.0 0:30 0 /usr/local/apache/bin/httpd -DSSL
1584 nobody 17 0 23748 16M 11972 R 3.8 1.0 0:38 0 /usr/local/apache/bin/httpd -DSSL
1588 nobody 17 0 23936 16M 11972 R 3.8 1.0 0:27 0 /usr/local/apache/bin/httpd -DSSL
1589 nobody 17 0 23660 16M 11972 R 3.8 1.0 0:32 0 /usr/local/apache/bin/httpd -DSSL
2948 nobody 17 0 23708 15M 11972 R 3.8 1.0 0:05 0 /usr/local/apache/bin/httpd -DSSL
3043 nobody 17 0 23764 16M 11972 R 3.8 1.0 0:05 0 /usr/local/apache/bin/httpd -DSSL
3119 nobody 17 0 23664 15M 11972 R 3.8 1.0 0:01 0 /usr/local/apache/bin/httpd -DSSL
2560 nobody 17 0 23632 15M 11972 R 3.5 1.0 0:22 0 /usr/local/apache/bin/httpd -DSSL
3117 nobody 17 0 23588 15M 11972 R 3.5 1.0 0:01 0 /usr/local/apache/bin/httpd -DSSL
3122 nobody 18 0 19368 11M 11972 R 3.5 0.7 0:00 0 /usr/local/apache/bin/httpd -DSSL
1570 nobody 17 0 23656 16M 11972 R 3.2 1.0 0:19 0 /usr/local/apache/bin/httpd -DSSL
1572 nobody 17 0 23684 16M 11972 R 3.2 1.0 0:35 0 /usr/local/apache/bin/httpd -DSSL
1580 nobody 17 0 23748 16M 11972 R 3.2 1.0 0:31 0 /usr/local/apache/bin/httpd -DSSL
1587 nobody 17 0 23688 16M 11972 R 3.2 1.0 0:27 0 /usr/local/apache/bin/httpd -DSSL
2503 nobody 17 0 23712 16M 11972 R 3.2 1.0 0:19 0 /usr/local/apache/bin/httpd -DSSL
2897 nobody 17 0 23768 16M 11972 R 3.2 1.0 0:14 0 /usr/local/apache/bin/httpd -DSSL
3105 nobody 17 0 23676 15M 11972 R 3.2 1.0 0:02 0 /usr/local/apache/bin/httpd -DSSL
3115 nobody 17 0 23584 15M 11972 R 3.2 1.0 0:02 0 /usr/local/apache/bin/httpd -DSSL
3111 nobody 17 0 23596 15M 11972 R 2.8 1.0 0:01 0 /usr/local/apache/bin/httpd -DSSL
3116 nobody 16 0 18960 11M 11972 S 1.6 0.7 0:01 0 /usr/local/apache/bin/httpd -DSSL
2559 root 17 0 4144 1192 3812 R 1.2 0.0 0:41 0 top
3131 nobody 15 0 17012 9696 11976 S 1.2 0.6 0:00 0 /usr/local/apache/bin/httpd -DSSL
3129 nobody 15 0 16644 9316 11972 S 0.9 0.6 0:00 0 /usr/local/apache/bin/httpd -DSSL
3130 nobody 15 0 17032 9676 11988 S 0.6 0.6 0:00 0 /usr/local/apache/bin/httpd -DSSL
1573 nobody 15 0 19820 12M 11976 S 0.3 0.8 0:23 0 /usr/local/apache/bin/httpd -DSSL
1578 nobody 15 0 21896 14M 11984 S 0.3 0.9 0:30 0 /usr/local/apache/bin/httpd -DSSL
1712 mysql 16 0 161M 35M 3756 S 0.3 2.3 0:00 0 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --us
2511 root 16 0 7028 2160 6388 S 0.3 0.1 0:00 0 sshd: root@pts/0
2625 mailnull 16 0 113M 109M 6120 S 0.3 7.2 0:14 0 /usr/bin/perl -I/usr/mailscanner/lib /usr/mai

[NWSTech]

you have a lot of ppl connecting to a secure site for some reason ?

[jNive]

try temporaily turning off keep-alives and reducing the max spare servers/max servers, and restart apache, see if that helps. also you can put limits to apache's memory and cpu usage i believe

[VapoRub]

try

netstat | grep SYN

see if you get any results in large amounts

[C.P]

Originally posted by jNive
try temporaily turning off keep-alives and reducing the max spare servers/max servers, and restart apache, see if that helps. also you can put limits to apache's memory and cpu usage i believe

Server and mysql has already been optimize. Usually at this time of hour CPU usage is below 1 or just a bit above 1.

right now is bouncing in an odd way. One minute it's around 1-2 CPU then next minute it jumps to 10-20 CPU.

This is normal:

03:01:05 up 1:38, 1 user, load average: 1.85, 4.86, 5.90
147 processes: 143 sleeping, 1 running, 2 zombie, 1 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 77.4% 0.0% 7.6% 0.0% 0.9% 0.0% 13.9%
Mem: 1546620k av, 1440308k used, 106312k free, 0k shrd, 43212k buff
1108220k active, 283944k inactive
Swap: 2097136k av, 0k used, 2097136k free 628628k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
3184 nobody 17 0 18544 12M 9900 S 65.6 0.8 0:46 0 /usr/local/apache/bin/httpd -DSSL
3175 nobody 15 0 17720 12M 9900 S 9.1 0.8 0:51 0 /usr/local/apache/bin/httpd -DSSL
4521 root 17 0 4144 1192 3812 R 3.8 0.0 0:09 0 top
4429 nobody 15 0 16628 11M 9900 S 0.9 0.7 0:07 0 /usr/local/apache/bin/httpd -DSSL
4571 nobody 15 0 16980 11M 9900 S 0.9 0.7 0:00 0 /usr/local/apache/bin/httpd -DSSL
33 root 15 0 0 0 0 SW 0.4 0.0 0:00 0 pdflush
3171 nobody 15 0 21212 15M 9900 S 0.4 1.0 0:25 0 /usr/local/apache/bin/httpd -DSSL
3179 nobody 15 0 18320 12M 9900 S 0.4 0.8 0:48 0 /usr/local/apache/bin/httpd -DSSL
3524 nobody 15 0 18304 12M 9900 S 0.4 0.8 0:40 0 /usr/local/apache/bin/httpd -DSSL
3649 nobody 15 0 17472 11M 9900 S 0.4 0.7 0:22 0 /usr/local/apache/bin/httpd -DSSL
4555 nobody 16 0 19300 13M 9900 S 0.4 0.9 0:04 0 /usr/local/apache/bin/httpd -DSSL
1 root 16 0 1504 504 1352 S 0.0 0.0 0:01 0 init [3]
2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
3 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
4 root 5 -10 0 0 0 SW< 0.0 0.0 0:00 0 events/0
5 root 13 -10 0 0 0 SW< 0.0 0.0 0:00 0 khelper
6 root 14 -10 0 0 0 SW< 0.0 0.0 0:00 0 kacpid
21 root 5 -10 0 0 0 SW< 0.0 0.0 0:00 0 kblockd/0
32 root 20 0 0 0 0 SW 0.0 0.0 0:00 0 pdflush
35 root 10 -10 0 0 0 SW< 0.0 0.0 0:00 0 aio/0
34 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 kswapd0
36 root 10 -10 0 0 0 SW< 0.0 0.0 0:00 0 xfslogd/0
37 root 5 -10 0 0 0 SW< 0.0 0.0 0:00 0 xfsdatad/0
38 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 xfsbufd
131 root 6 -10 0 0 0 SW< 0.0 0.0 0:00 0 ata/0
134 root 16 0 0 0 0 SW 0.0 0.0 0:00 0 kseriod
165 root 15 0 0 0 0 SW 0.0 0.0 0:01 0 kjournald
467 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 khubd
944 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
1344 root 15 0 1568 572 1400 S 0.0 0.0 0:00 0 syslogd -m 0
1348 root 15 0 1512 464 1348 S 0.0 0.0 0:00 0 klogd -x
1394 root 17 0 3640 1564 3420 S 0.0 0.1 0:00 0 /usr/sbin/sshd
1406 root 16 0 2128 936 1788 S 0.0 0.0 0:00 0 xinetd -stayalive -pidfile /var/run/xinetd.pid
1425 root 15 0 6104 2760 4704 S 0.0 0.1 0:00 0 chkservd
1479 mailnull 16 0 6604 1876 6144 S 0.0 0.1 0:00 0 /usr/sbin/exim -bd -q60m

Then a few seconds later like 30 seconds it jumps to this:

03:01:59 up 1:39, 1 user, load average: 10.04, 6.16, 6.26
159 processes: 129 sleeping, 27 running, 2 zombie, 1 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 94.1% 0.0% 4.3% 0.3% 1.2% 0.0% 0.0%
Mem: 1546620k av, 1522068k used, 24552k free, 0k shrd, 43316k buff
1189680k active, 283648k inactive
Swap: 2097136k av, 0k used, 2097136k free 630292k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
4343 nobody 16 0 21540 15M 9900 R 5.5 1.0 0:07 0 /usr/local/apache/bin/httpd -DSSL
1924 mysql 16 0 166M 39M 3756 R 5.2 2.6 0:12 0 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --us
3186 nobody 16 0 21588 15M 9900 R 5.2 1.0 0:47 0 /usr/local/apache/bin/httpd -DSSL
3164 nobody 16 0 21572 15M 9900 R 4.9 1.0 0:39 0 /usr/local/apache/bin/httpd -DSSL
3166 nobody 16 0 21616 15M 9900 R 4.9 1.0 0:46 0 /usr/local/apache/bin/httpd -DSSL
3172 nobody 16 0 21544 15M 9900 R 4.9 1.0 0:26 0 /usr/local/apache/bin/httpd -DSSL
3177 nobody 16 0 21680 15M 9900 R 4.9 1.0 0:44 0 /usr/local/apache/bin/httpd -DSSL
3181 nobody 16 0 21584 15M 9900 R 4.9 1.0 0:30 0 /usr/local/apache/bin/httpd -DSSL
3524 nobody 16 0 21556 15M 9900 R 4.9 1.0 0:40 0 /usr/local/apache/bin/httpd -DSSL
4327 nobody 16 0 21612 15M 9900 R 4.9 1.0 0:09 0 /usr/local/apache/bin/httpd -DSSL
4430 nobody 16 0 21568 15M 9900 R 4.9 1.0 0:10 0 /usr/local/apache/bin/httpd -DSSL
4547 nobody 16 0 21488 15M 9900 R 4.9 1.0 0:00 0 /usr/local/apache/bin/httpd -DSSL
4555 nobody 16 0 21472 15M 9900 R 4.9 1.0 0:05 0 /usr/local/apache/bin/httpd -DSSL
3176 nobody 16 0 21572 15M 9900 R 4.6 1.0 0:36 0 /usr/local/apache/bin/httpd -DSSL
3178 nobody 16 0 21552 15M 9900 R 4.6 1.0 0:40 0 /usr/local/apache/bin/httpd -DSSL
3649 nobody 16 0 21540 15M 9900 R 4.6 1.0 0:23 0 /usr/local/apache/bin/httpd -DSSL
3173 nobody 16 0 21608 15M 9900 R 4.3 1.0 0:37 0 /usr/local/apache/bin/httpd -DSSL
1862 mysql 15 0 166M 39M 3756 S 2.7 2.6 0:12 0 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --us
3185 nobody 16 0 20604 15M 9900 R 2.1 0.9 0:47 0 /usr/local/apache/bin/httpd -DSSL
4521 root 16 0 4144 1196 3812 R 2.1 0.0 0:10 0 top
3175 nobody 16 0 17724 12M 9900 R 1.8 0.8 0:51 0 /usr/local/apache/bin/httpd -DSSL
3187 nobody 15 0 17452 11M 9900 S 0.6 0.7 0:41 0 /usr/local/apache/bin/httpd -DSSL
4769 nobody 15 0 14916 9532 9900 S 0.6 0.6 0:00 0 /usr/local/apache/bin/httpd -DSSL
4770 nobody 15 0 11452 5944 9900 S 0.6 0.3 0:00 0 /usr/local/apache/bin/httpd -DSSL
4771 nobody 15 0 11392 5860 9900 S 0.6 0.3 0:00 0 /usr/local/apache/bin/httpd -DSSL
2442 mailnull 15 0 113M 109M 6120 S 0.3 7.2 0:21 0 /usr/bin/perl -I/usr/mailscanner/lib /usr/mailscanner/bin/
2444 mailnull 15 0 113M 109M 6120 S 0.3 7.2 0:20 0 /usr/bin/perl -I/usr/mailscanner/lib /usr/mailscanner/bin/
3171 nobody 16 0 21212 15M 9900 S 0.3 1.0 0:25 0 /usr/local/apache/bin/httpd -DSSL
3183 nobody 15 0 17488 11M 9900 S 0.3 0.7 0:33 0 /usr/local/apache/bin/httpd -DSSL
4429 nobody 15 0 18300 12M 9900 S 0.3 0.8 0:07 0 /usr/local/apache/bin/httpd -DSSL
4774 nobody 15 0 11264 5368 9900 S 0.3 0.3 0:00 0 /usr/local/apache/bin/httpd -DSSL
1 root 15 0 1504 504 1352 S 0.0 0.0 0:01 0 init [3]
2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
3 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
4 root 5 -10 0 0 0 SW< 0.0 0.0 0:00 0 events/0

It keeps going back and forth like that all night today. What the hell is going on?

[dollar]

It might help to install mod-status on apache and view the status page when the load gets high like that.

[mistwang]

Maybe your web site was hitting by a spider.

[zma229]

Hmm, first of all I would recommend mailscanner in a busy server. It will simply take load. Second suggestion is to try to install mod_dos also write some smart scritps to find the no of connections to apache using netstat. Also check for domlogs size. You may find some users with high domlogs size menas highest no of hits.

[C.P]

I've been able to find the cause of this. Well not actually the cause but where the high CPU came from.

The high CPU usage came directly from the forum. As stated I haven't touch the forum or made any changes to the forum. Http request and CPU usage has been normal for months until a couple of nights ago.

What I did to get CPU back to normal temporary was turn the forum from a public viewing forum to one that forces everyone to register or log in to view the forum. If you are a guest you won't be able to view the forum unless you log in.
This brought CPU back to normal.

What's the SSH command that you can view the latest visitors, IPs and the page that they are requesting in SSH? I remember a long time ago you can view the latest visitors and what page is being requested through SSH.

Also is this normal?

32428 mailnull 16 0 113M 109M 6120 S 0.4 7.2 0:27 0 /usr/bin/perl -I/usr/mailscanner/lib /usr/mailscanner/bin/
32489 mailnull 16 0 113M 109M 6120 S 0.4 7.2 0:26 0 /usr/bin/perl -I/usr/mailscanner/lib /usr/mailscanner/bin/

I notice that mailscanner seem to be running non stop in top.

[jsquires]

Your forum was probably being indexed by one of the many search engines. If you want part of your site to be listed I would suggest making a small or less used forum publicly available and make all other require a log in as you've done.

I had a lot of issues with the same thing when a member posted a picture on one of my forums and we got a ton of people googling for the title and coming from there.

After that google started hammering my server for a few weeks. Now I make sure that he public forum has very little within it to index.