i have a raq3i, and now and then (like twice a week) i receive 2 emails. one:

Over the past fifteen minutes, the CPU has been heavily loaded.

This will result in noticible performace loss. Consider moving some of the services to other Cobalt servers, or reduce the complexity of the CGI scripts running on the Cobalt server itself.

1 minute load average: 14.13
5 minute load average: 12.94
15 minute load average: 5.76


and the second is that the smtp service is shut down because of heavy load.

I have only 1 small site running on that server, and it has 512 mb ram. Whats the problem? How can i trace it? I really dunno whats up.. i dont think it can be a script because its only like twice a week, and all scripts on there run at least once a day.

Do you have formmail installed on your server by chance?

nope

Do a "top" and post it here, it might help us a bit more...;)

Okay, here ya go;

3:46am up 7 days, 22:01, 1 user, load average: 0.06, 0.33, 1.89
52 processes: 51 sleeping, 1 running, 0 zombie, 0 stopped
CPU states: 1.5% user, 0.7% system, 0.0% nice, 97.6% idle
Mem: 128040K av, 74328K used, 53712K free, 43580K shrd, 5124K buff
Swap: 131536K av, 12688K used, 118848K free 43644K cached

PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND
30333 admin 12 0 1052 1052 864 R 0 0.7 0.8 0:00 top
30004 httpd 1 0 4520 1568 1304 S 0 0.1 1.2 0:00 httpd
30335 httpd 1 0 4500 1544 1292 S 0 0.1 1.2 0:00 httpd
30337 httpd 2 0 4500 1544 1292 S 0 0.1 1.2 0:00 httpd
30338 httpd 2 0 4500 1544 1292 S 0 0.1 1.2 0:00 httpd
30339 httpd 2 0 4500 1544 1292 S 0 0.1 1.2 0:00 httpd
30341 httpd 2 0 4488 1532 1276 S 0 0.1 1.1 0:00 httpd
30342 httpd 3 0 4504 1548 1292 S 0 0.1 1.2 0:00 httpd
1 root 0 0 120 68 52 S 0 0.0 0.0 0:05 init
2 root 0 0 0 0 0 SW 0 0.0 0.0 0:02 kflushd
3 root 0 0 0 0 0 SW 0 0.0 0.0 0:09 kupdate
4 root 0 0 0 0 0 SW 0 0.0 0.0 0:00 kpiod
5 root 0 0 0 0 0 SW 0 0.0 0.0 0:04 kswapd
6 root -20 -20 0 0 0 SW< 0 0.0 0.0 0:00 mdrecoveryd
82 root 0 0 136 0 0 SW 0 0.0 0.0 0:00 safe_mysqld
123 mysql 0 0 1664 1320 840 S 0 0.0 1.0 0:12 mysqld
126 root 0 0 208 156 132 S 0 0.0 0.1 0:18 syslogd
135 root 0 0 392 0 0 SW 0 0.0 0.0 0:00 klogd

Looks 100% normal. Monitor your RaQ, and when the load gets high again, post THAT top output here.

You should do a ps -aux

Right at the time when your usage is high.
Look for cpu + ram and lenght of time of the process it has been running.

The kill the PID.

If it doesn kill,

type kill -9 XXXXXXXXX

xxx as the number of the pid.

Look for strange activities also. ie logs........

the thing is, when i receive the email, i go to my server, and its normal :?

Thats quite puzzling.

Only twice a week.

hmm......................
antyhing in the crons that only once twice a week?

Nope, only daily...


i havent had an email since i posted this btw... :rolleyes:


im going on a holiday today for 15 days so i wont be able to reply here untill i get back, sorry bout that

Originally posted by Cephren
You should do a ps -aux

Right at the time when your usage is high.
Look for cpu + ram and lenght of time of the process it has been running.

The kill the PID.

If it doesn kill,

type kill -9 XXXXXXXXX

xxx as the number of the pid.

Look for strange activities also. ie logs........

ok i did that, but i cant find anything weird.
here is the top at the time the server load is high(now);


5:47am up 36 days, 17:00, 1 user, load average: 21.17, 17.97, 14.16
89 processes: 50 sleeping, 37 running, 2 zombie, 0 stopped
CPU states: 50.8% user, 33.9% system, 0.0% nice, 15.1% idle
Mem: 128040K av, 89760K used, 38280K free, 152616K shrd, 2900K buff
Swap: 131536K av, 13008K used, 118528K free 11352K cached

PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND
15889 httpd 6 0 5812 4064 2956 S 0 2.3 3.1 0:00 httpd
14503 httpd 18 0 5292 3464 2408 R 0 1.7 2.7 0:00 httpd
15574 httpd 17 0 0 0 0 Z 0 1.7 0.0 0:00 httpd <defunct>
16800 admin 11 0 1088 1088 864 R 0 1.7 0.8 0:00 top
15531 httpd 19 0 5276 3448 2392 R 0 1.5 2.6 0:00 httpd
15573 httpd 16 0 5276 3448 2392 R 0 1.5 2.6 0:00 httpd
15577 httpd 16 0 5276 3420 2368 R 0 1.5 2.6 0:00 httpd
15750 httpd 16 0 5288 3460 2400 R 0 1.5 2.7 0:00 httpd
14468 httpd 18 0 5296 3468 3436 R 0 1.3 2.7 0:00 httpd
14469 httpd 18 0 5304 3480 2416 R 0 1.3 2.7 0:00 httpd
14623 httpd 15 0 5300 3476 2416 R 0 1.3 2.7 0:00 httpd
15298 httpd 16 0 5296 3472 2416 R 0 1.3 2.7 0:00 httpd
15575 httpd 18 0 5300 3468 2408 S 0 1.3 2.7 0:00 httpd
15578 httpd 18 0 5288 3460 2404 R 0 1.3 2.7 0:00 httpd
15612 httpd 15 0 5288 3460 2404 S 0 1.3 2.7 0:00 httpd
15613 httpd 18 0 5276 3440 2388 R 0 1.3 2.6 0:00 httpd
15652 httpd 18 0 5276 3428 2376 R 0 1.3 2.6 0:00 httpd
15698 httpd 15 0 5300 3472 2412 S 0 1.3 2.7 0:00 httpd
15702 httpd 16 0 5268 3416 2372 R 0 1.3 2.6 0:00 httpd
15752 httpd 15 0 5320 3472 2416 R 0 1.3 2.7 0:00 httpd
15753 httpd 18 0 5276 3424 2372 R 0 1.3 2.6 0:00 httpd
14617 httpd 16 0 0 0 0 Z 0 1.1 0.0 0:00 httpd <defunct>
14627 httpd 15 0 5300 3472 2412 R 0 1.1 2.7 0:00 httpd

and here is the ps-aux output:


and here is the ps -aux output:

[code]
[admin@www admin]$ ps -aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 1120 52 ? S Nov05 0:08 init
root 2 0.0 0.0 0 0 ? SW Nov05 0:20 [kflushd]
root 3 0.0 0.0 0 0 ? SW Nov05 1:22 [kupdate]
root 4 0.0 0.0 0 0 ? SW Nov05 0:00 [kpiod]
root 5 0.0 0.0 0 0 ? SW Nov05 1:28 [kswapd]
root 6 0.0 0.0 0 0 ? SW< Nov05 0:00 [mdrecoveryd]
root 82 0.0 0.0 1544 0 ? SW Nov05 0:00 [safe_mysqld]
mysql 119 0.0 1.1 17608 1496 ? S Nov05 0:40 /usr/sbin/mysqld
root 122 0.0 0.1 1172 156 ? S Nov05 0:48 syslogd -m 0
root 131 0.0 0.0 1484 0 ? SW Nov05 0:00 [klogd]
mysql 145 0.0 1.1 17608 1496 ? S Nov05 0:43 /usr/sbin/mysqld
mysql 146 0.0 1.1 17608 1496 ? S Nov05 0:03 /usr/sbin/mysqld
root 596 0.0 0.0 1156 120 ? S Nov05 0:02 crond
root 608 0.0 0.0 1136 56 ? S Nov05 0:05 inetd
root 617 0.0 0.0 6572 60 ? S Nov05 0:11 /usr/sbin/httpd -
postgres 653 0.0 0.3 5408 492 ? S Nov05 0:26 /usr/bin/postmast
root 678 0.0 0.0 8064 116 ? S Nov05 4:31 /usr/sbin/httpd -
root 733 0.0 0.7 3688 912 ? S Nov05 0:13 perl /usr/local/s
root 739 0.0 0.0 296 48 ? S Nov05 0:18 /sbin/lcdsleep
root 777 0.0 0.0 1136 0 ? SW Nov05 0:00 [getty]
root 22593 0.0 0.0 6920 44 ? S Dec11 0:00 /usr/sbin/httpd -
root 8482 0.0 0.2 2044 332 ? S 04:30 0:00 sendmail: rejecti
httpd 18688 0.2 1.9 8464 2516 ? S 05:28 0:02 /usr/sbin/httpd -
httpd 18834 0.2 1.7 8464 2276 ? S 05:29 0:02 /usr/sbin/httpd -
httpd 21869 0.2 1.6 8464 2120 ? S 05:31 0:02 /usr/sbin/httpd -
httpd 30759 0.0 0.1 10196 140 ? S 05:36 0:00 /usr/sbin/httpd -
httpd 30937 0.2 1.6 8464 2068 ? S 05:36 0:01 /usr/sbin/httpd -
root 31266 0.0 0.6 1556 772 ? S 05:36 0:00 in.telnetd
root 31480 0.0 0.8 2052 1128 pts/0 S 05:36 0:00 login -- admin
admin 31748 0.0 0.6 1556 880 pts/0 S 05:37 0:00 -bash
eriktc 4712 0.0 1.0 2292 1392 ? S 05:40 0:00 proftpd: eriktc -
httpd 5591 0.3 2.6 8464 3428 ? S 05:40 0:01 /usr/sbin/httpd -
httpd 7341 0.3 2.6 8464 3416 ? S 05:41 0:00 /usr/sbin/httpd -
httpd 8755 0.4 2.6 8464 3416 ? S 05:42 0:00 /usr/sbin/httpd -
httpd 9708 0.5 1.6 8464 2100 ? S 05:43 0:00 /usr/sbin/httpd -
httpd 9769 0.5 2.6 8464 3416 ? S 05:43 0:00 /usr/sbin/httpd -
httpd 10389 0.6 1.6 8464 2156 ? S 05:43 0:00 /usr/sbin/httpd -
httpd 11181 0.4 2.6 8464 3440 ? S 05:43 0:00 /usr/sbin/httpd -
httpd 11219 0.6 0.0 8464 0 ? SW 05:43 0:00 [httpd]
httpd 11256 0.6 0.0 8464 0 ? SW 05:43 0:00 [httpd]
httpd 11284 0.8 1.6 8464 2144 ? S 05:43 0:00 /usr/sbin/httpd -
httpd 11333 0.5 0.7 8464 936 ? S 05:43 0:00 /usr/sbin/httpd -
httpd 11369 0.8 0.6 8464 864 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 11373 0.8 1.6 8464 2148 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 11388 0.7 2.6 8464 3420 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 11390 0.5 0.0 8464 0 ? SW 05:44 0:00 [httpd]
httpd 11391 0.8 1.6 8464 2160 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 11437 0.8 1.1 8464 1524 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 11438 0.6 2.6 8464 3432 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 11440 0.8 2.6 8464 3432 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 11443 0.8 2.6 8468 3448 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 11444 0.8 2.6 8464 3432 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 11449 0.6 2.6 8464 3432 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 11538 0.6 2.6 8464 3444 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 11571 0.5 2.6 8464 3440 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 11590 0.5 2.6 8464 3432 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 11593 0.7 1.6 8528 2140 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12758 1.0 2.6 8464 3420 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12790 1.0 2.6 8464 3416 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12791 1.0 2.6 8464 3408 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12827 1.0 2.6 8468 3432 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12828 0.6 2.6 8464 3428 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12829 1.0 2.6 8464 3392 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12830 0.9 2.6 8464 3428 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12854 1.4 2.6 8464 3432 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12855 1.5 2.6 8464 3416 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12856 1.0 2.6 8464 3428 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12857 0.9 2.6 8464 3428 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12858 1.0 2.6 8464 3408 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12859 0.9 2.6 8468 3428 ? R 05:44 0:00 /usr/sbin/httpd -
httpd 12860 0.8 2.6 8464 3408 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12861 1.3 2.6 8464 3440 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12887 1.4 2.6 8464 3444 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12888 0.9 2.6 8468 3432 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12924 1.0 2.6 8464 3428 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12925 0.6 2.6 8464 3412 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12926 1.2 2.6 8464 3428 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12927 1.8 2.6 8464 3396 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12928 1.7 2.6 8464 3396 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12929 1.7 1.6 8528 2148 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12930 1.7 2.6 8464 3396 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12931 1.0 2.6 8464 3428 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12949 1.0 2.6 8464 3416 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12950 1.3 2.6 8464 3396 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12951 1.4 2.6 8464 3396 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12952 1.5 2.6 8464 3396 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 12953 1.4 2.6 8464 3444 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 13003 1.7 2.6 8464 3396 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 13038 1.5 2.6 8464 3396 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 13039 1.0 2.6 8456 3416 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 13040 1.7 1.6 8476 2148 ? D 05:44 0:00 /usr/sbin/httpd -
httpd 13041 1.5 2.6 8464 3396 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 13042 1.2 2.6 8464 3428 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 13043 1.2 2.6 8464 3440 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 13044 1.4 2.6 8464 3396 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 13045 1.5 2.6 8464 3396 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 13046 0.8 2.6 8464 3424 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 13047 1.5 2.6 8464 3396 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 13048 1.4 2.6 8464 3396 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 13091 0.9 2.6 8464 3416 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 13092 1.6 2.6 8464 3392 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 13093 1.0 2.6 8468 3432 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 13094 1.5 2.6 8464 3392 ? S 05:44 0:00 /usr/sbin/httpd -
httpd 13095 1.4 2.6 8464 3428 ? S 05:44 0:00 /usr/sbin/httpd -
root 13117 0.0 0.3 1156 396 ? S 05:44 0:00 CROND
root 13121 0.0 0.5 1516 664 ? S 05:44 0:00 /bin/bash -c /usr
root 13124 0.6 0.7 1612 964 ? S 05:45 0:00 /usr/local/sbin/s
postgres 13170 1.9 1.9 6056 2504 ? S 05:45 0:00 /usr/bin/postgres
httpd 13182 0.2 1.2 8096 1564 ? S 05:45 0:00 /usr/sbin/httpd -
httpd 13183 1.8 2.6 8468 3396 ? S 05:45 0:00 /usr/sbin/httpd -
httpd 13184 0.0 1.1 8080 1488 ? S 05:45 0:00 /usr/sbin/httpd -
httpd 13185 1.1 2.6 8464 3416 ? S 05:45 0:00 /usr/sbin/httpd -
httpd 13186 1.5 2.6 8464 3392 ? S 05:45

(its too long for 1 reply)

part 2 of ps-aux:


0:00 /usr/sbin/httpd -
httpd 13187 1.7 2.6 8464 3392 ? S 05:45 0:00 /usr/sbin/httpd -
httpd 13276 1.7 2.6 8464 3392 ? S 05:45 0:00 /usr/sbin/httpd -
httpd 13277 1.3 2.6 8464 3392 ? S 05:45 0:00 /usr/sbin/httpd -
httpd 13278 1.3 2.6 8464 3412 ? S 05:45 0:00 /usr/sbin/httpd -
httpd 13279 1.9 2.6 8464 3392 ? S 05:45 0:00 /usr/sbin/httpd -
httpd 13280 2.0 2.6 8464 3392 ? S 05:45 0:00 /usr/sbin/httpd -
httpd 13281 1.5 2.6 8464 3392 ? S 05:45 0:00 /usr/sbin/httpd -
httpd 13282 2.0 0.0 0 0 ? Z 05:45 0:00 [httpd <defunct>]
httpd 13363 2.2 2.6 8464 3388 ? S 05:45 0:00 /usr/sbin/httpd -
httpd 13364 1.8 2.6 8464 3388 ? S 05:45 0:00 /usr/sbin/httpd -
httpd 13539 6.0 2.6 8464 3408 ? S 05:45 0:00 /usr/sbin/httpd -
admin 13660 0.0 0.6 2520 880 pts/0 R 05:45 0:00 ps -aux
httpd 13727 0.0 2.6 8464 3396 ? R 05:45 0:00 /usr/sbin/httpd -
httpd 13728 0.0 1.6 8464 2164 ? R 05:45 0:00 /usr/sbin/httpd -

it looks like im just getting a lot of http requests at the same time, but when i look at my tracker stats the server only gets like 1200 uniques a day :confused:

holy ****, my log files are huge, some are 400 mb!!
how can i watch these, lol
i cant download all the 400 mb logs..

wow look at this log (just a few last lines) i think im getting attacked or something?

wtf can i do against this, they are all different ips :/

ok the server just stoped responding, but i got it rebooted and now it seems to be working fine again. I deleted the 400mb log file, and now its not loggin anymore..heh..what can i do about that?
The server is working fine though :)

damnit, now its messed up again..heavy server load :/

ok i now know what site it is...its site8, because now ihave suspended it and the server load is gone.

i cant delete that site, its like 5gb big :/ (mainly pics and movies)
maybe i can move that and then delete it..
hmm

ok i know whats going on, im getting brute force-attacked! :( :(

im gonna figure out what i can do about it now..

I think I am going through the same thing now, so if you figure it out, please post it. I've got my tech looking into my situation and if I make any ground, I'll let you know.

Good luck! Take a few Tylenol!

----- Jeremy ----

That is not a 'brute force' attack - it's Code Red or NIMDA or similar...only affects MS platforms...so breathe easy.

Brandon

Hello there,

The attack you are getting there is the Unicode-Exploit attack. Only Windows machines are vulnurable, so take it easy. And for the logs, make it rotate some more often.

Robin van Duiven