Web Hosting Talk : Web Hosting Main Forums : Hosting Security and Technology : Need help pinpointing a problem I had on my server

[Darvil]

Well just a while ago I was on cpanel doing something and decided to backup my forum database (this is not how I normally backup). Anyway while I was downloading the file. In a min or 2 the download stopped and I started getting IMs from people telling me that the forum was down.

The site was literally down. I manage to get in thru shell to see what was going on. This is what top told me when I ran the command.

top - 23:43:11 up 6 days, 3:29, 1 user, load average: 91.92, 92.91, 92.55
Tasks: 500 total, 1 running, 498 sleeping, 1 stopped, 0 zombie
Cpu(s): 4.2% us, 2.8% sy, 0.0% ni, 0.5% id, 92.5% wa, 0.0% hi, 0.0% si
Mem: 2074928k total, 2060484k used, 14444k free, 1100k buffers
Swap: 2096472k total, 1191716k used, 904756k free, 18372k cached

At first I thought I got hacked/dos/blah/blah

When I look at the process I noticed a MYSQL process that was using 97 percent of the CPU. I killed the process then restarted MYSQL service. Everything was back to normal after I did that.

top - 00:33:23 up 6 days, 4:19, 1 user, load average: 0.46, 3.68, 31.92
Tasks: 151 total, 1 running, 150 sleeping, 0 stopped, 0 zombie
Cpu(s): 7.0% us, 0.5% sy, 0.0% ni, 89.5% id, 3.0% wa, 0.0% hi, 0.0% si
Mem: 2074928k total, 844092k used, 1230836k free, 25924k buffers
Swap: 2096472k total, 444316k used, 1652156k free, 365472k cached

man that process really ate up the ram too.


Now here is where I'm not sure whats going on. I've recently updated cpanel to its latest updates. My suspicion is that it was the database backup download (thru cpanel) which started this whole thing that brought my server down to its knee. Is that remotely even a possibility? It doesn't seem to make sense. And if it is how can I resolve this issue? I don't want the server to crash if some of my friends decided to backup their forums (although its most likely me doing this).

If it wasn't what I think it is (which is more likely the possibility), what could it be?

I also ran the rootkit check and it didn't find anything.

Any ideas?


Thanks

[Techark]

How big is the database? Would have helped if you had captured the process and command line it was running so we could see if it was a mysql dump that was doing it or what.

[Darvil]

Hey Techart

That was a mistake I did. The database is 42 megs (gz).

It was /sbin/ directory. Is that a dump?

[ServerNinja]

Originally Posted by Darvil

Hey Techart

That was a mistake I did. The database is 42 megs (gz).

It was /sbin/ directory. Is that a dump?

Do you mean the process that consume huge memory was in /sbin directory??? Mysql dump wont be taken under the /sbin folder. Are you able to post the top results here so that we can have a look

[Darvil]

umm I was checking top today and the ram usage abnormally high.

I think perhaps me doing the backup might have tipped it over the edge. But I have no idea why the ram usage is so high. It doesn't make sense. I didn't have this problem before.

Here's the top result I just did



top - 16:18:00 up 6 days, 20:05, 1 user, load average: 0.92, 1.36, 1.33
Tasks: 200 total, 1 running, 199 sleeping, 0 stopped, 0 zombie
Cpu(s): 20.6% us, 1.8% sy, 0.0% ni, 75.0% id, 2.5% wa, 0.0% hi, 0.0% si
Mem: 2074928k total, 1904732k used, 170196k free, 30540k buffers
Swap: 2096472k total, 337336k used, 1759136k free, 1169384k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22922 nobody 15 0 56304 14m 2204 S 3 0.7 2:08.79 /usr/local/apache/bin/httpd
24746 nobody 15 0 54572 12m 2204 S 3 0.6 1:31.00 /usr/local/apache/bin/httpd
27276 nobody 15 0 56608 14m 2180 S 3 0.7 0:50.50 /usr/local/apache/bin/httpd
20573 root 16 0 22196 4520 876 S 3 0.2 20:18.85 ./sc_trans_linux
24893 nobody 15 0 54800 13m 2196 S 3 0.6 1:20.99 /usr/local/apache/bin/httpd
25266 nobody 15 0 54428 12m 2204 S 3 0.6 1:29.02 /usr/local/apache/bin/httpd
30334 nobody 15 0 53284 11m 2120 S 3 0.6 0:03.03 /usr/local/apache/bin/httpd
24459 nobody 15 0 56356 14m 2192 S 2 0.7 1:26.41 /usr/local/apache/bin/httpd
27937 nobody 15 0 56692 14m 2188 S 2 0.7 0:40.86 /usr/local/apache/bin/httpd
29758 nobody 15 0 55036 13m 2188 S 2 0.7 0:11.87 /usr/local/apache/bin/httpd
30474 nobody 17 0 52920 10m 2012 S 2 0.5 0:00.29 /usr/local/apache/bin/httpd
30486 nobody 17 0 52988 10m 2044 S 2 0.5 0:00.06 /usr/local/apache/bin/httpd
30488 nobody 15 0 53016 10m 1980 S 2 0.5 0:00.06 /usr/local/apache/bin/httpd
30485 nobody 15 0 53016 10m 1980 S 2 0.5 0:00.05 /usr/local/apache/bin/httpd
30487 nobody 15 0 49204 7220 1720 S 2 0.3 0:00.05 /usr/local/apache/bin/httpd
19244 mysql 15 0 177m 75m 3688 S 1 3.7 8:02.76 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/sar
16914 nobody 15 0 56576 14m 2184 S 1 0.7 1:28.50 /usr/local/apache/bin/httpd
26221 nobody 15 0 56364 14m 2204 S 1 0.7 1:10.12 /usr/local/apache/bin/httpd
28509 nobody 15 0 54440 12m 2184 S 1 0.6 0:32.59 /usr/local/apache/bin/httpd
29239 nobody 15 0 53624 11m 2188 S 1 0.6 0:26.87 /usr/local/apache/bin/httpd
29791 nobody 15 0 56144 14m 2164 S 1 0.7 0:14.84 /usr/local/apache/bin/httpd
30268 nobody 16 0 53552 11m 2132 S 1 0.6 0:03.01 /usr/local/apache/bin/httpd
30335 nobody 15 0 51276 9632 2060 S 1 0.5 0:01.92 /usr/local/apache/bin/httpd
30455 nobody 15 0 51280 9608 2032 S 1 0.5 0:00.54 /usr/local/apache/bin/httpd
30483 nobody 15 0 51280 9584 2012 S 1 0.5 0:00.08 /usr/local/apache/bin/httpd
30484 nobody 15 0 51252 9464 2012 S 1 0.5 0:00.08 /usr/local/apache/bin/httpd
4223 named 19 0 52544 5832 1444 S 0 0.3 16:23.45 /usr/sbin/named -u named
5695 root 16 0 46128 3484 1496 S 0 0.2 0:42.62 /usr/local/apache/bin/httpd
23291 nobody 15 0 56720 14m 2192 S 0 0.7 2:01.18 /usr/local/apache/bin/httpd
23323 nobody 15 0 56440 14m 2208 S 0 0.7 1:56.75 /usr/local/apache/bin/httpd
27908 nobody 15 0 54348 12m 2204 S 0 0.6 0:44.12 /usr/local/apache/bin/httpd
28016 nobody 15 0 54652 12m 2196 S 0 0.6 0:36.09 /usr/local/apache/bin/httpd
30482 nobody 15 0 51280 9584 2012 S 0 0.5 0:00.07 /usr/local/apache/bin/httpd
1 root 16 0 1868 468 436 S 0 0.0 0:04.11 init [3]


Any ideas why its using so much ram?

BTW the ./sc_trans_linux is a radio bot I run.


thanks

[ServerNinja]

A lot of apache processes

Can you please run the command
netstat -plan|grep :25|awk {'print $5'}|cut -d: -f 1|sort|uniq -c|sort -nk 1

and paste the results here? I doubts its a kinda DOS attack

[Darvil]

I don't think its a dos either

here's what I got

1 0.0.0.0
1 24.23.1.227
2 69.110.136.72
2 69.160.85.120
4 207.224.94.80
5 91.164.89.53

[ServerNinja]

hmmm... it shows a lot of apache processes on the server. can you try enabling phpsuexec to find out the "nobody" culprit?