Results 1 to 25 of 28
Thread: Strange Reading
-
05-27-2006, 10:05 AM #1Junior Guru
- Join Date
- May 2006
- Posts
- 237
Strange Reading
Hey all,
I have a strange reading showing up on the netstat readout for my linux box. It shows one of our nameservers connected to ede.nl.eu.undernet.org:ircd in a Close_Wait status.
Anyone know why this would be, and what ( if anything ) I should do about it?
It has only showed up recently, and so I am kind of puzzled.
Thanks.
-
05-27-2006, 02:42 PM #2Newbie
- Join Date
- May 2006
- Posts
- 24
Do you allow shell access to others? Remote connections through PHP/Perl? If not, then try to find which process it's coming from, and run some rootkit detection programs. I'd recommend running them anyways.
-
05-27-2006, 02:46 PM #3Junior Guru
- Join Date
- May 2006
- Posts
- 237
I don't allow shell access and I don't think I have ever set anything for or against remote connections with PHP or Perl. Where can I look to see if there are remote connections allowed?
-
05-27-2006, 03:08 PM #4Newbie
- Join Date
- May 2006
- Posts
- 24
If you can do a netstat -np you could possibly see the program using it.
-
05-27-2006, 08:43 PM #5Junior Guru
- Join Date
- May 2006
- Posts
- 237
My original netstat:
root@enterprise [~]# netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 1064 0 ns1.*.com:36014 ede.nl.eu.undernet.org:ircd CLOSE_WAIT
root@enterprise [~]# netstat -np
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 1064 0 XXX.XXX.XX.XXX:36014 193.109.122.67:6667 CLOSE_WAIT 1491/[kjournald]
tcp 187 0 XXX.XXX.XX.XXX:59671 193.109.122.67:6667 CLOSE_WAIT 1491/[kjournald]
-
05-27-2006, 09:23 PM #6Newbie
- Join Date
- May 2006
- Posts
- 24
There's a possiblility it's been compromised. Load up rkhunter and run it, and maybe a few others and see if it picks up anything.
-
05-27-2006, 10:56 PM #7Web Hosting Guru
- Join Date
- May 2005
- Posts
- 288
kjournald with pid 1491. Well, not impossible, but not likely either.
Sure looks like some process masq'ing as kjournald.
Maybe a look at /proc/1491 can give you a few tips ? /proc/1491/fd might also contain
interesting information.
-
05-27-2006, 11:12 PM #8Problem Solver
- Join Date
- Mar 2003
- Location
- California USA
- Posts
- 13,681
Looks like you have a psybnc or something like that running. Follow the suggestion about looking in /proc. If that is the case you need to look into securing your server more.
Steven Ciaburri | Industry's Best Server Management - Rack911.com
Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance
-
05-28-2006, 01:21 AM #9Junior Guru
- Join Date
- May 2006
- Posts
- 237
It seemed to be running multiple instances under the same PID . I simply killed it and it hasn't returned in the netstat readouts.
Thanks for your help guys Greatly appreciated.
-
05-28-2006, 02:00 AM #10Web Hosting Guru
- Join Date
- May 2005
- Posts
- 288
You really should give the machine a reboot to see if it comes back, and also to make sure the _correct_ kjournald is running.
-
05-28-2006, 02:15 AM #11Disabled
- Join Date
- Mar 2006
- Posts
- 30
maybe could be an: ircd or psybnc, something about the world of irc..
Look in your /var/tmp, and see if you have got something of strange..
-
05-28-2006, 04:43 PM #12Junior Guru
- Join Date
- May 2006
- Posts
- 237
How can I be sure the correct kjournald is running?
-
05-28-2006, 04:58 PM #13Junior Guru
- Join Date
- May 2006
- Posts
- 237
WHM shows one at the following:
PID 412 (kjournald)
PID 1441 (kjournald)
PID 1491 ([kjournald]) /usr/lib/sshdlib/[kjournald /usr/lib/sshdlib
[kjournald]
-
05-28-2006, 07:07 PM #14Web Hosting Guru
- Join Date
- May 2005
- Posts
- 288
Originally Posted by Calibur747
And yes, your host is compromised.. 412 is the real kjournald process.
You will also notice (if you use "ps auxww") that the real kjournald not only had a low pid, but it also has 0 RSS and VSZ.
/usr/lib/sshdlib/ ? Good bet you have some trace of the backdoor there, but not 100% sure. It can be anywhere.
So, in a nutshell, you are still compromised. And if the process has access to /usr/lib,
then you better rebuild your box from scratch, cause ... well, it is not your box anything. It is the hacker's.
-
05-29-2006, 12:35 AM #15Junior Guru
- Join Date
- May 2006
- Posts
- 237
Went to that directory and I found a bunch of files, including psybnc titled configs and scripts. I deleted this directory promptly.
Any other suggestions?
-
05-29-2006, 12:40 AM #16Disabled
- Join Date
- Jan 2006
- Posts
- 388
You will have to reformat, back up your data, then once it comes back up, have it secured, and restore data.
-
05-29-2006, 12:57 AM #17WHT Addict
- Join Date
- Apr 2006
- Posts
- 114
Ok your box has been definitely compromised. Please do a
ps auxww|grep kjournal
If the owner of this process is root (one kjournald will and will look like [kjournald]) you have no hope and the system will have to be rebuilt. You will need some experience in security to find out what happened. If the owner is not root there is some hope but you have to do the following:
- Find out how did it get there. If the process is owned by the apache user most likely it was some kind of cross scripting attack. Make sure you find out whats the vulnerable site and update it. It wont help if you rebuild the system before fixing this site because there are automated attacks so your systems will be compromised again. lsof is sometimes useful for this kind of problems.
- If the system was not compromised as root there is no need to rebuild the system but you still got to clean up the mess. See what kind of files the bad process has opened and how it affects your system. On compromises there is no specific cleanup that you have to do since the cleanup is specific to the attack.
- Although this is really bad, you can see it as a good thing since you will learn a lot from this process if its your first. Good luckIncrease email deliverability and know who clicked and opened your emails.
SendGrid, reliable SMTP service.
-
05-29-2006, 01:08 AM #18Problem Solver
- Join Date
- Mar 2003
- Location
- California USA
- Posts
- 13,681
I love how every one says that hes is root compromised and needs a os reload.
Not once did we see if it was running as root....
Getting a os reload is pointless unless he secures it, or has someone do it for him.Steven Ciaburri | Industry's Best Server Management - Rack911.com
Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance
-
05-29-2006, 01:14 AM #19Never Stop Working
- Join Date
- Nov 2004
- Location
- Silicon Valley
- Posts
- 576
As a important side note, simply deleting folders / files wont make your problems go away. If they got there to begin with, you need to keep tracing back the problem to its source, not just clean up its work
-
05-29-2006, 01:26 AM #20Web Hosting Master
- Join Date
- Oct 2005
- Posts
- 1,317
It is probably updating for some reason.
-
05-29-2006, 01:31 AM #21Newbie
- Join Date
- Mar 2005
- Location
- Fremont, California
- Posts
- 6
You don't think it might have to do with an ext3 (journaled)
file system, do you?
-
05-29-2006, 01:34 AM #22Problem Solver
- Join Date
- Mar 2003
- Location
- California USA
- Posts
- 13,681
Originally Posted by amacrae
root@enterprise [~]# netstat -np
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 1064 0 XXX.XXX.XX.XXX:36014 193.109.122.67:6667 CLOSE_WAIT 1491/[kjournald]
tcp 187 0 XXX.XXX.XX.XXX:59671 193.109.122.67:6667 CLOSE_WAIT 1491/[kjournald]
He needs to secure the server, the problem will never go away if he does not do that.Steven Ciaburri | Industry's Best Server Management - Rack911.com
Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance
-
05-29-2006, 01:45 AM #23Web Hosting Guru
- Join Date
- May 2005
- Posts
- 288
Originally Posted by Steven
/usr/lib/sshdlib
Considering nothing will create this directory by itself, during the compromise it was created. So, somehow, someone was able to create a directory inside /usr/lib.
So even if the process is not running as root, it is a sure bet that at some point the compromise involved privileged (root) access.
-
05-29-2006, 01:50 AM #24WHT Addict
- Join Date
- Apr 2006
- Posts
- 114
Originally Posted by StevenIncrease email deliverability and know who clicked and opened your emails.
SendGrid, reliable SMTP service.
-
05-29-2006, 01:51 AM #25Problem Solver
- Join Date
- Mar 2003
- Location
- California USA
- Posts
- 13,681
Originally Posted by morcego
Do you know how many servers I have seen that have had poor permissions or permissions changed on a directory in an attempt to "fix" a problem? Without more information we cannot give a definate answer as to if it was root compromised or not.Steven Ciaburri | Industry's Best Server Management - Rack911.com
Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance