maxihost
07-04-2004, 04:36 PM
Guys,
I´ve been with Ev1Servers since 2001. They have good uptime, 24/7 support and everything that could keep me with them since 2001.
But today, I am totally sick of their Restore Policy.
Today, they unplugged my server for AUP Violations and told us that my server got compromised.
We have proved them that the server was not compromised and if they could put the server back up to the network we could fix a little error that is caused by a cpanel bug.
I am sick of them and will move to another NOC as soon as possible. They unplug the server, take lot of time to investigate it, another hours to restore the server and re-install cPanel.
They just give fast explanations that the server cant back up without a restore, and every month is the same thing, I am totally SICK.
---
Ticket:
They unplugged it:
7/4/04 1:26:08 AM
NOC
The datacenter will now investigate the server and try and identify the exact method by which the server was used to launch this attack. If it was launched by a user, we might expect for that user to be removed. If the server has been compromised a restore might be required to remedy the situation and to prevent additional network disturbances to other customers.
(They always says that is it was launched by a user, they just back the server up and we remove the user, but this never happen, they always require restore)
We told them:
7/4/04 2:13:53 AM
This is most certaintly due to a end user uploaded script. Please advise of results.
They said:
7/4/04 4:21:15 AM
DataCenter
Dear customer,
Your server has been compromised and root access achieved.
user 'nobody' ran:
inetd
./par
./labs
./vad 194.85.83.97 (which is the IP of the server that got attacked)
in /var/tmp I found:
bg (an IRC bot)
I also found bg in /tmp
in /dev/shm (which requires root access) I found
bind*
I copied the contents of /dev/shm to /root/contents.of.dev.shm
Server is unplugged because of the security threat that this compromised server poses to our network and other servers.
Please order a restore online through the member’s section and open a ticket for this to be done. In the restore ticket, please specify whether you need your original HD left in the system for 48hrs to recover your data.
And we replied:
7/4/04 5:16:49 AM
The items discovered do not prove a root compromise. They only demonstrate that a non-root user was used to launch an outbound attack via 'nobody' account which is owned by the web server.
The items in /dev/shm can also be attributed to scripts uploaded via web server also. The files that the tech investigating backed up cannot be used to validate the ownership of these files as they were copied as 'root' most likely and have inherited these permissions.
Please check your findings.
We “strongly” disagree in EV1’s assessment that the server has been compromised on the ‘root’ level. You have noted the data written to the “tmp paths” (/tmp, /var/tmp and /dev/shm) was executed as user ‘nobody’, which is the apache user. However on the note that:
“in /dev/shm (which requires root access)”
This is not the case; /dev/shm is a globally read, write & execute path.
i.e: drwxrwxrwt 2 root root 40 Jul 2 00:06 shm/
Look on any Linux 2.4 system and you will find the same “rwx” permission for owner, group, world; likewise the ‘t’ (temporary) bit is set.
With that said; the packet dumps do not warrant that the attack was made using raw sockets (which requires root); it is an outbound attack to destination port 53 (client side port would be random high port [unprivileged]). We are not negating the fact the attack was a violation of policy but simply reinforcing that root access was not used to launch the attack.
The most probable cause of this whole situation is; from our experience, a vulnerable web site susceptible to injection exploits. An attacker simply crafts custom URL’s that inject outside code into a site and there-in downloads content for execution (which runs as apache user – nobody). It is quite easy for us; given access to the system too find the site with a simple ‘grep’ of the apache logs and there-in remove the site in question.
We do understand the fact that this issue is very serious and we plan to fully cooperate to resolve this issue, within practical bounds. The information provided does not prove that root access was compromised what so ever. Given our notes in the discrepancies of your investigation we ask that access to the system be permitted so we may handle the issue and ensure it does not happen again. We will not accept a restore as the only recourse since you can provide NO information on “root” owned binaries from the compromise (or there-in that uid 0 was set by any of the scripts executed by apache).
They:
7/4/04 6:20:54 AM
DataCenter
Dear customer,
vad* and pdr* were found in /usr/local/apache/proxy which requires root access.
This server has been compromised and root access has been achieved. It will need a restore to be allowed back onto the network
Us:
7/4/04 7:10:39 AM
Have you checked who owns these files though?.
7/4/04 7:15:07 AM
We have to disagree with your assessment again
If you check the default permissions on this folder that you note, /usr/local/apache/proxy on any cPanel server you will see that this is writable by any application running under the web server by default, i.e.
drwxr-xr-x 2 nobody nobody 4096 May 30 2003 proxy/
The presence of these files does not demonstrate that the end user had achieved root access.
Please reinvestigate and validate your findings.
And they finnaly:
7/4/04 8:56:55 AM
DataCenter
Dear Customer:
Upon reevaluation, we ran the chkrootkit, and found the login, pstree, and several other files infected. Server will require a restore before being brought back online.
DC
7/4/04 10:42:58 AM
DataCenter
Dear Customer:
As you have ordered a restore to clear up this matter, I am closing this ticket.
Thank you,
DC
----------------
What do you think ?
I have CHEETAWEB taking care of my servers, I dont think is their fault.
Anyone recommend a NOC that dont have this unfair policy to go with ?
I have 3 Dual Xeon 2.8 2GB RAM (2) HD 73GB SCSI
Thanks.
I´ve been with Ev1Servers since 2001. They have good uptime, 24/7 support and everything that could keep me with them since 2001.
But today, I am totally sick of their Restore Policy.
Today, they unplugged my server for AUP Violations and told us that my server got compromised.
We have proved them that the server was not compromised and if they could put the server back up to the network we could fix a little error that is caused by a cpanel bug.
I am sick of them and will move to another NOC as soon as possible. They unplug the server, take lot of time to investigate it, another hours to restore the server and re-install cPanel.
They just give fast explanations that the server cant back up without a restore, and every month is the same thing, I am totally SICK.
---
Ticket:
They unplugged it:
7/4/04 1:26:08 AM
NOC
The datacenter will now investigate the server and try and identify the exact method by which the server was used to launch this attack. If it was launched by a user, we might expect for that user to be removed. If the server has been compromised a restore might be required to remedy the situation and to prevent additional network disturbances to other customers.
(They always says that is it was launched by a user, they just back the server up and we remove the user, but this never happen, they always require restore)
We told them:
7/4/04 2:13:53 AM
This is most certaintly due to a end user uploaded script. Please advise of results.
They said:
7/4/04 4:21:15 AM
DataCenter
Dear customer,
Your server has been compromised and root access achieved.
user 'nobody' ran:
inetd
./par
./labs
./vad 194.85.83.97 (which is the IP of the server that got attacked)
in /var/tmp I found:
bg (an IRC bot)
I also found bg in /tmp
in /dev/shm (which requires root access) I found
bind*
I copied the contents of /dev/shm to /root/contents.of.dev.shm
Server is unplugged because of the security threat that this compromised server poses to our network and other servers.
Please order a restore online through the member’s section and open a ticket for this to be done. In the restore ticket, please specify whether you need your original HD left in the system for 48hrs to recover your data.
And we replied:
7/4/04 5:16:49 AM
The items discovered do not prove a root compromise. They only demonstrate that a non-root user was used to launch an outbound attack via 'nobody' account which is owned by the web server.
The items in /dev/shm can also be attributed to scripts uploaded via web server also. The files that the tech investigating backed up cannot be used to validate the ownership of these files as they were copied as 'root' most likely and have inherited these permissions.
Please check your findings.
We “strongly” disagree in EV1’s assessment that the server has been compromised on the ‘root’ level. You have noted the data written to the “tmp paths” (/tmp, /var/tmp and /dev/shm) was executed as user ‘nobody’, which is the apache user. However on the note that:
“in /dev/shm (which requires root access)”
This is not the case; /dev/shm is a globally read, write & execute path.
i.e: drwxrwxrwt 2 root root 40 Jul 2 00:06 shm/
Look on any Linux 2.4 system and you will find the same “rwx” permission for owner, group, world; likewise the ‘t’ (temporary) bit is set.
With that said; the packet dumps do not warrant that the attack was made using raw sockets (which requires root); it is an outbound attack to destination port 53 (client side port would be random high port [unprivileged]). We are not negating the fact the attack was a violation of policy but simply reinforcing that root access was not used to launch the attack.
The most probable cause of this whole situation is; from our experience, a vulnerable web site susceptible to injection exploits. An attacker simply crafts custom URL’s that inject outside code into a site and there-in downloads content for execution (which runs as apache user – nobody). It is quite easy for us; given access to the system too find the site with a simple ‘grep’ of the apache logs and there-in remove the site in question.
We do understand the fact that this issue is very serious and we plan to fully cooperate to resolve this issue, within practical bounds. The information provided does not prove that root access was compromised what so ever. Given our notes in the discrepancies of your investigation we ask that access to the system be permitted so we may handle the issue and ensure it does not happen again. We will not accept a restore as the only recourse since you can provide NO information on “root” owned binaries from the compromise (or there-in that uid 0 was set by any of the scripts executed by apache).
They:
7/4/04 6:20:54 AM
DataCenter
Dear customer,
vad* and pdr* were found in /usr/local/apache/proxy which requires root access.
This server has been compromised and root access has been achieved. It will need a restore to be allowed back onto the network
Us:
7/4/04 7:10:39 AM
Have you checked who owns these files though?.
7/4/04 7:15:07 AM
We have to disagree with your assessment again
If you check the default permissions on this folder that you note, /usr/local/apache/proxy on any cPanel server you will see that this is writable by any application running under the web server by default, i.e.
drwxr-xr-x 2 nobody nobody 4096 May 30 2003 proxy/
The presence of these files does not demonstrate that the end user had achieved root access.
Please reinvestigate and validate your findings.
And they finnaly:
7/4/04 8:56:55 AM
DataCenter
Dear Customer:
Upon reevaluation, we ran the chkrootkit, and found the login, pstree, and several other files infected. Server will require a restore before being brought back online.
DC
7/4/04 10:42:58 AM
DataCenter
Dear Customer:
As you have ordered a restore to clear up this matter, I am closing this ticket.
Thank you,
DC
----------------
What do you think ?
I have CHEETAWEB taking care of my servers, I dont think is their fault.
Anyone recommend a NOC that dont have this unfair policy to go with ?
I have 3 Dual Xeon 2.8 2GB RAM (2) HD 73GB SCSI
Thanks.