Web Hosting Talk : Web Hosting Main Forums : Colocation and Data Centers : SuperMicro IPMI Security

[brc_csf]

Hello,

We received a notification from our DataCenter about SPAM sent from our Supermicro's IPMI IP. This really worries me.

Anyone aware about any vulnerability ?

Tried to get SSH access to the BMC and check what is running on but I just get a prompt with:

Verbs :
cd
show
help
version
exit

We are not using default ADMIN password and have blocked anonymous access (since server was enabled - months ago).

Any hints?

Thanks

[FastServ]

Never heard of anything like this, I honestly didn't think it's possible.

/subscribed

[Dougy]

I don't know if that was possible but that being said, I keep my IPMI's on a private network with vpn only access

[brc_csf]

Originally Posted by Dougy

I don't know if that was possible but that being said, I keep my IPMI's on a private network with vpn only access

We thought about working this way but as we were always looking for the faster way to access IPMI on emergencies we are keeping them on public addresses.

I am really interested on finding out what happened (or is happening) This IP was never used before being assigned to IPMI (on 24/Jun/2010). We were listed at cbl.abuseat.org at 22/Oct/2010:

"
IP Address X.X.X.X is listed in the CBL. It appears to be infected with a spam sending trojan or proxy.


It was last detected at 2010-10-22 21:00 GMT (+/- 30 minutes), approximately 2 days, 15 hours, 59 minutes ago.
"


I wonder if this could be a spoofed SPAM header which resulted on incorrect classification. How trustable would this RBL be ?

This is a path were we start looking for other causes and not blame any Supermicro IPMI vulnerability.

If I could get root access to the BMC it would make stuff easier. I already e-mailed supermicro but from my previous experiences they might not reply.

[Dougy]

I get replies from SM support within 12 hours every single time

[FastServ]

Are you on a private VLAN? Reason I ask, is maybe someone hijacked your IP(s) and sent spam?

[brc_csf]

Originally Posted by FastServ

Are you on a private VLAN? Reason I ask, is maybe someone hijacked your IP(s) and sent spam?

I also thought about that but we have our own VLAN at Netriplex. We just work with VPS hosting (OpenVZ) so no clients would have enough privileges to hijack that IP.

[cresci]

a) We always got a reply when we needed to contact Supermicro.

b) As you said, you have a private VLAN, so IPs are not hijacked. Check.

c) Supermicro IPMI has some functions to send out warning emails. Have you verified if this is somehow being abused maybe by a security failure?

d) Supermicro IPMI has Guest access available that sometimes people forget to disable. Did you verify this?

e) Yes, the IP may have been spoofed. The CBL is pretty automatic, some email contents trigger it as well as other blacklists. It is harmless and not many people use the CBL directly, but it influences Spamhaus' ZEN RBL. The difference is that the CBL tries to work thoroughly with you, try emailing them and explaining the situation, and they may offer you some more hard evidence of that listing.

[brc_csf]

Originally Posted by cresci

a) We always got a reply when we needed to contact Supermicro.

b) As you said, you have a private VLAN, so IPs are not hijacked. Check.

c) Supermicro IPMI has some functions to send out warning emails. Have you verified if this is somehow being abused maybe by a security failure?

d) Supermicro IPMI has Guest access available that sometimes people forget to disable. Did you verify this?

e) Yes, the IP may have been spoofed. The CBL is pretty automatic, some email contents trigger it as well as other blacklists. It is harmless and not many people use the CBL directly, but it influences Spamhaus' ZEN RBL. The difference is that the CBL tries to work thoroughly with you, try emailing them and explaining the situation, and they may offer you some more hard evidence of that listing.

We've browsed all IPMI options trying to find anything different. Couldn't find anything (nothing different on alerts, we have none configured), we have no guest users. Default account was blocked when we first powered on this server.

I Wish I could have root on this Linux that is used on the IPMI device

[programguy]

unless mistaken, I do not see how they could say your ipmi sent out any spam.

However, I could see how they could see the IP address of your ipmi could have been used to send spam.

I would look at the server and see if it is compromised.

I would also look at my firewalls and block the ipmi IP from doing much of anything other than what is intended.

if spoofed, you really cannot do much. If your server has a trojan/virus/rootkit/etc that is sending mail via that ip address I would really look into it asap

[brc_csf]

Originally Posted by programguy

unless mistaken, I do not see how they could say your ipmi sent out any spam.

However, I could see how they could see the IP address of your ipmi could have been used to send spam.

I would look at the server and see if it is compromised.

I would also look at my firewalls and block the ipmi IP from doing much of anything other than what is intended.

if spoofed, you really cannot do much. If your server has a trojan/virus/rootkit/etc that is sending mail via that ip address I would really look into it asap

Unfortunately no one will be able to say that it was sent using IPMI until further investigations are done.

It is hard to believe a server would be hacked and the intruder would focus on hijacking another IP just to send SPAM. . All our servers just have SSH port opened and restricted (the vm guest software is than able to load vm's). I believe this would have already been noticed after 3 days.

I believe old SuperMicro's board would allow SSH access as root to its IPMI Linux (found some posts where people had access to everything on the fs). Still wasn't able to do it
I'd be really happy to audit it.

[plumsauce]

Originally Posted by brc_csf

We thought about working this way but as we were always looking for the faster way to access IPMI on emergencies we are keeping them on public addresses.

One possible solution is to put them on private ip space behind a VPN server. If money is a factor, you can use a consumer grade router with inbound VPN capabilities. No hard drive to fail, and the power bricks are usually reliable. If you have a choice, authentication via password + certificate is great as long as you remember to store the cert on all your portable devices. If you tether to your cell phone, you might be able to run the tunnel endpoint at the cellphone and then only need to install the cert on the cellphone.

For high bandwidth uses when you need to mount an iso for installs, mount the iso from an existing box that is also on the private network.

[plumsauce]

Originally Posted by brc_csf

IP Address X.X.X.X is listed in the CBL. It appears to be infected with a spam sending trojan or proxy.

Hang on a minute. Always carefully read notices and determine the basis of the notice.

CBL is the spamhaus "consumer black list".

The definition is something like "isp self reported residential ip allocations".

In other words, the owner of the isp voluntarily registers allocated blocks of ip space as being assigned to residential customers. Those blocks are added to the CBL. Only the owner of the allocation can have it removed.

[brc_csf]

Originally Posted by plumsauce

Hang on a minute. Always carefully read notices and determine the basis of the notice.

CBL is the spamhaus "consumer black list".

The definition is something like "isp self reported residential ip allocations".

In other words, the owner of the isp voluntarily registers allocated blocks of ip space as being assigned to residential customers. Those blocks are added to the CBL. Only the owner of the allocation can have it removed.

The definition is kinda confuse. This is what they say on "http://cbl.abuseat.org/lookup.cgi?ip=X.X.X.X&.submit=Lookup":

"IP Address X.X.X.X is listed in the CBL. It appears to be infected with a spam sending trojan or proxy.
It was last detected at 2010-10-22 21:00 GMT (+/- 30 minutes), approximately 2 days, 23 hours ago."

...

"
...


If this listing is of an unshared IP address, and the affected access is email, then, the computer corresponding to this IP address at time of detection (see above) is infected with a spambot, or, if it's a mail server, in some rare cases this can be a severe misconfiguration or bug.


...

"

[brc_csf]

Originally Posted by plumsauce

One possible solution is to put them on private ip space behind a VPN server. If money is a factor, you can use a consumer grade router with inbound VPN capabilities. No hard drive to fail, and the power bricks are usually reliable. If you have a choice, authentication via password + certificate is great as long as you remember to store the cert on all your portable devices. If you tether to your cell phone, you might be able to run the tunnel endpoint at the cellphone and then only need to install the cert on the cellphone.

For high bandwidth uses when you need to mount an iso for installs, mount the iso from an existing box that is also on the private network.

Would you recommend any model/brand? I believe this is the best way to go. Access to the VPN just by login/password is enough (logging all access). The services inside the VPN should be secure. I wish we could have something like SoftLayer.. Using their customer portal you just do a few clicks and you are inside the VPN (no matter if you are on Windows or Linux). That's what I'd like to have, something that would be compatible with Windows/Linux and it would be perfect to allow someone behind an HTTP proxy to access the VPN.