Page 1 of 6 1234 ... LastLast
Results 1 to 25 of 139

Hybrid View

  1. #1
    Join Date
    Nov 2005
    Posts
    305

    SuperMicro IPMI Security

    Hello,

    We received a notification from our DataCenter about SPAM sent from our Supermicro's IPMI IP. This really worries me.

    Anyone aware about any vulnerability ?

    Tried to get SSH access to the BMC and check what is running on but I just get a prompt with:

    Verbs :
    cd
    show
    help
    version
    exit

    We are not using default ADMIN password and have blocked anonymous access (since server was enabled - months ago).

    Any hints?

    Thanks

  2. #2
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,615
    Never heard of anything like this, I honestly didn't think it's possible.

    /subscribed
    Fast Serv Networks, LLC | AS29889 | DDOS Protected | Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  3. #3
    Join Date
    May 2006
    Location
    NJ, USA
    Posts
    6,645
    I don't know if that was possible but that being said, I keep my IPMI's on a private network with vpn only access
    AS395558

  4. #4
    Join Date
    Nov 2005
    Posts
    305
    Quote Originally Posted by Dougy View Post
    I don't know if that was possible but that being said, I keep my IPMI's on a private network with vpn only access

    We thought about working this way but as we were always looking for the faster way to access IPMI on emergencies we are keeping them on public addresses.

    I am really interested on finding out what happened (or is happening) This IP was never used before being assigned to IPMI (on 24/Jun/2010). We were listed at cbl.abuseat.org at 22/Oct/2010:

    "
    IP Address X.X.X.X is listed in the CBL. It appears to be infected with a spam sending trojan or proxy.


    It was last detected at 2010-10-22 21:00 GMT (+/- 30 minutes), approximately 2 days, 15 hours, 59 minutes ago.
    "


    I wonder if this could be a spoofed SPAM header which resulted on incorrect classification. How trustable would this RBL be ?

    This is a path were we start looking for other causes and not blame any Supermicro IPMI vulnerability.

    If I could get root access to the BMC it would make stuff easier. I already e-mailed supermicro but from my previous experiences they might not reply.

  5. #5
    Quote Originally Posted by brc_csf View Post
    We thought about working this way but as we were always looking for the faster way to access IPMI on emergencies we are keeping them on public addresses.
    One possible solution is to put them on private ip space behind a VPN server. If money is a factor, you can use a consumer grade router with inbound VPN capabilities. No hard drive to fail, and the power bricks are usually reliable. If you have a choice, authentication via password + certificate is great as long as you remember to store the cert on all your portable devices. If you tether to your cell phone, you might be able to run the tunnel endpoint at the cellphone and then only need to install the cert on the cellphone.

    For high bandwidth uses when you need to mount an iso for installs, mount the iso from an existing box that is also on the private network.
    edgedirector.com
    managed dns global failover and load balance (gslb)
    exactstate.com
    uptime report for webhostingtalk.com

  6. #6
    Join Date
    Nov 2005
    Posts
    305
    Quote Originally Posted by plumsauce View Post
    One possible solution is to put them on private ip space behind a VPN server. If money is a factor, you can use a consumer grade router with inbound VPN capabilities. No hard drive to fail, and the power bricks are usually reliable. If you have a choice, authentication via password + certificate is great as long as you remember to store the cert on all your portable devices. If you tether to your cell phone, you might be able to run the tunnel endpoint at the cellphone and then only need to install the cert on the cellphone.

    For high bandwidth uses when you need to mount an iso for installs, mount the iso from an existing box that is also on the private network.

    Would you recommend any model/brand? I believe this is the best way to go. Access to the VPN just by login/password is enough (logging all access). The services inside the VPN should be secure. I wish we could have something like SoftLayer.. Using their customer portal you just do a few clicks and you are inside the VPN (no matter if you are on Windows or Linux). That's what I'd like to have, something that would be compatible with Windows/Linux and it would be perfect to allow someone behind an HTTP proxy to access the VPN.

  7. #7
    Quote Originally Posted by brc_csf View Post
    Would you recommend any model/brand? I believe this is the best way to go. Access to the VPN just by login/password is enough (logging all access). The services inside the VPN should be secure. I wish we could have something like SoftLayer.. Using their customer portal you just do a few clicks and you are inside the VPN (no matter if you are on Windows or Linux). That's what I'd like to have, something that would be compatible with Windows/Linux and it would be perfect to allow someone behind an HTTP proxy to access the VPN.
    No particular recommendation on brands.

    You are probably looking at the models used for small branch offices and the like. Look for one that uses a hardware crypto accelerator chip. Be sure you understand exactly what types of VPN it offers, and what clients it supports.

    If you have a favourite router at home, you can try it from outside. If it all works, then buy another one. If it has wifi, be sure to disable the radio on the one you use at the datacenter. Also disable admin from the public side of the device. If you need to admin, vpn in, then bounce back from the private side. At the very least restrict admin from the public side to a static ip from your usual location.

    Some of the home routers are as good as the branch office models. It's a question of throughput. For ipmi, you don't need much.

    If you like to hack around with home router firmware, then most of the distributions can do vpn.

    A more restricted hardware list can be used with pfsense.org. The some of the purpose built mini-boards have crypto chips embedded. See the forums for details. Look for the bug on number of vpn channels simultaneously available. Still highly recommended. Installation is as easy as flashing a downloaded image onto a flash card and booting. No moving parts to fail at all.

    If you go with a high end VPN appliance be aware of the very high *annual* license fee model that they like to use. Not recommended for your application.
    edgedirector.com
    managed dns global failover and load balance (gslb)
    exactstate.com
    uptime report for webhostingtalk.com

  8. #8
    Quote Originally Posted by brc_csf View Post
    IP Address X.X.X.X is listed in the CBL. It appears to be infected with a spam sending trojan or proxy.
    Hang on a minute. Always carefully read notices and determine the basis of the notice.

    CBL is the spamhaus "consumer black list".

    The definition is something like "isp self reported residential ip allocations".

    In other words, the owner of the isp voluntarily registers allocated blocks of ip space as being assigned to residential customers. Those blocks are added to the CBL. Only the owner of the allocation can have it removed.
    Last edited by plumsauce; 10-25-2010 at 04:12 PM.
    edgedirector.com
    managed dns global failover and load balance (gslb)
    exactstate.com
    uptime report for webhostingtalk.com

  9. #9
    Join Date
    Nov 2005
    Posts
    305
    Quote Originally Posted by plumsauce View Post
    Hang on a minute. Always carefully read notices and determine the basis of the notice.

    CBL is the spamhaus "consumer black list".

    The definition is something like "isp self reported residential ip allocations".

    In other words, the owner of the isp voluntarily registers allocated blocks of ip space as being assigned to residential customers. Those blocks are added to the CBL. Only the owner of the allocation can have it removed.

    The definition is kinda confuse. This is what they say on "http://cbl.abuseat.org/lookup.cgi?ip=X.X.X.X&.submit=Lookup":

    "IP Address X.X.X.X is listed in the CBL. It appears to be infected with a spam sending trojan or proxy.
    It was last detected at 2010-10-22 21:00 GMT (+/- 30 minutes), approximately 2 days, 23 hours ago."

    ...

    "
    ...


    If this listing is of an unshared IP address, and the affected access is email, then, the computer corresponding to this IP address at time of detection (see above) is infected with a spambot, or, if it's a mail server, in some rare cases this can be a severe misconfiguration or bug.


    ...

    "

  10. #10
    Join Date
    Dec 2004
    Posts
    569
    Quote Originally Posted by plumsauce View Post
    CBL is the spamhaus "consumer black list".
    CBL = Composite block list

    The definition is something like "isp self reported residential ip allocations".

    In other words, the owner of the isp voluntarily registers allocated blocks of ip space as being assigned to residential customers. Those blocks are added to the CBL. Only the owner of the allocation can have it removed.
    I think you are confused with PBL = Policy Block List

  11. #11
    Quote Originally Posted by Maxnet View Post
    CBL = Composite block list
    I think you are confused with PBL = Policy Block List
    You're right, thanks for the reminder.
    edgedirector.com
    managed dns global failover and load balance (gslb)
    exactstate.com
    uptime report for webhostingtalk.com

  12. #12
    Join Date
    May 2006
    Location
    NJ, USA
    Posts
    6,645
    I get replies from SM support within 12 hours every single time
    AS395558

  13. #13
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,615
    Are you on a private VLAN? Reason I ask, is maybe someone hijacked your IP(s) and sent spam?
    Fast Serv Networks, LLC | AS29889 | DDOS Protected | Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  14. #14
    Join Date
    Nov 2005
    Posts
    305
    Quote Originally Posted by FastServ View Post
    Are you on a private VLAN? Reason I ask, is maybe someone hijacked your IP(s) and sent spam?
    I also thought about that but we have our own VLAN at Netriplex. We just work with VPS hosting (OpenVZ) so no clients would have enough privileges to hijack that IP.

  15. #15
    Join Date
    Aug 2003
    Location
    /dev/null
    Posts
    2,132
    a) We always got a reply when we needed to contact Supermicro.

    b) As you said, you have a private VLAN, so IPs are not hijacked. Check.

    c) Supermicro IPMI has some functions to send out warning emails. Have you verified if this is somehow being abused maybe by a security failure?

    d) Supermicro IPMI has Guest access available that sometimes people forget to disable. Did you verify this?

    e) Yes, the IP may have been spoofed. The CBL is pretty automatic, some email contents trigger it as well as other blacklists. It is harmless and not many people use the CBL directly, but it influences Spamhaus' ZEN RBL. The difference is that the CBL tries to work thoroughly with you, try emailing them and explaining the situation, and they may offer you some more hard evidence of that listing.

  16. #16
    Join Date
    Nov 2005
    Posts
    305
    Quote Originally Posted by cresci View Post
    a) We always got a reply when we needed to contact Supermicro.

    b) As you said, you have a private VLAN, so IPs are not hijacked. Check.

    c) Supermicro IPMI has some functions to send out warning emails. Have you verified if this is somehow being abused maybe by a security failure?

    d) Supermicro IPMI has Guest access available that sometimes people forget to disable. Did you verify this?

    e) Yes, the IP may have been spoofed. The CBL is pretty automatic, some email contents trigger it as well as other blacklists. It is harmless and not many people use the CBL directly, but it influences Spamhaus' ZEN RBL. The difference is that the CBL tries to work thoroughly with you, try emailing them and explaining the situation, and they may offer you some more hard evidence of that listing.


    We've browsed all IPMI options trying to find anything different. Couldn't find anything (nothing different on alerts, we have none configured), we have no guest users. Default account was blocked when we first powered on this server.

    I Wish I could have root on this Linux that is used on the IPMI device

  17. #17
    Join Date
    Jul 2009
    Posts
    451
    unless mistaken, I do not see how they could say your ipmi sent out any spam.

    However, I could see how they could see the IP address of your ipmi could have been used to send spam.

    I would look at the server and see if it is compromised.

    I would also look at my firewalls and block the ipmi IP from doing much of anything other than what is intended.

    if spoofed, you really cannot do much. If your server has a trojan/virus/rootkit/etc that is sending mail via that ip address I would really look into it asap

  18. #18
    Join Date
    Nov 2005
    Posts
    305
    Quote Originally Posted by programguy View Post
    unless mistaken, I do not see how they could say your ipmi sent out any spam.

    However, I could see how they could see the IP address of your ipmi could have been used to send spam.

    I would look at the server and see if it is compromised.

    I would also look at my firewalls and block the ipmi IP from doing much of anything other than what is intended.

    if spoofed, you really cannot do much. If your server has a trojan/virus/rootkit/etc that is sending mail via that ip address I would really look into it asap


    Unfortunately no one will be able to say that it was sent using IPMI until further investigations are done.

    It is hard to believe a server would be hacked and the intruder would focus on hijacking another IP just to send SPAM. . All our servers just have SSH port opened and restricted (the vm guest software is than able to load vm's). I believe this would have already been noticed after 3 days.

    I believe old SuperMicro's board would allow SSH access as root to its IPMI Linux (found some posts where people had access to everything on the fs). Still wasn't able to do it
    I'd be really happy to audit it.

  19. #19
    Join Date
    Jul 2008
    Location
    Dallas, TX
    Posts
    107
    Quote Originally Posted by cresci View Post
    a) We always got a reply when we needed to contact Supermicro.

    b) As you said, you have a private VLAN, so IPs are not hijacked. Check.

    c) Supermicro IPMI has some functions to send out warning emails. Have you verified if this is somehow being abused maybe by a security failure?

    d) Supermicro IPMI has Guest access available that sometimes people forget to disable. Did you verify this?

    e) Yes, the IP may have been spoofed. The CBL is pretty automatic, some email contents trigger it as well as other blacklists. It is harmless and not many people use the CBL directly, but it influences Spamhaus' ZEN RBL. The difference is that the CBL tries to work thoroughly with you, try emailing them and explaining the situation, and they may offer you some more hard evidence of that listing.
    I'm quoting this post because it is the most accurate out of anything in this thread. I'm pretty sure the CBL does not list based on your rDNS.

    I never knew you could SSH to the BMC before but I gave it a shot on 3 different SM motherboards.

    X7DCU (No SSH responding)
    X8DTU-F (Running BusyBox v1.1.3)
    X8SIL-F (Running ATEN SMASH v1.00)

    The IPMI interface you get is one running SMASH from ATEN. If you look on Page 87 of this PDF they have a whole guide on using the SMASH interface to manage the BMC.

    I wouldn't put it past someone to possibly hijack one of the BMCs running your standard BusyBox environment but doing this on one running SMASH would prove to be more difficult.

    If it was me I would just let the CBL listing expire and ignore it, unless it gets updated. You could also monitor the traffic coming out of that IP as well to see if any spam is actually being sent from it. Another option is to contact support@supermicro.com and have them login to the BMC and take a look.
    Last edited by Ryan G - Limestone; 10-26-2010 at 01:14 PM. Reason: more info about CBL

  20. #20
    Join Date
    Aug 2003
    Location
    /dev/null
    Posts
    2,132
    Without the Softlayer approach, you could maybe install on anysize server, a copy of ClearOS and make your private network and VPN out of it. No need for web access to the VPN, just a normal PPTP/L2TP account that you can setup with any Windows or Linux desktop client.

  21. #21
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,615
    I suppose if root SSH is open on the IPMI (I was not aware this was the case) there's nothing stopping someone from brute forcing it and using it to send or proxy SPAM.
    Fast Serv Networks, LLC | AS29889 | DDOS Protected | Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  22. #22
    Quote Originally Posted by FastServ View Post
    I suppose if root SSH is open on the IPMI (I was not aware this was the case) there's nothing stopping someone from brute forcing it and using it to send or proxy SPAM.
    Yes, it's amazing that vendors have not adopted rate limiting on failed logins. It was a standard part of the earliest versions of Novell Netware in the early nineties. It was a standard part of mainframe and midrange machines. It's been in every version of windows server since at least 3.51 and remains a favourite for Windows admins.

    Twenty years later, everyone seems to have forgotten the value of account lockout mechanisms. They act like it's never been done before while moaning about brute force dictionary attacks.

    And that's beside the fact that every account is either 'root' or 'admin' without the ability to change it to something else.

    For a brute forcer, that is like manna from the gods. Of two possible pieces of information, one is already known. The difficulty of a brute force multiplies exponentially with the number of pieces of the puzzle. But, oh no, "the account must always be 'root' it says so right here in the gospel according to the <insert unix variant here> man page.
    Last edited by plumsauce; 10-26-2010 at 02:28 AM.
    edgedirector.com
    managed dns global failover and load balance (gslb)
    exactstate.com
    uptime report for webhostingtalk.com

  23. #23
    Join Date
    Jun 2008
    Posts
    1,471
    If I remember correctly, you can create an account with any username and give it "root" privs and then delete the "root" or "admin" account on a SM IPMI. If not, you can at least change the root account to have a good password and never use it again.

  24. #24
    little known fact: you can get on the spam blacklist without even sending spam. If your server's hostname is set improperly, sometimes the blacklist companies will add you. I know someone who had a dedicated server, and their client set the hostname for the server to the person's actual name, instead of using a dns name. That server went straight onto the blacklist without even sending any email, simply because it relied to helo with a messed up hostname.
    IOFLOOD.com -- We Love Servers
    Phoenix, AZ Dedicated Servers in under an hour
    ★ Ryzen 9: 7950x3D ★ Dual E5-2680v4 Xeon ★
    Contact Us: sales@ioflood.com

  25. #25
    Join Date
    Aug 2003
    Location
    /dev/null
    Posts
    2,132
    funkywizard, you remembered me of something.
    If the reverse DNS of that particular IP address is in any form generic or contains the self IP address in the middle (like, hosted.by.xxx.com or ip.add.ress.somedomain.com or ip-add-res-s.domain.com or lastoctet.static.domain.com) then you get listed on the CBL also.

Page 1 of 6 1234 ... LastLast

Similar Threads

  1. Supermicro IPMI Issue
    By XFactorServers in forum Colocation, Data Centers, IP Space and Networks
    Replies: 9
    Last Post: 08-23-2010, 02:29 PM
  2. SuperMicro 's IPMI
    By Peter-SexyWing in forum Colocation, Data Centers, IP Space and Networks
    Replies: 16
    Last Post: 07-10-2010, 04:51 PM
  3. supermicro ipmi installation
    By phactor in forum Systems Management Requests
    Replies: 5
    Last Post: 04-02-2010, 02:57 PM
  4. Supermicro IPMI
    By opax in forum Colocation, Data Centers, IP Space and Networks
    Replies: 6
    Last Post: 04-29-2009, 12:13 PM
  5. Supermicro IPMI
    By DevelopAl in forum Colocation, Data Centers, IP Space and Networks
    Replies: 14
    Last Post: 03-10-2006, 02:17 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •