Page 4 of 6 FirstFirst 123456 LastLast
Results 76 to 100 of 139
  1. #76
    Join Date
    Nov 2005
    Posts
    305
    Quote Originally Posted by FastServ View Post
    Found a compromised BMC on a X8SIL-F the other day. Interesting because it uses the SMASH and not a normal Bash shell.

    Either way, turns out the guest account was being used as an SSH tunnel -- so that explains how some of you may have seen spam or other 'odd' activity originating from your BMC. This particular BMC attracted a multi-gigabit DDOS so they were certainly using it to piss somebody off, although we didn't see any spam reports.

    Due to the extreme limitations imposed by SMASH I wasn't able to dig much further. But it does appear they did more than tunneling -- they managed to add other users that weren't visible in the web interface. In addition to this several parts of the web interface were malfunctioning (sensor readings) and they had somehow blocked the local network from access. I can only imagine they somehow got access to a normal shell to be able to do these things.

    A firmware upgrade did not resolve this, it took a factory reset to get things back under control.

    And before anyone jumps all over me for having an IPMI device on a public interface, it was a colo client no one of our own systems.


    How did you find the "hidden" users? Would you guide us through the steps to check our servers?

  2. #77
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,615
    Quote Originally Posted by brc_csf View Post
    How did you find the "hidden" users? Would you guide us through the steps to check our servers?
    Pretty basic... I googled the IP and found several cracking sites sharing working logins that weren't visible in the web interface. Also, the CPU temp was reporting -30 degrees and the KVM was hit or miss...they messed things up pretty good.
    Fast Serv Networks, LLC | AS29889 | DDOS Protected | Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  3. #78
    Nice find. will need to check firmware builds for my x7spa, as 2.02 doesn't have this 'shell' available but 2.25 does.
    Last edited by Lockjaw; 05-03-2011 at 07:35 PM.

  4. #79
    Join Date
    Nov 2005
    Posts
    305
    Can we say that we are now sure that there is a vulnerability on Supermicro's IPMI ?

  5. #80
    Join Date
    Jun 2006
    Location
    Support Ticket Near You!
    Posts
    1,106
    There will always be a vulnerability, as Fastserv has already figured limit it to local access.

  6. #81
    Join Date
    Nov 2005
    Posts
    305
    News:

    One of our IPMI IPs was used today to connect and to host IRC servers and received a huge DDOS attack by a Botnet (1.4 Million packets per second) .. We received a report from our IDC after it happened.

    This motherboard is a X3440, not sure how to check the "IPMI" version. I advise everyone running this same motherboard to move IPMI to internal IPs asap if you don't want problems like this.

    Has supermicro discovered and/or released any updates to fix these issues?

  7. #82
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,615
    Quote Originally Posted by brc_csf View Post
    News:

    One of our IPMI IPs was used today to connect and to host IRC servers and received a huge DDOS attack by a Botnet (1.4 Million packets per second) .. We received a report from our IDC after it happened.

    This motherboard is a X3440, not sure how to check the "IPMI" version. I advise everyone running this same motherboard to move IPMI to internal IPs asap if you don't want problems like this.

    Has supermicro discovered and/or released any updates to fix these issues?
    Chances are someone got in using the default login (or grabbed the admin password in cleartext if you ever used the config backup). For giggles try searching the IP in google and see how many proxy lists you're on.

    Upgrade to latest firmware, reset to factory defaults, and quickly disable the anonymous account and change the admin password. Avoid using the config backup option as it stores the password in cleartext in an open location. Ideally you should have IPMI on a private network segment.
    Fast Serv Networks, LLC | AS29889 | DDOS Protected | Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  8. #83
    Join Date
    Nov 2005
    Posts
    305
    Quote Originally Posted by FastServ View Post
    Chances are someone got in using the default login (or grabbed the admin password in cleartext if you ever used the config backup). For giggles try searching the IP in google and see how many proxy lists you're on.

    Upgrade to latest firmware, reset to factory defaults, and quickly disable the anonymous account and change the admin password. Avoid using the config backup option as it stores the password in cleartext in an open location. Ideally you should have IPMI on a private network segment.
    I just read what I said on the last post and I meant to say the motherboard is a X8SIL-F (not a x3440 )

    We always use https and change the admin password as soon as we access IPMI for the first time (also, we always make sure that there are no other logins).

    Never used the config backup option

    I am sure there is a bug, we had previous problems with SPAM being sent from other IPMI cards. We have some dozens of these servers and I believe it is just a matter of time until it happens with the others.

    This one has already been moved to an internal IP and we are working to move the others. I hope that newer SM motherboards do not have similar issues.

  9. #84
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,615
    FYI the only problematic IPMI we had was also an X8SIL-F. No issues since upgrading and resetting the firmware.
    Fast Serv Networks, LLC | AS29889 | DDOS Protected | Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  10. #85
    Join Date
    Jul 2009
    Posts
    451
    are all the ipmi cards having issues onboard or add on cards?

  11. #86
    Join Date
    Nov 2005
    Posts
    305
    Onboard: SuperMIcro X8SIL-F

  12. #87
    Join Date
    Dec 2004
    Posts
    569
    Quote Originally Posted by brc_csf View Post
    We always use https and change the admin password as soon as we access IPMI for the first time (also, we always make sure that there are no other logins).
    Also made sure the passwords are unique per server?
    Be aware that they tend to be stored plain text, and are quite often retrievable from the host system.
    So one customer with bad intentions or compromised box is all it takes, if they are not different for each server.

  13. #88
    Join Date
    Apr 2010
    Posts
    493
    It's pretty simple, you can take over an IPMI card from the host OS. These devices only need to talk to a couple things. Put them on a secure network that only allows them to talk to a management firewall. For our dedicated we still have them sending snmp traps to us so we can be proactive and allow access via a customer portal. These it no reason for them to access the general internet, each other or anything but a firewall and a management box.

  14. #89
    Finally we've found the way how to get into the IPMI system... I don't really want to post it on public, because it will give the info of this vulnerability to the whole world. Just keep it on private network. I think pretty soon this issue will be completely fixed. Right now we are working with the customer on it. 2 motherboards reported. It is x8dtl-if(already fixed) and x8sti-f(in process). Both seems to have an easy cure for it.

  15. #90
    Join Date
    Aug 2008
    Posts
    536
    Quote Originally Posted by YuriyK View Post
    Finally we've found the way how to get into the IPMI system... I don't really want to post it on public, because it will give the info of this vulnerability to the whole world. Just keep it on private network. I think pretty soon this issue will be completely fixed. Right now we are working with the customer on it. 2 motherboards reported. It is x8dtl-if(already fixed) and x8sti-f(in process). Both seems to have an easy cure for it.
    Please report the bug to supermicro so they can solve this permanent.
    Regards,
    Yourwebhoster.eu [NL] based hosting
    Shared | Reseller | KVM VPS | Reseller VPS

  16. #91
    Quote Originally Posted by yourwebhostereu View Post
    Please report the bug to supermicro so they can solve this permanent.
    I am currently working with them on it. They are fixing it in newer ipmi firmware.

  17. #92
    Join Date
    Aug 2008
    Posts
    536
    Quote Originally Posted by YuriyK View Post
    I am currently working with them on it. They are fixing it in newer ipmi firmware.
    Thanks, please keep us updated when the new firmware is released.
    Regards,
    Yourwebhoster.eu [NL] based hosting
    Shared | Reseller | KVM VPS | Reseller VPS

  18. #93
    Join Date
    Nov 2005
    Posts
    305
    Quote Originally Posted by YuriyK View Post
    Finally we've found the way how to get into the IPMI system... I don't really want to post it on public, because it will give the info of this vulnerability to the whole world. Just keep it on private network. I think pretty soon this issue will be completely fixed. Right now we are working with the customer on it. 2 motherboards reported. It is x8dtl-if(already fixed) and x8sti-f(in process). Both seems to have an easy cure for it.

    Hello! How did you find that out? Sniffing?

    People will finally believe that I was NOT using default passwords

    Hope that this won't get into public. If you find a way of patching it, it would be interesting to let people in this thread (that are having issues) know how to fix.

  19. #94
    No sniffing. Just our customer was researching about this issue because of spam reports and found it.


    Quote Originally Posted by yourwebhostereu View Post
    Thanks, please keep us updated when the new firmware is released.
    Sure! For now I have only x8dtl-if firmware with fix.

  20. #95
    Join Date
    Dec 2004
    Posts
    569
    Quote Originally Posted by Ryan G - Limestone View Post
    I'm not sure about the H8SGL-F but on the X8SIL-F if you try to login with no user/pass entered it does not work. If you try to login with the username Anonymous it tells you, you need a password. If you try Anonymous/Anonymous it does not work either.
    It seems the anonymous account can be accessed over SSH by using an empty username, default password on a X9SCL-F "admin".

    $ ssh -l "" xx.xx.xx.xx
    @xx.xx.xx.xx's password:
    Auth User/Pass with PS...pass.

    ATEN SMASH-CLP System Management Shell, version 1.00
    Copyright (c) 2008-2009 by ATEN International CO., Ltd.
    All Rights Reserved


    ->
    So make sure you change the password of the "anonymous" account as well.

  21. #96
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,615
    As a matter of procedure we have always disabled the 'anonymous' account. I just tried SSH a few BMC's and they all seem to hang after the key exchange; I never get a password prompt. Would be nice to just disable SSH.
    Fast Serv Networks, LLC | AS29889 | DDOS Protected | Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  22. #97
    Join Date
    Dec 2004
    Posts
    569
    Quote Originally Posted by FastServ View Post
    I just tried SSH a few BMC's and they all seem to hang after the key exchange; I never get a password prompt.
    Your SSH client probably tries to login using public key authentication, which the BMC does not like.

    If that happens, try adding "-o PreferredAuthentications=password,keyboard-interactive" to your ssh command line.
    (I have that by default in my ssh_config, so my example didn't contain that)

  23. #98
    Join Date
    May 2006
    Location
    NJ, USA
    Posts
    6,645
    Quote Originally Posted by FastServ View Post
    As a matter of procedure we have always disabled the 'anonymous' account. I just tried SSH a few BMC's and they all seem to hang after the key exchange; I never get a password prompt. Would be nice to just disable SSH.
    Just realized my personal machine is running very dated firmware (over a year old) and ADMIN passowrd was the default..

    Flashed and updated now

    Welpp.. now my sensor readings are missing CPU temp as it doesnt work and system temp still doesnt work..
    AS395558

  24. #99
    Join Date
    Nov 2005
    Posts
    305
    I just realized that all our servers have this default password. I thought that the anonymous account would just work locally. I have already changed it for 20 servers, lot of password changes to do today.

  25. #100
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,615
    Quote Originally Posted by Maxnet View Post
    Your SSH client probably tries to login using public key authentication, which the BMC does not like.

    If that happens, try adding "-o PreferredAuthentications=password,keyboard-interactive" to your ssh command line.
    (I have that by default in my ssh_config, so my example didn't contain that)
    Ah yes, you're right. Looks like disabling the anonymous (e.g. "" in ssh) account seems to work, tried both 'ADMIN' and blank password:

    Auth User/Pass with PS...fail...Please reconnect!.
    Fast Serv Networks, LLC | AS29889 | DDOS Protected | Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

Page 4 of 6 FirstFirst 123456 LastLast

Similar Threads

  1. Supermicro IPMI Issue
    By XFactorServers in forum Colocation, Data Centers, IP Space and Networks
    Replies: 9
    Last Post: 08-23-2010, 02:29 PM
  2. SuperMicro 's IPMI
    By Peter-SexyWing in forum Colocation, Data Centers, IP Space and Networks
    Replies: 16
    Last Post: 07-10-2010, 04:51 PM
  3. supermicro ipmi installation
    By phactor in forum Systems Management Requests
    Replies: 5
    Last Post: 04-02-2010, 02:57 PM
  4. Supermicro IPMI
    By opax in forum Colocation, Data Centers, IP Space and Networks
    Replies: 6
    Last Post: 04-29-2009, 12:13 PM
  5. Supermicro IPMI
    By DevelopAl in forum Colocation, Data Centers, IP Space and Networks
    Replies: 14
    Last Post: 03-10-2006, 02:17 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •