Results 1 to 37 of 37

Thread: Broken IPMI

  1. #1

    Broken IPMI

    This is probably the third or fourth time I've seen this in two years. Basically a server goes down for some reason (whether network attack or CPU abuse, etc.), IPMI works briefly, and then suddenly is stops functioning completely. Sometimes it happens right after an IKVM reset, sometimes I just try to log in to find it unusable. The web interface refuses to load from then on. I've seen it happen on two different, common, SuperMicro boards.

    I'm running CentOS 6 and have ipmitool installed, but it doesn't seem to offer any solution. What might cause IPMI to act like this and is there any fix short of replacing the motherboard?
    Last edited by Nick A; 01-14-2014 at 08:55 AM.

  2. #2
    Join Date
    May 2002
    Location
    Raleigh, NC
    Posts
    699
    Are you using the dedicated IPMI nic, or sharing NIC #1? Have you confirmed the firmware is updated to the latest version? And, lastly, is IPMI on a non-publicly accessible IP?
    Tranquil Hosting

  3. #3
    Dedicated NIC, not latest version, non-publicly accessible IP (not sure what bearing that may have, but curious).

  4. #4
    Join Date
    May 2006
    Location
    NJ, USA
    Posts
    6,456
    Quote Originally Posted by Nick A View Post
    non-publicly accessible IP (not sure what bearing that may have, but curious).
    DOS, hacked, etc..
    simplywww: directadmin and cpanel hosting that will rock your socks
    Need some work done in a datacenter in the NYC area? NYC Remote Hands can do it.

    Follow my "deals" Twitter for hardware specials.. @dougysdeals

  5. #5
    Yeah, neither of those.

  6. #6
    Join Date
    Nov 2006
    Location
    Pune, India
    Posts
    1,417
    Nick
    Which board are you using? We have seen this happen on X10SLM+-F
    LeapSwitch Networks Pvt. Ltd. - Managed VPS / Dedicated Servers India
    ASN 132335 - India - USA - Spain - Portugal - Ukraine - Germany
    █ Shared, Reseller, VPS, Dedicated Servers, Colocation
    Pay via - PayPal, Payza, Skrill, Credit/Debit Cards (USD / INR)

  7. #7
    X9SCL+-F and X9DRL-IF.
    RamNode - #1 SSD VPS
    High Performance SSD and SSD-Cached VPS
    NYC - LA - ATL - SEA - NL - 1Gbps - IPv6 - DDoS Protection - AS3842
    Get your super fast VPS today! - www.ramnode.com

  8. #8
    Join Date
    Apr 2009
    Location
    San Jose
    Posts
    69
    In my experience Supermicro makes pretty decent hardware but their software capabilities are pretty weak. The IPMI firmware going all the way back to the X7 series has been plagued with bugs. They appear to have source code control issues and you will sometimes see things like version numbers changing but firmware dates staying the same and vice versa. In my view they don't take reports of software bugs seriously or just don't have the talent to solve them. Eventually they will just stop doing updates for older version even though there are still obvious reproducible bugs.

  9. #9
    Quote Originally Posted by progent View Post
    In my experience Supermicro makes pretty decent hardware but their software capabilities are pretty weak. The IPMI firmware going all the way back to the X7 series has been plagued with bugs. They appear to have source code control issues and you will sometimes see things like version numbers changing but firmware dates staying the same and vice versa. In my view they don't take reports of software bugs seriously or just don't have the talent to solve them. Eventually they will just stop doing updates for older version even though there are still obvious reproducible bugs.
    Do you think a firmware upgrade would fix this?

  10. #10
    Join Date
    Apr 2009
    Location
    San Jose
    Posts
    69
    Quote Originally Posted by Nick A View Post
    Do you think a firmware upgrade would fix this?
    Maybe. Sometimes newer versions fix issues you may have but its also not too uncommon for new bugs to be introduced.

    I understand how frustrating it can be that you out-of-band management that you resort to in emergencies you can't trust.

    The suggestion to make sure your IPMI is not directly exposed to the public Internet in any way is good one. Putting it in private IP space that is only VPN accessible seems to cut down on some of the issues that happen with certain versions.

  11. #11
    Quote Originally Posted by ishan View Post
    Nick
    Which board are you using? We have seen this happen on X10SLM+-F
    We also have one X10SLM-F IPMI not working...

  12. #12
    Join Date
    Jan 2010
    Posts
    308
    Quote Originally Posted by Nick A View Post
    This is probably the third or fourth time I've seen this in two years. Basically a server goes down for some reason (whether network attack or CPU abuse, etc.), IPMI works briefly, and then suddenly is stops functioning completely. Sometimes it happens right after an IKVM reset, sometimes I just try to log in to find it unusable. The web interface refuses to load from then on. I've seen it happen on two different, common, SuperMicro boards.

    I'm running CentOS 6 and have ipmitool installed, but it doesn't seem to offer any solution. What might cause IPMI to act like this and is there any fix short of replacing the motherboard?
    Don't run SuperMicro. Their IPMI implementations are utter crap. I swore off SuperMicro 4 years ago because of their terrible IPMI. It's not worth any perceived "cost savings." FYI, Dell will beat any SuperMicro price if you work with the right sales person and buy more than a handful of servers.

  13. #13
    Join Date
    Aug 2004
    Location
    Dallas, TX
    Posts
    3,507
    Quote Originally Posted by Nick A View Post
    This is probably the third or fourth time I've seen this in two years. Basically a server goes down for some reason (whether network attack or CPU abuse, etc.), IPMI works briefly, and then suddenly is stops functioning completely. Sometimes it happens right after an IKVM reset, sometimes I just try to log in to find it unusable. The web interface refuses to load from then on. I've seen it happen on two different, common, SuperMicro boards.
    Bad RAM? We have a lot of supermicro and have not seen the issues you refer to.
    Dallas Colocation by Incero, 8 years and counting!
    e: sales(at)incero(dot)com 855.217.COLO (2656)
    Colocation & Enterprise Servers, SATA/SAS/SSD, secure IPMI/KVM remote control, 100% U.S.A. Based Staff
    SSAE 16, SAS70, Redundant Power & Network, Fully Diverse Fiber

  14. #14
    Join Date
    Nov 2006
    Location
    USA
    Posts
    762
    Quote Originally Posted by Nick A View Post
    This is probably the third or fourth time I've seen this in two years. Basically a server goes down for some reason (whether network attack or CPU abuse, etc.), IPMI works briefly, and then suddenly is stops functioning completely. Sometimes it happens right after an IKVM reset, sometimes I just try to log in to find it unusable. The web interface refuses to load from then on. I've seen it happen on two different, common, SuperMicro boards.

    I'm running CentOS 6 and have ipmitool installed, but it doesn't seem to offer any solution. What might cause IPMI to act like this and is there any fix short of replacing the motherboard?
    Have you tried performing an IPMI device and configuration reset using the Supermicro ipmicfg utility?

  15. #15
    Join Date
    Jun 2010
    Location
    AZ
    Posts
    383
    We had this happen on one of our X9SCL -F boards. It was about 2 1/2 years old and the IPMI sorta worked but eventually became unreachable. In the bios we couldn't even change the IP settings for the IPMI

    I could have paid a good amount of money for the DC tech's to reflash the bios and ipmi by hand but instead I sent it to supermicro as we still had labor warrenty covered (hardware expired)

    They fixed it and said a part of the bios or IPMI became corrupted and they reflashed bios + ipmi and tested it to make sure it worked. It took three weeks though (it was christmas/new years time) so you may want to have a replacement board shipped in or do the cross ship option.
    -Ryan K
    Quality Minecraft hosting at a price you can afford!
    Managed Dedicated and VPS hosting.
    www.evlgaming.com

  16. #16

    Security

    They had some serious security issue:

    https://community.rapid7.com/communi...ulnerabilities

  17. #17
    Join Date
    Nov 2005
    Location
    Nassau, The Bahamas
    Posts
    74
    There was a recent hack on Supermicro IPMI, did you know about it?
    Secure Hosting | Premium Cloud & Dedicated hosting solutions since 2001
    email: sales(at)securehost(dot)com +1-242-502-8700 24/7 support by phone/helpdesk
    Located in The Bahamas & Bermuda: Redundant Cooling, Power & Network, PCI DSS compliant, 7ms from USA

  18. #18
    This issue has nothing to do with IPMI hacks. Our IPMI is not accessible to the public as stated previously.

    I'll see about updating the firmware and let you guys know how it goes.
    RamNode - #1 SSD VPS
    High Performance SSD and SSD-Cached VPS
    NYC - LA - ATL - SEA - NL - 1Gbps - IPv6 - DDoS Protection - AS3842
    Get your super fast VPS today! - www.ramnode.com

  19. #19
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,571
    I've never seen it do that unless it was flashed with bad firmware or the mainboard/ps is going out. My guess - freak failures aside - there's a L1-L3 issue on your private network.
    Last edited by FastServ; 01-15-2014 at 08:00 PM.
    Fast Serv Networks, LLC | AS29889 | Fully Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  20. #20
    Quote Originally Posted by FastServ View Post
    My guess - freak failures aside - there's a L1-L3 issue on your private network.
    Like what?
    RamNode - #1 SSD VPS
    High Performance SSD and SSD-Cached VPS
    NYC - LA - ATL - SEA - NL - 1Gbps - IPv6 - DDoS Protection - AS3842
    Get your super fast VPS today! - www.ramnode.com

  21. #21
    Join Date
    Mar 2009
    Posts
    55
    We had similar issues with some supermicro servers. Bios and IPMI update fixed the problem.

  22. #22
    Are you sure IPMI is set to dedicated mode though? Even if it's the dedicated port it might not be using it. Could be failover/shared mode.
    Hyperconnezion - your fast path to success
    IT management, Cloud and Data Center Services in Asia Pacific

  23. #23
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,571
    Quote Originally Posted by concerto49 View Post
    Are you sure IPMI is set to dedicated mode though? Even if it's the dedicated port it might not be using it. Could be failover/shared mode.
    Good call, worth a shot forcing it to the dedicated port and not allowing it to switch around on its own.
    Fast Serv Networks, LLC | AS29889 | Fully Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  24. #24
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,571
    Quote Originally Posted by scurvy View Post
    Don't run SuperMicro. Their IPMI implementations are utter crap. I swore off SuperMicro 4 years ago because of their terrible IPMI. It's not worth any perceived "cost savings." FYI, Dell will beat any SuperMicro price if you work with the right sales person and buy more than a handful of servers.
    Dell's IPMI implimentation (DRAC) is full of bugs and vulnerabilities as well, if not more. That said I've never had any serious issues with IPMI in the last few years that a BMC reset via the web interface couldn't resolve.
    Fast Serv Networks, LLC | AS29889 | Fully Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  25. #25
    Setting it to dedicated fixed one of the three servers currently experiencing this issue. Thank you for pointing that out.
    RamNode - #1 SSD VPS
    High Performance SSD and SSD-Cached VPS
    NYC - LA - ATL - SEA - NL - 1Gbps - IPv6 - DDoS Protection - AS3842
    Get your super fast VPS today! - www.ramnode.com

  26. #26
    Quote Originally Posted by Nick A View Post
    Setting it to dedicated fixed one of the three servers currently experiencing this issue. Thank you for pointing that out.
    No problems - what other issues are you having with the other 2? Where are we at? Is it up or down? As in does the IP respond? Is it just showing a blank screen?
    Hyperconnezion - your fast path to success
    IT management, Cloud and Data Center Services in Asia Pacific

  27. #27
    Quote Originally Posted by concerto49 View Post
    No problems - what other issues are you having with the other 2? Where are we at? Is it up or down? As in does the IP respond? Is it just showing a blank screen?
    One IP responds to pings, one does not. Neither web console loads over a browser.
    RamNode - #1 SSD VPS
    High Performance SSD and SSD-Cached VPS
    NYC - LA - ATL - SEA - NL - 1Gbps - IPv6 - DDoS Protection - AS3842
    Get your super fast VPS today! - www.ramnode.com

  28. #28
    Join Date
    Feb 2002
    Location
    New York, NY
    Posts
    4,612
    I love Supermicro's hardware, but I can't recall them ever having reliable IPMI. It's a lot better than it used to be, but I still prefer to have an external serial console server and remote reboot PDUs, just in case.
    Scott Burns, President
    BQ Internet Corporation
    Remote Rsync and FTP backup solutions
    *** http://www.bqbackup.com/ ***

  29. #29
    Join Date
    Feb 2006
    Location
    Kepler 62f
    Posts
    14,873
    Quote Originally Posted by Nick A View Post
    Our IPMI is not accessible to the public as stated previously.
    I know several hosts that have this behind a VPN. Safer that way.
    And no need to waste more IPv4.
    || Need a good host?
    || See my Suggested Hosts List || Editorial: EIG/Site5/Arvixe/Hostgator Alternatives
    ||

  30. #30
    Quote Originally Posted by Nick A View Post
    One IP responds to pings, one does not. Neither web console loads over a browser.
    nmap the one that responds. What ports do you see open?

    Does ipmitool work remotely against them? (ipmitool mc info -I lan -H IPADDRESS -U whatever -P whatever)

    If ipmitool works remotely, try a cold reset (ipmitool mc reset cold _I lan...).

    Does your switch show the non-working controllers mac address at all? Do you have access to the OS? You could try resetting it via ipmitool again, just locally.


    I would love to be able to throw my own firmware on these things, but that's not an easy process :/

  31. #31
    Ok, the cold reset option seems to have triggered a favorable response on one of them. Combined with resetting the ipaddr and a few other things, that one works again.

    Now I'm down to one last server throwing fits. It apparently lost the ipaddr setting at some point and gives me the following error when I try to set it:

    # ipmitool lan set 1 ipaddr <IP address>
    Setting LAN IP Address to <IP address>
    LAN Parameter Data does not match! Write may have failed.

    I tried a cold reset and a few other tricks I know of to no avail. Any other ideas?
    RamNode - #1 SSD VPS
    High Performance SSD and SSD-Cached VPS
    NYC - LA - ATL - SEA - NL - 1Gbps - IPv6 - DDoS Protection - AS3842
    Get your super fast VPS today! - www.ramnode.com

  32. #32
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,571
    Try ipmicfg instead of ipmitool.
    Fast Serv Networks, LLC | AS29889 | Fully Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  33. #33
    Quote Originally Posted by Nick A View Post
    Ok, the cold reset option seems to have triggered a favorable response on one of them. Combined with resetting the ipaddr and a few other things, that one works again.

    Now I'm down to one last server throwing fits. It apparently lost the ipaddr setting at some point and gives me the following error when I try to set it:

    # ipmitool lan set 1 ipaddr <IP address>
    Setting LAN IP Address to <IP address>
    LAN Parameter Data does not match! Write may have failed.

    I tried a cold reset and a few other tricks I know of to no avail. Any other ideas?
    I'd probably try a hard power cycle (aka unplug from wall) of the machine. I've occasionally seen IPMI controllers get screwed up enough where nothing else helps :/

  34. #34
    Quote Originally Posted by devicenull View Post
    I'd probably try a hard power cycle (aka unplug from wall) of the machine. I've occasionally seen IPMI controllers get screwed up enough where nothing else helps :/
    That did it! All three are operational again.

    Thanks everyone.
    RamNode - #1 SSD VPS
    High Performance SSD and SSD-Cached VPS
    NYC - LA - ATL - SEA - NL - 1Gbps - IPv6 - DDoS Protection - AS3842
    Get your super fast VPS today! - www.ramnode.com

  35. #35
    Join Date
    Jul 2008
    Location
    Manhattan, NY Seattle,WA
    Posts
    1,392
    Hehe, I was about to suggest this for you. We've had to do this on a few systems.
    Sales/Support - sales @ purevoltage.com / 1-855-787-8658
    PureVoltage.com Premium Colocation, Dedicated & VPS Hosting, along with Remote Hands NY
    Enterprise Hardware with 6 Global Locations - Seattle | Dallas | Chicago | Los Angeles | New York | Amsterdam

  36. #36
    Join Date
    Mar 2005
    Location
    Indiana, USA
    Posts
    858
    Quote Originally Posted by devicenull View Post
    I'd probably try a hard power cycle (aka unplug from wall) of the machine. I've occasionally seen IPMI controllers get screwed up enough where nothing else helps :/
    I've had this happen too... that is why we use switched PDUs on top of IPMI.
    Jason Canady
    Unlimited Net, LLC AS11990
    Responsive, Reliable, Secure
    Dedicated Server Hosting

  37. #37
    Join Date
    Aug 2003
    Location
    Taiwan
    Posts
    1,061
    Quote Originally Posted by Nick A View Post
    That did it! All three are operational again.

    Thanks everyone.
    Seems to be same issue as mine months ago and cold reset fixed it
    www.hostinginside.com
    Taiwan Dedicated Server
    Taiwan, US, UK & Germany XEN Based VPS with RAID 10

Similar Threads

  1. Replies: 0
    Last Post: 03-03-2012, 05:45 PM
  2. Glibc broken, whole server broken
    By chielsen in forum Hosting Security and Technology
    Replies: 16
    Last Post: 07-25-2004, 02:25 AM
  3. Oh My God This Thread Is Broken! Links are broken too! WHT is broken!
    By websterworld in forum Web Hosting Lounge
    Replies: 67
    Last Post: 07-12-2004, 10:13 PM
  4. Is *.*.name Broken?
    By jmf314159 in forum Domain Names
    Replies: 1
    Last Post: 06-29-2004, 06:30 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •