Results 1 to 16 of 16

Thread: Bricked IPMI

  1. #1
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,615

    Bricked IPMI

    Anyone ever uploaded the wrong IPMI firmware and bricked the IPMI interface?

    After the incorrect firmware was loaded, the web interface became unresponsive and the BMC is unresponsive via ipmicfg or ipmitool - factory reset or reboot just hangs indefinitely.

    Resisting the urge to power cycle the box in case IPMI hangs the POST process.

    Am I right in guessing the mainboard will need to be RMA'd?

    *Note - let this be a warning/disclaimer to those offering unrestricted customer access to IPMI
    Last edited by FastServ; 12-04-2013 at 02:04 PM.
    Fast Serv Networks, LLC | AS29889 | DDOS Protected | Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  2. #2
    Join Date
    Dec 2005
    Posts
    3,110
    I'm guessing this is a Supermicro -F board with IPMI inbuilt?

    It may not have been the wrong firmware, I've had problems updating some lately even with the right firmware. Not been able to schedule an outage yet to get a proper look at it.

  3. #3
    Join Date
    Dec 2004
    Posts
    569
    On some boards I also have seen the update file upload fail repeatedly when using HTTPS to access the webinterface.
    And then you can no longer login because an upgrade is in progress...
    Required a power cycle, and uploading through insecure http.

  4. #4
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,615
    Quote Originally Posted by PCS-Chris View Post
    I'm guessing this is a Supermicro -F board with IPMI inbuilt?

    It may not have been the wrong firmware, I've had problems updating some lately even with the right firmware. Not been able to schedule an outage yet to get a proper look at it.
    Yes it's a -F.

    I know it wasn't the correct firmware at least according to SM's site, but it was 'similar' e.g. correct format but a higher number intended for newer boards.

    Would be nice if there was some way to force another firmware load or even a reset from the command line, but from what I can tell if the BMC isn't responding there's nothing you can do.

    Even 'modprobe ipmi_si' hangs indefinitely.

    Quote Originally Posted by Maxnet View Post
    On some boards I also have seen the update file upload fail repeatedly when using HTTPS to access the webinterface.
    And then you can no longer login because an upgrade is in progress...
    Required a power cycle, and uploading through insecure http.
    This seems to not be the case. It was via HTTP and the upload completed (progress all the way to 100). It said wait for the reboot, but it never rebooted. Additional attempts to upload firmware ended up with an empty page instead of a firmware versions page. Eventually, the web interface became unresponsive along with ipmicfg although it still pings.
    Fast Serv Networks, LLC | AS29889 | DDOS Protected | Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  5. #5
    I did this to a board as well, and the IPMI controller is completely dead at this point. Supermicro told me it needs to be RMA'd to fix it.

    If you've got physical access, they generally have a 'IPMI health LED' on the motherboard. If that's not flashing, I believe the IPMI controller is not actually executing code. If the IPMI controller isn't actually executing code, there's no way to tell it to write new firmware.

    You might have luck with some of the DOS based tools, but I doubt it.

    The only alternative I can think of is flashing the IPMI controller via some sort of external programmer and FlashRom. This would likely require figuring out which chip actually contains the firmware, and hoping that FlashRom supports it.

  6. #6
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,615
    Quick question, does the POST complete with a non-functioning IPMI? You know, where it says "waiting for IPMI...blah blah". At this point I'm over trying to get IPMI back, but am more concerned about the board's ability to POST in the case of a power cycle.
    Fast Serv Networks, LLC | AS29889 | DDOS Protected | Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  7. #7
    With the IPMI controller enabled (there's a jumper on the board), mine will not complete POST. With an old BIOS, it hung indefinitely midway through. With the latest BIOS, it just restarts continuously saying something like 'IPMI controller initialized, machine will now reset'.

    With the IPMI controller disabled, it boots fine.

  8. #8
    Join Date
    Mar 2005
    Location
    Indiana, USA
    Posts
    937
    I have an X9SCL+-F board that is stuck on 2.38 and I'd like to upgrade to 3.15. I upgraded a bunch of boards to 3.15, but this one just won't go. In fact you can reset the device but all configuration pages are blank. I do see the new Power Consumption option in the menu though. It half way upgraded, so I don't know what happened. The box is currently leased to a client, so I can't really power cycle it or access the terminal to do ipmitool stuff.
    Jason Canady • Unlimited Net, LLC
    812.669.0551 • sales [at] unlimitednet.us
    Midwest Hosting Solutions • AS11990
    Dedicated Server Hosting

  9. #9
    Join Date
    Jun 2010
    Location
    AZ
    Posts
    383
    We updated ours a few weeks ago with the correct firmware and it bricked the IPMI. It still posts but we pulled the board and put in a replacement. Its on its way to supermicro X9SCL-F.

    Weird part is the firmware upgrade says it completed successfully, but it wouldn't ping afterward, and the bios doesn't show any config options either to reconfigure the ipmi network.

    I'll try to post back and let you know if the RMA folks say anything.
    -Ryan K
    Quality Minecraft hosting at a price you can afford!
    Managed Dedicated and VPS hosting.
    evlbox.com

  10. #10
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,615
    Before upgrading any firmware, make sure to factory reset/default before doing so. Had a couple X9 boards brick where I didn't do that. My suspicion is that the BMC may get compromised or flaky over time and the defaults/reset before upgrading seems to help.
    Fast Serv Networks, LLC | AS29889 | DDOS Protected | Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  11. #11
    Join Date
    May 2013
    Location
    USA
    Posts
    931
    Quote Originally Posted by FastServ View Post
    Quick question, does the POST complete with a non-functioning IPMI? You know, where it says "waiting for IPMI...blah blah". At this point I'm over trying to get IPMI back, but am more concerned about the board's ability to POST in the case of a power cycle.
    Old post, but I thought I'd add my two cents as this thread is a top result in Google for bricked Supermicro IPMIs.

    In my experience with an X8 generation Supermicro motherboard, the motherboard will POST even if the IPMI is bricked -- but it will take a long time, somewhere around 5-10 minutes.

    However, if you've bricked the IPMI by uploading the wrong firmware or the flash was interrupted, I doubt you'll be able to recover it. I've never been able to use the Supermicro DOS utilities to recover an IPMI that was bricked by a failed firmware flash.
    ▄▀▄ Brian Harrison, Lead Engineer - Reprise Hosting (AS62838)
    ▄▀▄ Deals on cheap dedicated server hosting. IPMI included! Unmetered bandwidth.
    ▄▀▄ Website migration, 24/7/365 support, basic server setup, 15 day money back.
    ▄▀▄ Looking for DEALS on self-managed cheap VPS hosting? Visit VPSHostingDEAL.com

  12. #12
    Join Date
    Oct 2000
    Posts
    1,769
    Power cycle the system, boot from a DOS boot disk, and use the DOS flash utility to re-flash with the proper firmware. Should fix it.
    [QuickPacket™] [AS46261]
    Located in Ashburn, VA, Los Angeles, CA, Chicago, IL, and Manchester, UK
    Since 2003 - 20+ Years! Dedicated Servers, Co-location, DDoS Filtering, Data Backup & More!

  13. #13
    Join Date
    May 2013
    Location
    USA
    Posts
    931
    Quote Originally Posted by qps View Post
    Power cycle the system, boot from a DOS boot disk, and use the DOS flash utility to re-flash with the proper firmware. Should fix it.
    Didn't work for me actually (X8 gen board). After a bad firmware flash, the yafukcs.exe DOS firmware flashing utility doesn't even recognize that there is a BMC.
    ▄▀▄ Brian Harrison, Lead Engineer - Reprise Hosting (AS62838)
    ▄▀▄ Deals on cheap dedicated server hosting. IPMI included! Unmetered bandwidth.
    ▄▀▄ Website migration, 24/7/365 support, basic server setup, 15 day money back.
    ▄▀▄ Looking for DEALS on self-managed cheap VPS hosting? Visit VPSHostingDEAL.com

  14. #14
    Join Date
    Oct 2000
    Posts
    1,769
    Quote Originally Posted by BrianHarrison View Post
    Didn't work for me actually (X8 gen board). After a bad firmware flash, the yafukcs.exe DOS firmware flashing utility doesn't even recognize that there is a BMC.
    Have you tried unplugging power, letting it sit for a while, then plugging back in and trying again?
    [QuickPacket™] [AS46261]
    Located in Ashburn, VA, Los Angeles, CA, Chicago, IL, and Manchester, UK
    Since 2003 - 20+ Years! Dedicated Servers, Co-location, DDoS Filtering, Data Backup & More!

  15. #15
    Join Date
    Sep 2015
    Location
    United States
    Posts
    968
    Quote Originally Posted by qps View Post
    Have you tried unplugging power, letting it sit for a while, then plugging back in and trying again?
    Sometimes when something is bricked; it's really bricked. You can use DOS recovery and jump all the pins you want but in the end it's just still bricked.

    Really, it's simple mathematics: the time invested in trying to fix a bricked BMC versus just replacing the board. If "best effort" recovery doesn't work, the cost to try and fix the board will likely exceed the cost of replacing the board.

    Worse yet is that some boards will be able to be recovered from a bad flash with the recovery utilities but other boards of the same exact model (and perhaps batch) won't. It's just the nature of electronics.

    My advice is that if "best effort" recovery doesn't work, replace the board.
    Cloud Mosaic by NortheBridge
    High Performance Global Cloud | Contact Us: hello@northebridge.com
    Apps & Stacks: WordPress | Magento | Drupal | Ghost | Redmine | LAMP
    Node.js | Gitlab | MEAN | Nginx | Django | cPanel | Plesk | Windows & Linux

  16. #16
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681
    Old thread but typically we just send to SM, they fix it and send it back.
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

Similar Threads

  1. Replies: 0
    Last Post: 03-03-2012, 05:45 PM
  2. IPMI
    By loveweb in forum Colocation, Data Centers, IP Space and Networks
    Replies: 38
    Last Post: 05-19-2010, 02:49 PM
  3. Ipmi
    By Dualism in forum Colocation, Data Centers, IP Space and Networks
    Replies: 2
    Last Post: 04-16-2007, 10:46 AM
  4. What is IPMI, how does it help?
    By voipfc in forum Dedicated Server
    Replies: 1
    Last Post: 03-04-2007, 11:39 AM
  5. IPMI or not?
    By gate2vn in forum Colocation, Data Centers, IP Space and Networks
    Replies: 52
    Last Post: 10-04-2006, 02:11 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •