Results 1 to 16 of 16
Thread: Bricked IPMI
-
12-04-2013, 01:59 PM #1Randy
- Join Date
- Aug 2006
- Location
- Ashburn VA, San Diego CA
- Posts
- 4,615
Bricked IPMI
Anyone ever uploaded the wrong IPMI firmware and bricked the IPMI interface?
After the incorrect firmware was loaded, the web interface became unresponsive and the BMC is unresponsive via ipmicfg or ipmitool - factory reset or reboot just hangs indefinitely.
Resisting the urge to power cycle the box in case IPMI hangs the POST process.
Am I right in guessing the mainboard will need to be RMA'd?
*Note - let this be a warning/disclaimer to those offering unrestricted customer access to IPMILast edited by FastServ; 12-04-2013 at 02:04 PM.
Fast Serv Networks, LLC | AS29889 | DDOS Protected | Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
Since 2003 - Ashburn VA + San Diego CA Datacenters
-
12-04-2013, 03:41 PM #2Web Hosting Master
- Join Date
- Dec 2005
- Posts
- 3,110
I'm guessing this is a Supermicro -F board with IPMI inbuilt?
It may not have been the wrong firmware, I've had problems updating some lately even with the right firmware. Not been able to schedule an outage yet to get a proper look at it.
-
12-04-2013, 04:34 PM #3Web Hosting Master
- Join Date
- Dec 2004
- Posts
- 569
On some boards I also have seen the update file upload fail repeatedly when using HTTPS to access the webinterface.
And then you can no longer login because an upgrade is in progress...
Required a power cycle, and uploading through insecure http.Maxnet
Offering automated dedicated server provisioning software
-
12-04-2013, 07:01 PM #4Randy
- Join Date
- Aug 2006
- Location
- Ashburn VA, San Diego CA
- Posts
- 4,615
Yes it's a -F.
I know it wasn't the correct firmware at least according to SM's site, but it was 'similar' e.g. correct format but a higher number intended for newer boards.
Would be nice if there was some way to force another firmware load or even a reset from the command line, but from what I can tell if the BMC isn't responding there's nothing you can do.
Even 'modprobe ipmi_si' hangs indefinitely.
This seems to not be the case. It was via HTTP and the upload completed (progress all the way to 100). It said wait for the reboot, but it never rebooted. Additional attempts to upload firmware ended up with an empty page instead of a firmware versions page. Eventually, the web interface became unresponsive along with ipmicfg although it still pings.Fast Serv Networks, LLC | AS29889 | DDOS Protected | Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
Since 2003 - Ashburn VA + San Diego CA Datacenters
-
12-04-2013, 10:07 PM #5Junior Guru
- Join Date
- Dec 2013
- Posts
- 194
I did this to a board as well, and the IPMI controller is completely dead at this point. Supermicro told me it needs to be RMA'd to fix it.
If you've got physical access, they generally have a 'IPMI health LED' on the motherboard. If that's not flashing, I believe the IPMI controller is not actually executing code. If the IPMI controller isn't actually executing code, there's no way to tell it to write new firmware.
You might have luck with some of the DOS based tools, but I doubt it.
The only alternative I can think of is flashing the IPMI controller via some sort of external programmer and FlashRom. This would likely require figuring out which chip actually contains the firmware, and hoping that FlashRom supports it.
-
12-04-2013, 10:19 PM #6Randy
- Join Date
- Aug 2006
- Location
- Ashburn VA, San Diego CA
- Posts
- 4,615
Quick question, does the POST complete with a non-functioning IPMI? You know, where it says "waiting for IPMI...blah blah". At this point I'm over trying to get IPMI back, but am more concerned about the board's ability to POST in the case of a power cycle.
Fast Serv Networks, LLC | AS29889 | DDOS Protected | Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
Since 2003 - Ashburn VA + San Diego CA Datacenters
-
12-04-2013, 10:34 PM #7Junior Guru
- Join Date
- Dec 2013
- Posts
- 194
With the IPMI controller enabled (there's a jumper on the board), mine will not complete POST. With an old BIOS, it hung indefinitely midway through. With the latest BIOS, it just restarts continuously saying something like 'IPMI controller initialized, machine will now reset'.
With the IPMI controller disabled, it boots fine.
-
12-05-2013, 12:23 PM #8
I have an X9SCL+-F board that is stuck on 2.38 and I'd like to upgrade to 3.15. I upgraded a bunch of boards to 3.15, but this one just won't go. In fact you can reset the device but all configuration pages are blank. I do see the new Power Consumption option in the menu though. It half way upgraded, so I don't know what happened. The box is currently leased to a client, so I can't really power cycle it or access the terminal to do ipmitool stuff.
Jason Canady • Unlimited Net, LLC
812.669.0551 • sales [at] unlimitednet.us
Midwest Hosting Solutions • AS11990
Dedicated Server Hosting
-
12-13-2013, 02:45 AM #9Aspiring Evangelist
- Join Date
- Jun 2010
- Location
- AZ
- Posts
- 383
We updated ours a few weeks ago with the correct firmware and it bricked the IPMI. It still posts but we pulled the board and put in a replacement. Its on its way to supermicro X9SCL-F.
Weird part is the firmware upgrade says it completed successfully, but it wouldn't ping afterward, and the bios doesn't show any config options either to reconfigure the ipmi network.
I'll try to post back and let you know if the RMA folks say anything.-Ryan K
Quality Minecraft hosting at a price you can afford!
Managed Dedicated and VPS hosting.
evlbox.com
-
12-14-2013, 03:55 PM #10Randy
- Join Date
- Aug 2006
- Location
- Ashburn VA, San Diego CA
- Posts
- 4,615
Before upgrading any firmware, make sure to factory reset/default before doing so. Had a couple X9 boards brick where I didn't do that. My suspicion is that the BMC may get compromised or flaky over time and the defaults/reset before upgrading seems to help.
Fast Serv Networks, LLC | AS29889 | DDOS Protected | Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
Since 2003 - Ashburn VA + San Diego CA Datacenters
-
11-15-2016, 03:29 AM #11Web Hosting Master
- Join Date
- May 2013
- Location
- USA
- Posts
- 931
Old post, but I thought I'd add my two cents as this thread is a top result in Google for bricked Supermicro IPMIs.
In my experience with an X8 generation Supermicro motherboard, the motherboard will POST even if the IPMI is bricked -- but it will take a long time, somewhere around 5-10 minutes.
However, if you've bricked the IPMI by uploading the wrong firmware or the flash was interrupted, I doubt you'll be able to recover it. I've never been able to use the Supermicro DOS utilities to recover an IPMI that was bricked by a failed firmware flash.▄▀▄ Brian Harrison, Lead Engineer - Reprise Hosting (AS62838)
▄▀▄ Deals on cheap dedicated server hosting. IPMI included! Unmetered bandwidth.
▄▀▄ Website migration, 24/7/365 support, basic server setup, 15 day money back.
▄▀▄ Looking for DEALS on self-managed cheap VPS hosting? Visit VPSHostingDEAL.com
-
11-15-2016, 10:59 AM #12Web Hosting Master
- Join Date
- Oct 2000
- Posts
- 1,769
Power cycle the system, boot from a DOS boot disk, and use the DOS flash utility to re-flash with the proper firmware. Should fix it.
[QuickPacket™] [AS46261]
Located in Ashburn, VA, Los Angeles, CA, Chicago, IL, and Manchester, UK
Since 2003 - 20+ Years! Dedicated Servers, Co-location, DDoS Filtering, Data Backup & More!
-
11-16-2016, 07:55 PM #13Web Hosting Master
- Join Date
- May 2013
- Location
- USA
- Posts
- 931
▄▀▄ Brian Harrison, Lead Engineer - Reprise Hosting (AS62838)
▄▀▄ Deals on cheap dedicated server hosting. IPMI included! Unmetered bandwidth.
▄▀▄ Website migration, 24/7/365 support, basic server setup, 15 day money back.
▄▀▄ Looking for DEALS on self-managed cheap VPS hosting? Visit VPSHostingDEAL.com
-
11-16-2016, 08:04 PM #14Web Hosting Master
- Join Date
- Oct 2000
- Posts
- 1,769
[QuickPacket™] [AS46261]
Located in Ashburn, VA, Los Angeles, CA, Chicago, IL, and Manchester, UK
Since 2003 - 20+ Years! Dedicated Servers, Co-location, DDoS Filtering, Data Backup & More!
-
11-17-2016, 06:49 AM #15
Sometimes when something is bricked; it's really bricked. You can use DOS recovery and jump all the pins you want but in the end it's just still bricked.
Really, it's simple mathematics: the time invested in trying to fix a bricked BMC versus just replacing the board. If "best effort" recovery doesn't work, the cost to try and fix the board will likely exceed the cost of replacing the board.
Worse yet is that some boards will be able to be recovered from a bad flash with the recovery utilities but other boards of the same exact model (and perhaps batch) won't. It's just the nature of electronics.
My advice is that if "best effort" recovery doesn't work, replace the board.██ Cloud Mosaic by NortheBridge
██ High Performance Global Cloud | Contact Us: hello@northebridge.com
██ Apps & Stacks: WordPress | Magento | Drupal | Ghost | Redmine | LAMP
██ Node.js | Gitlab | MEAN | Nginx | Django | cPanel | Plesk | Windows & Linux
-
12-09-2016, 11:18 AM #16Problem Solver
- Join Date
- Mar 2003
- Location
- California USA
- Posts
- 13,681
Old thread but typically we just send to SM, they fix it and send it back.
Steven Ciaburri | Industry's Best Server Management - Rack911.com
Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance
Similar Threads
-
[INFINITIE.net] E3 8GB|500GB|IPMI - $85 Dual E5645 Hex Core 48GB|2x500GB|IPMI - $399
By JohnSH in forum Dedicated Hosting OffersReplies: 0Last Post: 03-03-2012, 05:45 PM -
IPMI
By loveweb in forum Colocation, Data Centers, IP Space and NetworksReplies: 38Last Post: 05-19-2010, 02:49 PM -
Ipmi
By Dualism in forum Colocation, Data Centers, IP Space and NetworksReplies: 2Last Post: 04-16-2007, 10:46 AM -
What is IPMI, how does it help?
By voipfc in forum Dedicated ServerReplies: 1Last Post: 03-04-2007, 11:39 AM -
IPMI or not?
By gate2vn in forum Colocation, Data Centers, IP Space and NetworksReplies: 52Last Post: 10-04-2006, 02:11 AM