Server BMCs can need to be rebooted every so often

However, as this fleet was used for computation, after a while they noticed that whenever the BMC stopped working, performance of the system increased by almost 10% or so. Definitely a non-neglible amount. So they kept the machines running in the broken BMC state for as much as possible.

Hardware arrived a day or two late, got it unpacked and carted upstairs. Spent a few days installing it (had to build a deployment environment on my Mac in a VM because the servers had no CDROM drive, only NetBoot and USB; our centralised imaging system hadn't been built with "installs outside of core DC network" in mind).

Discovered that the machines had been ordered with iDRACs, not DRACs. Damn things did not work until the box was booted into Linux and a driver to interface/activate the board was loaded. This, of course, worked wonderfully when the kernel locked up with a panic.

The machines were being used to test custom kernel modifications. Panics and lockups were common. Every time it happened, the engineers in the US had to ask "smart hands" to go reset the machine in the cage.

Experiment was a success though. Turned into the foundation for how a very large edge network was built.

This is one of those things that I'd probably be willing to spend a ton of time analysing to find the root cause if I could. Dump the memory image, debug the code, and figure out why it's just that (and possibly other) keys. My guess just from the description is that a bitflip happened exactly to the entry of a character map translation table or similar.

Incidentally, at a previous gig, the software that handled issuing BMC commands had a BMC reboot as the first step in any script it ran.

The device in question didn't had NTP so to get accurate BMC logs timing it had to be synced from host OS

BMC is them fucking up the watchdog implementation, that worked on wall time.

Which means if you changed time in-between OS sending watchdog ping and BMC checking for timeout, you got system reboot.

But it was intermittent enough that you might set the cron to update the BMC clock (because you want accurate time on it for log correlation) and be lucky enough time to not connect to the cause when it finally happens.

Also we generally observed that BMCs got better, 10 years server were lucky if it was working and not crashing every few weeks (all mostly on IBM gear), to the point we had to stop alerting on that, and it was unreliable as method for OOB access, so far anything newer than few years have been fine. Althought I have no idea how you make modern hardware take minutes to return list of sensors(I'm looking at you Lenovo)

And the worst are those that use java applets or webstart and require a ancient java version.

An idea I have floating around in the back of my head is to just use a raspberry pi as a bmc on consumer grade hardware, I am sure it would turn out more complicated than this, but basically just hook the power button, i2c, and other headers to the pi's gpio. now you have a bmc that runs an off the shelf os.

Restart your management devices!

* [ASRock BMC] The BMC firmware updater sometimes causes the NICs to get stuck in a bad state where every NIC has the same MAC address. This can be resolved via a proprietary UEFI application for reflashing the correct MAC address.

* [Dell iDRAC] Local authentication randomly stops working due to some tmpfs running out of space (can see the message if there's an active SSH session). IPMI/SSH occasionally works well enough to issue a reboot command, but when it doesn't, sending the BMC reset comand to /dev/ipmi0 in the server OS is needed.

* [Dell iDRAC] Setting an asset tag via the IPMI DCMI command has a 15 character limit. If that limit is exceeded, the success response is still returned, but when querying the asset tag, random junk data longer than 15 bytes is returned. If I had to guess, I bet there was an sprintf() call somewhere in there :). This was fixed in newer iDRAC firmware. Now, it stores the last 15 bytes of the asset tag instead of returning an error.

* [Lenovo IMM] The shift or alt key sometimes gets stuck on the emulated keyboard without having used the remote console since the last BMC reboot. Can't be fixed by repressing the button, neither physically nor via the virtual keyboard. BMC reboot required.

* [Lenovo IMM] Booting the UEFI shell sometimes crashes both the system and the BMC.

* [Lenovo IMM] BMC and BIOS update sometimes claims to have succeeded, but didn't actually take effect.

* [Lenovo IMM] Rebooting the BMC via the web UI or SSH sometimes doesn't work. Making 50+ simultaneous requests to the login page is enough to crash and restart some component that allows the BMC reboot command to work again though.

* [Supermicro BMC] The BIOS update sometimes doesn't fully upload, but claims that it did. It still parses the header, so the new/old version fields look correct. Sometimes, rebooting the BMC and reflashing works. Other times, only a USB drive + USB keyboard + a recovery key combination fixes it.

* [Supermicro BMC] The remote console sometimes completely fails to initialize (though I've only seen this on servers where the BMC uptime was measured in years). Not just a blank screen. The GPU device was just... gone.

* [Supermicro BMC] Various IPMI commands just lie about successful execution. For example, setting the asset tag via the FRU succeeds, but has no effect. Those commands require toggling a write lock bit via an OEM command, which I only found by reverse engineering. Other commands, like the set asset tag DCMI command, leave the data in a corrupted state until a BMC reboot.

And finally, not really a bug, but an interesting thing about the Lenovo IMM. Instead of exposing information via standard IPMI features, like FRU or DCMI commands, the Lenovo IMM implements a virtual filesystem over OEM IPMI commands. These are commands, like (my naming) open_ro, open_rw, get_size, read, write, and close. They sometimes fail too. I think I ended up making all commands retry up to 10 times with a 5 second delay. At least Lenovo gets error return values right :).

To query the asset tag, you have to open_ro the "config.efi" file, get the size (because read-until-EOF doesn't always work), do a read loop, and close the file. Then, you have the XML data from the file you can query (20 seconds later due to retries). (If anyone ever needs to deal with the Lenovo IMM programmatically, I'd highly recommend the pyghmi [1] library. Wish I knew about it before reverse engineering their proprietary commands...)

[1] https://opendev.org/x/pyghmi

AFAIK larger vendors like Dell and HP have their own thing.

The latest iDRACs (with the fancy blue GUI) work a little better. I have no numbers to back all that, its just a feeling, maybe the problems are yet to come.

Given the Dell pricing, they should be better. But i've heard from collegues that ILO and others are not much better either.

I'm doing a few timing sensitive projects involving hardware timestamping in the NICs. Does this mess with say timing variability? I've disabled the BMCs out of paranoia but I don't know the topology inside.

I also remember a long series of machines (Supermicro I think) whose BMCs would stop responding after a few weeks if there was too much traffic on the network.

Also those BMC that hijack silently eth0, breaking your server's bonded connections, when the main BNC connector get disconnected for some reason...

We had IDSes that were happy, up, network interface counts were climbing, but they were STONE DEAF to the network traffic we were actually interested in.

Such a silly comment. If your BMC is updated then it will use a recent version of Linux.