IPMI and the Old Server Nobody Wants to Reboot

There is a machine in most homelabs that everybody is slightly afraid of. It has been up for four hundred days. It runs three things nobody has documented. Its kernel has security fixes waiting that require a reboot, and the reason it has been up for four hundred days is that nobody is entirely confident it will come back.

Mine is a second-hand rack server, bought for the price of a nice dinner, running the storage that everything else depends on. Its uptime was a source of pride for about a year and then quietly became a source of dread. The thing that eventually made it rebootable was a dedicated ethernet port on the back that I had ignored for two years, wired to a network segment nobody else can reach, giving me a console and a power button from anywhere.

That is what IPMI is for, and it is the single best argument for buying old enterprise gear over a shiny mini PC. It is also a security liability of such magnificence that people have written papers about it. Both things are entirely true.

What the BMC actually is

The Baseboard Management Controller is a small computer that lives on your server’s motherboard and has nothing to do with your server. It has its own ARM processor, its own memory, its own network stack, and its own Linux, and it is running right now even though the machine is powered off. It sees the host’s video output, presents virtual USB devices to it, reads every sensor on the board, and can assert the power and reset lines.

This is worth sitting with for a second. There is a second computer inside your computer, permanently powered whenever the PSU has mains, with complete control over the first one, running firmware last updated when the machine was new. It does not care what OS you run. It does not care whether your OS has crashed. It answers when nothing else does.

That property is exactly why it is useful and exactly why it is dangerous. IPMI is the reason I can reboot the scary server from a train. It is also a permanently-powered, rarely-patched, network-attached computer with total control over my storage, and the correct posture towards it is deep suspicion.

Setting it up so it is useful

First, decide how it is wired. Most server boards offer two modes: a dedicated management port, or sharing the first onboard NIC. Use the dedicated port. Shared mode means BMC traffic and host traffic arrive on the same cable, which makes isolating the BMC a matter of VLAN configuration rather than physics, and physics is more reliable than my ability to not make mistakes.

Configure it from the host itself, which spares you hunting for the default address:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

sudo apt install ipmitool
sudo modprobe ipmi_si ipmi_devintf

# Confirm the BMC is talking
sudo ipmitool mc info

# Static address on the isolated management segment
sudo ipmitool lan set 1 ipsrc static
sudo ipmitool lan set 1 ipaddr 10.10.99.20
sudo ipmitool lan set 1 netmask 255.255.255.0
sudo ipmitool lan set 1 defgw ipaddr 10.10.99.1

# Force the dedicated port (channel/option numbers vary by vendor)
sudo ipmitool raw 0x30 0x70 0x0c 1 0

sudo ipmitool lan print 1

Note the missing gateway you might expect to add. If the management segment has no route anywhere, the BMC cannot reach the internet and the internet cannot reach it. A default gateway on a BMC is a decision to be made deliberately, and my decision is no.

Now the credentials, because this is where the horror stories start:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# See what you inherited
sudo ipmitool user list 1

# Kill the vendor default
sudo ipmitool user disable 2

# Create your own, admin on channel 1
sudo ipmitool user set name 3 labadmin
sudo ipmitool user set password 3
sudo ipmitool channel setaccess 1 3 link=on ipmi=on callin=on privilege=4
sudo ipmitool user enable 3

Second-hand server boards routinely arrive with the factory account still enabled and the password printed on a sticker or, worse, documented in a manual anyone can download. ADMIN/ADMIN is a real default on a very popular family of boards. If you bought used, assume the previous owner also left a user behind, and read the full list rather than skimming it.

Serial-over-LAN, which is the part worth having

The Java KVM applet in a 2011 BMC is a genuine ordeal. It needs a Java runtime nobody has installed, then a certificate exception, then an unsigned-applet override, and by then you could have driven to the machine. Some vendors added an HTML5 console in later firmware; if yours has it, use it.

If it does not, skip the whole mess and use serial-over-LAN. It is text, it works in a terminal, and it is the interface I actually use:

1
2
3
4
5

# Host side: put the console on ttyS1 (verify with `dmesg | grep ttyS`)
# /etc/default/grub
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS1,115200n8"
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=1"

1
2
3
4
5
6

sudo update-grub
sudo systemctl enable --now [email protected]

# From your laptop, over the mesh, into the management segment
ipmitool -I lanplus -H 10.10.99.20 -U labadmin -a sol activate
# (~. to escape)

That gives you the boot messages, the GRUB menu and a login prompt on a machine whose network stack never came up. It is the difference between “I will reboot it when I am next physically there” and “I will reboot it now and watch what happens”.

The -a flag prompts for the password rather than taking it on the command line, which keeps it out of your shell history. -I lanplus selects IPMI 2.0 with proper encryption; plain -I lan is version 1.5 and sends things in the clear.

The everyday commands, which are the ones that make the scary server unscary:

1
2
3
4
5

ipmitool -I lanplus -H 10.10.99.20 -U labadmin -a power status
ipmitool -I lanplus -H 10.10.99.20 -U labadmin -a power cycle
ipmitool -I lanplus -H 10.10.99.20 -U labadmin -a chassis bootdev bios
ipmitool -I lanplus -H 10.10.99.20 -U labadmin -a sdr list          # all sensors
ipmitool -I lanplus -H 10.10.99.20 -U labadmin -a sel list          # event log

sdr list is a free monitoring feed — every fan, every temperature, every voltage rail, read from outside the OS. sel elist shows the hardware event log with timestamps, and it is the first place to look when a machine has rebooted for no reason you can find in journalctl. A correctable-ECC storm at 04:12 explains a great deal. It pairs naturally with the sort of monitoring for boxes with no agent that most homelabs skip.

Virtual media, and the bare-metal restore you can do from the sofa

The capability that surprised me most is virtual media. The BMC can present an ISO on your laptop to the host as a USB CD drive, over the network, before any OS exists. You can install a distribution on a machine you cannot see, from a different building, with no USB stick and nobody standing at the rack.

Most vendors expose this only through the web interface, which is the same Java-encrusted interface I have just told you to avoid. Some support it over ipmitool; more useful, many support it over a plain HTTP or NFS mount that the BMC pulls from itself:

1
2

# Point the BMC at an ISO served from the OOB box, then boot from it once
ipmitool -I lanplus -H 10.10.99.20 -U labadmin -a   chassis bootdev cdrom options=persistent=false

The persistent=false is the part to get right. It sets the boot device for the next boot only, so a machine that fails its install falls back to the disk instead of looping the installer forever while you are three hundred miles away.

This is what makes a genuine bare-metal restore possible without travel, and it changes what “disaster recovery” means for a homelab. The rehearsal is the same one I keep banging on about — practise the restore before you need it, because discovering that your BMC’s virtual media has been broken since a firmware reset is a discovery to make on a quiet Sunday.

The security situation, stated plainly

BMC firmware is some of the worst-audited code that will ever run on your network. The reasons are structural: it is written by the board vendor rather than a software company, it is derived from a reference implementation shared across the industry, it ships and is then abandoned, and the update process is frightening enough that nobody performs it. Public research has repeatedly found authentication bypasses, unauthenticated information disclosure, and remote code execution across multiple vendors’ BMCs.

The protocol itself has a design problem that no patch fixes. IPMI 2.0’s RAKP authentication handshake returns a salted hash of the user’s password to anyone who asks, before authentication. That is in the specification. An attacker who can reach the port can request the hash and crack it offline at their leisure, and a BMC where you can reach port 623 is a BMC where you have a password-cracking exercise rather than an access control.

There is exactly one mitigation that works, and it is not a firmware update: the BMC must not be reachable from anything that could be compromised. Not the LAN. Not the IoT VLAN. Certainly not the internet — search engines for internet-connected devices index exposed BMCs by the tens of thousands, and every one of them is somebody’s very bad day in waiting.

What I do, and what I would suggest:

• A management segment with no route to anything, following the same reasoning as keeping untrusted devices off the network that matters — the logic is identical, and the BMC belongs on the untrusted side of it.
• Reachable only through the out-of-band rescue box, which sits on both segments and is itself only reachable over an authenticated mesh.
• Vendor default accounts disabled outright. Renaming them achieves nothing.
• The BMC’s web interface treated as compromised by default. I use ipmitool and touch the web UI once a year.
• Firmware updated if the vendor still ships updates, which for a 2011 board they do not.

Troubleshooting

Error: Unable to establish IPMI v2 / RMCP+ session — wrong password, or the account lacks privilege on that channel. ipmitool user list 1 from the host tells you which. It is almost never a network problem when the port is open.

SOL activates and shows nothing. Console redirection is not enabled in the BIOS, or your console= line points at the wrong ttyS. Check dmesg | grep ttyS on the host to see which port the BMC actually presents — ttyS0 and ttyS1 are both common and the vendor documentation is often wrong.

SOL is garbled. Baud mismatch. The BIOS setting, the GRUB setting and the getty must all agree. 115200 everywhere.

The BMC stops responding but the host is fine. BMCs hang. It is routine. ipmitool mc reset cold from the host restarts it without touching the OS. Remember this one — the panic when the management port dies is out of all proportion to the fix.

Power commands accepted, nothing happens. Some boards need power off then power on rather than power cycle, and some ignore commands while the host is mid-POST. chassis power reset is the blunter instrument.

Fans at full speed after a firmware reset. Vendor fan tables reset to “safe”, which means “aircraft”. Some boards let you set fan mode over raw IPMI commands; the raw bytes are vendor-specific and searching for your exact model number is the only way.

Is it worth it?

If you already have a server with a management port: yes, absolutely, spend the hour. It converts your least-touchable machine into one you can reboot from a train, and it gives you a hardware event log that will one day explain something otherwise inexplicable. The only real cost is doing the network isolation properly, and you should be doing that anyway.

If you are choosing between old enterprise gear and a mini PC: this is the strongest thing on the server’s side of the ledger, and it is not enough on its own. That old machine idles at a hundred watts or more, and the maths on that is grim — grim enough that I gave it its own article. IPMI is a genuine capability that mini PCs simply do not have. It is also the reason your electricity bill has a step change in it.

If you cannot isolate it: do not enable it. A BMC on your flat LAN is worse than no BMC. That sounds dramatic and I mean it literally — you would be adding an unpatchable computer with total control over your storage to the same network as everything else you own, in exchange for the convenience of not walking to the rack. Walk to the rack.