Fail-Safe Remote Access: Building an Out-of-Band Path

I was in a different country when my home router decided that a firmware update was a good idea at 3am. It came back up with a factory configuration. Everything behind it — the VPN endpoint, the servers, the mesh overlay’s exit node, the ability to fix any of it — was on the far side of a device that had cheerfully forgotten who it was. My beautiful layered remote-access setup, with its hardware keys and its mesh and its careful ACLs, had precisely one thing in common across every layer: it all went through that box.

That is the lesson, and it is not about routers. Every remote-access method you have depends on some component, and if all your methods depend on the same component, you have one method wearing several hats. An out-of-band path is a second route in that shares as little as possible with the first. This is a piece about building one that is deliberately, aggressively boring.

The dependency audit, which is the actual work

Before buying anything, write down what your primary path depends on. Be honest and be granular. Mine, at the time:

• Mains power to the rack
• The ISP’s fibre line
• The ISP-supplied ONT
• My router
• The managed switch
• The machine running the VPN endpoint
• That machine’s OS booting successfully
• The mesh overlay’s control plane being reachable

Eight components, and a failure of any one takes out every access method I had. The VPN, the mesh, the tunnel, SSH — all of them, simultaneously. I had built layers that looked independent and were not, because they all traversed the same physical and logical chain.

Now write the second path’s list and count the overlaps. The goal is not zero overlap, which is impossible — mains power is going to be on both lists unless you are considerably more serious than me. The goal is that the likely failures are not shared. Ranked by how often they actually bite:

1. Something I did. A firewall rule, a config typo, an upgrade. Overwhelmingly the most common, and the most embarrassing.
2. A machine that will not boot. Failed kernel, full disk, dead SSD.
3. The router. Firmware, crash, config loss.
4. The ISP. Line down, CGNAT change, IP rotation.
5. Power. Cut, or a UPS that lied about its battery.

An out-of-band path that shares the router covers exactly one of those five. That is why the useful answer almost always involves a different physical network.

The tiers, honestly ranked

Tier zero: a second machine on the same network. A small always-on box — a Pi, a mini PC, an old laptop — running its own SSH and its own mesh client, doing nothing else, updated rarely, configured once. This covers failure mode two entirely and is nearly free. It is not out-of-band in any meaningful sense, because it shares the router and the line, and it is still the highest-value thing on this list because “the main server will not boot” is the failure you will actually have. Everyone should have this. Most people stop here and that is a defensible place to stop.

Tier one: a cellular modem. A 4G/5G dongle or a small router with a data SIM, plugged into a low-power always-on box. This is the first genuinely independent path — different physical medium, different provider, different everything up to the point where it enters your flat. A low-usage data SIM costs a few pounds a month and the box costs less than a night out. This is the tier I would push most people towards.

Tier two: hardware management. IPMI, iDRAC, iLO, or a discrete KVM-over-IP. Power cycling and console access on a machine whose OS is thoroughly wedged. This covers the “kernel panic on boot” case that nothing else touches, and it comes with genuine security baggage that I will get to. Enterprise gear has this built in; if you run an old server with a management port, you already have it and possibly have not configured it.

Tier three: a networked PDU. Remote power cycling, per outlet. The nuclear option, and the one that resolves “the thing is hung so hard nothing responds”. Second-hand units are affordable and dreadfully insecure, which is fine when they are not routable from anywhere interesting.

Tier four: a friendly human. A flatmate, a neighbour, a partner, and a laminated card by the rack saying “if I ring you, hold this button for ten seconds”. Free, effective, and by an enormous margin the fastest recovery for a physical problem. Do not be too proud for this tier. It is the only one on the list that can also unplug something that is on fire.

The serial console, which is the cheapest tier-two there is

Between “a second machine on the network” and “buy a KVM-over-IP” sits an option people forget exists, because it looks like 1994. A USB-to-serial cable from the rescue box into the server’s serial header, and getty on the far end, gives you a console that survives the network stack being gone, the SSH daemon being dead, and the disk being full.

1
2
3
4
5
6
7
8

# On the target machine: console on serial, and keep the local one
# In /etc/default/grub:
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200n8"
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0"

sudo update-grub
sudo systemctl enable --now [email protected]

Then from the rescue box, screen /dev/ttyUSB0 115200 and you are looking at the boot messages, the GRUB menu, and a login prompt. You can pick an older kernel. You can watch a panic scroll past instead of guessing at it. You can log in when the network never came up at all.

The cable costs about a tenner. The catch is that plenty of mini PCs have no serial header at all, which is one of the quiet advantages of the old rack-mount gear people are so keen to replace. If your machines have the header, wire it up — it is the highest ratio of capability to money on this entire list.

What the cellular box looks like

The design principle is that the out-of-band box must be simpler than the thing it rescues. Every service you add to it is another way for it to fail on the day you need it. Mine is a Pi with a data SIM, and it runs a mesh client and sshd. That is the complete inventory.

1
2
3
4
5
6

# On the OOB box: mesh client, tagged, with a subnet route
# to the management network, and key expiry disabled in the console
sudo tailscale up \
  --advertise-tags=tag:oob \
  --advertise-routes=192.168.1.0/24 \
  --accept-dns=false

--accept-dns=false matters more than it looks. If the box takes DNS from the tailnet and the tailnet is having a bad day, your rescue path loses name resolution at the worst possible moment. Give it a static resolver and static addresses for everything it needs to reach. Names are a dependency; the rescue box should have as few as possible.

Then the piece everyone forgets:

1
2
3
4
5
6
7

# /etc/systemd/system/oob-watchdog.service — reboot if the SIM path dies
[Unit]
Description=OOB connectivity watchdog

[Service]
Type=oneshot
ExecStart=/usr/local/bin/oob-check.sh

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

#!/usr/bin/env bash
# /usr/local/bin/oob-check.sh
set -u
FAILFILE=/var/lib/oob-fails
fails=$(cat "$FAILFILE" 2>/dev/null || echo 0)

if ping -c3 -W5 1.1.1.1 >/dev/null 2>&1; then
    echo 0 > "$FAILFILE"
    exit 0
fi

fails=$((fails + 1))
echo "$fails" > "$FAILFILE"
logger -t oob-check "connectivity check failed ($fails)"

# Three consecutive failures over 15 minutes: cycle the modem, then reboot
if [ "$fails" -ge 3 ]; then
    echo 0 > "$FAILFILE"
    systemctl restart ModemManager
    sleep 60
    ping -c3 -W5 1.1.1.1 >/dev/null 2>&1 || systemctl reboot
fi

Paired with a timer running every five minutes. Crude, and it has recovered the box from a stuck modem three times. The failure mode of a rescue device is that it silently stops being reachable and you find out when you need it — so it must be capable of resuscitating itself, and you must know when it cannot.

Which brings us to the only part that is genuinely mandatory.

Monitor the path you never use

An untested out-of-band path is a story you tell yourself. It will be broken when you need it, because it has been broken for four months and nothing told you.

Put a dead-man’s-switch check on it: the box pings an external endpoint on a schedule, and if the ping stops arriving, you get told. A self-hosted Healthchecks instance does this well, with the enormous caveat that it must not live on the network you are monitoring. Host it elsewhere or use the hosted free tier. A monitor inside the failure domain it monitors is decoration.

And then actually use the thing. Once a quarter, deliberately: connect via the out-of-band path, do something real through it, confirm it works end to end. Put it in the calendar. This is the same discipline as testing your backups and it fails for the same reason — the check is boring, nothing is wrong today, and so it never happens.

The security problem you are creating

A second way in for you is a second way in for everyone else. This is where people build a rescue path that is also their largest vulnerability.

Management interfaces are appalling. BMC firmware is a genre of software with a long, colourful history of authentication bypasses, and the vendor stopped shipping updates for your board years ago. It must never touch a routable network. Put it on an isolated segment reachable only through the mesh, following the same logic as VLAN segmentation for anything you do not trust. Change the default credentials. Assume the web interface is exploitable, because it probably is.

PDUs are worse. Many second-hand units speak plain HTTP and have telnet on by default. Same treatment: isolated segment, no route from anywhere, reachable only via the rescue box.

The rescue box is a target. It runs old software, because you configured it once and left it, which was the entire design goal. That tension is real and the resolution is to give it as little as possible to attack: no public listener, mesh only, unattended security updates on, nothing else installed. Being reachable only over an authenticated overlay is what makes “rarely updated” survivable.

Troubleshooting

The SIM works when tested and not in an emergency. Data SIMs get deactivated for inactivity. Force traffic across it periodically — the watchdog’s pings do this as a side effect, which is half of why it exists.

Mesh client on the OOB box is offline after months. Node key expiry. Disable it for this device specifically. This is the single most common way a rescue path dies quietly.

IPMI console is blank or the Java applet will not launch. Old BMCs need old browsers and older Java. Check for an HTML5 console in the firmware; if there is not one, ipmitool gives you serial-over-LAN from the command line and dodges the entire mess.

Cellular is up, mesh will not connect. Carrier-grade NAT on the mobile network is blocking direct paths. It should relay and still work, slowly. If it does not, the carrier is filtering; a different APN sometimes helps.

Everything is fine and you still cannot get in. Check whether your rescue box gets its DNS or its NTP from the network that is down. Certificate and token validation fails on a box whose clock has drifted, and a rescue box that has been offline for a week has a drifted clock.

Is it worth it?

The Pi-on-a-SIM tier: yes, unreservedly, for anyone who travels or runs anything they care about. Under fifty pounds and a few pounds a month, and it converts “I am locked out until I fly home” into “I am mildly annoyed for ten minutes”.

IPMI: if you already have it, configure it properly and isolate it. Buying dedicated KVM-over-IP hardware for a homelab is hard to justify unless your machines are physically far away.

A PDU: only when you have actually experienced a hang so complete that nothing else worked. Wait for the pain. It will let you know.

The honest verdict is that the dependency audit at the top is worth more than any hardware on this list. Most people discover, on writing it down, that they have four access methods and one point of failure — and that the cheapest fix is a thirty-pound computer with its own SIM, sitting in a corner, doing almost nothing, for years.