UPS Monitoring With NUT and Graceful Shutdowns

Buying a UPS feels like solving the power problem. It isn’t. An unmonitored UPS buys you eight minutes of runtime and then drops every machine in the rack exactly as hard as the mains did, except later. The battery has converted an instantaneous outage into a delayed one, and charged you €200 for the delay. What actually protects your data is the bit most people skip: something watching the battery, and something telling the machines to lie down properly before it runs out.

That something, on Linux, is Network UPS Tools. NUT is old, its documentation reads like a hardware manual from 2003, and its terminology is genuinely confusing the first time through. It is also the piece of software that has saved my ZFS pools during three multi-hour outages, so I have made peace with it.

Why the shutdown order matters more than the runtime

The instinct is to buy a bigger battery. Resist it. Runtime scales badly — doubling your VA rating roughly doubles the price and buys you maybe eight extra minutes — and the outages that hurt are the ones lasting hours, where no plausible battery helps. The battery’s job is to buy enough time for an orderly shutdown, and orderly shutdown takes about three minutes if you have thought about it and never happens if you haven’t.

The order is what matters. Consider a modest rack: a NAS holding the storage, a hypervisor running VMs whose disks live on that NAS over NFS, and a couple of standalone containers hosts. If the NAS goes down first, every VM has its storage yanked mid-write, and you discover which of your filesystems are actually crash-consistent. If the hypervisor goes down first — VMs quiesced, guest agents flushing buffers, NFS mounts unmounted cleanly — the NAS can then shut down with nothing in flight.

So the sequence you want, roughly:

1. On battery, immediately: stop discretionary work. Cancel the backup job, pause the scrub, stop the transcode. These are the things burning the most watts and holding the most open files.
2. After a low-battery threshold: shut down the compute layer. VMs first, then their hosts. Give guests a real timeout — 60 to 120 seconds — because a Windows VM will take its time and killing it early is exactly the failure you bought the UPS to avoid.
3. Last: the storage layer, then the UPS itself.

That last step surprises people. You want the UPS to cut its own output after the shutdown completes, because otherwise this happens: power returns for ninety seconds, machines boot, power dies again, and now the battery is at 4% and the machines are mid-boot with no runtime left. Telling the UPS to kill its outlets and only restore them when mains is stable is the difference between one clean shutdown and a boot loop that eats your filesystem.

Sizing the battery you actually need

Before any of this, work out what you are protecting. UPS units are sold in VA, which is apparent power, and the number you care about is watts, which is real power. The relationship depends on the power factor of your load, and modern switching supplies with active PFC sit around 0.95–0.99, while the UPS vendor’s marketing assumes something closer to 0.6. A “1500 VA” unit is frequently a 900 W unit, and the small print says so.

Take your measured idle draw — you have measured it, following idle power draw — and add the peak of whatever might plausibly be running when the lights go out. A rack idling at 90 W with a 160 W transient during a backup wants headroom to about 250 W. On a 900 W unit that is a 28% load, which is a comfortable place to live: the runtime curve is steep and non-linear, so a UPS at 25% load runs roughly four times as long as the same unit at 75%, rather than three times.

There is a related trap. Do not put a laser printer, a space heater, or anything else with a large inrush on the battery outlets. The UPS will either trip on overload during the printer’s fuser cycle or spend its life at a load factor that murders the battery. Those go in the surge-only sockets, which exist precisely for this and which everyone ignores.

Finally, accept the runtime you get. Line-interactive consumer units give you six to fifteen minutes on a modest rack, and the honest planning assumption is that you have three usable minutes after the grace period expires. Design the shutdown to fit inside three minutes and the battery stops being the constraint.

What NUT’s three daemons actually do

The terminology is the main barrier. NUT splits into three parts and the names are unhelpful:

• upsdrvctl / the driver — talks to the physical UPS over USB or serial. One driver per UPS. This is the piece that needs to know your hardware.
• upsd — a network server that exposes the driver’s readings to clients over TCP 3493. Even on a single machine, the client talks to the UPS through this daemon.
• upsmon — the client that watches upsd, decides when to panic, and runs the shutdown. Every machine you want to protect runs one of these.

That splits the world into two roles. The machine with the USB cable plugged into the UPS runs all three and is called the master (netserver mode). Every other machine on the same battery runs upsmon alone and is called a slave (netclient mode). The master waits for all slaves to report themselves down before it shuts itself off. That handshake is the whole reason NUT exists rather than a shell script.

A working configuration

Start on the master. Plug in the USB cable and let NUT identify the hardware:

1

sudo nut-scanner -U

1
2
3
4
5
6
7

[nutdev1]
        driver = "usbhid-ups"
        port = "auto"
        vendorid = "0764"
        productid = "0501"
        product = "CP1500EPFCLCD"
        bus = "003"

Take that verbatim into /etc/nut/ups.conf, giving it a name you will type often:

1
2
3
4
5
6
7

[rackups]
    driver = "usbhid-ups"
    port = "auto"
    desc = "Rack UPS"
    # Kill the outlets after shutdown completes; restore only on stable mains.
    offdelay = 60
    ondelay = 120

ondelay must exceed offdelay, and on most hardware the values get rounded to a granularity of the UPS’s choosing. Getting these backwards is the classic way to build a device that immediately powers itself back on into the outage it just escaped.

/etc/nut/nut.conf sets the role:

1

MODE=netserver

/etc/nut/upsd.users defines who may talk to the daemon. The monitor user needs upsmon master; anything read-only gets a separate account:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

[monmaster]
    password = "use-a-real-generated-secret-here"
    upsmon master

[monslave]
    password = "a-different-generated-secret"
    upsmon slave

[metrics]
    password = "third-generated-secret"
    actions = SET
    instcmds = none

Then /etc/nut/upsmon.conf, which is where the actual policy lives:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

MONITOR rackups@localhost 1 monmaster use-a-real-generated-secret-here master

# How many slaves must report down before the master shuts itself off
MINSUPPLIES 1
SHUTDOWNCMD "/sbin/shutdown -h +0"
POWERDOWNFLAG /etc/killpower

# Run our own script on every state change
NOTIFYCMD /usr/sbin/upssched
NOTIFYFLAG ONBATT   SYSLOG+WALL+EXEC
NOTIFYFLAG ONLINE   SYSLOG+WALL+EXEC
NOTIFYFLAG LOWBATT  SYSLOG+WALL+EXEC

# Shut down when the UPS says low battery, OR after this many seconds on battery
FINALDELAY 5

And on each slave, the same file with a single MONITOR line pointing at the master’s address and the slave keyword:

1
2

MONITOR [email protected] 1 monslave a-different-generated-secret slave
SHUTDOWNCMD "/sbin/shutdown -h +0"

Restart the stack and confirm it can see the battery:

1
2

sudo systemctl restart nut-server nut-monitor
upsc rackups@localhost

1
2
3
4
5
6
7
8
9

battery.charge: 100
battery.runtime: 2130
battery.voltage: 27.1
device.mfr: CyberPower
input.voltage: 232.0
output.voltage: 232.0
ups.load: 18
ups.status: OL
ups.realpower.nominal: 900

ups.status: OL means online. OB is on battery, LB is low battery, OL CHRG is mains restored and charging. Those strings are what upsmon acts on.

Timers, and why upssched exists

upsmon fires an event the moment power drops, which is too eager — a two-second brownout should not trigger a shutdown. upssched adds timers, so you can express “if we are still on battery after 120 seconds, then start reacting”:

/etc/nut/upssched.conf:

1
2
3
4
5
6
7
8
9

CMDSCRIPT /etc/nut/upssched-cmd
PIPEFN /run/nut/upssched.pipe
LOCKFN /run/nut/upssched.lock

AT ONBATT * START-TIMER onbatt-grace 120
AT ONLINE * CANCEL-TIMER onbatt-grace
AT ONBATT * START-TIMER onbatt-long 600
AT ONLINE * CANCEL-TIMER onbatt-long
AT LOWBATT * EXECUTE lowbatt-now

And the script it calls, which is where your own policy goes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

#!/bin/sh
# /etc/nut/upssched-cmd
case "$1" in
    onbatt-grace)
        # 2 minutes on battery: stop discretionary work, notify.
        logger -t upssched "on battery 120s: pausing backups"
        systemctl stop restic-backup.timer
        zpool scrub -p tank 2>/dev/null
        curl -fsS -m 10 https://ntfy.example.com/rack -d "UPS on battery 2min" || true
        ;;
    onbatt-long)
        # 10 minutes: this is a real outage. Shed the compute layer.
        logger -t upssched "on battery 600s: shutting down VMs"
        /usr/local/sbin/shed-compute.sh
        ;;
    lowbatt-now)
        logger -t upssched "LOW BATTERY: full shutdown"
        /usr/local/sbin/shed-compute.sh
        ;;
esac

Notice the || true on the notification. A dead notification endpoint should never abort your shutdown script, which is the sort of thing you only learn by having it happen. The same reasoning applies to the alerting habits in Uptime Kuma: self-hosted monitoring — the monitoring must degrade more gracefully than the thing it monitors. If any of this systemd unit and timer wiring is unfamiliar, systemd without fear covers the mechanics.

Getting the readings into your monitoring

Once NUT is talking to the hardware, it is a shame to leave the data inside upsc. The UPS knows its own load in watts, its battery charge, its input voltage, and how many times it has switched to battery this month — which is a genuinely interesting number if you have ever wondered whether your mains is as clean as you assume.

A NUT exporter for Prometheus is a small container that logs into upsd with the read-only account defined earlier and republishes the variables as metrics. Point it at the master, scrape it every 30 seconds, and you get three panels worth having: battery charge over time (which shows the battery ageing across years), UPS load in watts (a free whole-rack power meter you already own), and a counter of transfers to battery.

That third one is the sleeper. My own rack logged eleven transfers in a month that I never noticed, each lasting under two seconds — brownouts short enough that nothing rebooted and long enough that the UPS did its job silently. Without the counter I would have believed the mains was flawless. It also gives you an early warning: a battery that starts taking a visible charge dip on a two-second transfer is a battery approaching the end of its life, and you would rather learn that from a graph than from a failed shutdown.

Alert on two things only. Battery charge below 90% while ups.status reports OL means the battery is failing to hold charge on mains, which is a real fault. And an outage lasting more than your grace timer means machines are about to start shutting down, which you want to know about while you can still do something. Everything else — every brownout, every self-test — belongs on a graph rather than in your notifications.

Troubleshooting

Can't connect to UPS [rackups] (usbhid-ups-rackups): Connection refused. The driver isn’t running or can’t claim the USB device. Check permissions first — NUT runs as an unprivileged user and needs a udev rule granting access to the HID device. The packaged rules cover common vendors; obscure hardware needs a rule adding the vendor ID. sudo upsdrvctl -D start in the foreground shows the real error, which the systemd unit swallows.

The UPS disappears every few days. Almost always a USB problem rather than a NUT one. Cheap UPS HID implementations are flaky, and autosuspend on the USB port makes it worse. Disable autosuspend for that device via udev, and if it still drops, try a different port — I have had a UPS that was stable on the motherboard’s rear USB 2.0 header and dropped weekly on a front-panel USB 3.0 port. Losing the connection also raises a genuine alarm state in upsmon, so you will hear about it.

Slaves never shut down. Firewall. upsd listens on 3493 and by default only on localhost — set LISTEN 0.0.0.0 3493 in upsd.conf and confirm the port is reachable from the slave before you blame the config. Test with upsc [email protected] from the slave; if that works, upsmon will work.

Everything shuts down but the UPS never cuts its outlets. The POWERDOWNFLAG file exists to tell the shutdown sequence to run upsdrvctl shutdown at the very last moment, after the filesystems are read-only. On systems where the packaged systemd unit for this is missing or masked, the flag gets written and nothing reads it. Check for nut-driver-enumerator and the upsdrvctl shutdown invocation in your distribution’s shutdown path.

battery.runtime is wildly optimistic. Most consumer UPS units estimate runtime from a curve calibrated at the factory, and a three-year-old battery no longer matches that curve. Do not build your timers around the reported runtime. Build them around a number you measured yourself.

Test it, properly

Pull the plug. The actual mains lead, on a Saturday morning, with everything running as it normally does. Watch the sequence, time it, and confirm the machines come back cleanly. upsmon -c fsd forces a shutdown for testing the software path, but it will not tell you that your battery, at its current age, delivers four minutes instead of the fourteen it advertises.

Do this annually. Batteries lose capacity, and the rehearsal argument is the same one as in testing your backups: an untested recovery is a hypothesis.

The verdict

NUT is worth the afternoon it costs, with a caveat. If your homelab is one machine on one UPS, you can get 80% of the benefit from your distribution’s simpler UPS daemon or even the vendor’s own tool, and NUT’s master/slave ceremony is overhead you don’t need. The moment you have two machines on one battery with a dependency between them — which describes almost every rack with a NAS in it — NUT becomes the only sane option, because the ordering problem is real and shell scripts with sleep in them solve it badly.

The setup takes an afternoon. The configuration files are unlovely and the terminology is a museum piece. In exchange you get a rack that puts itself to bed in the right order at 3 a.m. while you sleep through it, which is the entire point of the exercise. My own battery is now old enough that it delivers about six minutes rather than the eleven it claims, and because I measured that, the timers are built around six.