SNMP Is Not Dead: Monitoring the Dumb Boxes

Every modern monitoring guide assumes you can install an agent. Drop a Node Exporter on the box, run Netdata, ship metrics over OTLP — all of it presumes a general-purpose computer you control, running an OS you can put software on. Then you look at your rack and realise half the things you’d like to watch are nothing of the sort. The managed switch. The UPS keeping the whole lot alive. The network printer. The PDU. The NAS with its locked-down appliance firmware. None of them will run your agent. Most of them will barely admit they run software at all.

These are the dumb boxes, and they are exactly the devices you most want to hear from — because a switch dropping packets or a UPS on failing batteries is the kind of problem that takes everything else down with it. The protocol that watches them has been doing the job since 1988, gets sneered at regularly, and remains completely irreplaceable for this exact task. SNMP is not dead. For the appliances, it’s often the only door in.

Why it survives

SNMP — the Simple Network Management Protocol — persists for a reason that has nothing to do with elegance and everything to do with ubiquity. Practically every piece of network and infrastructure hardware built in the last thirty years speaks it. Your switch exposes its per-port traffic counters, error counts and link states over SNMP. Your UPS exposes battery charge, load percentage, input voltage and “am I currently on battery” over SNMP. Printers expose toner levels; PDUs expose per-outlet power draw; the NAS exposes disk and volume health. The vendor didn’t have to build a Prometheus exporter, because they built the SNMP interface once, decades ago, and it still works.

You are not going to change this. There is no world where a UPS manufacturer ships a Netdata collector. So the pragmatic move is to accept that SNMP is the lingua franca of dumb boxes, learn just enough of it to poll what you need, and translate that into whatever monitoring stack you already run. Which turns out to be entirely doable, and once wired up, those appliance metrics sit right alongside the per-second metrics from your real servers as if they’d always belonged there.

What SNMP actually is

The model is worth ten minutes because it explains every frustration that follows. An SNMP-capable device exposes a tree of values. Each value has an address called an OID (Object Identifier), which is a dotted string of numbers like 1.3.6.1.2.1.2.2.1.10.3. That OID identifies exactly one thing — in this case, the total inbound bytes on interface index 3. You query the device for an OID and it hands back the value. That’s the whole protocol in one sentence: ask for a numbered thing, get its current value.

The numbers are unreadable, so there are MIBs (Management Information Bases) — text files that map the numeric OIDs to human names and describe their types and meaning. With the right MIB loaded, 1.3.6.1.2.1.2.2.1.10.3 becomes ifInOctets.3, which you can at least read. MIBs are where a lot of the pain lives, because vendors ship their own for device-specific metrics and you have to go find them.

Authentication comes in versions. v1 and v2c use a community string — a plaintext shared password, conventionally public for read-only access — sent in the clear. v3 adds real authentication and encryption. The default public community on an internet-exposed device is a genuine security problem, and we’ll come back to it, but on a locked-down management network v2c is what most homelabs actually use.

Poking it by hand first

Before automating anything, prove the device answers, using the Net-SNMP command-line tools. snmpwalk traverses the tree from an OID downward and prints everything it finds, which is how you discover what a device actually exposes:

1
2
3
4
5
6
7

$ snmpwalk -v2c -c public 192.168.1.10 1.3.6.1.2.1.1
SNMPv2-MIB::sysDescr.0 = STRING: 24-Port Gigabit Managed Switch
SNMPv2-MIB::sysUpTime.0 = Timeticks: (18452300) 2 days, 3:15:23.00
SNMPv2-MIB::sysName.0 = STRING: rack-switch-01

$ snmpget -v2c -c public 192.168.1.10 1.3.6.1.2.1.2.2.1.8.3
IF-MIB::ifOperStatus.3 = INTEGER: up(1)

If those return values, SNMP is enabled and reachable and you’re in business. snmpwalk on the interface subtree (1.3.6.1.2.1.2) will dump every port’s counters, and reading through that output is how you find the specific OIDs worth polling. If they time out, SNMP is disabled on the device, the community string is wrong, or a firewall is blocking UDP 161 — which is the whole troubleshooting tree, and we’ll get to it.

Getting it into your stack

Hand-polling proves it works; you don’t want to do it every minute forever. For a Prometheus-based setup, the standard bridge is the snmp_exporter. It sits between Prometheus and your devices: Prometheus scrapes the exporter over HTTP as usual, and the exporter translates that into SNMP polls against the target and hands the results back as Prometheus metrics.

The exporter is driven by a generated config that maps the OIDs you care about into named metrics. You describe the MIBs and OIDs you want in a generator file, run the generator, and it produces the snmp.yml the exporter reads. Prometheus then scrapes it with the target device passed as a parameter:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

scrape_configs:
  - job_name: snmp-switch
    static_configs:
      - targets:
          - 192.168.1.10   # the switch
    metrics_path: /snmp
    params:
      module: [if_mib]      # the module built by the generator
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: snmp-exporter:9116   # the exporter's address, which polls the device

That relabel dance looks fiddly, and it is, but the shape is always identical: rewrite things so Prometheus connects to the exporter while passing the device as the target parameter. Copy it once and it works for every device.

If your stack is Telegraf-based rather than Prometheus, its SNMP input plugin does the same job more directly — you list the fields and OIDs in the Telegraf config and it polls on the interval, no separate exporter to generate:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

[[inputs.snmp]]
  agents = ["udp://192.168.1.10:161"]
  version = 2
  community = "public"
  interval = "60s"

  [[inputs.snmp.field]]
    name = "uptime"
    oid = "1.3.6.1.2.1.1.3.0"

  [[inputs.snmp.table]]
    name = "interface"
    oid = "1.3.6.1.2.1.2.2"
    [[inputs.snmp.table.field]]
      name = "ifDescr"
      oid = "1.3.6.1.2.1.2.2.1.2"
      is_tag = true

Whichever bridge you use, the endgame is the same: appliance metrics flowing into the same store as everything else, ready to be graphed and — the part that matters — alerted on. A UPS reporting on battery should absolutely trip an alert, and once the metric is in Prometheus, an Alertmanager rule makes that trivial.

The MIB and OID pain, named honestly

The standard stuff — interfaces, system info, the IF-MIB and SNMPv2-MIB trees — is well-trodden and the snmp_exporter ships knowing about it. The pain arrives with vendor-specific metrics. That UPS’s battery runtime, that PDU’s per-outlet current, that switch’s PoE budget live under the vendor’s private enterprise OID branch, described by a MIB you have to download from the vendor and feed to the generator. Finding the right MIB, getting it to parse, and identifying the exact OID for the one value you want is genuinely the most tedious part of SNMP monitoring, and pretending otherwise helps no one.

The approach that keeps it bearable: snmpwalk the whole device once, read through what comes back, and pick out the handful of OIDs that actually matter. You do not need the entire MIB — you need battery charge, load, and on-battery status from the UPS, and three OIDs is a manageable thing to pin down even when the MIB is a mess.

Security: public is a loaded gun

The default public read-only community string is fine on a genuinely isolated management network and dangerous anywhere it can be reached from. SNMP v1/v2c sends the community string in plaintext, so anyone who can sniff the traffic has it, and a device answering public to the world leaks its entire configuration tree to any scanner.

The defensible positions, in order: keep SNMP on a management network that untrusted hosts can’t reach, and firewall UDP 161 to only the polling host; change the community string away from public to something non-obvious even on that network; and where the device supports it, use SNMP v3 with authentication and encryption so the credentials and the data are both protected on the wire. For a homelab, restricting who can reach port 161 does most of the work, but never leave public answering on anything with a route to the outside.

Troubleshooting

snmpwalk times out. Work outward from the device. SNMP may simply be disabled — many switches ship with it off, and you enable it in the web UI. If it’s on, the community string is wrong (case-sensitive, and not always public). If both are right, a firewall is dropping UDP 161 between you and the device — SNMP is UDP, so nothing helpfully refuses the connection; it just silently goes nowhere.

The walk works from your laptop but the exporter gets nothing. The exporter is polling from a different host than you tested from, and that host either can’t reach the device or is blocked by the firewall rule you scoped to your laptop. Test with snmpwalk from the exporter’s host to confirm the path, and widen the firewall rule to include the poller.

Metrics appear but the values are nonsense or counters wrap. Interface byte counters in the base IF-MIB are 32-bit and wrap quickly on a gigabit link. Use the 64-bit high-capacity counters (ifHCInOctets, ifHCOutOctets) from IF-MIB for anything faster than 100Mbit, and let rate() compute throughput from the ever-increasing counter rather than reading a raw number.

The generator can’t find a vendor MIB. MIBs have dependencies — one imports another — and a missing import stops the parse. Download the vendor’s full MIB bundle rather than a single file, put them all where the generator looks, and the imports resolve.

A device stops answering under load. Cheap appliances have weak management CPUs, and polling too many OIDs too often can overwhelm the very device you’re watching. Lengthen the scrape interval and poll only what you need. A switch is there to switch packets; its SNMP agent is an afterthought and will behave like one.

The verdict

SNMP is nobody’s favourite protocol and it doesn’t need to be. It is the pragmatic, universal answer to a question modern tooling pretends doesn’t exist: how do you monitor the boxes you can’t install anything on? The dumb boxes are frequently the most consequential things in the rack — the switch and the UPS underpin everything else — and going blind on them because the protocol is unfashionable is a poor trade. The setup has real friction, mostly in the MIB-and-OID archaeology, and the security defaults demand respect. But once a device is wired into your stack, its metrics behave like any other: graphed, trended, and alerted on. I’d rather know my UPS is on battery from a phone alert than from the whole rack going dark, and SNMP is what makes that possible. Wire the appliances in alongside a good status page for the services on top, and you’ve closed the last blind spot in the homelab.