Scrutiny: SMART Monitoring Before the Drive Dies

Every hard drive on the market ships with SMART (Self-Monitoring, Analysis, and Reporting Technology) built into its firmware, quietly counting things like reallocated sectors and pending sector errors, and almost nobody looks at it until a drive has already failed. I was in that camp for years — smartctl -a felt like a wall of cryptic numbers I’d only consult forensically, after the fact, to confirm what I already knew. Scrutiny changed that by taking the data smartd was already collecting and turning it into a dashboard I actually glance at, plus alerts that fire before a drive dies rather than a post-mortem after it has.

Why SMART Data Alone Isn’t Enough

The problem with raw SMART data isn’t that it’s unavailable, it’s that it’s unreadable at a glance and easy to misread even when you do look. smartctl -a /dev/sda dumps dozens of attributes, most of which are irrelevant to predicting failure, several of which use vendor-specific raw value encodings that mean different things on different drive models, and none of which are compared against anything — there’s no “this attribute is fine on a Seagate but a warning sign on a WD” built in. Worse, a drive’s SMART overall-health self-assessment (PASSED/FAILED) is notoriously unreliable on its own; plenty of drives report PASSED right up until they die, because that field only fails once a threshold is crossed that manufacturers set conservatively.

What actually predicts failure is trend across specific attributes rather than the pass/fail summary. Backblaze’s long-running public drive-failure studies (based on hundreds of thousands of drives in their own fleet) found that a small handful of SMART attributes correlate strongly with near-term failure: reallocated sector count (5), reported uncorrectable errors (187), command timeout (188), current pending sector count (197), and uncorrectable sector count (198). A drive with any of these climbing, even slowly, is meaningfully more likely to fail soon than one with all zeros. Scrutiny’s value is exactly this: it knows which attributes matter, tracks them over time so you see a trend rather than a single reading, and normalises the noisy vendor-specific bits so you’re comparing like with like across a mixed array of drive models.

Architecture: smartd Does the Reading, Scrutiny Does the Understanding

Scrutiny doesn’t replace smartd or reimplement SMART polling — it consumes what smartd (part of the smartmontools package, the same tool underlying that raw smartctl -a output) already collects, and layers a web dashboard, history, and alerting on top. The collector runs on each machine with disks to monitor, and either pushes to a central Scrutiny instance or the standalone all-in-one image runs collector and web UI together for a single-box setup.

A docker-compose all-in-one setup, good for a single NAS or server:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

services:
  scrutiny:
    image: ghcr.io/analogj/scrutiny:master-omnibus
    container_name: scrutiny
    cap_add:
      - SYS_RAWIO
    ports:
      - "8080:8080"
    volumes:
      - /run/udev:/run/udev:ro
      - scrutiny-config:/opt/scrutiny/config
      - scrutiny-influxdb:/opt/scrutiny/influxdb
    devices:
      - /dev/sda
      - /dev/sdb
      - /dev/sdc
      - /dev/nvme0n1
    restart: unless-stopped

volumes:
  scrutiny-config:
  scrutiny-influxdb:

Each devices entry needs to be listed explicitly because Scrutiny (via smartctl underneath) has to open a raw device handle to read SMART data, hence SYS_RAWIO. It stores history in an embedded InfluxDB instance so the dashboard can show trend lines across weeks and months, alongside whatever the current values happen to be.

For a multi-host setup — collector on each NAS or server, one central web UI — split it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# on each remote host with disks: collector only
services:
  scrutiny-collector:
    image: ghcr.io/analogj/scrutiny:master-collector
    cap_add:
      - SYS_RAWIO
    volumes:
      - /run/udev:/run/udev:ro
    devices:
      - /dev/sda
      - /dev/sdb
    environment:
      COLLECTOR_API_ENDPOINT: "http://scrutiny.mylab.local:8080"
    restart: unless-stopped

The collector runs on a schedule (daily by default, configurable) and pushes results to the central COLLECTOR_API_ENDPOINT, which lets you monitor drives across every box in the house from one dashboard instead of SSHing into each to run smartctl by hand.

Reading the Dashboard: What Actually Matters

Once it’s running, the dashboard gives every drive a status of Passed, Failed (SMART), or Failed (Scrutiny) — that last one is the useful addition: Scrutiny can flag a drive as failing based on its own attribute thresholds even when the drive’s own SMART overall-health assessment still says PASSED, because it’s watching the specific attributes known to predict failure rather than trusting the manufacturer’s conservative built-in threshold.

The attributes worth actually watching, ranked by how directly they correlate with impending failure in practice:

• Reallocated Sector Count (5) — the drive has already remapped bad sectors to spare ones. Any nonzero value here is worth watching closely; a rising trend is a strong signal.
• Current Pending Sector Count (197) — sectors the drive suspects are bad but hasn’t yet remapped, often because it hasn’t had a successful read to confirm. A nonzero value that persists across multiple readings (rather than clearing after a scrub) is a bad sign.
• Reported Uncorrectable Errors (187) and Uncorrectable Sector Count (198) — reads that failed even with error correction. These indicate the drive is losing the fight against its own physical degradation.
• Command Timeout (188) — the drive took too long to respond to a command. Isolated occurrences can be cabling or power issues rather than the drive itself; a steadily rising count usually isn’t.
• Power-On Hours (9) — useful context for interpreting the other numbers: a drive with three bad-sector events at 60,000 hours is a different conversation than one at 3,000 hours.

Scrutiny’s per-attribute history graphs are the actual value-add over raw smartctl output — seeing reallocated sectors go from 0 to 8 over three months tells a completely different story than seeing a static reading of 8, and you only get that story with data collected and graphed over time.

Alerting

A dashboard nobody checks is worthless, so wire up notifications rather than relying on remembering to look:

1
2
3
4
5
6

# scrutiny.yaml, mounted into the config volume
notify:
  urls:
    - "discord://webhook-id/webhook-token"
  filters:
    - notify-level: 1  # 0=all, 1=warn+fail, 2=fail-only

Scrutiny supports the shoutrrr notification library under the hood, which covers most common targets — email, Discord, Slack, ntfy, ones I’d suggest checking against your own alerting setup if you already run something like the stack in Grafana and Prometheus. If you’re already centralising alerts through that stack, Scrutiny also exposes a metrics endpoint compatible with Prometheus scraping, which is the tidier option if you don’t want a second, separate alerting channel just for disks.

Scrutiny vs Just Reading smartctl Yourself

It’s fair to ask whether a dashboard is really necessary when smartctl -a gives you the same underlying numbers for free. In my experience the answer comes down to two things a manual check can’t give you: trend and consistency of attention. A single smartctl -a reading tells you a drive’s reallocated sector count is currently 3 — it doesn’t tell you whether that’s been 3 for two years (fine, stable) or 3 for two weeks after being 0 for the previous three years (a rapidly developing problem worth acting on immediately). Scrutiny’s history graphs answer that question at a glance; reconstructing it manually means keeping your own log of smartctl output over time, which nobody actually does consistently by hand.

The second piece is that manual checks require remembering to run them, and remembering doesn’t scale across a dozen drives across several machines. I know this because I used to genuinely intend to spot-check drives monthly and, in practice, went six months or more between checks more than once. An automated collector running daily and a dashboard I glance at during my normal morning coffee routine closes that gap without relying on my own discipline, which is exactly the kind of thing worth automating away.

Troubleshooting

Drive shows up but attributes are all blank. Usually a permissions or capability issue — confirm SYS_RAWIO is granted and the specific /dev/sdX device is passed through in the compose file. Test directly first with docker exec scrutiny smartctl -a /dev/sda to confirm the container can actually read the drive before blaming Scrutiny’s parsing.

NVMe drives show no SMART data at all. NVMe SMART/health data uses a different command set than SATA SMART, and older smartmontools versions handled it poorly. Confirm the omnibus image is reasonably current, and check the device is passed through as the whole controller (/dev/nvme0n1) rather than a single partition.

Scrutiny flags a drive as failing but smartctl -H says PASSED. This isn’t a bug, it’s the entire point — Scrutiny’s thresholds on specific predictive attributes are intentionally stricter than the drive’s own conservative self-assessment. Cross-check the specific attribute it flagged (usually visible on the drive detail page) against the Backblaze-informed thresholds above before dismissing it as a false positive.

Collector runs but nothing appears on the central dashboard. Check COLLECTOR_API_ENDPOINT is reachable from the collector host and isn’t blocked by a firewall rule — the collector pushes over plain HTTP by default, so also confirm nothing between the two hosts is silently dropping the POST.

A drive with concerning attributes still passes every scrub. Bad sectors that have already been reallocated (attribute 5) are, by definition, no longer being read from the original bad location — a scrub reads the remapped location and finds good data. That doesn’t mean the drive is fine; a growing reallocation count is a mechanical or media degradation signal regardless of whether current reads succeed.

Alerts fire once and never again for the same issue. Check the notify-level filter and confirm you haven’t set it to only alert on state transitions — some configurations only notify when a drive’s status changes (healthy to warning), so a drive that’s been sitting in “warning” for weeks won’t re-alert unless you explicitly configure periodic reminders. For anything I actually care about, I’d rather get a slightly noisier daily digest than silently stop hearing about a known problem.

Temperature: The Attribute Everyone Forgets

Reallocated sectors and pending sectors get the attention, but drive temperature (attribute 194) deserves a mention because it’s both easy to monitor and a genuine contributor to long-term failure rates. Backblaze’s data shows the relationship isn’t simply “hotter is worse” — drives running consistently cold aren’t necessarily safer either — but sustained operation well above a drive’s rated range (typically 0–60°C, with most manufacturers recommending staying under 40–45°C in practice) does correlate with elevated failure rates over years of operation. Scrutiny graphs this alongside the failure-predictive attributes, which is useful for catching a slowly failing case fan or a NAS enclosure with genuinely inadequate airflow before it cooks every drive in the array simultaneously — a failure mode that defeats any amount of parity or RAID because it doesn’t respect vdev boundaries.

Verdict

Scrutiny doesn’t add any capability that wasn’t already sitting in smartd’s output — it makes existing data legible and actionable, which for SMART monitoring is most of the value that matters. Running it array-wide costs almost nothing (a small container, negligible resource use) and turns “check drive health” from a thing you do reactively after a scary noise into a dashboard you glance at weekly and alerts that catch a climbing reallocated-sector count while you still have time to order a replacement and migrate data calmly. Pair it with whichever recovery mechanism you’re actually running — SnapRAID + MergerFS or ZFS for Mortals — so that by the time Scrutiny flags a drive, you already know exactly what replacing it involves.