Netdata in an Afternoon: Per-Second Metrics, Zero Config

Most monitoring setups sample every fifteen seconds. Prometheus defaults to it, and for a great many things fifteen seconds is fine. Then one day a service stutters — a CPU spike that lasts four seconds, a disk that saturates for a moment, a burst of network that fills a buffer — and your fifteen-second dashboard shows a serene flat line, because the event began and ended entirely between two samples. You know something happened. You have no picture of it. The resolution ate the evidence.

Netdata is the answer to that specific frustration. It collects metrics at one-second resolution by default, auto-detects everything running on the box without you telling it anything, and renders it all in a live dashboard you can have open within about two minutes of deciding to try it. The tagline it has earned is “zero-config”, and unusually for a tagline, it is close to true.

Why one second changes what you can see

High resolution matters for one concrete reason: the interesting failures in a homelab are often short, and a sampling interval longer than the event is blind to it by definition. A backup job that pins a disk for six seconds, a nightly cron that spikes CPU, a container that briefly thrashes memory before the OOM killer steps in — these are precisely the events that a fifteen-second average smooths into invisibility.

At one-second granularity you see the shape of things. You watch the spike rise and fall. You catch the correlation — CPU jumps at the same instant disk I/O does — that tells you which process is the cause rather than which is the symptom. When you are staring at a live chart trying to work out why a box feels sluggish, the difference between one-second and fifteen-second data is the difference between watching the problem happen and being told, vaguely, that a problem occurred. Pair that live picture with logs aggregated in Loki and you can line up “the spike happened at 03:14:07” against the exact log line from that second.

The cost of all this resolution is real and I’ll come to it — high-frequency data is expensive to keep for long — but the default trade Netdata strikes is deliberately weighted toward “show me right now, in detail” rather than “keep two years of coarse history”. For interactive troubleshooting that is the correct priority.

Zero-config means it already knows

The thing that makes Netdata feel like magic on first run is auto-detection. It ships with hundreds of collectors, and on startup it goes looking for what you are running. Postgres on its default socket? Detected, with dozens of database metrics. Nginx with the stub-status endpoint enabled? Detected. Docker containers, systemd services, the smart data off your drives, the temperatures off your sensors, network interfaces, disk queues — it finds them and starts charting them without a config file in sight.

This inverts the usual metrics workflow. With Prometheus you decide what to scrape, install an exporter for each thing, and write the scrape config; you get exactly the metrics you asked for and nothing you didn’t. With Netdata you get everything the machine can tell you about itself immediately, and the discovery is the point — you find out a metric exists because it’s already on a chart, rather than having to know to go looking for it. Both models are valid. For “I want eyes on this box this afternoon”, Netdata’s find-everything approach wins outright.

Standing it up

There is a one-line kickstart installer that runs Netdata natively on the host, which is the setup I actually prefer for a monitoring agent — it sees the host’s processes and hardware directly, without the indirection of a container namespace.

1
2

wget -O /tmp/netdata-kickstart.sh https://get.netdata.cloud/kickstart.sh \
  && sh /tmp/netdata-kickstart.sh --stable-channel

The installer sets up the agent, wires in a systemd service, and starts collecting. Point a browser at port 19999 and the dashboard is already live and populated. If you would rather containerise it, the compose route works, though it needs a few host mounts to see enough of the machine to be useful:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

services:
  netdata:
    image: netdata/netdata:stable
    container_name: netdata
    pid: host
    network_mode: host
    restart: unless-stopped
    cap_add:
      - SYS_PTRACE
      - SYS_ADMIN
    security_opt:
      - apparmor:unconfined
    volumes:
      - netdataconfig:/etc/netdata
      - netdatalib:/var/lib/netdata
      - netdatacache:/var/cache/netdata
      - /etc/passwd:/host/etc/passwd:ro
      - /etc/group:/host/etc/group:ro
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro

volumes:
  netdataconfig:
  netdatalib:
  netdatacache:

Either way you are looking at charts in a couple of minutes, which is the entire promise. The container version’s long list of host mounts and capabilities is the price of a containerised process seeing enough of the host to monitor it — on a dedicated box, the native install sidesteps all of that.

The dashboard, and its firehose problem

The Netdata dashboard is dense — deliberately, gloriously dense. Every metric it collects has a chart, grouped into sections, all of it updating live at one-second cadence as you watch. The first time you scroll through it you understand your machine better than you did an hour ago, because things you never thought to measure are just there.

The flip side is that the firehose can overwhelm. There is so much on screen that a newcomer doesn’t know where to look. The trick is to treat the default dashboard as a reference you scroll to when investigating something specific, and to lean on the anomaly indicators for the “what should I be worried about” question. Netdata has built-in machine-learning anomaly detection — it trains a model per metric on the normal shape of your data and flags dimensions behaving unusually — so you can let it point you at the odd one out rather than eyeballing a thousand charts.

Health alarms out of the box

Netdata ships with a large set of pre-configured health alarms — disk filling, RAM exhausted, a service down, an interface flapping, drives reporting SMART errors — that are active from the moment it starts. You do not write these; they come with sensible defaults and you tune the ones that matter.

Alarms are defined in simple config files. Here is the shape of a disk-space alarm so you can see how readable they are:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

 template: disk_space_usage
       on: disk.space
    class: Utilization
     type: System
component: Disk
     calc: $used * 100 / ($avail + $used)
    units: %
    every: 60s
     warn: $this > 80
     crit: $this > 90
    delay: up 1m down 15m multiplier 1.5 max 1h
     info: percentage of disk space used

The delay line is the part worth understanding, because it is how Netdata avoids alert spam: it can require a condition to persist before raising, and hold before clearing, so a metric hovering on the boundary doesn’t machine-gun you with warnings. That hysteresis is the same principle behind any alerting setup that doesn’t cry wolf — the alarm should fire on a real, sustained condition and stay quiet on a flicker.

Notifications route through a long list of methods — email, Telegram, Discord, ntfy, Slack, PagerDuty and more — configured once in health_alarm_notify.conf. As with any monitor, decide up front which alarms are allowed to wake you and which merely log.

Growing past one box: streaming

One Netdata agent watches one machine. When you have several, you do not want to open five dashboards, and you do not want a week of one-second data eating the SD card on a Raspberry Pi. Netdata’s answer is streaming: a child agent ships its metrics to a parent agent, which stores the history and presents everything centrally. The children can keep almost no local data — they become thin collectors — while the parent, on a box with real disk, holds the retention.

This parent-child model is how a lightweight per-second agent scales to a fleet without the per-node storage cost getting silly. It also means the little machines stay light, which matters when one of them is a Pi doing three other jobs. You configure it with a shared API key in stream.conf on both ends; the parent lists the children it will accept, and the children point at the parent.

The retention reality

Here is the honest catch, and it follows directly from the whole per-second premise: one-second data is voluminous, so Netdata’s default retention is short compared to a coarse-grained metrics store. Left alone it keeps a few days to a couple of weeks depending on how much you’re collecting and how much disk the database engine is allowed.

Netdata handles this with tiered storage — the database engine keeps high-resolution data for a recent window, then automatically rolls older data into progressively coarser tiers (per-minute, then per-hour), so you keep long-term trends cheaply while only the recent past stays second-by-second. You configure the tiers’ disk budgets in netdata.conf. The mental model to hold: Netdata is superb for “what is happening now and over the last few days in fine detail”, and with tiering configured it will also give you the long-term trend line — it is simply weighted toward the recent and detailed, which is what it is for.

Troubleshooting

A service you’re running isn’t auto-detected. The collector exists but the app isn’t exposing what the collector needs. Nginx needs its stub_status endpoint enabled; Postgres needs a reachable socket and permissions. Check the collectors’ status at the agent’s http://host:19999/api/v1/info and the debug log; the collector usually says exactly why it couldn’t connect.

The dashboard is slow or the agent is using a lot of CPU/RAM. Almost always retention and collection breadth. You are keeping too much high-resolution data, or collecting per-second from something that doesn’t warrant it. Trim the tiers’ disk budget in netdata.conf, or lower the update frequency of the heaviest collectors.

Alarms are firing constantly for boundary conditions. Tune the warn/crit thresholds and, more importantly, the delay line so a condition must persist before it alerts. The defaults are general; your disk that lives at 82% full permanently just needs its warning threshold moved.

A child’s metrics don’t appear on the parent. Streaming is a shared-secret handshake. Confirm the API key matches on both ends, that the parent’s stream.conf accepts that key, and that the child can actually reach the parent’s port. The agent error log names the rejection.

Container install shows almost nothing. The host mounts or capabilities are missing, so the containerised agent can’t see the host’s /proc, /sys or Docker socket. Compare your compose against the mounts above; the empty charts map directly to the missing mount.

The verdict

Netdata is the tool I install when I want to understand a machine right now with the least possible ceremony. Nothing else gets you from “fresh box” to “a thousand live, per-second metrics with anomaly detection and working alarms” in the time it takes to make a coffee. For interactive troubleshooting, for that afternoon when a server feels off and you need to watch it breathe, it is unmatched. The trade to go in with clear eyes about is retention: it is built around the recent and the detailed, and long-term history means configuring the storage tiers deliberately. I run it alongside a longer-horizon metrics store rather than instead of one — Netdata for the live, high-resolution picture and the fast alarms, something coarser for the two-year trend. For the price of an afternoon, it is one of the highest-return installs in the whole homelab monitoring toolkit, and it slots in comfortably next to the dumb boxes you monitor over SNMP to give you eyes on the smart ones.