Netdata: Real-Time Metrics With Zero Config

I have written enough Prometheus config in my life to have a settled opinion about it, and the opinion is that it is worth every line — right up until the moment someone hands you a machine you have never seen and asks why it is slow. At that point the last thing you want is to write a scrape config, deploy an exporter, import a dashboard and wait fifteen minutes for enough samples to draw a trend line. You want to look at the thing. Now.

Netdata is what you install for that. You run a one-liner, you open a browser, and roughly ninety seconds later you are looking at several thousand metrics at one-second resolution, correctly grouped, correctly labelled, with alarms already configured. Nobody wrote a config file. Nobody imported a dashboard. It found your PostgreSQL, your nginx, your ZFS pool, your NVMe temperature and the fan speed on your motherboard, and it charted all of them.

The first time this happens it feels like cheating. The second time you start wondering what the catch is. There is one, and it took me a while to find it, and it turns out to be about time rather than resources.

What “zero config” actually means

Netdata ships with roughly 800 collectors and an auto-discovery loop that runs continuously. It reads /proc and /sys for everything the kernel exposes, then goes hunting: it scans the process table, spots a listening PostgreSQL, tries the default socket, and if it connects, it starts collecting. Same for Redis, nginx, Apache, MySQL, Docker, systemd units, ZFS, NUT, Pi-hole, and a very long tail of things you forgot were running.

The result is the highest metrics-per-minute-of-effort ratio of anything in this space, and it is not close. On a plain Debian box with nothing special installed, a fresh Netdata gives you:

• Per-core CPU broken down by mode, including steal and IOwait
• Per-interface network throughput, packets, errors, drops and TCP retransmits
• Per-disk IOPS, throughput, latency, and utilisation percentage
• Per-container CPU, memory, network and disk, automatically, with real names
• Memory broken into a dozen categories including page faults and OOM kills
• systemd unit states, restarts and per-unit resource consumption
• Drive SMART attributes, if smartd is running

That last one overlaps with a dedicated tool — Scrutiny does SMART monitoring properly with proper failure prediction — but the point is that you did not ask for it and it turned up anyway.

The per-second resolution is the other half of the pitch, and it matters more than it sounds. Prometheus at a 30-second scrape interval cannot see a 4-second latency spike; it does not exist in the data. Netdata sees it, charts it, and lets you hover over the exact second. When you are chasing something intermittent, that resolution is the difference between finding it and shrugging.

Installing it

The official one-liner works and I use it on bare metal:

1
2
3

wget -qO /tmp/kickstart.sh https://get.netdata.cloud/kickstart.sh
sha256sum /tmp/kickstart.sh   # compare against the published checksum
sh /tmp/kickstart.sh --stable-channel --disable-telemetry --dont-start-it

Piping a remote script straight into a shell is a habit worth losing, hence the two-step. --dont-start-it lets you edit the config before the first run, which saves a restart.

In compose, where most of my services live:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

services:
  netdata:
    image: netdata/netdata:v2.2.6
    container_name: netdata
    hostname: lab-01
    restart: unless-stopped
    pid: host
    network_mode: host
    cap_add:
      - SYS_PTRACE
      - SYS_ADMIN
    security_opt:
      - apparmor:unconfined
    environment:
      - DO_NOT_TRACK=1
      - NETDATA_EXTRA_DEB_PACKAGES=smartmontools
    volumes:
      - netdata-config:/etc/netdata
      - netdata-lib:/var/lib/netdata
      - netdata-cache:/var/cache/netdata
      - /etc/passwd:/host/etc/passwd:ro
      - /etc/group:/host/etc/group:ro
      - /etc/localtime:/etc/localtime:ro
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /etc/os-release:/host/etc/os-release:ro
      - /var/log:/host/var/log:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro

volumes:
  netdata-config:
  netdata-lib:
  netdata-cache:

Look at that volume list and be honest with yourself about what it means. Netdata in a container needs pid: host, the host’s /proc and /sys, SYS_PTRACE, SYS_ADMIN and the Docker socket to do its job. It is a container in the packaging sense and a host agent in the security sense. That is a defensible trade for a monitoring agent — it is the same trust you extend to node-exporter, only more so — but do not tell yourself the container is a boundary here. It is not.

Set hostname: explicitly. Without it the container reports a random hex ID and every chart is labelled a3f9c1e0b2, which is useless the moment you have two hosts.

The dbengine, and where the RAM goes

Netdata’s storage is called the dbengine and it works in tiers, each a downsampled roll-up of the one below:

Every tier is written continuously, so a query for last Tuesday reads tier 1 and a query for the last hour reads tier 0, transparently. This is downsampling done properly, and it is free in the open-source build, which is worth noting given what other vendors charge for it.

The knobs live in netdata.conf:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

[db]
    mode = dbengine
    storage tiers = 3
    dbengine tier 0 retention size = 1GiB
    dbengine tier 0 retention time = 14d
    dbengine tier 1 retention size = 1GiB
    dbengine tier 1 retention time = 3mo
    dbengine tier 2 retention size = 1GiB
    dbengine tier 2 retention time = 2y
    update every = 1

[ml]
    enabled = yes

[global]
    memory mode = dbengine
    process scheduling policy = idle

Netdata’s default footprint on a typical host is roughly 150–400 MB resident, which is more than node-exporter’s 15 MB and vastly less than the several gigabytes people assume from the sheer number of charts. If it matters, update every = 2 halves the tier-0 cost at the price of the one-second resolution that is the whole reason you installed it. I would sooner cut retention.

The [ml] block is the anomaly detection, and it is the feature I expected to dismiss and did not. Netdata trains a small unsupervised model per metric on the host itself, continuously, and flags dimensions behaving unlike their own recent history. It costs a few percent of a core. It has twice pointed me at a genuinely odd disk-latency pattern I would never have plotted, and it has also flagged my backup window every single night for a year because 3 a.m. genuinely does look anomalous compared to 3 p.m. Treat it as a hint generator with a permanent asterisk.

Streaming: the answer to “one dashboard per host”

The obvious objection to Netdata is that eleven machines means eleven dashboards on eleven ports, and correlating anything across them means eleven browser tabs and a lot of squinting.

The answer is streaming. A child agent collects locally and ships everything to a parent, which stores it, renders it and alerts on it. Children can then be configured with almost no local retention at all, which drops their footprint substantially.

On each child, in stream.conf:

1
2
3
4
5
6
7

[stream]
    enabled = yes
    destination = 192.168.1.10:19999
    api key = 0c9f7b4e-3a1d-4e8f-9b2c-6d5a8e1f3c7b
    timeout seconds = 60
    buffer size bytes = 10485760
    reconnect delay seconds = 5

On the parent, in the same file:

1
2
3
4
5
6

[0c9f7b4e-3a1d-4e8f-9b2c-6d5a8e1f3c7b]
    enabled = yes
    allow from = 192.168.1.*
    default history = 3600
    default memory mode = dbengine
    health enabled by default = auto

Generate the API key with uuidgen — it is a shared secret and the example above is a placeholder, so please do not paste it. Streaming is plain TCP by default and you should either keep it on a trusted segment or put it through your existing mesh VPN. Netdata supports TLS on the stream; configuring it is fiddly and worth it if the link crosses anything you do not own.

Set the children to memory mode = ram with a small history once streaming works. They then use around 40 MB each and the parent holds all the history, which is the arrangement that makes a fleet sensible.

Alarms, and the noise

Netdata ships several hundred pre-configured health alarms. This is either the best or the worst thing about it depending on the week.

Out of the box it will tell you about disk space trends, network errors, RAM pressure, disk latency, container restarts and about a hundred other conditions, with sensible thresholds, without you writing anything. That is a genuinely enormous head start.

It will also, out of the box, tell you at 04:00 that 10min_disk_utilization on the backup volume has exceeded 98%, which is what a backup does, and that 1m_ipv4_tcp_resets_sent is elevated, which is what happens when a laptop closes its lid. Netdata’s defaults are tuned for a production server that is doing exactly one thing. A homelab box doing nine things looks pathological by those standards.

Silencing them is per-alarm, in health.d/:

1
2
3
4
5

# /etc/netdata/health.d/disk-quiet.conf
template: 10min_disk_utilization
      on: disk.util
   hosts: lab-nas
      to: silent

Budget an hour, one week in, to walk the alarm log and silence everything that has fired without ever meaning anything. This is exactly the same discipline as tuning Alertmanager so it does not cry wolf, and skipping it produces exactly the same outcome: an alert channel you have muted, which is worse than having no alerts, because you believe you have alerts.

The Cloud question

Netdata Cloud is a hosted layer offering a single sign-on view across all your nodes, mobile notifications and a room/space organisation model. The agent is GPL and fully functional without it. The company behind it is a company, and the free Cloud tier’s feature set has moved over the years.

You do not need it. A parent agent with streaming gives you the unified dashboard locally, and the parent’s own web UI is the same UI. What Cloud adds for a homelab is push notifications to a phone and access from outside your network without a VPN — both of which you probably already solve another way.

What you should know is that the default install does try to nudge you toward claiming the node, and that the agent phones home for version checks and anonymous telemetry unless you say otherwise. DO_NOT_TRACK=1, --disable-telemetry on the kickstart, and this in netdata.conf:

1
2
3
4

[global]
    anonymous statistics = no
[registry]
    enabled = no

Set those and the agent stays on your network, permanently. I have run it that way for years without friction.

Troubleshooting

Every chart is labelled with a hex string. No hostname: in the compose file. The container’s hostname is its ID.

No container metrics appear. The Docker socket mount is missing, or cgroups discovery cannot see the container paths. On cgroups v2 with a non-standard Docker root, set [plugin:cgroups] path to /sys/fs/cgroup explicitly.

Charts stop at exactly one hour of history. memory mode is ram with the default history = 3600. Switch to dbengine.

RAM climbs to a gigabyte and stays. Check storage tiers and the retention sizes. The defaults are generous and they are per tier, so three tiers at 1 GiB each is 3 GiB of page cache the agent will happily use. It is mostly reclaimable cache, and free -m will lie to you about that, which is a Linux problem rather than a Netdata one.

The dashboard is fast locally and treacle over a VPN. The default UI streams updates over WebSocket once per second for every visible chart. Over a slow link, reduce visible charts or set the dashboard’s update rate down from 1s in the UI.

smartd data missing in the container. Add smartmontools via NETDATA_EXTRA_DEB_PACKAGES as above and confirm the container can see /dev/sd*, which needs explicit device passthrough. This one is genuinely awkward and is a fair argument for a bare-metal install on the NAS.

The honest verdict

The catch I mentioned at the start: Netdata is superb at “what is happening right now” and mediocre at “what has been happening since March”. Tier 2 gives you two years of hourly data, so the history exists, but the query surface is a dashboard rather than a query language. There is no PromQL. When I want to correlate power draw against disk temperature against container count over six months, Prometheus and Grafana is the tool, and it always will be. Netdata exports to Prometheus via /api/v1/allmetrics?format=prometheus if you want both, which is a legitimate architecture and one I ran for a year.

The other catch is that it is large. Compared to something like Beszel, which does the small honest subset in 30 MB, Netdata is a lot of machine. You will use perhaps 5% of the charts. The other 95% cost you nothing until the day one of them is the one you need, which is the whole argument.

Who should install it: anyone diagnosing a slow machine this afternoon, and anyone whose honest alternative is no monitoring at all because the Prometheus setup has been on the todo list since October. Install it on that box now. It takes ninety seconds and it will tell you something you did not know.

Who should skip it: anyone who has already built a metrics pipeline they trust and does not want a second source of truth with different numbers on it. Two monitoring systems that disagree slightly is its own special kind of Tuesday. I have described that flavour of self-inflicted complexity before as technical debt in a home lab, and running Netdata alongside Prometheus purely because both are nice is a textbook example.