Prometheus and Grafana Without the Enterprise Baggage

Every guide to Prometheus and Grafana seems to be written for someone running four hundred nodes across three availability zones. They open with federation, move on to Thanos, and by the third paragraph you are being advised about sharding strategies for a workload that consists of a NAS, two mini PCs and a Raspberry Pi that mostly runs a DNS sinkhole. For its intended audience that advice is sound. It simply addresses an entirely different problem from the one you have.

Prometheus scales down further than almost anyone admits. A single container, a 200-line config file and about 400 MB of RAM will comfortably monitor a dozen machines and thirty services with a year of history. The parts that make it heavy in production — the high availability, the long-term storage layers, the query federation — are optional, and you can leave every one of them on the shelf. What follows is the stack I actually run, why each piece is there, and the specific ways it will misbehave.

Why bother when you already have uptime checks

A black-box monitor answers one question: is the service responding? That is a genuinely valuable question, and something like Uptime Kuma answers it in ten minutes with no config file. I run one, and I would not stop.

The trouble is that “responding” and “healthy” diverge long before anything goes down. A disk fills at a steady 2 GB a day for six weeks and then the database refuses writes at three in the morning. A container leaks 40 MB of RSS an hour and gets OOM-killed every eleven days, which looks like a random reboot until you plot it. A drive starts reallocating sectors months before SMART throws a hard failure. None of that is visible to a monitor that only asks for a 200 OK.

Metrics are the tool for the “how much, and which way is it trending” question. Prometheus stores numbers over time and lets you ask them questions afterwards, including questions you had not thought of when you installed it. That last property is the whole point. You cannot go back and collect the data you did not gather.

What Prometheus actually does

Prometheus is a pull-based time-series database with a query language bolted to it. On a schedule — every 15 seconds by default — it fetches a plain-text page of numbers from each target and appends them to local storage. That is the entire model.

The page looks like this, and you can curl it yourself:

1
2
3
4
5

# HELP node_filesystem_avail_bytes Filesystem space available to non-root users in bytes.
# TYPE node_filesystem_avail_bytes gauge
node_filesystem_avail_bytes{device="/dev/sda2",fstype="ext4",mountpoint="/"} 4.2949673e+10
node_memory_MemAvailable_bytes 6.442450944e+09
node_cpu_seconds_total{cpu="0",mode="idle"} 918273.44

Three things follow from that design, and they explain most of Prometheus’s behaviour.

Pull means Prometheus needs to reach your targets. There is no agent shipping data outward. Prometheus opens the connection. This is lovely inside a flat home network and irritating the moment a machine sits behind NAT somewhere else, at which point you need the Pushgateway or a tunnel.

Everything is a labelled number. No strings, no log lines, no traces. If you want text, you need something else alongside it, which is why Loki exists and pairs with it so naturally.

Storage is local by default. The TSDB writes to a directory on disk. That directory is your entire history, it is not replicated, and if the machine dies you lose it. For a homelab this is a completely acceptable trade — the metrics are diagnostic, they are not the family photos — as long as you have decided it consciously.

The compose file

Here is a working stack: Prometheus, Grafana, node-exporter for host metrics and cAdvisor for per-container metrics.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68

services:
  prometheus:
    image: prom/prometheus:v3.2.1
    container_name: prometheus
    restart: unless-stopped
    user: "65534:65534"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=365d'
      - '--storage.tsdb.retention.size=40GB'
      - '--web.enable-lifecycle'
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prom-data:/prometheus
    ports:
      - "127.0.0.1:9090:9090"

  grafana:
    image: grafana/grafana-oss:11.5.2
    container_name: grafana
    restart: unless-stopped
    user: "472:472"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/gf_admin
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_ANALYTICS_REPORTING_ENABLED=false
    volumes:
      - grafana-data:/var/lib/grafana
    secrets:
      - gf_admin
    ports:
      - "127.0.0.1:3000:3000"

  node-exporter:
    image: prom/node-exporter:v1.9.0
    container_name: node-exporter
    restart: unless-stopped
    command:
      - '--path.rootfs=/host'
    pid: host
    volumes:
      - /:/host:ro,rslave
    ports:
      - "9100:9100"

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.52.1
    container_name: cadvisor
    restart: unless-stopped
    privileged: true
    devices:
      - /dev/kmsg
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    ports:
      - "8080:8080"

volumes:
  prom-data:
  grafana-data:

secrets:
  gf_admin:
    file: ./secrets/grafana_admin_password

Two decisions in there deserve defending. Both HTTP ports bind to 127.0.0.1, so the only way in is a reverse proxy with real authentication in front — Prometheus ships no login screen whatsoever, and its /api/v1/admin/tsdb/delete_series endpoint is exactly as friendly as it sounds. And retention is capped by size as well as time, because retention.time alone will happily let the TSDB grow past whatever disk you have.

The config file it reads:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

global:
  scrape_interval: 30s
  scrape_timeout: 10s
  evaluation_interval: 30s
  external_labels:
    lab: home

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

  - job_name: node
    static_configs:
      - targets:
          - '192.168.1.10:9100'
          - '192.168.1.11:9100'
          - '192.168.1.12:9100'
        labels:
          role: server
      - targets: ['192.168.1.30:9100']
        labels:
          role: nas

  - job_name: cadvisor
    static_configs:
      - targets: ['192.168.1.10:8080']
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'container_tasks_state|container_memory_failures_total'
        action: drop

A 30-second scrape interval rather than the default 15 halves your sample count for free. You will not miss anything in a homelab at that resolution; a CPU spike lasting eight seconds is noise you were going to ignore anyway.

The metric_relabel_configs block at the bottom is the important part, and I will come back to it.

Exporters, and the things that do not speak Prometheus

node-exporter gives you roughly a thousand series per host: CPU per core per mode, memory, filesystems, network, load, systemd unit states, entropy, thermal zones. It is the single highest-value container in the stack.

Beyond that, most self-hosted software has grown a /metrics endpoint. Traefik, Caddy, Gitea, Vaultwarden, Home Assistant, Nextcloud, MinIO, Redis, PostgreSQL via postgres_exporter — all fine. For the boxes that predate the idea entirely, the SNMP exporter is still very much alive and will translate a managed switch or a UPS into something Prometheus understands.

For the awkward remainder there is blackbox_exporter (probes HTTP, TCP, ICMP and DNS from the outside) and the textfile collector, which is my favourite piece of the whole ecosystem. Drop a file of metrics into a directory and node-exporter serves it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# /etc/cron.hourly/zfs-textfile
cat > /var/lib/node_exporter/textfile/zfs.prom.$$ <<'METRICS'
# HELP zfs_pool_capacity_percent Pool capacity in percent.
# TYPE zfs_pool_capacity_percent gauge
METRICS
zpool list -Hp -o name,capacity | while read -r name cap; do
  echo "zfs_pool_capacity_percent{pool=\"$name\"} $cap" \
    >> /var/lib/node_exporter/textfile/zfs.prom.$$
done
mv /var/lib/node_exporter/textfile/zfs.prom.$$ \
   /var/lib/node_exporter/textfile/zfs.prom

Any shell script that can print a number becomes a first-class metric. The write-to-temp-then-mv dance matters: node-exporter may read the file mid-write otherwise, and you get parse errors on a random 1-in-40 scrape.

Grafana, and the dashboard restraint problem

Grafana is where most people go wrong, myself very much included. The temptation is to import dashboard 1860 — the excellent node-exporter full dashboard — see 47 panels of gorgeous graphs, and feel like the job is done. Six months later you have never opened it, because 47 panels is not a thing a human looks at.

The dashboards that survive in my setup have between four and nine panels and answer one question each. One “is anything about to run out of disk” board with a single table sorted ascending by free space. One “what is this container doing” board. One board per genuinely important service. When I want something exotic I write the PromQL in the Explore tab and throw it away afterwards.

The five queries that earn their keep:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Root filesystem free, per host
node_filesystem_avail_bytes{mountpoint="/",fstype!="tmpfs"} / 1e9

# Days until the root filesystem fills, based on the last week's trend
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[7d], 86400 * 30) < 0

# CPU busy percent, per host
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Container memory, top 10
topk(10, container_memory_working_set_bytes{name!=""}) / 1024 / 1024

# Anything that has restarted in the last hour
changes(node_boot_time_seconds[1h]) > 0

predict_linear is the one that pays for the whole exercise. It fits a line to the last week of a series and tells you where it lands in thirty days. It has warned me about a slowly filling log partition twice, both times about a fortnight before it would have mattered.

Once you have queries you trust, wire the important ones into Alertmanager — though tuning alerts so they do not cry wolf is a separate discipline entirely, and a badly tuned alert is worse than no alert.

The cardinality trap

Here is the failure mode that will bite you, and it is worth understanding before it does.

Prometheus keeps an index of every unique combination of metric name and labels in memory. Each combination is a series. Roughly, the RAM cost is a few kilobytes per active series plus the chunk buffer. Ten thousand series is nothing. A million series is a very unhappy small server.

Cardinality explodes multiplicatively. A metric with a path label on a service with unbounded URLs, or a user_id label, or a container_id label that changes on every deploy, will generate a fresh series for every distinct value forever. cAdvisor is a repeat offender — it emits per-container series that churn each time a container is recreated, and those dead series stay in the index until their retention window expires.

Check where you stand:

1
2
3
4
5
6

# Total active series
curl -s localhost:9090/api/v1/query --data-urlencode \
  'query=prometheus_tsdb_head_series' | jq '.data.result[0].value[1]'

# Which metric names are the worst offenders
curl -s localhost:9090/api/v1/status/tsdb | jq '.data.seriesCountByMetricName[:10]'

If a metric name you have never queried is contributing 80,000 series, drop it at scrape time with a metric_relabel_configs block like the one in the config above. Dropping metrics you do not use is the single most effective tuning lever in the entire stack, and almost nobody pulls it. My own stack drops about 30% of what cAdvisor offers and I have never once missed any of it.

When the honest answer is that you genuinely want the cardinality and a year of it, that is the point at which VictoriaMetrics starts to look appealing — it eats the same data at a fraction of the RAM and the disk.

Sizing the disk before it sizes you

The arithmetic here is unusually friendly, so it is worth doing once rather than finding out later.

A Prometheus sample compresses to roughly 1.5 to 2 bytes on disk after the TSDB’s delta-of-delta and XOR encoding has had its way with it. That is a genuinely remarkable number and it is why local storage works at all. The formula is:

1

bytes = retention_seconds / scrape_interval * active_series * bytes_per_sample

Fifteen thousand series at a 30-second interval for a year works out at about 30 GB. Double it for the head block, the WAL and general slack, and 64 GB of disk buys you a comfortable year of history for a small rack. That is a cheap SSD. Sizing surprises come almost entirely from cardinality rather than from retention — halving your series count saves exactly as much as halving your history, and costs you nothing you were using.

One caveat: the TSDB deletes whole two-hour blocks, so disk usage sawtooths rather than declining smoothly, and retention.size is enforced by dropping the oldest block once the limit is crossed. You will briefly exceed the number you set. Leave 20% headroom and stop thinking about it.

Troubleshooting

A target shows DOWN with “connection refused”. The exporter is bound to localhost inside its container. Check docker port node-exporter and confirm it is listening on 0.0.0.0, then confirm the host firewall lets 9100 through from the Prometheus host.

A target shows DOWN with “context deadline exceeded”. The scrape took longer than scrape_timeout. cAdvisor on a machine with sixty containers can genuinely take twelve seconds to render its page. Raise the timeout for that job specifically and consider --docker_only on cAdvisor.

Grafana shows “No data” but the query works in Prometheus. Nine times in ten this is the datasource URL. Inside compose, Grafana must reach http://prometheus:9090, using the service name. localhost means Grafana’s own container.

The TSDB refuses to start after an unclean shutdown. Prometheus replays the write-ahead log at boot and a truncated WAL segment stops it dead. The log names the file. Move the offending segment out of wal/ and restart; you lose the last couple of minutes of data. Set stop_grace_period: 2m in compose so this stops happening — the default 10 seconds is not enough for Prometheus to flush a large head block.

Memory climbs steadily and then the container gets OOM-killed. Cardinality, ninety-five percent of the time. Run the TSDB status query above. The other five percent is a query — a Grafana panel with rate(...[30d]) over a wide selector will happily try to load a gigabyte of samples to answer one graph.

Metrics vanish after a container update. Check whether the volume is actually a named volume. A bind mount to a path that does not exist gets created as an empty directory owned by root, Prometheus running as nobody cannot write to it, and it starts with a fresh empty TSDB and no complaint loud enough to notice.

The honest verdict

Prometheus and Grafana take an afternoon to stand up and about three months to actually get good at. The query language is genuinely strange — rate() before sum(), always, and the reason why involves counter resets — and you will write wrong PromQL with total confidence for a while. The dashboards you import will overwhelm you. The disk will fill.

Against that: nothing else gives you the ability to answer a question you did not know you had six months ago. When a machine started swapping last November I could look at exactly when it began, correlate it against a container that had been updated that week, and fix it in twenty minutes. Without the history that is a day of guessing.

Who should run it: anyone with more than about three machines who has already been surprised by an outage they could have seen coming. Who should skip it: anyone with one box and no ambitions, for whom Netdata gives you 80% of the answer with none of the config. The stack here is worth the effort precisely at the point where you stop being able to hold the whole system in your head — and if you are reading a 2,000-word article about metrics, you probably passed that point some time ago.