VictoriaMetrics: When Prometheus Gets Too Hungry for Your Hardware

I love Prometheus. I’ve said as much before. But there’s a moment in the life of a growing homelab where the love turns slightly conditional, and that moment usually arrives when you check htop and find Prometheus quietly chewing through two gigabytes of RAM to remember some numbers about your fridge thermometer. It’s not that Prometheus is badly written — it’s that it was designed for ephemeral, short-retention monitoring of large fleets, and it makes memory and disk trade-offs that suit a data centre better than a fanless box in a cupboard.

This is where VictoriaMetrics quietly walks in and asks you to hold its beer.

What it actually is

VictoriaMetrics is a time-series database that speaks Prometheus fluently. It ingests data in the Prometheus format, it answers queries in PromQL (and a superset called MetricsQL), and it can be scraped and queried by exactly the tools you already use. From Grafana’s point of view, pointing at VictoriaMetrics instead of Prometheus is a one-line data-source change — the dashboards don’t know or care.

The pitch is simple: same interface, far less hardware. It compresses data more aggressively, uses memory more frugally, and writes to disk in a way that’s kinder to both spinning rust and SD cards. On the same workload that has Prometheus sweating, the single-binary version of VictoriaMetrics will typically sit there using a few hundred megabytes and a fraction of the disk.

The single binary is the whole point

Forget the clustered version for a homelab — that’s for people storing billions of series, and you are not that person. The single-node binary is one executable, one data directory, and a handful of flags. Here’s the shape of it in compose:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

services:
  victoriametrics:
    image: victoriametrics/victoria-metrics:latest
    command:
      - '--storageDataPath=/storage'
      - '--retentionPeriod=12'        # months, not days
      - '--httpListenAddr=:8428'
    volumes:
      - vm_data:/storage
    ports:
      - "8428:8428"

volumes:
  vm_data:

Note that retentionPeriod=12 — that’s twelve months. Long retention is where VictoriaMetrics genuinely shines and where vanilla Prometheus starts crying. Keeping a year of history on Prometheus means either a lot of disk or bolting on Thanos or Cortex, both of which add real operational weight. Here it’s a single flag, and the compression keeps the footprint sane.

Two ways to feed it

You have a choice. You can keep your existing Prometheus scraping everything and just tell it to remote_write a copy of every sample to VictoriaMetrics — useful as a long-term store sitting behind a Prometheus you don’t want to disturb:

1
2

remote_write:
  - url: http://victoriametrics:8428/api/v1/write

Or — and this is what I eventually did — you drop Prometheus entirely and let VictoriaMetrics do the scraping itself via its bundled agent, vmagent. vmagent reads a Prometheus-style scrape config, so your existing scrape_configs move across untouched:

1
2
3
4
5
6
7

  vmagent:
    image: victoriametrics/vmagent:latest
    command:
      - '--promscrape.config=/etc/prometheus.yml'
      - '--remoteWrite.url=http://victoriametrics:8428/api/v1/write'
    volumes:
      - ./prometheus.yml:/etc/prometheus.yml

vmagent is also dramatically lighter than Prometheus’s own scraper, and it can buffer to disk if the database is briefly unreachable — handy when you’re restarting things.

A complete single-node stack

Here’s the full compose file I actually run, tying vmagent and VictoriaMetrics together with Grafana. It’s the whole monitoring stack for a small home lab in one file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

services:
  victoriametrics:
    image: victoriametrics/victoria-metrics:latest
    container_name: victoriametrics
    restart: unless-stopped
    command:
      - '--storageDataPath=/storage'
      - '--retentionPeriod=12'          # months
      - '--httpListenAddr=:8428'
      - '--selfScrapeInterval=30s'
    volumes:
      - vm_data:/storage
    ports:
      - "127.0.0.1:8428:8428"

  vmagent:
    image: victoriametrics/vmagent:latest
    container_name: vmagent
    restart: unless-stopped
    command:
      - '--promscrape.config=/etc/prometheus.yml'
      - '--remoteWrite.url=http://victoriametrics:8428/api/v1/write'
      - '--remoteWrite.tmpDataPath=/vmagent-buffer'
    volumes:
      - ./prometheus.yml:/etc/prometheus.yml:ro
      - vmagent_buffer:/vmagent-buffer
    depends_on:
      - victoriametrics

  grafana:
    image: grafana/grafana-oss:latest
    container_name: grafana
    restart: unless-stopped
    volumes:
      - grafana_data:/var/lib/grafana
    ports:
      - "127.0.0.1:3000:3000"

volumes:
  vm_data:
  vmagent_buffer:
  grafana_data:

Note the loopback port bindings — nothing here should face the public internet, and I’d put Grafana behind a reverse proxy with TLS rather than exposing 3000 directly. The remoteWrite.tmpDataPath gives vmagent a real disk buffer, so a database restart doesn’t drop samples on the floor. The matching prometheus.yml is a bog-standard Prometheus scrape config — node-exporter, cadvisor, whatever you already point Prometheus at — copied across verbatim.

Sizing and retention on real hardware

The reason I bother with any of this is footprint, so let’s be concrete. On a small fleet — a dozen or so scrape targets, a few thousand active time series — a single-node VictoriaMetrics comfortably sits in a few hundred megabytes of RAM where Prometheus wanted a couple of gigabytes for the same job. The disk story is even better: VictoriaMetrics’ compression regularly lands under a byte per sample on typical infrastructure metrics, so a full year of history that would have Prometheus reaching for Thanos fits on a modest SSD.

Two flags do most of the sizing work. --retentionPeriod is how long data lives; set it in months (or 1y, 3y) and forget it. If you’re memory-constrained on a really small box, --memory.allowedPercent caps how much of system RAM VictoriaMetrics will use for caches — handy when the same fanless machine is also running other services and you don’t want the database greedily claiming everything. Start generous and tighten only if the box is genuinely under pressure; over-restricting the cache just trades RAM for disk I/O.

If you’re weighing whether a beefier machine would be simpler than all this tuning, I’ve argued both sides of that in the home lab upgrade trap — sometimes the right answer is more RAM, but “swap the software for something lighter” is very often the cheaper win.

Backups: don’t skip this

A metrics database you can’t restore is a graph you’re going to lose. VictoriaMetrics ships vmbackup/vmrestore for proper snapshots, but the single-node quick version is a snapshot API plus a copy:

1
2
3
4
5

# trigger a consistent on-disk snapshot
curl -s http://127.0.0.1:8428/snapshot/create | jq -r .snapshot

# the snapshot appears under the storage path; archive it off the box
tar czf "vm-$(date +%F).tar.gz" -C /var/lib/docker/volumes/…/storage/snapshots <name>

Snapshots are hard-linked and cheap to create, so a nightly one costs almost nothing until data actually diverges. Ship the archive somewhere off the machine — the same rule applies to every stateful service I run, and it’s the difference between “the SSD died” being an inconvenience versus a catastrophe.

Troubleshooting the move

Graphs go flat after switching data source. Almost always the Grafana data-source URL or type is wrong. Set the type to Prometheus and the URL to http://victoriametrics:8428 (no trailing path), then use Save & test. If PromQL queries return nothing, check that vmagent is actually scraping — hit http://127.0.0.1:8428/api/v1/query?query=up and confirm you get series back.

vmagent isn’t ingesting. Look at its logs first (docker compose logs vmagent). The usual culprits are a malformed prometheus.yml (vmagent validates strictly and will tell you the line) or an unreachable remoteWrite.url. Because vmagent buffers to disk, a backlog after an outage is normal and drains on its own.

Counter rates look slightly off versus Prometheus. MetricsQL handles counter resets and staleness a little differently. For most dashboards you won’t notice, but if a specific panel disagrees, that’s the corner case to check — rewrite the expression in MetricsQL’s rate() and compare.

High memory despite the reputation. A genuinely large or high-churn workload (lots of series being created and destroyed) can still use real RAM. Check the vm_cache_size_bytes metrics, and if churn is the problem, fix the labels generating it rather than throwing hardware at it.

Querying feels identical, mostly

In Grafana, set the data source type to Prometheus and the URL to http://victoriametrics:8428 and everything works. Your old PromQL queries run unchanged. MetricsQL adds a few conveniences — rate() that handles counter resets more gracefully, WITH templates so you stop copy-pasting the same subexpression — but you can ignore all of it and pretend you’re still writing PromQL.

The alerting story is handled by vmalert, a companion binary that evaluates the same recording and alerting rules Prometheus uses and forwards firing alerts to Alertmanager. So your existing alert rules also survive the move.

One genuinely nice touch for a home lab is vmui, the built-in query interface baked into VictoriaMetrics itself. Browse to /vmui on port 8428 and you get a query box, a plot, and — the bit I actually use — a cardinality explorer that tells you which metrics and label pairs are eating the most storage. When your database is bigger than you expected, that page usually finds the culprit in about thirty seconds: some exporter emitting a high-cardinality label like a per-request ID or a timestamp-in-a-label, quietly multiplying your series count. Fix the label at the source and the footprint collapses. Prometheus makes you install extra tooling to get that visibility; here it’s just there.

The migration itself, if you’re moving existing history rather than starting fresh, is handled by vmctl, which reads directly from a Prometheus TSDB snapshot and replays it into VictoriaMetrics. So you don’t even lose the metrics you’ve already collected — point vmctl prometheus at the snapshot directory, let it stream across, and your year-to-date graphs are intact on the new backend.

Where it isn’t a free lunch

Honesty time. VictoriaMetrics is developed by a company, and while the core is genuinely open source under Apache 2.0, some enterprise features (downsampling, certain cluster bits) are not. For a homelab that’s irrelevant — everything you need is in the open core — but it’s worth knowing the model.

It’s also subtly different from Prometheus under load and in edge cases: histogram handling, staleness, and a few MetricsQL-versus-PromQL quirks can surprise you if you’re porting complex dashboards. And because it’s “Prometheus-compatible” rather than “Prometheus,” the occasional third-party tool that pokes Prometheus’s internal API directly won’t be happy. These are corner cases, but they exist.

There’s also a mindset adjustment. Prometheus’s model — each server scrapes and stores its own data, self-contained — is beautifully simple, and VictoriaMetrics with vmagent splits that into a scraper and a store. For a single small node the difference is cosmetic, but it means one more moving part to reason about, and if you were relying on Prometheus’s own web UI for ad-hoc queries you’ll be nudged towards Grafana or vmui instead. None of this is hard; it’s just different enough that you should spend an afternoon getting comfortable before you decommission the Prometheus you trust.

The verdict

If you’re running a handful of nodes and Prometheus is using more RAM than you’d like, or you want a year of metrics without standing up Thanos, switch. This is the same “reach for the leaner tool” instinct that makes me skip heavyweight abstractions when the job doesn’t need them — much like the argument in Helm charts demystified for knowing when not to add a layer. VictoriaMetrics is the closest thing to a free upgrade I’ve found in the monitoring world: same dashboards, same queries, same alerts, a fraction of the resources, and far longer retention thrown in. I migrated a memory-starved mini-PC to it over a wet Sunday afternoon, the graphs looked identical, and the RAM graph for the box itself dropped off a cliff. That’s the most satisfying kind of change — the one nobody but you ever notices.