VictoriaMetrics: When Prometheus Storage Gets Silly

There is a specific evening in every self-hoster’s life when the monitoring stack becomes the thing that needs monitoring. Mine arrived when I noticed that Prometheus was the largest single consumer of RAM on the box — larger than the media server, larger than the database behind it, larger than the Kubernetes control plane it was supposedly watching. It had 3.1 GB resident and a 90 GB TSDB directory, and the only thing it did with all that was let me look at a CPU graph roughly once a fortnight.

The obvious move is to cut retention and drop metrics, and you should absolutely do both — I covered the cardinality lever in the Prometheus and Grafana stack post and it is the cheapest win available. But there comes a point where you have dropped everything you can bear to drop and the numbers are still silly. That is where VictoriaMetrics earns its place.

What VictoriaMetrics actually is

VictoriaMetrics is a time-series database that speaks Prometheus. It accepts Prometheus remote-write, it scrapes Prometheus-format endpoints, it answers PromQL over the Prometheus HTTP API, and Grafana talks to it using the Prometheus datasource type with the URL changed. From the outside it is a drop-in.

From the inside it is a different animal. Prometheus uses a per-series chunked TSDB optimised for the read-recent, write-append pattern of an alerting system. VictoriaMetrics uses an LSM-tree-ish design with its own columnar compression, an inverted index that lives mostly on disk rather than in RAM, and aggressive per-block encoding that exploits the fact that most homelab metrics barely change from one sample to the next.

The practical outcome, on my hardware and with my data, was this:

Your numbers will differ — compression ratio depends enormously on how noisy your metrics are, and a series of pure random floats compresses like a series of pure random floats. But a 4-to-8x disk reduction and a 3-to-4x memory reduction is the range everyone reports, and it matched what I saw closely enough that I stopped double-checking.

That restart figure is the one I underrated. Prometheus replaying a large write-ahead log is genuinely slow, and every unclean shutdown means a minute and a half of no monitoring at exactly the moment you want monitoring most.

The two migration paths

You have a choice, and it determines how much of your existing setup survives.

Path one: keep Prometheus, add remote-write. Prometheus continues to scrape everything exactly as it does today, and forwards every sample to VictoriaMetrics. You then cut Prometheus’s local retention to something tiny — two days, enough to cover a VictoriaMetrics outage — and point Grafana at VictoriaMetrics for anything historical.

This is the reversible option. If VictoriaMetrics disappoints you, delete the remote-write block and put retention back. It costs you a little RAM — the remote-write queue has a real memory footprint — and buys you a lot of peace of mind. This is how I ran it for six weeks before committing.

Path two: replace Prometheus with vmagent. vmagent is VictoriaMetrics’s scraper. It reads the same prometheus.yml you already have, scrapes the same targets, and writes to VictoriaMetrics. It uses about 60 MB of RAM, has no local TSDB at all, and supports a disk-backed queue so that a VictoriaMetrics restart does not lose samples.

This is where you end up. Prometheus leaves the building entirely, and the stack becomes vmagent (scrape) → VictoriaMetrics (store, query) → vmalert (rules) → Grafana (draw).

Start with path one. Move to path two once you have stopped worrying.

The compose file

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

services:
  victoriametrics:
    image: victoriametrics/victoria-metrics:v1.111.0
    container_name: victoriametrics
    restart: unless-stopped
    command:
      - '-storageDataPath=/storage'
      - '-retentionPeriod=24'
      - '-httpListenAddr=:8428'
      - '-memory.allowedPercent=60'
      - '-search.maxQueryDuration=60s'
      - '-dedup.minScrapeInterval=30s'
    volumes:
      - vm-storage:/storage
    ports:
      - "127.0.0.1:8428:8428"

  vmagent:
    image: victoriametrics/vmagent:v1.111.0
    container_name: vmagent
    restart: unless-stopped
    depends_on:
      - victoriametrics
    command:
      - '-promscrape.config=/etc/prometheus/prometheus.yml'
      - '-remoteWrite.url=http://victoriametrics:8428/api/v1/write'
      - '-remoteWrite.tmpDataPath=/vmagent-buffer'
      - '-remoteWrite.maxDiskUsagePerURL=2GB'
      - '-promscrape.suppressScrapeErrors=false'
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - vmagent-buffer:/vmagent-buffer
    ports:
      - "127.0.0.1:8429:8429"

volumes:
  vm-storage:
  vmagent-buffer:

-retentionPeriod=24 means twenty-four months. This tripped me up for an embarrassingly long afternoon: the unit is months by default, and -retentionPeriod=30 gives you two and a half years rather than a month. Suffixes exist (30d, 52w, 5y) and you should use them.

-memory.allowedPercent=60 caps how much of the host’s RAM VictoriaMetrics will use for caches. It is a cache budget rather than a hard limit — the process can still exceed it — but it is the knob that stops a co-tenanted box from being eaten alive. On a dedicated machine leave it alone.

-dedup.minScrapeInterval=30s deduplicates samples that land within the same 30-second window. Set it to your scrape interval. If you ever run two vmagents scraping the same targets for redundancy, this is what makes that work.

The maxDiskUsagePerURL on vmagent is the bit that makes path two safe. If VictoriaMetrics goes down for maintenance, vmagent keeps scraping and spools to disk, then replays on reconnect. You lose nothing for a restart, and up to 2 GB worth of buffering for a longer outage.

Migrating existing Prometheus data across is a separate tool, vmctl, and it is a one-liner that takes a few hours for 90 GB:

1
2
3
4
5
6

docker run --rm -it \
  -v /var/lib/docker/volumes/prom-data/_data:/prom:ro \
  victoriametrics/vmctl:v1.111.0 \
  prometheus --prom-snapshot=/prom/snapshots/20250330T090000Z-abc123 \
             --vm-addr=http://192.168.1.10:8428 \
             --vm-concurrency=4

Take the snapshot first with curl -XPOST http://localhost:9090/api/v1/admin/tsdb/snapshot — vmctl reads a snapshot rather than a live TSDB, and pointing it at a live one produces exciting nonsense.

MetricsQL, and where PromQL surprises you

VictoriaMetrics answers PromQL. Then it answers a superset called MetricsQL, and the superset includes some behaviour changes that are improvements right up until they are not.

The friendly additions are genuinely good. rate() and friends do not need a range vector selector — rate(node_cpu_seconds_total) just works, using the step. Missing lookbehind windows default sensibly. There is histogram_quantile over VictoriaMetrics’s own histogram buckets, keep_last_value(), range_avg(), and WITH expressions that let you name subqueries:

1
2
3
4
5
6
7
8

WITH (
  cpu_busy = 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100),
  mem_used = 100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)
)
union(
  label_set(cpu_busy, "metric", "cpu"),
  label_set(mem_used, "metric", "mem")
)

Writing that in stock PromQL is a copy-paste crime scene.

The surprise is the one everybody hits: MetricsQL fills gaps that PromQL leaves empty. VictoriaMetrics extrapolates across missing data points within the lookbehind window, which means a series that stops reporting keeps drawing a flat line for a while rather than going blank. For dashboards this is usually what you wanted. For an alert rule of the form “alert when this metric is absent”, it is a quiet disaster — your absent() fires later than you expect, or on a rough day it does not fire at all.

If that matters, -search.disableAutoCacheReset and careful use of default_rollup will get you closer to Prometheus’s literal behaviour, and honestly the better fix is to alert on up == 0 and time() - last_scrape_timestamp > 300 instead of leaning on absence semantics. Which you should be doing anyway — alert rules that fire on the right thing are the difference between a monitoring stack you trust and a notification channel you have muted.

Recording rules and alerts run in vmalert, which reads your existing Prometheus rule files unchanged and forwards firing alerts to Alertmanager. That migration is a volume mount and nothing else.

Backups, and the thing Prometheus never gave you

Prometheus’s answer to “back up the TSDB” is a snapshot API and a tar command, and it is fine, and nobody does it because a Prometheus TSDB is enormous.

VictoriaMetrics ships vmbackup, which does incremental backups to a filesystem path or an S3-compatible bucket, and vmrestore, which puts them back. Because the data is a tenth of the size, backing up your metrics history stops being an absurd proposition:

1
2
3
4
5
6
7
8
9

# Nightly, via a systemd timer
docker run --rm \
  -v vm-storage:/storage:ro \
  -v /mnt/backup/vm:/backup \
  victoriametrics/vmbackup:v1.111.0 \
  -storageDataPath=/storage \
  -snapshot.createURL=http://192.168.1.10:8428/snapshot/create \
  -dst=fs:///backup/latest \
  -origin=fs:///backup/latest

The -origin pointing at the previous backup is what makes it incremental — it only ships changed blocks. My nightly run moves about 40 MB and finishes in under a minute. Point -dst at s3://bucket/path and it works the same way against MinIO or anything else that speaks S3.

Whether metrics history deserves backup is a real question with a defensible “no”. I keep it because power consumption trends and drive-health history are the two datasets I have actually gone back three years for.

Single-node, and the cluster version you should ignore

VictoriaMetrics ships in two shapes and choosing wrong will cost you a weekend.

Single-node is one binary, one storage directory, one port. It handles somewhere north of a million samples per second on decent hardware, which is several orders of magnitude past anything a homelab will ever generate. Everything in this post uses it.

Cluster splits the same job across vminsert, vmstorage and vmselect, adds a /select/0/prometheus style path prefix to every URL, and gives you horizontal scaling plus multi-tenancy. It exists for people ingesting from thousands of hosts.

Roughly every second tutorial online uses the cluster version, because the people writing tutorials work at companies that need it. If you deploy cluster in a homelab you inherit three moving parts, a URL scheme that breaks every copy-pasted Grafana config, and a replication factor to reason about, in exchange for capacity you will never touch. My rack of eleven machines produces around 1,600 samples per second at peak. Single-node treats that as an idle Tuesday.

The one genuine reason to look at cluster is multi-tenancy — separate metric namespaces with separate retention, which single-node does not do at all. If you are hosting metrics for someone else, that is your answer. Otherwise, one container.

Downsampling deserves a mention here because it is the feature people go looking for and then find behind the enterprise licence. The open-source build keeps every sample at full resolution for the whole retention period; there is no built-in “keep 30-second data for a month, then 5-minute data for a year”. Given the compression ratios, this matters far less than it sounds — full-resolution two-year retention on my data costs 11 GB, which is less than the downsampled Prometheus setup it replaced. But if you were counting on downsampling to make a plan work, check the licence before you build around it.

Troubleshooting

Grafana shows “No data” after switching the datasource URL. Check the URL has no trailing path. VictoriaMetrics single-node serves the Prometheus API at the root — http://victoriametrics:8428, with no /prometheus suffix. The cluster version does use a prefix, every blog post about it uses the cluster version, and this eats an hour of everyone’s life exactly once.

vmagent scrapes nothing and logs nothing useful. You almost certainly have -promscrape.suppressScrapeErrors at its default. Turn it off, as in the compose above, and the failures become loud. vmagent also serves its target list at :8429/targets, which is the same page Prometheus gives you and just as useful.

Ingestion stalls with “too many outstanding rows”. VictoriaMetrics is behind on merges, usually on slow spinning disk. It recovers on its own; if it does not, the storage is genuinely too slow for the ingest rate. Metrics want an SSD. This is the one hardware opinion I will state flatly.

Queries that worked in Prometheus return subtly different numbers. Check for a [5m] window shorter than twice your scrape interval. Prometheus needs at least two samples in the window to compute a rate and returns nothing otherwise; MetricsQL will extrapolate and hand you a number. Both are defensible; they are not the same number.

Memory grows past -memory.allowedPercent. That flag governs caches, and query execution allocates outside it. A wide query — {__name__=~".+"} over a month, which is exactly what a careless Grafana variable produces — can allocate gigabytes. Set -search.maxUniqueTimeseries and -search.maxQueryDuration to fence it in.

vmctl migration finishes but the data is not queryable. You migrated a live TSDB rather than a snapshot, or you migrated a snapshot and then queried a time range outside -retentionPeriod. VictoriaMetrics silently drops incoming samples older than the retention window, which is entirely sensible and completely invisible.

The honest verdict

The case against is short and real. VictoriaMetrics is a single vendor’s product with an open-source core and a paid enterprise tier, and some features — downsampling, multi-tenancy, several backup niceties — live behind that line. Prometheus is a CNCF graduated project with no such gradient. If you want your homelab to mirror what you run at work, and work runs stock Prometheus, there is value in the muscle memory. And the documentation, while thorough, is written by people who assume you already know what an inverted index is.

The case for is the table at the top of this post. My monitoring stack went from the largest RAM consumer on the box to a rounding error, my metrics retention went from six months to two years on the same disk, and restarts stopped being an event. I ran the earlier version of this comparison a while back when I first found Prometheus getting too hungry for the hardware, and everything I liked then has only improved since.

Who should switch: anyone whose Prometheus is over about 20,000 active series, anyone monitoring a rack of dumb boxes over SNMP where the series count multiplies per port, and anyone who wants more than a year of history on hardware they can afford to leave running.

Who should stay put: anyone under 10,000 series on a machine with RAM to spare. Prometheus at that scale is boring and boring is the highest praise available in infrastructure. Swapping a working, well-understood component for a better one you do not understand yet is how homelabs eat weekends. Do it when the numbers get silly, and not one day before.