Loki for Logs: Grep at Scale Without the Elasticsearch Tax

For years my log strategy was ssh and grep. Something breaks, I guess which box it broke on, I journalctl -u whatever and squint, and eventually I find the line that matters. It works right up until the moment a request crosses three services and you have no idea which of the three logs holds the answer, at which point you are opening four terminal tabs and correlating timestamps by eye like an animal.

The obvious fix is log aggregation: ship every line from every machine to one place you can search. The obvious tool for that, for the better part of a decade, was Elasticsearch — the E in the ELK stack. And Elasticsearch is genuinely excellent at full-text search. It is also a Java application that will happily eat several gigabytes of RAM before it has indexed a single interesting log line, because it builds a full inverted index of every word in every message so that any query is fast. That trade is worth making if you are a search company. In a homelab where the log volume is modest and the hardware is finite, it is a tax I stopped wanting to pay.

Loki is Grafana Labs’ answer to that tax, and the pitch is refreshingly blunt: it deliberately does less indexing than Elasticsearch, which is exactly why it is cheap to run.

Why Loki is cheap: it only indexes labels

Here is the one idea that makes Loki make sense. Elasticsearch indexes the content of your logs — every token in every message becomes searchable, which is powerful and expensive. Loki indexes only a small set of labels you attach to each stream — things like job, host, container — and stores the actual log text as compressed chunks in object storage. The message bodies are never tokenised or indexed at all.

When you query, Loki uses the tiny label index to narrow down to the handful of streams that could match, then it scans the compressed chunks for those streams, applying your filter as it goes. It is, essentially, a very fast distributed grep over exactly the log streams you pointed it at. The model borrows its whole shape from Prometheus — the same label selectors, the same mental model — which is deliberate, because it means the metrics you already collect and your logs speak the same language. If you run Prometheus and Grafana already, Loki drops into that world without a fight.

The consequence of the label-only index is a storage bill that is a fraction of Elasticsearch’s, and memory use that lets Loki run comfortably on a machine you would never dare point Elasticsearch at. The price you pay is that a query which cannot be narrowed by labels — “find the string deadbeef across everything, forever” — has to scan more data than an inverted index would. In practice you learn to label sensibly and that cost stays small. For a homelab it is close to free.

Standing it up

Three containers get you the whole thing: Loki to store and query, an agent to ship logs, and Grafana to look at them. Promtail is Grafana’s own log-shipping agent and the path of least resistance in mid-2024.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

services:
  loki:
    image: grafana/loki:3.0.0
    command: -config.file=/etc/loki/config.yml
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yml:/etc/loki/config.yml
      - loki-data:/loki
    restart: unless-stopped

  promtail:
    image: grafana/promtail:3.0.0
    command: -config.file=/etc/promtail/config.yml
    volumes:
      - ./promtail-config.yml:/etc/promtail/config.yml
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    restart: unless-stopped

  grafana:
    image: grafana/grafana:11.0.0
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
    restart: unless-stopped

volumes:
  loki-data:
  grafana-data:

Loki 3.0 landed in April 2024 and ships a sane single-binary mode out of the box, so you do not need to understand the distributor/ingester/querier microservice diagram to run it at home. A minimal loki-config.yml using the local filesystem for storage looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kind: inmemory

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 720h   # 30 days

filesystem storage is fine for a single box; if you later grow into an S3-compatible bucket (MinIO on your own hardware, say), you change the object_store and Loki keeps working. Start on the filesystem and don’t over-engineer it.

Shipping the logs

Promtail’s job is to tail files, attach labels, and push lines to Loki’s /loki/api/v1/push endpoint. The config below scrapes system logs and, more usefully, Docker container logs — with the container name promoted to a label so you can query per-service.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets: [localhost]
        labels:
          job: varlogs
          host: mylab-01
          __path__: /var/log/*.log

  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 15s
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'

The positions.yaml file is Promtail remembering where it got to in each file, so a restart doesn’t re-ship everything or skip lines. The host label is the one bit worth setting deliberately per machine — it is how you tell one box’s logs from another’s once several of them are pushing to the same Loki.

A word on labels, because getting this wrong is the one way to make Loki miserable. Labels are cheap only while they have low cardinality. host, job, container, level — a handful of possible values each — are perfect. A label whose value is a request ID, a user ID, or a timestamp will generate a distinct stream for every value and blow up the index, which is the one failure mode that turns Loki from cheap into painful. Keep the searchable-but-high-cardinality stuff inside the log line and filter on it at query time, which Loki is fast at anyway.

Querying with LogQL

LogQL is Loki’s query language, and if you know PromQL it will feel like home. A query starts with a label selector in braces, then adds line filters and parsers with pipes. Point Grafana at Loki as a data source, open the Explore view, and start typing.

1
2
3
4
5
6
7
8

# every error line from one host
{host="mylab-01"} |= "error"

# 5xx responses from the reverse proxy, parsed as JSON
{container="reverse-proxy"} | json | status >= 500

# error rate per host over 5 minutes — a metric, from logs
sum by (host) (rate({job="varlogs"} |= "error" [5m]))

That last query is the quiet superpower. Because Loki understands time and labels the way Prometheus does, you can turn a log stream into a metric on the fly — count errors per minute, alert when the rate spikes — without ever writing that metric into your application. The line between “logs” and “metrics” gets pleasantly blurry, which is the same instinct behind the three-signal OpenTelemetry approach: correlate the signals instead of siloing them.

The |= operator is a plain substring match; |~ takes a regex; != and !~ exclude. Parsers like | json, | logfmt and | pattern pull structured fields out of the line so you can filter on them numerically. You build queries incrementally, watching the result narrow, which after years of piping grep into grep feels like a genuine upgrade.

Retention and disk

Loki’s retention is set by retention_period in the config above, and it is worth deciding deliberately rather than letting logs accumulate until a disk fills at 3am. Thirty days is my default for a homelab: long enough to investigate “why did the thing break last week”, short enough that the filesystem storage stays small. Compaction and deletion are handled by Loki’s compactor, which you enable in the config; on the single-binary setup it runs in-process.

The storage numbers are the whole reason to be here. Compressed log chunks are small, and because there is no giant inverted index sitting alongside them, the on-disk footprint for a typical homelab’s worth of logs is measured in low gigabytes for a month of history. That is a different universe from what the same logs cost in Elasticsearch.

Troubleshooting

The failures are few and cluster around a handful of causes.

Promtail is running but nothing appears in Loki. Check Promtail’s own logs first (docker logs promtail). The usual culprit is that it cannot reach Loki — a wrong url, or Loki not up yet — in which case you’ll see push errors. If pushes are succeeding but you see nothing in Grafana, your query’s label selector doesn’t match any stream: open Explore, click the label browser, and confirm what labels actually exist rather than guessing at their spelling.

“Per-stream rate limit exceeded” or ingestion errors. Loki caps how much a single stream can ingest by default. If one very chatty service trips this, either raise ingestion_rate_mb in limits_config or, better, ask why one stream is producing that much. Frequently it is a service stuck in a crash-restart loop spraying the same error, which is a real problem the log volume is politely telling you about.

“Too many outstanding requests” or slow, sweeping queries. This is almost always a query that couldn’t be narrowed by labels and is scanning weeks of chunks. Add a tighter time range and a label selector before the line filter. Loki is a fast grep over selected streams; it is a slow grep over all streams, and the fix is to select.

Cardinality explosion. If the index directory balloons and queries crawl, you have put a high-cardinality value into a label. Find the offending stream in the label browser, move that value out of the label and into the line body (filter on it at query time), and re-deploy Promtail. This is the single most common self-inflicted wound.

Timestamps look wrong / logs out of order. Loki historically rejected out-of-order writes; 3.0 accepts them by default within a window, but if you see rejected lines, it is usually a container logging in a timezone Promtail is reinterpreting. Let Loki apply the ingestion time unless you have a specific reason to parse the log’s own timestamp.

The verdict

Loki is the log aggregator I recommend to anyone running more than two machines who has felt the pain of ssh-and-grep across them. It is cheap on RAM, cheap on disk, and it speaks the same label language as your metrics, so it slots into a Grafana-based setup with almost no friction. The honest caveat is that it is not a full-text search engine — if your actual job is searching enormous corpora of unstructured text for arbitrary strings, Elasticsearch earns its weight and Loki does not pretend to compete. For the homelab job of “collect every log in one place and let me find the line that matters in seconds”, Loki is the tool I reach for, and it pairs naturally with per-second metrics from something like Netdata and threshold alerts from a properly tuned Alertmanager to give you the full picture when something goes wrong at an inconvenient hour. It replaced my terminal-tab correlation ritual entirely, and I have not missed it once.