Loki for Logs When ELK Is Overkill

The first time I tried to run an ELK stack at home, Elasticsearch demanded a 4 GB heap before it would consider starting, Logstash added another gigabyte of JVM on top, and Kibana arrived with a setup wizard that wanted to know about my index lifecycle management policy. This was in service of searching perhaps 200 MB of logs a day from a handful of containers. The stack was using more memory than everything it was watching, combined, and I turned it off within a fortnight.

The problem it solves is real. When something breaks at 2 a.m. across four machines, ssh plus journalctl plus docker logs is a slow and error-prone way to find out what happened, and the logs you most want are the ones that scrolled off an hour ago. What I wanted was the ability to ask “show me everything from every machine between 02:10 and 02:20” and get an answer. Elasticsearch’s answer to that question is superb, and it costs 8 GB of RAM.

Loki’s insight is that most of what Elasticsearch does for logs is wasted effort.

Loki indexes almost nothing, deliberately

Elasticsearch builds a full-text inverted index over every log line. Every token in every message becomes a searchable term. That index is what costs the RAM and the disk, and it is what makes “find the word timeout anywhere in six months of logs” return in 40 milliseconds.

Loki indexes only the labels — the small set of key-value pairs describing where a log stream came from, like {host="lab-01", container="postgres", level="error"}. The log lines themselves are compressed into chunks and stored as opaque blobs. There is no index over their contents at all.

When you query, Loki uses the label index to work out which chunks could possibly match, fetches those chunks, and then brute-force greps through them in parallel. It is exactly as crude as it sounds, and it works, because a label selector plus a time range usually narrows a hundred gigabytes down to eighty megabytes before the grep starts. Grepping eighty megabytes is instant.

There is a second-order effect worth naming. Because the index is tiny, it fits in RAM trivially, which means Loki has no equivalent of Elasticsearch’s shard-heap-pressure death spiral — the failure mode where the JVM spends more time in garbage collection than in query execution and the whole cluster goes yellow at the worst possible moment. Loki’s failure modes are boring: it runs out of disk, or a query takes too long and gets killed. Both are legible at 2 a.m. in a way that “the heap is fragmented” never is.

The design borrows its entire mental model from Prometheus — which is why a Loki query looks like a PromQL query with a pipe in it, and why it slots into the same Grafana next to the Prometheus stack as if it were always meant to be there. It was. Same team, same label semantics, deliberately.

The trade is stark and you should understand it before you commit. “Show me errors from the postgres container yesterday” is fast. “Find the string deadbeef anywhere in any log from any host in the last year” is a full table scan, and Loki will honestly attempt it, and it will take minutes and hammer your disk. Elasticsearch answers that instantly. If unbounded ad-hoc search across all history is your daily workflow, pay the RAM tax and run Elasticsearch. If, like most of us, you always know roughly which service and roughly when, Loki is a tenth of the resources for the same answer.

The compose file

Loki 3.x runs perfectly well as a single binary in monolithic mode. Ignore every diagram showing ingesters, distributors and queriers as separate services — that is the microservices deployment, and it exists for people ingesting terabytes a day.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

services:
  loki:
    image: grafana/loki:3.4.2
    container_name: loki
    restart: unless-stopped
    user: "10001:10001"
    command: -config.file=/etc/loki/config.yml
    volumes:
      - ./loki-config.yml:/etc/loki/config.yml:ro
      - loki-data:/loki
    ports:
      - "127.0.0.1:3100:3100"

  alloy:
    image: grafana/alloy:v1.7.4
    container_name: alloy
    restart: unless-stopped
    command:
      - run
      - --server.http.listen-addr=0.0.0.0:12345
      - --storage.path=/var/lib/alloy/data
      - /etc/alloy/config.alloy
    volumes:
      - ./config.alloy:/etc/alloy/config.alloy:ro
      - alloy-data:/var/lib/alloy/data
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/log:/var/log:ro
    ports:
      - "127.0.0.1:12345:12345"

volumes:
  loki-data:
  alloy-data:

The collector there is Grafana Alloy. If you have read older Loki guides you will know Promtail, which did the same job for years; Grafana put it into long-term support in early 2025 and Alloy is the successor. Promtail still works and will keep working for a good while, so there is no emergency, but a new install should start on Alloy — the config language is stranger and the component model is genuinely better.

Loki’s config is where the “overkill” risk creeps back in, so here is a deliberately small one:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  log_level: info

common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2024-04-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 90d
  ingestion_rate_mb: 8
  ingestion_burst_size_mb: 16
  max_streams_per_user: 5000
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  allow_structured_metadata: true

compactor:
  working_directory: /loki/compactor
  retention_enabled: true
  delete_request_store: filesystem
  compaction_interval: 10m

analytics:
  reporting_enabled: false

Two of those lines are load-bearing beyond their size. retention_enabled: true on the compactor is what actually deletes old logs — retention_period on its own does precisely nothing without it, and people discover this when their disk fills at exactly day 91. And schema: v13 with store: tsdb is the current index format; older guides show boltdb-shipper, which still works and is slower and larger. A fresh install should never touch it.

Getting logs in with Alloy

Alloy’s config language is a pipeline of components wired together by referencing each other’s exports. It reads oddly at first and then makes complete sense:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

discovery.docker "containers" {
  host = "unix:///var/run/docker.sock"
  refresh_interval = "10s"
}

discovery.relabel "docker_labels" {
  targets = discovery.docker.containers.targets

  rule {
    source_labels = ["__meta_docker_container_name"]
    regex         = "/(.*)"
    target_label  = "container"
  }
  rule {
    source_labels = ["__meta_docker_container_log_stream"]
    target_label  = "stream"
  }
}

loki.source.docker "containers" {
  host       = "unix:///var/run/docker.sock"
  targets    = discovery.relabel.docker_labels.output
  labels     = { host = "lab-01" }
  forward_to = [loki.process.drop_noise.receiver]
}

loki.source.journal "systemd" {
  max_age    = "12h"
  labels     = { host = "lab-01", job = "systemd-journal" }
  forward_to = [loki.process.drop_noise.receiver]
}

loki.process "drop_noise" {
  stage.drop {
    expression = ".*health check succeeded.*"
    drop_counter_reason = "healthcheck_spam"
  }
  stage.regex {
    expression = "(?P<level>DEBUG|INFO|WARN|ERROR|FATAL)"
  }
  stage.labels {
    values = { level = "level" }
  }
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
  external_labels = {}
}

That mounts the Docker socket read-only, discovers every running container, ships its stdout and stderr, and does the same for the systemd journal. The stage.drop block throws away health-check noise before it ever costs you disk, which on my setup removes about 40% of total log volume for zero information loss.

Mounting the Docker socket into a container is a real privilege escalation — read-only helps very little, since the socket is an API rather than a file. If that bothers you, and it reasonably might, a socket proxy that whitelists only the container-list and log endpoints is the mitigation.

LogQL in five minutes

A LogQL query is a label selector, then a pipeline:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# Everything from one container
{container="postgres"}

# Errors only, case-insensitive
{container="postgres"} |~ "(?i)error"

# Parse JSON logs and filter on a field
{container="traefik"} | json | status >= 500 | line_format "{{.RequestPath}} {{.status}}"

# Errors per minute per container, as a graph
sum by (container) (rate({host="lab-01"} |~ "(?i)error" [1m]))

# Extract a duration from unstructured text and take the 99th percentile
quantile_over_time(0.99,
  {container="api"}
  | regexp "took (?P<ms>[0-9]+)ms"
  | unwrap ms [5m]
) by (container)

That last one is the trick that makes Loki more than a log viewer: it turns text into numbers, and numbers go on the same Grafana dashboard as your Prometheus metrics, on the same time axis, with the same time picker. Correlating “the latency spike” with “the log line that explains it” stops being an act of manual archaeology.

The filter operators are |= (contains), != (does not contain), |~ (regex match) and !~ (regex non-match). Put the cheapest filter first — Loki evaluates left to right, and |= "error" |~ "(?i)connection.*refused" is dramatically faster than the reverse, because the plain string match eliminates 99% of lines before the regex engine wakes up.

The label cardinality trap

This is the mistake, and everyone makes it once.

Every unique combination of label values creates a stream. Each stream gets its own chunk. Loki keeps active chunks in memory until they are full or idle, so a thousand streams is a thousand small in-flight buffers.

The temptation is to make useful things into labels: request_id, user, trace_id, path. Do that and you get one stream per request. Ingestion collapses, memory explodes, and the error you see is Maximum active stream limit exceeded, which is Loki protecting you from yourself.

The rule is: a label is only a label if you would use it in a selector, and if it has fewer than a few dozen values. host, container, job, level — all fine. Everything else belongs inside the log line, where a filter expression will find it perfectly quickly. Loki 3.x also added structured metadata, which stores high-cardinality fields alongside the line without indexing them, and that is the right home for a trace ID.

Ten containers on five hosts with four levels is 200 streams. That runs on 300 MB of RAM and I have never had to think about it since.

Where the chunks live

The config above uses filesystem for both the index and the chunks, which means Loki writes everything to a directory on the local disk. For a single-node homelab this is the right answer and I will defend it against the internet.

Loki’s object-store backends — S3, GCS, Azure, and anything S3-compatible like MinIO or Garage — exist so that a cluster of queriers can all read the same chunks. With one Loki container, there is no cluster, and pointing it at a MinIO instance running on the same box adds a network hop and a whole extra failure domain in exchange for nothing.

The genuine reason to use an object store is offsite retention. Chunks are immutable, compressed and content-addressed, which makes them exceptionally well behaved for backup. If you want your logs to survive the machine, either point the storage config at a remote bucket:

1
2
3
4
5
6
7
8
9

common:
  storage:
    s3:
      endpoint: s3.example.com
      bucketnames: loki-chunks
      region: eu-west-1
      s3forcepathstyle: true
      access_key_id: ${LOKI_S3_KEY}
      secret_access_key: ${LOKI_S3_SECRET}

…or leave it on the filesystem and let your existing backup tool sweep the directory, which is what I do. Restic against /loki/chunks deduplicates well and the incremental is tiny, because old chunks never change.

One sizing note. Compressed Loki chunks land at roughly 10:1 against raw text for typical application logs — gzip-class ratios, since log lines are enormously repetitive. My eleven machines generate about 1.4 GB of raw log a day, which becomes roughly 150 MB of chunks, which is 13.5 GB for ninety days. Add the TSDB index at perhaps 3% of chunk size and the whole thing rounds to 14 GB. Compare that to Elasticsearch, where the full-text index typically lands somewhere between 40% and 120% of the original data size, on top of storing the document itself.

That ratio is the entire argument for Loki compressed into one paragraph. You give up instant unbounded search. You get an order of magnitude back in disk, and another order of magnitude in RAM.

Troubleshooting

entry out of order errors, endlessly. Loki 3.x accepts out-of-order writes within a window, but two collectors shipping the same file with the same labels will still fight. Check you have not left Promtail running alongside Alloy after a migration. I did exactly this and spent an hour blaming the ingester.

Logs appear in Grafana with a delay of several minutes. That is chunks not yet flushed. It is normal. chunk_idle_period defaults to 30 minutes, which sounds alarming until you realise the querier reads in-memory chunks too. If you genuinely see minutes of lag on queries, check the clock on the shipping host — a 90-second clock skew makes logs land in the future and vanish from your time range.

Maximum active stream limit exceeded. Cardinality, as above. Query sum(count_over_time({host=~".+"}[5m])) by (container) in Explore and look for the label doing it.

The disk fills despite retention_period. The compactor is not running with retention_enabled: true, or delete_request_store is unset, in which case the compactor starts and silently declines to delete anything. Check /loki/compactor exists and is writable by the container’s user.

Alloy starts, reports healthy, ships nothing. Visit :12345/graph — Alloy renders its component pipeline as a live diagram showing which components have targets and which are receiving data. A dangling forward_to shows up instantly. This UI is the best thing about Alloy and almost nobody knows it is there.

Queries over a wide range time out. You are asking Loki to grep more than it can grep. Narrow the labels, narrow the time, or raise querier.max_query_parallelism. Or accept that this is the shape of the trade you signed up for.

The honest verdict

Loki is not a replacement for a security log platform. It has no correlation rules, no threat intel, no detection content — that is what something like Wazuh is for, and the two coexist perfectly happily. Loki is where you look when you already know something is wrong and you want to know what.

For that job, on a homelab, it is close to ideal. My install has been running for well over a year on 300 MB of RAM with 90 days of logs from eleven machines in 14 GB of disk. The equivalent Elasticsearch would want a dedicated box. I first wrote up this stack a while back when I was comparing it against the Elasticsearch tax and again in the Splunk-affordability piece; the version that survived contact with reality is the small one described here.

Who should run it: anyone with more than two machines who has ever ssh’d into three boxes in a row hunting the same error. Who should skip it: anyone with one host, where journalctl -f plus a bigger journal retention is genuinely the correct answer and costs nothing. The honest floor for Loki being worth its config file is about the point where you stop remembering which machine runs which service.