OpenTelemetry at Home: Traces, Metrics, Logs

For years my homelab observability was three unrelated tools that happened to share a Grafana dashboard. Prometheus scraped metrics endpoints. Loki tailed logs. And when a request was slow, I had no idea why, because nothing told me which service in the chain was actually the bottleneck — I’d grep logs across four containers by timestamp and hope the clocks agreed. That last part is what traces are for, and it’s the piece most homelabs never bother with, because setting up distributed tracing used to mean picking a vendor-specific agent and hoping it played nicely with everything else.

OpenTelemetry is what changed that calculus. It’s a CNCF project — donated by Google and others, now the second-most active CNCF project after Kubernetes itself — that defines one vendor-neutral format and API for all three observability signals: traces, metrics, and logs. Instrument your code once with the OpenTelemetry SDK, and you can point the output at Grafana Tempo, Jaeger, Prometheus, Datadog, or all of them at once, without touching application code. The piece that makes this practical rather than theoretical is the OpenTelemetry Collector, a single binary that receives telemetry, transforms it, and routes it wherever you want.

Why bother with tracing at home

Metrics tell you that something is slow — request latency p99 climbed, error rate ticked up. Logs tell you what happened inside one service at one point in time. Neither tells you where in the chain the time actually went when a request touches your reverse proxy, then an API container, then a database, then an external call. A trace does: it’s a tree of spans, each span a timed unit of work, linked by a shared trace ID that follows the request across every service boundary.

The value shows up the first time a request is slow for a reason you wouldn’t have guessed. I had a case where my Grafana and Prometheus stack showed elevated latency on a small internal API, and every log line looked normal — no errors, no obviously slow queries. The trace showed the actual picture: 380ms of a 420ms request was spent in a single downstream call to an authentication service that itself looked fine in isolation, because its own metrics only measured server-side processing time, not the network round-trip plus connection setup. Nothing in logs or metrics alone would have pointed there. The trace made it obvious in one glance at a waterfall diagram.

The three signals and how OTel treats them

Traces are spans stitched together by trace and span IDs, capturing latency and causality across service boundaries. This is OpenTelemetry’s original focus and still its strongest area.

Metrics are numeric measurements over time — counters, gauges, histograms — the same shape Prometheus already popularised. OTel’s metrics API can export in OTLP (OpenTelemetry’s native protocol) or expose a Prometheus-compatible scrape endpoint directly, so you don’t have to abandon an existing Prometheus setup to adopt it.

Logs are the newest signal to standardise, and honestly the least mature part of the spec, but it’s caught up a lot — structured log records with trace/span IDs attached automatically, so you can jump from a slow trace straight to the log lines emitted during that exact span. That correlation is the actual payoff of doing all three signals through one framework instead of three disconnected tools: click a slow span, see the logs from precisely that request, not a firehose of everything the service logged that minute.

The Collector: the piece that does the real work

You could point every application directly at your tracing backend, but that couples your code to one destination and duplicates configuration everywhere. The Collector sits in between: applications export telemetry to it over OTLP, and the Collector receives, batches, filters, transforms, and exports to one or more backends. Change backends, add sampling, or redact sensitive fields, and you edit one YAML file instead of redeploying every instrumented service.

A Collector config has three stages — receivers, processors, exporters — wired together into pipelines:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
  resource:
    attributes:
      - key: deployment.environment
        value: home
        action: insert

exporters:
  otlp/tempo:
    endpoint: tempo.mylab.local:4317
    tls:
      insecure: true
  prometheusremotewrite:
    endpoint: http://prometheus.mylab.local:9090/api/v1/write
  loki:
    endpoint: http://loki.mylab.local:3100/loki/api/v1/push

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, resource]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch, resource]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch, resource]
      exporters: [loki]

Three pipelines, one receiver each, fanning out to Tempo for traces, Prometheus (via remote-write) for metrics, and Loki for logs — the same two backends most self-hosted Grafana users already run, per the Loki and Grafana/Prometheus posts linked above. The memory_limiter processor matters more than it looks — without a cap, a Collector under a sudden burst of telemetry (a crash-looping container logging in a tight loop, say) will happily eat all available RAM before it ever drops data.

Run it as a container:

1
2
3
4
5

docker run -d --name otel-collector \
  -p 4317:4317 -p 4318:4318 \
  -v $(pwd)/otel-collector-config.yaml:/etc/otelcol/config.yaml \
  otel/opentelemetry-collector-contrib:latest \
  --config /etc/otelcol/config.yaml

Use the -contrib image rather than the core image the moment you need anything beyond the basics — the Prometheus remote-write and Loki exporters above both live in contrib, not core.

Instrumenting an application

For anything written in a mainstream language, you rarely hand-write spans. Auto-instrumentation libraries wrap common frameworks (HTTP servers, database clients, message queues) and emit spans automatically. A minimal Python example, manual instrumentation for clarity:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="otel-collector.mylab.local:4317", insecure=True))
)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)

def handle_request(request):
    with tracer.start_as_current_span("handle_request") as span:
        span.set_attribute("http.method", request.method)
        with tracer.start_as_current_span("fetch_user"):
            user = fetch_user(request.user_id)
        with tracer.start_as_current_span("render_response"):
            return render(user)

Each nested start_as_current_span becomes a child span automatically — the tracer tracks the active span via context, so fetch_user and render_response show up as children of handle_request in the resulting trace, with no manual ID plumbing required. For most frameworks (Flask, Django, Express, Spring) the opentelemetry-instrument auto-instrumentation wrapper does this for the framework’s own request handling without any code changes at all — you install the instrumentation package, prefix your start command with opentelemetry-instrument, and point OTEL_EXPORTER_OTLP_ENDPOINT at the Collector via environment variable.

Sampling: don’t ship every trace

At home you probably don’t need to sample at all — traffic volumes are low enough that keeping 100% of traces costs nothing meaningful. But it’s worth knowing the knob exists before you accidentally point a noisy service (a health-check endpoint hit every few seconds) at your Collector and wonder why Tempo’s storage is filling up with spans nobody will ever look at:

1
2
3

processors:
  probabilistic_sampler:
    sampling_percentage: 20

Drop that into a pipeline and you keep one in five traces, chosen at random per trace ID so a whole trace is kept or dropped together rather than fragmented. For anything you actually care about tracking closely (errors, slow requests), tail-based sampling in the Collector can keep 100% of the interesting traces and sample down the boring ones — more setup, and only worth it once volume actually becomes a problem.

Migrating gradually instead of ripping out Prometheus

None of this requires abandoning a working Prometheus and Loki setup on day one, which is the usual objection I hear when I bring OTel up. The prometheusremotewrite exporter in the Collector config above writes into the exact same Prometheus instance your existing dashboards already query — nothing downstream needs to change. If you’d rather keep scraping instead of push-based remote-write, the Collector can also expose a prometheus exporter that Prometheus scrapes on its usual interval, so you can adopt OTel purely for the traces-and-logs correlation while leaving your metrics pipeline untouched.

The practical migration path I’d actually recommend: stand up the Collector and point one chatty service’s tracing at it first, leave metrics and logs exactly as they are, and get comfortable reading Tempo waterfalls before touching anything else. Once tracing has earned its keep, migrate metrics collection service by service — OTel’s Prometheus-compatible exporter and your existing /metrics scrape configs can run side by side indefinitely, so there’s no hard cutover forcing the pace. Logs are the signal I’d migrate last; the correlation benefit (jumping from a slow span straight to its log lines) only pays off once tracing is already solid, and OTel’s log SDKs are the newest and least battle-tested part of the stack across most language ecosystems.

Context propagation across process boundaries

The mechanism that makes a trace a trace, rather than a pile of unrelated spans that happen to share a timestamp, is context propagation, and it’s worth understanding because it’s the part most likely to quietly fail without any error message. When service A calls service B over HTTP, the active span’s trace ID and span ID get encoded into a traceparent header on the outgoing request, following the W3C Trace Context standard OTel adopted rather than inventing its own. Service B’s instrumentation reads that header on the way in and creates its own spans as children of the ID it received, rather than starting a fresh trace. Break that header — a proxy that strips unrecognised headers, a load balancer that doesn’t pass them through, a message queue hop where nobody thought to propagate context into the message envelope — and you get two disconnected traces instead of one continuous one, with no error logged anywhere, because from each service’s point of view it did exactly what it was told: it created a span, it just didn’t know it should have been a child of another one.

This is the single most common reason someone’s first OTel deployment produces a wall of orphaned root spans instead of the connected waterfall they expected. Auto-instrumentation for HTTP clients and servers in mainstream frameworks handles propagation transparently, which covers the large majority of real traffic. The gaps are the boundaries auto-instrumentation doesn’t reach automatically: a raw socket connection, a cron job that kicks off a chain of work with no inbound request to inherit context from, or a queue consumer that needs the context explicitly extracted from message metadata rather than an HTTP header. For those cases, the OTel API exposes inject and extract functions specifically for carrying context across a boundary the SDK doesn’t already understand, and it’s a few lines of code at each custom boundary rather than a structural limitation.

Resource attributes and why service.name discipline matters

Every span, metric, and log record in OTel carries a set of resource attributes describing where it came from — service.name, service.version, deployment.environment, and anything else you choose to attach via the Collector’s resource processor or the SDK directly. These attributes are what let Grafana correlate a slow trace with the metrics and logs from the same service, and the correlation only works if the same service consistently reports the same service.name across all three signals. A service that’s api in its traces but api-server in its logs (because someone hand-rolled the log shipper’s config separately from the tracing SDK) breaks the very correlation that’s OTel’s main selling point over three disconnected tools. Setting OTEL_SERVICE_NAME once as an environment variable at the container or systemd-unit level, and letting every signal inherit it from there rather than hardcoding it separately per SDK call, avoids this category of bug entirely.

Troubleshooting

Collector starts but no traces show up in Tempo. Check the Collector’s own logs first (docker logs otel-collector) — a misconfigured exporter endpoint fails silently from the application’s point of view since the app already handed the span off successfully; the failure happens downstream. docker logs on the Collector itself will show connection refused or TLS errors against the backend.

Traces appear but spans aren’t nested correctly, everything shows as a root span. Almost always a context propagation problem — the trace context (a traceparent header, for HTTP) isn’t being forwarded between services. Auto-instrumentation handles this for supported HTTP client/server libraries automatically; a hand-rolled HTTP call or a message queue hop needs the context injected and extracted manually via the OTel propagation API.

Collector OOMs under a burst of telemetry. This is what memory_limiter is for — if it’s missing from your pipeline, add it and set limit_mib comfortably under the container’s memory limit, so the processor starts refusing new data before the OS OOM-killer takes the whole process down.

Metrics show up in Prometheus with weird or missing labels. The resource processor is where you inject consistent labels (service name, environment) across all three signals. If your dashboards can’t correlate a trace to its metrics, check that both are tagged with the same service.name resource attribute — OTel expects this to be set at the SDK level (OTEL_SERVICE_NAME env var is the easiest way) and it needs to match exactly, case included.

High cardinality blowing up Prometheus storage after switching to OTel metrics. OTel’s auto-instrumentation sometimes attaches per-request attributes (a full URL path with an ID in it, for instance) as metric labels by default. Add a processor to strip or template high-cardinality attributes before they hit the Prometheus exporter, the same discipline you’d apply hand-writing /metrics endpoints.

Is it worth it

If your homelab is a handful of services with no request chains between them, plain Prometheus and Loki cover you fine and OpenTelemetry’s extra moving part — the Collector — is overhead you don’t need. Once you have more than two or three services actually calling each other (a reverse proxy in front of an API in front of a database, say), tracing stops being a nice-to-have the first time you spend twenty minutes cross-referencing logs by timestamp to find a slow hop that a trace would have shown you in one glance. The Collector is a genuinely small piece of infrastructure for what it buys: one instrumentation format, three correlated signals, and the freedom to swap backends later without touching application code.