OpenTelemetry at Home: Traces for a Two-Service App

Metrics tell you a request was slow. Logs tell you what each service said while it was being slow. Neither tells you where the time went when a request passes through more than one service, and that gap is the whole reason distributed tracing exists.

You hit this the moment you split an app in two. Say you have a small front-end API that takes a request and calls a back-end service to do the real work. A user reports the thing is slow. Your metrics show the front-end’s p95 latency climbing. Your logs show the front-end received the request and, some time later, returned a response. Was the delay in the front-end’s own code? In the network hop? In the back-end? In something the back-end called? You are back to correlating timestamps across two log streams by eye, which is exactly the archaeology tracing abolishes.

OpenTelemetry is the vendor-neutral standard for producing traces (and metrics and logs, but traces are where it earns its keep). I’ve written before about the broader three-signal picture; this piece is narrower and more practical. We are going to instrument a two-service app, watch a single request become a trace that spans both services, and read it in Jaeger. Two services is the smallest system where tracing pays off, and it is the perfect size to learn on.

What a trace actually is

A trace is the story of one request as it moves through your system. It is made of spans, and each span is one unit of work with a start time, a duration, a name, and some attributes. When the front-end handles a request, that’s a span. When it calls the back-end, that call is a child span. When the back-end does its work, that’s another span, nested inside. Draw them on a timeline and you get a waterfall: a bar for each span, nested and offset, and the long bar is your culprit. You see where the time went.

The magic that makes spans across two separate processes belong to the same trace is context propagation. When the front-end calls the back-end, the OpenTelemetry instrumentation injects a traceparent HTTP header — the W3C Trace Context standard — carrying the trace ID and the current span ID. The back-end’s instrumentation reads that header and knows: I am part of this trace, and my work is a child of that span. Without propagation you get two disconnected single-service traces; with it you get one continuous story. Almost every tracing problem you will hit is a propagation problem, so hold onto that idea.

The pieces

Four things run in this setup:

• Two application services, each with the OpenTelemetry SDK producing spans.
• The OpenTelemetry Collector — a single process that receives spans from your apps over OTLP (the OpenTelemetry Protocol), optionally processes them, and exports them onward. Your apps talk only to the Collector; the Collector talks to the backend. This indirection means you can swap tracing backends without touching a line of app code.
• A tracing backend to store and visualise the traces. Jaeger is the easy starting point; Grafana Tempo is the choice if you want it to live alongside a Grafana stack.

Here is the whole thing in compose. The apps send OTLP to the Collector; the Collector exports to Jaeger.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

services:
  frontend:
    build: ./frontend
    ports:
      - "8080:8080"
    environment:
      OTEL_SERVICE_NAME: frontend
      OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4317
      BACKEND_URL: http://backend:9000

  backend:
    build: ./backend
    environment:
      OTEL_SERVICE_NAME: backend
      OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4317

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.104.0
    command: ["--config=/etc/otel/config.yaml"]
    volumes:
      - ./otel-collector.yaml:/etc/otel/config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP

  jaeger:
    image: jaegertracing/all-in-one:1.58
    ports:
      - "16686:16686"   # Jaeger UI

The OTEL_SERVICE_NAME on each app is what names it in the trace waterfall, so set it to something you’ll recognise. The OTEL_EXPORTER_OTLP_ENDPOINT points both apps at the Collector’s gRPC port.

The Collector config

The Collector is configured as a pipeline: receivers take data in, processors optionally transform it, exporters send it out. A minimal traces pipeline that receives OTLP and forwards to Jaeger looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/jaeger]

The batch processor is the one you always want — it groups spans before export so you’re not making a network call per span. Modern Jaeger accepts OTLP directly on 4317, so the Collector just forwards. If you were exporting to Tempo instead, you’d change one exporter block and nothing else, which is precisely the decoupling the Collector buys you.

Instrumenting the apps

The fastest way in is auto-instrumentation, where the SDK hooks the common libraries — your web framework, your HTTP client — and produces spans for you without code changes. In Python you install the packages and run your app under opentelemetry-instrument:

1
2
3
4
5

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install     # pulls instrumentations for your libs

# run the app wrapped — spans are produced automatically
opentelemetry-instrument python app.py

With that alone, the front-end’s incoming request becomes a span, its outbound call to the back-end becomes a child span with the traceparent header injected automatically, and the back-end’s handling becomes a further child span. You have a two-service trace and you wrote no tracing code.

When you want detail the framework can’t see — the specific slow database query, a chunk of business logic — you add a manual span:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

from opentelemetry import trace

tracer = trace.get_tracer("backend.worker")

def do_expensive_work(item_id):
    with tracer.start_as_current_span("do_expensive_work") as span:
        span.set_attribute("item.id", item_id)
        result = crunch(item_id)          # the actual work
        span.set_attribute("result.rows", len(result))
        return result

That span nests inside whatever span is active when the function runs, so it appears as a child bar in the waterfall, complete with the attributes you set. This is how a trace goes from “the back-end took 400ms” to “the back-end took 400ms and 380ms of it was do_expensive_work on item 4212” — which is an actionable statement.

Reading the trace

Open the Jaeger UI on port 16686, pick the frontend service, and find your request. You get the waterfall: a top-level span for the front-end, a child for the HTTP call to the back-end, a child of that for the back-end’s handler, and your manual span nested inside. The horizontal position and length of each bar is time. The long bar is the answer.

This is the payoff for all the setup. The question that used to take twenty minutes of cross-referencing two log files — “which service ate the latency?” — now takes one glance at a waterfall. And because each span carries attributes, you can see where the time went and, more tellingly, on what: which item ID, which query, which downstream call. Line the trace ID up against your aggregated logs (log the trace ID and you can jump straight from a slow span to its log lines) and against the per-second host metrics for that instant, and you have the full three-dimensional picture of one bad request.

When a span fails, the trace says so

Tracing earns extra keep on the requests that go wrong. When an operation throws, the OpenTelemetry SDK marks that span’s status as an error and records the exception as a span event, with the exception type and message attached. In the Jaeger waterfall the failed span shows up flagged in red, so a trace of a broken request points you straight at the span that threw — and because context propagated across both services, you see it even when the error originated deep in the back-end and only surfaced as a generic 500 at the front-end. Recording the exception on a span is a couple of lines:

1
2
3
4
5
6

        try:
            result = crunch(item_id)
        except Exception as exc:
            span.record_exception(exc)
            span.set_status(trace.Status(trace.StatusCode.ERROR))
            raise

That turns a trace into a debugging tool for failures: it tells you which service and which operation failed, with the exception already in hand, before you have opened a single log file. For a request that crosses two services and comes back with an unhelpful error, this is the fastest route to the real cause I know.

Sampling, before it bites you

One decision to make consciously: sampling. Tracing every single request is fine at homelab volume, and I’d start there for the visibility. At higher throughput you don’t keep every trace — you sample, keeping a representative fraction — because storing a span for every request of a busy service gets expensive fast. OpenTelemetry supports head sampling (decide at the start of the request) and, via the Collector, tail sampling (decide after the trace completes, so you can keep all the slow or errored traces and drop the boring fast ones). For a two-service homelab app, sample everything and enjoy the completeness; just know the lever exists before your traffic grows into needing it.

Troubleshooting

The two services show up as two separate single-service traces. Context propagation isn’t happening. The front-end must be instrumented such that its outbound HTTP call injects the traceparent header, and the back-end must read it. With auto-instrumentation this is automatic if both the client library and the server framework are instrumented — check opentelemetry-bootstrap actually installed the instrumentation for your HTTP client, and that a hand-rolled or unusual client isn’t bypassing it.

No traces reach Jaeger at all. Follow the path backwards. Is the app exporting? Enable the SDK’s debug logging or add a logging exporter to the Collector so you can confirm spans arrive at the Collector. If they reach the Collector but not Jaeger, it’s the exporter block — wrong endpoint, or TLS mismatch (Jaeger’s OTLP port over plaintext needs insecure: true). The Collector’s own logs will show export failures explicitly.

Spans appear but with no useful detail. Auto-instrumentation gives you the framework-level spans and stops there. The gap between “the handler ran” and “this specific operation was slow” is filled by manual spans around the code you care about. If a trace is all top-level bars and no nesting, you haven’t instrumented the interesting internals yet.

Wrong or duplicated service names. Every service needs a distinct OTEL_SERVICE_NAME. If two services share one, they collapse together in the UI and the waterfall becomes unreadable. Set it explicitly per service, as in the compose above.

Clock skew makes the waterfall look impossible. If a child span appears to start before its parent, the two hosts’ clocks disagree. Traces are drawn from timestamps taken on each machine, so run NTP everywhere. On one box this never bites; across several it will.

The verdict

Tracing is the observability signal people reach for last and then wish they’d had first. For a single-service app it is arguably overkill — logs and metrics cover you. The instant you have two services talking to each other, the “where did the time go?” question becomes genuinely hard to answer any other way, and OpenTelemetry answers it cleanly. The setup cost is real: a Collector to run, a backend to stand up, and instrumentation to add, which is more ceremony than dropping in a metrics agent. But the standard is vendor-neutral, the auto-instrumentation gets you most of the way for almost no code, and the first time a waterfall points straight at the slow span you’d have spent an evening hunting, it pays for itself. For a homelab with a two-service app you actually care about, spend the afternoon. Instrument it while it’s small, and you’ll have the map ready before the system grows complicated enough to need it.