Dagger: CI/CD Pipelines as Code That Run Anywhere

The single most demoralising thing about CI is the feedback loop. You edit a YAML file, push, wait three minutes for a runner to spin up, watch it fail on a typo, fix the typo, push again. Repeat until either the pipeline goes green or you go home. I once burned an entire afternoon — eleven pushes, I counted — chasing a pipeline failure that turned out to be a misindented with: block, because the only way to run the pipeline was to push and pray. The pipeline existed only inside the CI provider, so the provider was the only thing that could execute it. Dagger’s whole pitch is that this is daft, and they’re right.

What Dagger actually is

Dagger is a portable execution engine for pipelines. You describe your build, test and deploy steps as code — in Go, Python, TypeScript, PHP, or via a shell-like CLI — and Dagger compiles that into a DAG of containerised operations executed by BuildKit under the hood. Because every step runs in a container, the pipeline produces the same result on your laptop as it does on a GitHub Actions runner, a GitLab job, or a Jenkins box. Your CI provider stops being the thing that defines the pipeline and becomes a dumb trigger that runs dagger call.

Two properties make this more than a gimmick. First, it’s content-addressed and aggressively cached — every operation is keyed by its inputs, so unchanged steps don’t re-run. Run it twice and the second run is mostly cache hits. Second, the pipeline logic is real code in a real language: you get functions, types, loops, tests, your editor’s autocomplete, and a debugger. After years of YAML that you can only validate by submitting it, that shift alone is worth a hard look.

A pipeline you can actually run locally

Here’s a function written in Go using the Dagger SDK. It builds a binary, runs the tests, and returns the test output — all inside containers Dagger manages.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

package main

import "context"

type Ci struct{}

// Test runs the Go test suite against the source directory.
func (m *Ci) Test(ctx context.Context, src *Directory) (string, error) {
	return dag.Container().
		From("golang:1.24").
		WithDirectory("/app", src).
		WithWorkdir("/app").
		WithMountedCache("/go/pkg/mod", dag.CacheVolume("go-mod")).
		WithExec([]string{"go", "test", "./...", "-v"}).
		Stdout(ctx)
}

The WithMountedCache line is the part to notice: it persists the Go module cache across runs as a named cache volume, so the second invocation doesn’t re-download the dependency tree. That single line is the difference between a 90-second test run and a 6-second one.

Now the part that matters. You invoke that from your terminal:

1
2
3
4
5
6
7

$ dagger call test --src=.
=== RUN   TestParseConfig
--- PASS: TestParseConfig (0.00s)
=== RUN   TestRoundTrip
--- PASS: TestRoundTrip (0.02s)
PASS
ok      example.com/app   0.231s

No push. No waiting for a runner. The exact same code path your CI will execute, running on your machine, in seconds. When it’s green locally, it’s green in CI — because it’s the same engine running the same containers.

Composing real pipelines

The toy example builds and tests. Real pipelines chain stages, and Dagger’s model is that a function takes containers or directories in and returns new ones out, so you compose them like any other functions. A build-then-publish flow looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

// Publish builds the image and pushes it to a registry, returning the ref.
func (m *Ci) Publish(ctx context.Context, src *Directory, tag string) (string, error) {
	build := dag.Container().
		From("golang:1.24").
		WithDirectory("/app", src).
		WithWorkdir("/app").
		WithExec([]string{"go", "build", "-o", "/out/app", "./cmd/app"})

	return dag.Container().
		From("gcr.io/distroless/static").
		WithFile("/app", build.File("/out/app")).
		WithEntrypoint([]string{"/app"}).
		Publish(ctx, "registry.example.com/app:"+tag)
}

Note the multi-stage build falls out naturally: build is one container, the final image is another, and you copy one file between them. No Dockerfile syntax to learn, no --target flags — it’s a function returning a File. That composability is what people mean when they call Dagger “elegant,” and they’re not wrong, once it lands.

Wiring it into your provider

Your CI config shrinks to almost nothing. It installs the Dagger CLI and calls the function. Here’s the whole GitHub Actions job:

1
2
3
4
5
6
7
8
9

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dagger/dagger-for-github@v8
        with:
          version: "latest"
          call: test --src=.

That’s it. The logic lives in ci/main.go, version-controlled and refactorable like any other code. Want to switch from GitHub to GitLab next year? You rewrite ten lines of trigger config and your actual pipeline is untouched. That portability is the headline benefit, and it’s real — I’ve moved a project between two providers and the migration was an afternoon, not a fortnight. The same “define it once, run it anywhere” instinct shows up in reproducible dev environments with devcontainers: the closer your local environment is to CI, the less of your life you spend debugging the gap between them.

How the caching actually works, and why it changes how you write pipelines

Once you understand Dagger’s caching, you start writing pipelines differently, so it’s worth a paragraph. Every operation — a From, a WithExec, a WithDirectory — produces a content-addressed layer keyed by its inputs. Change nothing about a step’s inputs and the engine returns the cached result instead of running it. This is the same principle that makes Docker layer caching work, but applied to your entire pipeline graph, including test runs.

The practical consequence: order your operations from least-likely-to-change to most-likely-to-change. Copy your dependency manifest and install dependencies before you copy your source code, exactly as you would in a well-written Dockerfile. Then a source change invalidates only the test step, not the dependency install. Get this backwards — copy all your source first, then install — and every one-character edit re-downloads your entire dependency tree, and you’ve thrown away the speed advantage that justified Dagger in the first place.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

func (m *Ci) base(src *Directory) *Container {
	return dag.Container().
		From("node:22").
		// Manifest first — changes rarely, so the install caches.
		WithFile("/app/package.json", src.File("package.json")).
		WithFile("/app/package-lock.json", src.File("package-lock.json")).
		WithWorkdir("/app").
		WithExec([]string{"npm", "ci"}).
		// Source last — changes constantly, invalidates only what follows.
		WithDirectory("/app", src)
}

The other thing the cache buys you is honest parallelism. Because operations declare their inputs, the engine knows which steps are independent and runs them concurrently for free — lint, unit tests, and a type-check that share a base container but don’t depend on each other execute in parallel without you orchestrating anything. In YAML you’d hand-write a job matrix and a needs: graph; in Dagger it falls out of the data dependencies. That’s the quiet productivity win people don’t mention: you describe what depends on what, and concurrency is the engine’s problem, not yours.

One caveat on the cache that catches people: it’s keyed on inputs the engine can see. If a step reads a file you didn’t pass in, or depends on the current time, or pulls a floating tag, the cache key won’t reflect the change and you’ll get a stale result that looks like a bug. The discipline is to make every input explicit — pass directories and secrets in as arguments, pin base images by digest when determinism matters, and treat “but it cached the old thing” as a signal that you smuggled in a hidden dependency rather than evidence the cache is broken.

The catch, because there’s always a catch

Dagger is not free of friction.

• You’re now running a BuildKit engine. There’s a persistent dagger-engine container, and on a fresh runner the first invocation pays a startup tax before caching kicks in. On ephemeral runners with no persistent cache, you lose a chunk of the speed advantage unless you wire up a remote cache.
• More power, more rope. Writing pipelines as Go or Python is genuinely more capable than YAML, but a junior who could copy-paste a GitHub Actions snippet now has to read SDK code. That’s a real onboarding cost.
• The ecosystem is younger. The Daggerverse of pre-built modules is growing but still patchier than the mature marketplace of an established CI provider. You’ll write more from scratch.
• A conceptual hump. People expect a pipeline to be a list of steps; Dagger wants you to think in composable functions returning containers and directories. It’s lovely once it lands and bewildering before it does — the first week feels like learning to cook by being handed a chemistry set. The people who bounce off Dagger are almost always the ones who never made that mental shift.

Troubleshooting the things that actually bite

A few real-world snags and the fixes:

• First run is glacial, every run after is too. Your cache isn’t persisting. Locally that means the engine container is being recreated; in CI it means there’s no shared cache backend. Point Dagger at a remote cache (an OCI registry or an S3-compatible store via the engine’s cache config) so ephemeral runners reuse layers.
• dagger call can’t reach a private registry. Pass credentials as a Secret — never bake them into a string argument, because string args land in the cache key and the engine logs. Use --token=env:REGISTRY_TOKEN so the value is read from the environment at runtime and treated as a secret.
• “Cannot connect to the Docker daemon.” Dagger needs a container runtime. On a runner that’s fine, but in some sandboxed environments you need Docker-in-Docker or a rootless runtime configured first.
• Cache hits when you expected a rebuild. This is content-addressing working as designed — if nothing in the inputs changed, the step is skipped. If you genuinely need a fresh run (pulling latest of a base image, say), add a cache-busting input or pin the digest so changes are visible to the engine.

The recurring theme: Dagger is honest about why it does or doesn’t re-run a step, which is more than most YAML pipelines manage. When the same image verification discipline matters downstream, pairing this with signing and verifying your container images with Sigstore and cosign closes the loop from “built reproducibly” to “provably the thing we built.”

Verdict

Dagger is worth it if your pipeline is complex enough that the push-wait-fail loop is costing you real hours, or if you’re allergic to vendor lock-in. The local-execution story alone justifies it for anyone debugging a gnarly multi-stage build — being able to run the whole thing on your laptop, set a breakpoint, and step through it is a quality-of-life upgrade you don’t give back.

If your CI is “run the tests, build a container, push it” and it already works on three lines of YAML, Dagger is a solution to a problem you don’t have — leave it, and don’t let the novelty tax you. But for teams maintaining serious, branching, multi-environment pipelines, or anyone who’s been burned by a pipeline that only exists inside one vendor’s walls, I’d reach for it on any new project of meaningful size. Budget a week for the team to actually absorb the model rather than copy-pasting examples and hoping — that week is the whole investment, and it pays back the first time you fix a pipeline bug without pushing once.