Earthly: Containerized Build Pipelines That Combine Dockerfile and Makefile
Reproducible builds with a syntax you already half-know, that run the same on your laptop and in CI
Contents
Every project I’ve maintained eventually grows a Makefile full of build, test, lint, and release targets that work beautifully on my machine and nowhere else. The targets assume a particular toolchain, a particular OS, particular environment variables. Then CI does something subtly different, and you maintain two parallel descriptions of the same build forever. Earthly is the tool that finally made me delete that duplication. It’s what you get if a Makefile and a Dockerfile had a child who insisted everything run in a container.
The pitch
An Earthfile looks like a Dockerfile with named targets. Each target runs in its own container, inherits a base, and can produce artifacts or images. Because every step runs in a container, the build is the same on your laptop, on a colleague’s laptop, and in CI — there is no “but my Go version is different.” And because it’s built on BuildKit, caching is content-addressed and aggressive: unchanged steps are skipped, and independent targets run in parallel automatically.
Here’s a realistic Earthfile for a Go service:
| |
The +name syntax references another target. +build/vors-api means “the vors-api artifact produced by the build target.” SAVE ARTIFACT ... AS LOCAL writes a file back to your actual filesystem; SAVE IMAGE produces a Docker image. You run any target by name:
| |
Run earthly +docker and it builds, packages, and tags the image — reusing the cached deps layer it already computed for test. That’s the whole trick: shared base targets mean go mod download happens once even though three targets depend on it.
If you’ve never met BuildKit, the short version is that it’s the modern build engine Docker shells out to under the hood. It builds a content-addressed graph of every step, runs independent branches in parallel, and only re-executes a step when its inputs actually change. Earthly is, more or less, a friendlier front end on top of that engine — it’s the same caching model that makes docker build fast, exposed through a syntax that lets you name and depend on targets the way a Makefile does. The win is that you stop thinking about layers as an accident of line ordering and start thinking about them as a dependency graph you control.
The why, before the how
It’s worth being clear about the actual problem this solves, because “another build tool” is a hard sell. The problem is drift. You have a Makefile that works on your machine, and you have a CI pipeline that does almost the same thing in slightly different ways — a different Go version, a different golangci-lint, an apt package the runner happens to have that your colleague’s doesn’t. The two descriptions diverge slowly, and the day they diverge enough to matter is always the day you’re trying to ship something. You end up debugging the pipeline instead of the code, pushing commits titled “fix CI” into a black box you can’t run locally.
Earthly collapses that into a single description that runs in a container, so “works on my machine” and “works in CI” become the same statement. The container is the contract. This is the same instinct behind reaching for Dagger’s pipelines-as-code that run anywhere — both tools attack pipeline drift by putting the logic somewhere you can execute it identically everywhere, rather than trusting a YAML file you can only test by pushing. Where they differ is mostly taste: Dagger gives you a real programming language and an SDK; Earthly gives you a declarative Earthfile that any Dockerfile-literate person can read on sight.
Parallelism and the BUILD keyword
A top-level target can orchestrate others. This is where it starts replacing CI YAML:
| |
earthly +all runs lint, test, and docker — and because they share the deps base but are otherwise independent, BuildKit runs them concurrently up to your local resource limits. No fan-out/fan-in YAML, no manually declaring “needs” between jobs. The dependency graph is implicit in the FROM and +target references.
CI that’s just earthly +all
The reason I bother is that CI becomes thin. A GitHub Actions workflow drops to essentially one meaningful line:
| |
The --ci flag enforces a clean, push-disabled run with strict caching. The logic lives in the Earthfile, which a developer can run identically with earthly +all before pushing. When CI fails, you reproduce it locally in one command. That alone has saved me more debugging than I can count — no more pushing “fix CI” commits to poke at a black box.
This is also why Earthly plays nicely with running pieces of your pipeline locally before they ever reach a runner. If you’ve used act to run GitHub Actions locally, you’ll recognise the appeal — but where act emulates the Actions runner so you can test the workflow YAML, Earthly sidesteps the question by moving the real work out of the YAML entirely. The workflow becomes a thin shim whose only job is to call earthly, and there’s far less workflow left to get wrong.
Caching and remote caches
Locally, caching is automatic and content-addressed: change one line in go.mod and only the deps target and everything downstream of it rebuilds. The interesting part is making that cache survive across CI runs, because a fresh runner starts cold every time. Earthly supports pushing its cache to a registry so the next run pulls warm layers instead of recomputing them:
| |
With that wired up, an unchanged dependency layer is fetched rather than rebuilt, and a typical CI run that did ten minutes of cold work drops to a couple of minutes of “pull cache, build the bit that changed.” Getting the remote cache configured is, honestly, the least pleasant part of adopting Earthly — see the troubleshooting section — but once it’s done you mostly forget it exists.
Gotchas and troubleshooting
It is not free of friction, and the failure modes are worth knowing before you hit them at 2am.
The daemon fills the disk. Earthly needs a BuildKit daemon, which it manages for you in a container. On a long-lived machine that cache grows without bound until a build dies with no space left on device. The fix is to prune it on a schedule:
| |
On constrained CI runners, set a cache size cap so BuildKit evicts old entries automatically instead of filling the volume — earthly config global.buildkit_max_parallelism and the cache-size settings in earthly config are where you’ll spend a few minutes the first time.
Cold CI runs feel slow and you blame Earthly. The first run on a fresh runner with no remote cache rebuilds everything, and it’s tempting to conclude the tool is slow. It isn’t — you just haven’t given it a warm cache. Wire up --remote-cache (above) before judging CI performance, and compare a second run, not the first.
SAVE ARTIFACT AS LOCAL writes nothing. A common confusion: artifacts only land on your real filesystem when the target that produces them is actually built as an output, not merely depended on as a FROM. If +build/vors-api is consumed by +docker but you never asked for the local copy, you won’t see the file. Run the producing target directly, or add an explicit BUILD +build.
Mounting secrets the wrong way. Don’t COPY a token into a layer — it’s baked into the cache and leaks. Earthly has a dedicated secrets mechanism that mounts a value for a single RUN without persisting it:
| |
This matters for the same reason it matters everywhere else in your stack: a credential that ends up in a cached layer is a credential you’ve published. If you care about that (you should), the same discipline that earns you a backup plan you can actually restore from applies to build artifacts — assume anything written to disk or a registry is forever.
Rootless and nested-container surprises. If you’re running Earthly inside another container (a Kubernetes-based runner, say), BuildKit wants either privileged mode or a carefully configured rootless setup. The rootless path works but is the kind of thing you test once and document, because the error messages when it’s misconfigured are unhelpful.
The adoption question
There’s a real elephant here. The company behind Earthly wound down its commercial operations, and the project is no longer under active development. You should treat it as a stable, capable open-source tool that has stopped moving rather than something with a vendor’s roadmap behind it. For my purposes that’s acceptable: the Earthfile format is simple, the behaviour is predictable, and BuildKit underneath is maintained independently and going nowhere. But it does change the calculus. I’d happily adopt it for a personal project or an internal tool; I’d think harder before betting a team’s whole release process on a frozen codebase, and I’d keep the Earthfile simple enough that porting the logic to plain docker build plus a thin Makefile would be a bad afternoon rather than a bad month.
Verdict
Earthly is worth it if you have a polyglot monorepo, a build that’s painful to reproduce, or a CI config that has drifted from how anyone actually builds locally. The single biggest win is that “reproduce CI on your machine” becomes one command, and the second-biggest is that the dependency graph between your build steps becomes something you declare instead of something you maintain by hand across two files. It’s overkill for a tidy single-language project where the native tooling already gives you reproducible builds, and it adds a daemon you have to babysit a little.
Given the frozen-codebase caveat, my honest recommendation is to try it on one annoying service — the one whose CI you dread touching — before committing the whole org. If it makes that service’s build boring, you’ll know within an afternoon, and the Earthfile you write is small and legible enough that walking it back later costs you very little. For the messy, real-world projects where I’ve reached for it, where half the bugs were environment differences rather than actual defects, it earned its place and then some. The fact that it’s no longer chasing a venture roadmap is, for a tool that does one thing well and predictably, almost reassuring.
Related Content
