Multi-Arch Images with buildx

The first time I mixed a Raspberry Pi into a cluster that was otherwise amd64 boxes, I hit the classic wall: exec format error. The image pulled fine, Docker didn’t complain, and then the container refused to start because the binary inside was compiled for x86_64 and the Pi is arm64. Nothing in docker pull warns you about this — it happily hands you an incompatible image if that’s the only tag that exists.

The fix isn’t cross-compiling everything by hand, and it isn’t maintaining two separate Dockerfiles. It’s docker buildx, which has shipped with Docker for years now and does two things well: it can build for architectures your host CPU doesn’t natively support, and it can push a single tag that resolves to the right image no matter which architecture pulls it. That second part — the manifest list — is the bit people usually don’t understand until something breaks.

Why this matters once you have mixed hardware

A homelab that starts as one x86 box tends to grow a Pi or two: a k3s node for Kubernetes on a Raspberry Pi, a dedicated Pi-hole, maybe a Pi doing nothing but running a single Go binary because it’s silent and sips 3 watts. The moment you add a second architecture to a cluster, every image you build yourself needs to work on both, or you end up hand-tagging myapp:amd64 and myapp:arm64 and remembering which node needs which — which is exactly the kind of bookkeeping that quietly breaks six months later when you’ve forgotten the convention.

Multi-arch images solve this at the registry level. You push one tag, myapp:latest, and it’s actually a manifest list — a small JSON document pointing at multiple platform-specific image manifests. When a Pi does docker pull myapp:latest, the registry (or rather, the client, reading the manifest list) picks the linux/arm64 entry automatically. Your amd64 server pulling the same tag gets the linux/amd64 entry. Same tag, same compose file, same Kubernetes manifest, correct binary every time.

Two ways to get non-native architectures built

There are two fundamentally different approaches, and picking the right one matters for build time.

QEMU emulation is the zero-setup option. buildx uses binfmt_misc and QEMU’s user-mode emulation to run arm64 instructions on your amd64 build host, transparently, inside the build. It requires no additional hardware and works out of the box on most Docker Desktop installs and on Linux hosts once you register the QEMU binfmt handlers. The cost is speed — compiling anything non-trivial under emulation is noticeably slower, sometimes 5-10x, because every instruction is being translated rather than executed natively.

Native builders skip emulation entirely by routing the arm64 portion of the build to an actual arm64 machine — a build farm node, a cloud arm64 instance, or, in a homelab, literally one of your Pis acting as a remote builder. buildx supports adding remote nodes to a single builder instance, so a docker buildx build --platform linux/amd64,linux/arm64 fans out: the amd64 layer builds locally, the arm64 layer builds on the Pi over SSH, and the results get combined into one manifest list at the end. Faster for anything beyond trivial builds, at the cost of needing a real arm64 box sitting around — which, if you’re already running a Pi cluster, you have.

For anything more than a quick shell-script wrapper image, I use the Pi as a native builder. For genuinely small utility images, QEMU emulation is fine and one less thing to maintain.

Setting up buildx

Modern Docker includes buildx by default (docker buildx version to check). If it’s missing, install the plugin:

1
2
3
4
5
6

# Debian/Ubuntu
sudo apt install docker-buildx-plugin

# register QEMU handlers for emulated builds (one-time, per host, survives reboot
# via a systemd unit if you use the persistent tag)
docker run --privileged --rm tonistiigi/binfmt --install all

Create a builder that isn’t the default docker driver — the default driver can’t do multi-platform builds because it’s tied to the local Docker engine’s single architecture:

1
2

docker buildx create --name multiarch --driver docker-container --use
docker buildx inspect --bootstrap

To add your Pi as a native node instead of relying purely on emulation:

1
2
3
4
5
6
7

# on your build host, with SSH access to the Pi already working
docker buildx create --append --name multiarch \
  --node pi-builder \
  --platform linux/arm64 \
  ssh://[email protected]

docker buildx inspect multiarch

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

Name:          multiarch
Driver:        docker-container

Nodes:
Name:          multiarch0
Endpoint:      unix:///var/run/docker.sock
Platforms:     linux/amd64, linux/arm64*, linux/arm/v7*   (* = emulated)

Name:          pi-builder
Endpoint:      ssh://smarc@pi-builder.mylab.local
Platforms:     linux/arm64

The asterisk marks emulated platforms on the primary node; the Pi node has no asterisk because it’s native. buildx will prefer the native node for linux/arm64 work once both are available in the same builder.

Building and pushing a manifest list

1
2
3
4
5
6

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  --tag registry.example.com/myapp:1.4.0 \
  --tag registry.example.com/myapp:latest \
  --push \
  .

--push is required for multi-platform builds — buildx can’t --load a multi-arch result into the local Docker engine’s image store, because that store only ever holds one platform per tag. The image has to go straight to a registry, where the manifest list format actually lives. If you want to test locally before pushing, build single-platform first (--platform linux/arm64 --load) and run it under QEMU, or just push to a local/throwaway registry tag.

Check what actually landed:

1

docker buildx imagetools inspect registry.example.com/myapp:1.4.0

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

Name:      registry.example.com/myapp:1.4.0
MediaType: application/vnd.oci.image.index.v1+json
Digest:    sha256:9f2a...

Manifests:
  Name:      registry.example.com/myapp:1.4.0@sha256:1a2b...
  Platform:  linux/amd64

  Name:      registry.example.com/myapp:1.4.0@sha256:3c4d...
  Platform:  linux/arm64

That’s the manifest list — one index digest, two platform-specific children. Anyone pulling myapp:1.4.0 gets routed automatically.

Build cache across architectures

The naive multi-platform build re-runs every layer for every architecture on every invocation, which gets old fast once a Dockerfile has a few slow steps (a full apt-get layer, a go mod download, a npm install). buildx supports exporting and importing cache separately from the image itself, and pointing it at your registry means both a local rebuild and a CI run can reuse the same cache:

1
2
3
4
5
6
7

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  --tag registry.example.com/myapp:latest \
  --cache-to type=registry,ref=registry.example.com/myapp:buildcache,mode=max \
  --cache-from type=registry,ref=registry.example.com/myapp:buildcache \
  --push \
  .

mode=max caches every intermediate layer rather than just the final one, which costs more registry storage but means a change late in the Dockerfile doesn’t force a rebuild of unrelated early layers on the other architecture. Without an explicit cache export, buildx’s docker-container driver has no cache to draw on between separate invocations — the default local Docker engine cache only applies to single-platform builds using the classic builder, not the container-based multi-platform one.

Dockerfile gotchas for multi-arch builds

Most Dockerfiles are already architecture-agnostic if they’re doing normal package-manager installs, but a few patterns break under multi-arch:

1
2
3
4
5
6

# BAD: hardcodes an amd64-only binary URL
RUN curl -L https://example.com/tool-amd64.tar.gz -o tool.tar.gz

# GOOD: use buildx's automatic TARGETARCH build arg
ARG TARGETARCH
RUN curl -L https://example.com/tool-${TARGETARCH}.tar.gz -o tool.tar.gz

buildx automatically populates TARGETARCH (amd64, arm64, arm), TARGETOS, and TARGETPLATFORM for each platform in the build, without you needing to pass them explicitly on the command line. Any Dockerfile step that downloads a prebuilt binary, references an architecture-specific base image tag, or shells out to something that only exists on x86 needs this treatment.

Wiring it into CI

The pattern in GitHub Actions is nearly identical to the manual commands above, just with the official actions doing the builder setup:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

name: build-and-push
on:
  push:
    tags: ["v*"]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-qemu-action@v3
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: registry.example.com
          username: ${{ secrets.REGISTRY_USER }}
          password: ${{ secrets.REGISTRY_TOKEN }}
      - uses: docker/build-push-action@v6
        with:
          context: .
          platforms: linux/amd64,linux/arm64
          push: true
          tags: |
            registry.example.com/myapp:${{ github.ref_name }}
            registry.example.com/myapp:latest

Hosted GitHub Actions runners are amd64-only, so setup-qemu-action is doing the emulation setup here — there’s no native arm64 runner unless you’re on a paid tier or self-hosting one. For a homelab CI pipeline running on your own hardware, you can register your Pi as a buildx node the same way as the manual example and skip emulation for the arm64 leg entirely, which is worth doing once your image builds start taking more than a couple of minutes under QEMU.

Testing before you trust the manifest list

A manifest list existing doesn’t guarantee both platform images actually work — it only guarantees the registry will hand out the right one. I’ve pushed arm64 images that pulled fine on a Pi and then crash-looped, because a dependency baked into the image at build time silently fell back to a slower or incomplete code path on arm64 without erroring, and nothing about the build itself flagged it. docker buildx build --platform linux/arm64 --load locally under QEMU emulation and running the container is a reasonable smoke test before you push anything tagged latest — slow, because it’s still emulated, but enough to catch an obviously broken entrypoint before it reaches real hardware. The more thorough version is a canary rollout: push the new tag, deploy it to one arm64 node first if you’re running a multi-node k3s cluster with a mix of Pi and amd64 workers, and only roll it out cluster-wide once that node’s confirmed healthy. Kubernetes’ own nodeAffinity and topology.kubernetes.io/arch node labels make this easy to script — you can pin a canary Deployment to a specific architecture without touching the image or the manifest list at all, since the scheduler already knows which nodes are which architecture from the kubelet-reported labels.

Why not just build separate images per architecture

It’s a fair question, since nothing stops you from tagging myapp:amd64 and myapp:arm64 by hand and skipping buildx and manifest lists entirely — plenty of projects did exactly that before multi-platform tooling matured. The problem shows up downstream, in every consumer of the image rather than in the build itself. A Kubernetes Deployment, a docker-compose file, an Ansible playbook — anything referencing the image by tag now needs architecture-aware logic baked in, or a human remembering to pick the right suffix for each node. That’s fine for one image on two machines. It stops being fine the moment you’re running a dozen self-built images across a mixed-architecture cluster, because now every deployment manifest needs the same conditional logic duplicated, and updating one image’s tag format means touching every consumer of it. The manifest list moves that complexity to exactly one place — the registry — and makes every downstream reference architecture-agnostic permanently. It’s the same argument for unifying behind one abstraction that shows up elsewhere in a homelab: better to solve a problem once, centrally, than to solve it repeatedly at every call site.

There’s also a subtler cost to the hand-tagged approach that only shows up over time: drift. Two separately built images from the same source at nominally the same version can end up with different installed dependency versions if the build happened on different days, or if a base image tag moved between the two builds. A single buildx build --platform linux/amd64,linux/arm64 invocation builds both platforms from the same build context and the same cache state in one pass, which at least guarantees both images were built from an identical source tree at an identical moment — hand-tagging separately, especially across two different build hosts, gives you no such guarantee unless you’re careful to pin every input.

Troubleshooting

exec format error even after building multi-arch. Almost always means you built and pushed with --load for a single platform by habit, overwriting the manifest list with a single-platform image. Re-run with --push and both --platform values in one invocation — building each architecture in separate commands and pushing them under the same tag will make the second push clobber the first rather than merge into a manifest list.

QEMU emulated build hangs or is absurdly slow on anything doing heavy compilation. Expected for large C/C++/Rust builds under emulation. Either accept the wait, split the build so compilation happens via a cross-compiler toolchain instead of emulation (Go and Rust both support this natively with GOARCH/--target), or add a native arm64 builder node.

docker buildx create --append with an SSH node fails with a permission or key error. The build host needs passwordless SSH to the remote node (key-based, no passphrase prompt, since buildx can’t interact with one) and Docker installed and running on the remote end — buildx doesn’t install Docker for you, it just connects to an existing daemon over SSH.

Base image doesn’t have an arm64 variant. Check the image’s tags on its registry page — most official images (alpine, debian, python, node) are multi-arch already, but small community images sometimes aren’t. If you’re stuck with an amd64-only base, you’re back to QEMU emulation for that stage, or finding an alternative base image.

Manifest list shows the wrong digest for one platform after a partial rebuild. buildx build cache is platform-specific; if you rebuild only intending to change one architecture’s output you still need to pass all --platform values in the same invocation, since each build creates a fresh manifest list from scratch rather than patching the existing one.

A note on Apple Silicon dev machines

The other place this comes up unexpectedly is development. An Apple Silicon Mac is arm64, and if you build images on it with plain docker build and push straight to a registry your amd64 server pulls from, you’ve reproduced the exact exec format error this post opened with — just with the roles reversed, an arm64-native build accidentally landing on an amd64-only production box. The fix is identical: buildx build --platform linux/amd64,linux/arm64 from the Mac, same as from any other build host, produces the correct manifest list regardless of which architecture actually did the building. It’s a detail worth knowing before it costs you a confusing half hour tracing a working local container that mysteriously refuses to start once deployed.

Is it worth it

If your fleet is entirely one architecture, skip this — plain docker build is simpler and there’s no manifest list to reason about. The moment a Pi, an Apple Silicon dev machine building images that need to run on a server, or any arm64 box enters the picture, buildx stops being optional. The CI YAML above is barely longer than a single-arch build once it’s set up — it’s a setup step people skip until the exec format error forces the issue.