Distroless and Chainguard Images in a Homelab

I scanned a small Go service of mine last year — about 900 lines, one binary, no dependencies worth naming. The scanner reported forty-one vulnerabilities. Every single one was in Debian packages I had never asked for and the binary never touched: a package manager, a shell, libssl for a program that speaks plain HTTP behind a proxy, and the locale data for languages I do not speak. My code contributed nothing to that number. The base image contributed all of it.

That is the argument for distroless, and it is a good one. The counter-argument arrives about three weeks later, at 23:00, when the service is misbehaving and docker exec -it thing sh returns OCI runtime exec failed: exec: "sh": executable file not found.

Both of these things are true. Here is how I ended up splitting the difference.

What “distroless” actually means

It is a misleading name. A distroless image still contains files from a distribution — libc, CA certificates, timezone data, a /etc/passwd with a nonroot user. What it omits is everything that exists for a human: no shell, no package manager, no coreutils, no ps, no curl. The image is your binary, its runtime dependencies, and the minimum scaffolding the kernel needs to start it.

Google’s originals live at gcr.io/distroless/* and come in a small family:

• static — nothing but CA certs, timezone data and /etc/passwd. For fully static binaries. About 2 MB.
• base — adds glibc. For Go binaries built with cgo, or Rust with the default linker.
• cc — adds libgcc and libstdc++. For C++ things.
• java, python3, nodejs — language runtimes, which are large enough that “distroless” starts to feel aspirational.

Chainguard’s are the more interesting modern option. They are built from Wolfi, a Linux undistro designed for containers — glibc-based, no kernel, apk-flavoured packaging, rebuilt continuously so that patched versions land within hours rather than at a distribution’s release cadence. The images at cgr.dev/chainguard/* are minimal by default, ship with an SBOM baked in, and are signed.

The practical difference between the two families is upkeep. Google’s distroless images are rebuilt when Google rebuilds them. Chainguard’s are rebuilt constantly, which is what makes the “zero known CVEs” claim on their marketing hold up more often than you would expect.

Building against them

The pattern is a multi-stage build, and the important part is that your builder stage can be as fat as you like — none of it survives:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

FROM golang:1.24-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/app ./cmd/app

FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/app /app
USER nonroot:nonroot
EXPOSE 8080
ENTRYPOINT ["/app"]

The result:

1
2
3
4

$ podman images | grep myapp
localhost/myapp   latest   8f2c1a44b   8.41 MB
$ trivy image localhost/myapp
Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)

Forty-one to zero, and the image went from 118 MB to 8.4 MB. The :nonroot tag matters — it sets the default user to UID 65532, which means the image cannot run as root even if someone forgets to say so, and it pairs naturally with rootless Podman.

The Chainguard equivalent, if you want continuous rebuilds under you:

1
2
3
4
5
6
7
8

FROM cgr.dev/chainguard/go:latest AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /out/app ./cmd/app

FROM cgr.dev/chainguard/static:latest
COPY --from=build /out/app /app
ENTRYPOINT ["/app"]

The debugging problem, and the four answers

There is no shell. This is the point and it is also genuinely inconvenient, and anyone telling you otherwise has never had a container misbehaving in a way its logs did not explain.

Debug tags. Both families publish :debug variants with BusyBox included. Google’s is gcr.io/distroless/static-debian12:debug; Chainguard’s is cgr.dev/chainguard/static:latest-dev. Rebuild against the debug tag, poke around, rebuild against the real one. It works, it requires a rebuild and redeploy, and it means the thing you debugged is not quite the thing that was broken.

Ephemeral debug containers. The proper answer, if the runtime supports it: attach a container with a full toolbox into the same namespaces as the running one.

1
2
3
4
5

$ podman run -it --rm \
    --pid=container:myapp \
    --network=container:myapp \
    --cap-add=SYS_PTRACE \
    nicolaka/netshoot

You now have a shell with tcpdump, strace, dig and everything else, seeing the same processes and the same network stack as the distroless container. The filesystem is netshoot’s, and the target’s is reachable at /proc/1/root/. On Kubernetes the equivalent is kubectl debug -it pod/myapp --image=nicolaka/netshoot --target=myapp.

nsenter from the host. Blunt and effective:

1
2
3

$ pid=$(podman inspect --format '{{.State.Pid}}' myapp)
$ sudo nsenter -t $pid -n -p ps aux
$ sudo ls -l /proc/$pid/root/

You are on the host with root, so nothing is hidden. Reading the container’s filesystem through /proc/<pid>/root is often all you needed.

Copy the file out. podman cp myapp:/data/config.yaml - prints it. No shell required for the 60% of debugging that is “what does that file say”.

Between ephemeral containers and nsenter, the missing shell has cost me almost nothing in the last year. The first month was worse, because the reflex is to type exec sh and the reflex takes a while to unlearn.

What the CVE count is really measuring

The forty-one findings on my Go service deserve a closer look, because “distroless took it to zero” is a sentence that flatters the wrong thing.

Not one of those forty-one was reachable. The binary was a static Go program that opened a socket and served JSON. It never invoked the shell. It never called into libssl. The package manager sat on disk being a file. A vulnerability in apt’s HTTP method matters if something runs apt, and nothing was ever going to. On any honest reachability analysis, my exposure from those forty-one was approximately zero before I changed a line.

So what did I gain? Two things, and only one of them is about vulnerabilities.

The smaller gain is that the number went away. This sounds trivial and is not, because a scanner reporting forty findings every week trains you to stop reading scanner output, and then the one that matters goes past unread. Getting a component to genuinely zero means any future finding is signal. That is a workflow benefit dressed up as a security one, and I will take it — it is the same argument I made about alert volume in the context of Trivy and container scanning, where the tool’s value collapses the moment its output becomes wallpaper.

The larger gain is the one nobody puts in the marketing: the post-exploitation environment disappears. Which brings us to the part that actually changed my mind, at the end.

Alpine, and the argument you will have

Somebody always says: Alpine is 5 MB, it has a shell, and it has almost no CVEs either. Why bother with any of this?

It is a reasonable position and for a lot of homelabs it is the right one. Alpine’s CVE count is low because musl and BusyBox are small and the package set is tight. You get a shell, apk when you need to add something, and an image most maintainers already publish. The gap to distroless is smaller than the gap from Debian to Alpine.

Two things separate them. Alpine uses musl, and musl’s differences from glibc are real — DNS resolution behaves differently under some configurations, thread stack sizes are smaller by default, and a handful of Python wheels and Java workloads have had genuine trouble. If you have ever debugged a container that resolves some hostnames and not others, you have met this. Chainguard’s images are glibc-based specifically to avoid that whole category, which is a considerable part of their appeal for anything that is not a static Go binary.

And Alpine has the toolkit. BusyBox is a shell, wget, nc, find and about three hundred other applets in one binary. That is enormously convenient for you and equally convenient for anyone who lands code execution in your container. apk add on a running container with network access is a full compromise toolkit two commands away.

My rule: Alpine for the builder stage, where the toolkit helps and nothing ships. Static or distroless for the runtime stage, where the toolkit only ever helps someone else. If that sounds like a lot of ceremony for a homelab, it is four lines of Dockerfile, and I would rather have the argument once than every time.

Where the payoff actually is

Here is the part the enthusiastic blog posts skip: for a homelab, this mostly does not apply to the software you run.

You do not build your document scanner, your media server, your password manager or your feed reader. Their maintainers publish images, those images are built on Debian or Alpine or Ubuntu, and the choice is not yours. Rebuilding somebody’s Python application on a distroless base is a real project — you must reproduce their build, their runtime deps, their entrypoint script, their migrations — and then maintain that fork forever, chasing every upstream release. I tried it once for a service I cared about, and abandoned it after the second upstream release broke my Dockerfile in a way that took an hour to diagnose. The security benefit did not survive contact with the maintenance cost.

The payoff is real in exactly two places.

Your own code. Anything you write and build yourself — a Go CLI, a Rust service, the little webhook shim, a static site’s server — should be on static:nonroot and there is no argument against it. The Dockerfile is four lines longer and the image has no attack surface to speak of. This is free.

The base of things you already build. If you have CI producing images at all, whether it is a Kaniko job or a plain builder, the base image is a choice you are already making. Making it a good one costs one line.

For everything else, the honest move is to scan what you run and fix what matters, which is the argument in Trivy and container scanning, combined with keeping images current. A Debian-based image that gets pulled weekly is in better shape than a distroless image you built in March and forgot.

The Chainguard catch

Chainguard’s free tier gives you :latest and the most recent release. Older tagged versions — the ability to pin go:1.23.4 rather than go:latest — are behind a paid subscription aimed at companies, and the pricing reflects that.

For a homelab this is more annoying than fatal, and it does bite. Building against :latest means your builds are not reproducible: the same Dockerfile today and in a month produces different images, and when a build breaks you cannot bisect because the input moved. That is the opposite of everything I believe about pinning, which I have argued elsewhere at some length.

Two workarounds. Pin by digest — cgr.dev/chainguard/static@sha256:... — which works indefinitely because the digest is immutable, and update it deliberately with a tool that opens a PR. Or use Google’s distroless, which is versioned by Debian release and free forever, and accept a slower patch cadence in exchange. I use Chainguard for builder stages, where :latest is fine because the builder does not ship, and Google’s distroless pinned by digest for runtime stages. That is a compromise born of the licensing rather than the engineering, and I would rather it were not necessary.

The SBOM you get whether you wanted it or not

Chainguard images carry a signed SBOM as an attestation attached to the image, which you can pull without pulling the image itself:

1
2
3

$ cosign verify-attestation     --type spdxjson     --certificate-oidc-issuer=https://token.actions.githubusercontent.com     --certificate-identity-regexp='https://github.com/chainguard-images/.*'     cgr.dev/chainguard/static:latest | jq -r '.payload' | base64 -d | jq '.predicate.packages[].name'
ca-certificates-bundle
wolfi-baselayout

Two packages. That is the whole image. Running the same query against a typical application image built on Debian returns somewhere north of ninety, and the exercise of reading that list is more educational than any argument I can make: you will find a Perl interpreter, apt, and a mail transfer agent’s client library in a container whose entire job is serving JSON.

Whether the attestation is useful to you depends on whether you would ever act on it. In a company with a compliance function, yes. In my house, the SBOM’s value is diagnostic — when a CVE lands with a scary name, one query tells me whether the affected package exists in anything I run, which beats guessing. That is worth having and it is not worth building a pipeline for.

The signature is the more interesting half. Verifying that an image came from the build system it claims to have come from, before it runs, closes off a class of registry tampering entirely, and it costs a cosign verify in the pull path. That deserves its own treatment; the short version is that minimal images make it tractable, because there is almost nothing in them to reason about.

Troubleshooting

“exec format error” or “no such file or directory” on a binary you can see. The binary is dynamically linked and the image has no linker. CGO_ENABLED=0 for Go; for Rust, target x86_64-unknown-linux-musl; or move up to distroless/base, which has glibc. ldd on the build stage tells you what it wants.

“x509: certificate signed by unknown authority”. static includes CA certificates, but if you built your own scratch-adjacent image, copy them: COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/.

Timezone is UTC and will not change. TZ=Europe/Copenhagen does nothing without the zoneinfo database. Copy /usr/share/zoneinfo from the builder, or accept UTC — which for a server is arguably the correct answer anyway.

Entrypoint script does not run. There is no shell to run it. Distroless images have no /bin/sh, so ENTRYPOINT ["/entrypoint.sh"] fails and the shell form of ENTRYPOINT fails too. Whatever the script did — waiting for a database, templating a config — has to move into the binary or into systemd/compose dependency ordering.

The image will not write anywhere. :nonroot runs as UID 65532 and the filesystem is owned by root. Either COPY --chown=65532:65532, or mount a volume, or better, use a tmpfs for scratch space and keep the image read-only.

Is it worth it?

For code you build: yes, unconditionally, and the effort is about fifteen minutes.

For code other people build: almost never, unless you were going to maintain a fork anyway. The maintenance cost of rebasing somebody’s application onto a minimal image outlives your enthusiasm for it, and the CVE count you were chasing was mostly unreachable packages that a scanner counted because counting is what scanners do.

The part that changed how I think was less about vulnerabilities than about intent. A container with a shell, a package manager and curl in it is a container where an attacker who gets code execution has a working environment waiting for them — the exact “living off the land” toolkit that makes the difference between a nuisance and a foothold, which is the mechanism behind most of what I described in supply chain attacks. Removing the shell removes the toolkit. The zero in the scanner output is a side effect. The empty toolbox is the thing worth having.