Rootless Podman for the Paranoid

The pitch for rootless containers is easy to say and hard to feel: if the container escapes, it escapes into an unprivileged user account instead of onto your host as root. The distance between those two outcomes is the entire value proposition, and for a long time I filed it under “nice in theory, painful in practice” — which was accurate in 2020 and stopped being accurate somewhere around Podman 4.

I now run most of my containers rootless. The daemonless argument I made in Podman for people who distrust the Docker daemon is only half the story; this is the other half, and it is the half that matters. It cost me two evenings of learning things that are obvious in retrospect, and I have not regretted it. Here is what actually happens, what it protects you from, and what it does not.

What rootless actually means

When you run podman run as an unprivileged user, Podman creates a user namespace. Inside it, your process believes it is root — UID 0, full capabilities, the lot. Outside it, the kernel sees an unprivileged process owned by you. The mapping between those two views is the whole mechanism, and it is configured in two files:

1
2
3
4

$ cat /etc/subuid
smarc:100000:65536
$ cat /etc/subgid
smarc:100000:65536

That says: the user smarc may map 65,536 UIDs starting at 100000. So container UID 0 becomes host UID 100000, container UID 1 becomes 100001, and so on. A process running as root inside the container that manages to break out is a process owned by host UID 100000, which owns nothing, is in no groups, and cannot read your home directory.

Most distributions create these entries when the user account is created. If yours did not, usermod --add-subuids 100000-165535 --add-subgids 100000-165535 smarc sorts it, and then podman system migrate tells Podman to rebuild its storage with the new mapping. Do this before you have containers you care about; the migration is disruptive.

Verify from inside:

1
2
3
4
5
6

$ podman run --rm alpine id
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon)
$ podman run --rm -d --name test alpine sleep 300
$ ps -o user,pid,cmd -p $(pgrep -f "sleep 300")
USER       PID CMD
100000    4812 sleep 300

Root inside. Nobody outside. That is the demo, and it remains satisfying.

The four things that break

Volume permissions. This is the one that sends people back to Docker. You bind-mount a directory from your home, and the container writing as its own root creates files owned by 100000 on the host — which your own user cannot delete. Or the reverse: the container cannot read files you own, because your UID 1000 maps to nothing inside its namespace.

The clean fix is --userns=keep-id, which maps your host UID to the same UID inside the container:

1
2
3

$ podman run --rm -v ~/data:/data:Z --userns=keep-id alpine touch /data/hello
$ ls -l ~/data/hello
-rw-r--r-- 1 smarc smarc 0 Jun 11 10:22 /home/smarc/data/hello

The messy fix, for when the image insists on running as a specific UID, is --userns=keep-id:uid=1000,gid=1000 to place your UID at the container’s expected value. And when you have already made a mess, podman unshare runs a command inside your user namespace so you can clean up:

1

$ podman unshare chown -R 1000:1000 ~/data

That command is the single most useful thing in this article. It looks like it needs sudo. It does not.

Named volumes sidestep the whole problem, because Podman handles the mapping. Where I can use a named volume, I do.

Ports below 1024. An unprivileged process cannot bind them, so -p 80:80 fails. The right answer is publishing high and putting a reverse proxy in front, which you almost certainly already have. The wrong-but-tempting answer is lowering net.ipv4.ip_unprivileged_port_start to 80 in sysctl, which quietly lets any user on the box bind privileged ports — a fine trade on a single-user machine and a poor one anywhere else. It belongs in the same category as the other knobs I went through in sysctl hardening: defensible with a reason, and never by reflex.

Networking is different. Rootless containers get their networking from a userspace proxy. Podman 5 uses pasta by default, which replaced slirp4netns and is dramatically faster, but both share one visible consequence: the container sees a source address that is not the real client’s. If your service logs client IPs, or applies rate limiting per IP, or you ban abusive clients based on its logs, you will see one address for everything. pasta handles this better than slirp did — with -p publishing it can preserve the source address in many cases — and it remains the first thing to check when your logs go strange.

No ping, sometimes. ICMP from inside a rootless container needs net.ipv4.ping_group_range to include your GID. Most distributions set this now. When your container’s health check uses ping and mysteriously fails, this is why, and the fix is a sysctl one line long.

Storage, briefly

Rootless Podman needs a way to do overlay mounts without privileges. On a modern kernel it uses native overlayfs in a user namespace and there is nothing to configure. On older ones it falls back to fuse-overlayfs, which is noticeably slower on write-heavy workloads.

1
2
3
4

$ podman info --format '{{.Store.GraphDriverName}}'
overlay
$ podman info --format '{{.Store.GraphOptions}}'
map[]

An empty GraphOptions with the overlay driver means native. If you see overlay.mount_program=/usr/bin/fuse-overlayfs, you are on the fuse path — fine for most things, worth knowing about if a build is inexplicably slow.

Storage lives in ~/.local/share/containers/storage, which means it is inside your home directory, which means it is inside whatever backup covers your home directory. That has caught out more than one person who suddenly had 40 GB of container layers in their nightly snapshot. Exclude it. Images are reproducible; snapshotting them is a waste of disk.

Putting it together

Rootless on its own is a command you type. Rootless plus systemd is a deployment. The two compose properly through Quadlet, which I covered in Podman Quadlets — the same unit file works under systemctl --user, and the result is a container running as an unprivileged user, in its own namespace, supervised by a user-level init, with no root process anywhere in the chain:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# ~/.config/containers/systemd/miniflux.container
[Unit]
Description=Miniflux
After=network-online.target

[Container]
Image=docker.io/miniflux/miniflux:2.2.5
PublishPort=127.0.0.1:8081:8080
Volume=miniflux-data.volume:/data:Z
UserNS=keep-id
NoNewPrivileges=true
DropCapability=ALL
ReadOnly=true
Tmpfs=/tmp

[Service]
Restart=always

[Install]
WantedBy=default.target

DropCapability=ALL and ReadOnly=true are worth the two lines. Rootless already means the container’s root has no real power on the host; dropping capabilities means it has little power inside its own namespace either, which raises the cost of a two-stage escape. Most well-behaved images run fine with both. The ones that do not will tell you loudly on first start, and the loud failure is a useful signal about the image.

And remember loginctl enable-linger, or all of this vanishes when you log out.

Proving it to yourself

Security claims deserve a demonstration rather than a paragraph. The classic Docker escape is one command:

1
2
3
4
5

$ docker run --rm -v /:/host -it alpine chroot /host sh
# whoami
root
# cat /etc/shadow | head -1
root:$6$rounds=...

Any user in the docker group can run that, on any host with the daemon, and be root in under a second. Adding a user to the docker group is granting them passwordless root, and the group name does an excellent job of disguising that.

The rootless equivalent:

1
2
3
4
5
6
7

$ podman run --rm -v /:/host -it alpine chroot /host sh
# whoami
root
# cat /etc/shadow
cat: can't open '/etc/shadow': Permission denied
# touch /root/pwned
touch: /root/pwned: Permission denied

Root inside, and the kernel still checks every access against host UID 100000, which owns nothing. The mount succeeded because I asked for it and I can read those files myself; the privileged ones stay closed. That is not a subtle difference, and running both commands back to back on the same machine is more persuasive than any amount of prose about namespaces.

Worth being clear about what this demo does not show. It does not show the container being unable to reach the network, read my own home directory, or burn every core on the box. It shows one specific class of catastrophe becoming a permission error.

The performance question

Everyone asks and the answer is dull, which is the answer you want.

CPU-bound work inside a rootless container runs at native speed. The user namespace is a kernel data structure consulted at permission-check time; it costs nothing measurable to a process doing arithmetic. Disk I/O on a named volume with native overlay is likewise indistinguishable — the same overlayfs the root daemon uses, mounted in a different namespace.

Networking used to be the exception and mostly is not any more. The old slirp4netns implemented a TCP/IP stack in userspace and copied every packet through it, which cost somewhere around half your throughput on a fast link and a great deal of CPU on a slow one. pasta, the default since Podman 5, is dramatically better — it bypasses the userspace stack for most flows and, on a gigabit LAN, I cannot distinguish it from rootful in a file transfer. If you are pushing 10 GbE through a rootless container, measure it. Everyone else can stop worrying.

The place you may notice something is a build. If you are on a kernel old enough to force fuse-overlayfs, a large multi-stage build with many layers spends real time in FUSE context switches, and a build that takes ninety seconds rootful can take three minutes rootless. Check the driver with podman info as above; if it says native overlay, the question is closed.

And there is a startup cost of a few tens of milliseconds per container while the namespace and the network are set up. For a long-running service this is beneath notice. For a workflow that runs a container per invocation — a CI runner, a podman run inside a loop — it adds up in a way you will feel, and it is the one honest performance argument against rootless I know of.

The Docker comparison, fairly

Docker has rootless mode too, and it has since 2019. It works, I have run it, and it is a worse experience for a reason that has nothing to do with the quality of the implementation.

Docker’s rootless mode moves the daemon into your user namespace. You now have a long-running process, owned by you, supervising your containers, reachable over a socket in your runtime directory. Everything Docker does architecturally is preserved; only the UID changed. The daemon must be running for your containers to run, it must be started at login or via lingering, and if it dies your containers die with it.

Podman’s rootless mode has no daemon to relocate, because there was never one. podman run forks a conmon per container and exits. The container’s parent is conmon; conmon’s parent is systemd or nothing. This is why Quadlet works as well as it does — there is no supervisor to reconcile with systemd’s supervision, so systemd simply is the supervisor.

There is also a separate Docker feature called userns-remap that is easy to confuse with rootless mode and is a different thing entirely: the daemon still runs as root, but containers get mapped into a subordinate UID range. It defends the host from the container. It does nothing about the daemon, the socket, or the docker group, which is where the real exposure lives. Useful, and answering a different question.

If you are deep in Docker and rootless is the only thing you want, Docker’s rootless mode is a smaller change than a migration to Podman and gets you most of the isolation. I would still argue for Podman, on the grounds that the thing you are trying to remove is a privileged supervising process, and moving that process to a different UID is a smaller removal than not having one.

Troubleshooting

“lchown: operation not permitted” pulling an image. Your subuid range is too small. The image contains a UID above your allocation — 65,536 covers almost everything, but some images use UID 100000+ internally out of misplaced enthusiasm. Widen the range in /etc/subuid, then podman system migrate.

“cannot clone: operation not permitted”. Unprivileged user namespaces are disabled. Check sysctl kernel.unprivileged_userns_clone on Debian-derived systems and user.max_user_namespaces. Some hardened kernels ship these off deliberately. Turning them on is a real trade-off — see below.

Container starts as root under systemd but not by hand. You have both a root and a user copy of the same Quadlet file. Both generators ran.

SELinux denials on every volume. The :Z suffix relabels the host directory to the container’s context, and without it a mount fails opaquely on an enforcing system. :z shares the label between containers; :Z makes it private. Get this wrong and the temptation is to set the whole thing permissive, which is exactly what SELinux without turning it off exists to argue against.

Everything works, then breaks after a reboot with no changes. /tmp cleanup removed Podman’s runtime directory. Setting XDG_RUNTIME_DIR correctly, which lingering does for you, fixes it permanently.

The honest limits

Rootless Podman is a meaningful improvement over running containers as root, and it is not a sandbox. Three caveats worth internalising.

The user namespace itself is attack surface. unshare(CLONE_NEWUSER) available to unprivileged users has been the entry point for a genuine list of kernel privilege escalations, which is why some distributions gate it. You are trading “a container escape lands on root” for “a container escape lands on an unprivileged user, plus a slightly larger kernel surface exposed to that user”. For a homelab that trade is clearly correct. For a machine where unprivileged local users are already your threat model, think about it properly.

Rootless does nothing about the kernel. A container escaping into UID 100000 that then exploits a kernel bug is root anyway. The namespace is a wall, and walls are for raising cost.

And rootless says nothing about what the application does with the network. A compromised container running as nobody can still scan your LAN, hit your other services, and exfiltrate whatever it can read. Isolation of identity is one axis. Segmentation is another, and it needs separate work.

Is it worth it?

Yes, with a caveat about which containers.

For anything that terminates untrusted input — anything reachable from outside, anything processing files other people sent you — rootless is the correct default and the friction is now low enough that arguing about it is a waste of an evening. Podman 5 with pasta and native overlay has removed most of what made this annoying three years ago.

For a handful of infrastructure containers that genuinely need host networking, privileged device access, or to manipulate the network stack, rootless will fight you and lose you time. I run those as root Quadlets with capabilities dropped to the specific set they need, and I keep that list short enough to recite from memory. Six containers, all of which I chose deliberately.

The version of this that fails is the all-or-nothing one, where you migrate everything in a weekend, hit the volume permission wall on the third stack, and go back to Docker with a story about how Podman is not ready. Move one service. Learn podman unshare. Move the next one when the first has been boring for a fortnight.