Impermanence: Wiping Root on Every Boot

Graham Christensen wrote a blog post in 2020 called “Erase your darlings”, and it did something rare: it took an idea that sounds like a stunt and made it obviously correct within about four paragraphs. The idea is that your root filesystem gets rolled back to an empty snapshot on every single boot. Everything you did not explicitly declare — every stray file, every config you edited at 2am, every cache some installer decided to drop in /var — is gone the next time the machine comes up.

I have run this on two machines for eighteen months. It is the single best test I know of whether your infrastructure is actually described anywhere, and it fails you loudly and early rather than three years later during a restore. It is also, for most people, more discipline than the payoff justifies. Both of those deserve explaining.

Why a declarative distro is not enough

If you run NixOS you might reasonably think you already have this. You describe the machine in a file, nixos-rebuild builds it, the system in /nix/store is exactly what you wrote. Where is the room for drift?

Everywhere outside the store, is where. NixOS generates /etc and owns /nix, and it leaves /var, /home, /root and /srv as ordinary mutable directories, because software needs somewhere to write. So over a year your “declarative” machine accumulates: a Docker daemon’s entire state tree, a Jellyfin library database, systemd’s journal, a ~/.ssh/known_hosts full of machines that no longer exist, a /root/.bash_history documenting your mistakes, and a /var/lib/something created by a service you enabled, tried, disabled and forgot.

None of that is in your configuration. If the SSD dies, none of it comes back. You believed you had a reproducible machine and you had a reproducible system running on top of an undocumented pile — which I wrote about as the central trade in the NixOS honest trade-off, and impermanence is the answer to the half I left open there.

Impermanence removes the pile. Root is wiped, so anything that survives a reboot survives because you said it should, in a file, in git. The set of things you must back up becomes exactly the set of things you listed. That is a genuinely different claim from “I take snapshots”.

How the wipe works

The mechanics are simpler than they sound. You need a filesystem with cheap snapshots — ZFS or btrfs — and a hook that runs in the initrd before the root filesystem is mounted.

Immediately after installing the OS, while root is pristine, you take a snapshot called blank. On every boot, the initrd rolls root back to blank. The kernel and initrd live on a separate ESP; /nix lives on a separate dataset that is never rolled back, so the store survives and the system boots into whatever generation you selected.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# ZFS variant. The dataset layout matters more than the code.
#   rpool/local/root   -> wiped on boot, has a @blank snapshot
#   rpool/local/nix    -> the store, never wiped
#   rpool/safe/persist -> everything that survives, snapshotted properly
#   rpool/safe/home    -> your files

boot.initrd.postDeviceCommands = lib.mkAfter ''
  zfs rollback -r rpool/local/root@blank
'';

fileSystems."/persist" = {
  device = "rpool/safe/persist";
  fsType = "zfs";
  neededForBoot = true;   # the impermanence module reads it early
};

The neededForBoot flag is load-bearing and easy to miss. Bind mounts from /persist are set up early in the boot, so /persist has to be mounted before that happens. Without it you get a machine that boots into a system with no SSH host keys and no machine-id, which on a headless box means a trip to find a monitor.

The btrfs equivalent deletes and recreates the root subvolume from a read-only snapshot instead of rolling back, and there is a well-known script for it that moves the old root aside into a dated directory first — useful for the first month, when you will want to look at what got deleted and ask whether you meant it.

Declaring what survives

The nix-community/impermanence module turns the surviving set into a list. It bind-mounts each path from /persist into place at boot, so the software sees an ordinary directory and has no idea anything unusual is happening.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

{
  imports = [ inputs.impermanence.nixosModules.impermanence ];

  environment.persistence."/persist" = {
    hideMounts = true;

    directories = [
      "/var/log"
      "/var/lib/nixos"          # uid/gid stability — do not skip this
      "/var/lib/systemd/coredump"
      "/var/lib/docker"
      "/var/lib/jellyfin"
      "/etc/NetworkManager/system-connections"
    ];

    files = [
      "/etc/machine-id"
      "/etc/ssh/ssh_host_ed25519_key"
      "/etc/ssh/ssh_host_ed25519_key.pub"
      "/etc/ssh/ssh_host_rsa_key"
      "/etc/ssh/ssh_host_rsa_key.pub"
    ];

    users.smarc = {
      directories = [
        "projects"
        { directory = ".ssh"; mode = "0700"; }
        { directory = ".gnupg"; mode = "0700"; }
        ".local/share/direnv"
      ];
      files = [ ".bash_history" ];
    };
  };
}

Read that list as a document. It is the complete answer to “what is on this machine that I would miss”, written down, in git, reviewed every time you add a service. Getting to a correct version of that list is the actual work, and it takes a couple of weeks of reboots.

/var/lib/nixos is the entry people miss and then spend an evening on. It holds the uid/gid allocation table. Wipe it and NixOS reassigns user IDs on the next boot, so the files in /persist owned by uid 1000 now belong to somebody else, and your permissions quietly rearrange themselves. Put it in the list on day one.

Building the list, which takes two weeks

Nobody writes a correct persistence list up front. The process is empirical and mildly humiliating, and it goes like this.

Start with the minimum: /etc/machine-id, the SSH host keys, /var/lib/nixos, /var/log, and your home directory. Reboot. Something breaks. Find out what wrote where, add the path, reboot again. Repeat for a fortnight until a week goes by without a surprise.

The tool that makes this tractable is a diff of the running root against the blank snapshot, run just before you reboot. On ZFS that is a one-liner, and the output is the list of every file the machine created since boot that is about to be destroyed:

1
2

# What has been written to root since the blank snapshot?
zfs diff rpool/local/root@blank | grep -vE '/(tmp|run|var/tmp)/' | less

Run that after a week of normal use and read every line. Some of it is obviously junk — caches, sockets, PID files, things whose absence at boot is a feature. Some of it is state you would miss. And a few lines will make you stop, because they will be evidence of something you configured by hand and had forgotten entirely.

The judgement call on each line is one question: if this file disappeared, would the machine still do its job? If the answer is yes, let it die. Most homelabbers persist far too much on the first pass, out of nervousness, and end up with a /persist that is a slightly tidier version of the pile they were trying to eliminate. The list gets better when you delete entries from it, and the only way to know an entry is unnecessary is to remove it and reboot.

The btrfs equivalent is less elegant — there is no zfs diff — so the standard approach moves the old root subvolume aside into a dated directory and lets you rummage through it afterwards with find before a timer deletes anything older than thirty days. Slower to read, same information.

Troubleshooting

SSH host key changed on every boot. The host keys were regenerated because you did not persist them, and every client screams about a man-in-the-middle. Persist the four files above. If you have already booted a few times, clear the stale entry from your workstation’s known_hosts.

A service starts fresh every boot with an empty database. Its state directory is under /var/lib and is not in your list. systemctl show <unit> -p StateDirectory tells you where it writes; add that path and reboot.

Logs vanish, so you cannot debug why logs vanish. Persist /var/log early. Alternatively ship logs off the box, which you should be doing anyway — the machine that erases itself is a poor place to keep the evidence of what it did.

Permissions are wrong on a persisted directory. Bind mounts carry the source’s ownership, and the source was created by root. Use the attribute-set form with an explicit mode and user, rather than fixing it by hand — a manual chown is exactly the kind of undeclared change this whole exercise exists to abolish.

The machine boots to an emergency shell after a config change. Usually a missing neededForBoot on /persist, or a rollback command referencing a dataset that no longer has a @blank snapshot. Boot the previous generation from the menu, which still works and is the reason this is survivable at all.

Docker containers lose everything. /var/lib/docker holds images, volumes and container state. Persist it, and accept that your carefully declared machine now has a large opaque blob in it that impermanence cannot inspect. This is the honest limit of the technique.

What it actually teaches you

The value shows up in the first fortnight, and it is unlike what I expected. You reboot, something is broken, and you find out that a thing you believed was configured was actually something you typed once. My list of discoveries included: a manually installed Python package a monitoring script depended on, a /etc/hosts entry from a debugging session two years earlier that had been silently propping up a service, and a cron job that existed nowhere in any repository.

Every one of those would have surfaced during a restore, at the worst possible moment, with the least available patience. Impermanence surfaces them on a Tuesday evening when the stakes are a reboot. That is the whole product: it converts a rare catastrophic discovery into a frequent trivial one.

It also makes your backup story exact. /persist and /home are the datasets that matter; everything else is derivable from git. Snapshot those two, replicate them off-site, and you are covered — which is a much clearer statement than most homelabs can make, and it makes actually implementing the 3-2-1 rule a small job rather than an audit.

The caveat is the one I keep repeating: a snapshot of /persist is still a snapshot. It lives on the same pool as the thing it protects, and a pool that dies takes both. Snapshots are not backups applies with full force here, and arguably more, because the tidiness of the setup is seductive enough to make you feel finished.

Doing this without NixOS

The technique is not exclusive to NixOS, though NixOS is the only place it feels natural, and the reason is worth being precise about: the wipe is only survivable when something else can rebuild what was wiped. On NixOS that is the store and your configuration. Elsewhere you have to supply the rebuilder yourself.

On Fedora Silverblue or any ostree-based system, you already have most of the property by a different route. The root is read-only and versioned, /etc is a three-way merge against the deployment, and /var persists. It is impermanence with the defaults inverted — the system is immutable and your state is mutable, and you get the atomic upgrade and rollback without writing a list. The cost is that /var remains an undocumented pile, so you get the reproducible system and none of the audit.

On an ordinary Debian or Arch box, you can do the literal thing: a btrfs root with a read-only @blank subvolume, a systemd unit or an initramfs hook that recreates root from it before the switch-root, and bind mounts from a /persist subvolume driven by entries in /etc/fstab. It works. It is also entirely on you, because nothing rebuilds the wiped root’s contents except the package manager’s idea of the world, and /etc on Debian is full of files that packages wrote and expect to find again. In practice this means persisting most of /etc, at which point you have gone to considerable trouble to wipe /usr, which never changes anyway.

The honest summary: outside a distribution that can regenerate root from a declaration, impermanence is mostly ceremony. If you want the benefit on Debian, you get much closer with a good Ansible playbook and the willingness to actually reinstall the machine from it twice a year. That reinstall is the same test, run less often, at a cost you can afford — and it is the same discipline as rehearsing a bare-metal restore, which is the version of this idea that works on any distribution ever made.

The verdict

Impermanence is worth it on a machine you would rebuild, running services you have described, on a filesystem you already trust. That is a narrow target, and if you hit it the setup costs a weekend and pays back permanently.

It is a bad idea on a workstation where you experiment. The whole point of a workstation is that you install things to see what happens, and a machine that deletes your experiments nightly turns curiosity into paperwork. I tried it, lasted five weeks, and went back to an ordinary root with snapshots.

It is also a bad idea if you are new to NixOS. Add it in month six, once your configuration is genuinely complete and you have stopped fighting the language. Doing it in week one means every ordinary beginner mistake presents as a boot failure, and you will not know which of your two new problems you are looking at.

The people it fits are the ones who already suspect their servers know things that their repositories do not. If that sentence made you uncomfortable, the technique is for you, and the discomfort is the point.