SOPS and age: Secrets in Git Without Regret

SOPS is easy to start and easy to regret. The first hour is delightful: you generate an age key, encrypt a file, commit it, and your repository finally describes a system that can actually boot. The regret arrives in month six, when you have eleven encrypted files, three machines with different keys, one of which you cannot find, a .sops.yaml that somebody — you — wrote from a blog post, and a secret that needs rotating across all of it.

I have written about how SOPS works and I still think it is the correct default for homelab secrets. This is the other article: the decisions that determine whether you are still happy with it next year. Almost all of them are made in the first twenty minutes and almost none of them are obvious at the time.

Creation rules are the architecture

.sops.yaml is where SOPS looks to decide who can decrypt what, based on the file’s path. It feels like configuration. It is actually your access-control model, and changing it later means re-encrypting everything it touches.

The version everyone writes first:

1
2

creation_rules:
  - age: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p

One key, everything, done. It works and it hurts later, because “everything” now includes the secret you would like your CI runner to read and the secret you would very much like it never to see.

The version worth typing instead:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

keys:
  - &admin_laptop age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
  - &admin_yubikey age1yubikey1qwt50d05nh5vutpdzmlg5wn80xq5negm4uj9ghv0snvdd3jjxwxdvpg7owg
  - &backup_offline age1lggyhqrw2nlhcxprm67z43rta597azn8gknawjehu9d9dl0jq3yqqvfafg
  - &host_prometheus age17we5s0hlk5svnzlq8jgvyf5pyawn3xgqspjy4whwnkuq5xpr0dvsu7cnvj
  - &host_media age1kj7ztlf4d3v8lqf2v3lmgv5ke6zhwzn8xg0m2vqhg9zgh5wf9v5s9x4qwl

creation_rules:
  # Infrastructure-wide: every admin key, plus the offline copy.
  - path_regex: secrets/global/.*\.ya?ml$
    key_groups:
      - age: [*admin_laptop, *admin_yubikey, *backup_offline]

  # Per-host: that host, plus admins. The media box cannot read Prometheus's secrets.
  - path_regex: secrets/hosts/prometheus/.*$
    key_groups:
      - age: [*admin_laptop, *admin_yubikey, *backup_offline, *host_prometheus]

  - path_regex: secrets/hosts/media/.*$
    key_groups:
      - age: [*admin_laptop, *admin_yubikey, *backup_offline, *host_media]

Three decisions are embedded there and each one saves you a specific future afternoon.

A per-host key. Each machine gets its own age identity and can only decrypt its own secrets. The alternative — one key on every host — means a compromise of the noisiest, least-patched box in your lab hands over the credentials for all the others. This is the single highest-value line in the file.

An offline recipient. backup_offline is a key on an encrypted USB stick in a drawer. It never touches a machine. It exists so that a laptop failure plus a forgotten passphrase is an inconvenience rather than the permanent loss of every secret you own. Add it on day one, because adding it later means re-encrypting every file.

YAML anchors. &name and *name mean a key appears once and is referenced everywhere. When you rotate a key, you edit one line. Without anchors, you edit thirty and miss one, and the file you missed is the one you need in the outage.

Encrypt values, and be careful what counts as a value

By default SOPS encrypts every value and leaves keys readable. That is usually right, and occasionally too aggressive — a compose file where image: and ports: are ciphertext is unreadable and undiffable for no benefit.

encrypted_regex fixes it:

1
2
3
4
5

creation_rules:
  - path_regex: stacks/.*/secrets\.ya?ml$
    encrypted_regex: '^(password|passwd|secret|token|key|api_?key|.*_PASSWORD|.*_TOKEN)$'
    key_groups:
      - age: [*admin_laptop, *backup_offline]

Now the structure is legible in a diff and only the sensitive values are sealed. The risk is obvious once stated: a secret named something your regex does not match gets committed in plaintext, and it looks fine, because SOPS does not warn you that it encrypted nothing. DATABASE_URL containing postgres://user:hunter2@db/app is the classic: it looks like a URL, your regex ignores it, and there is a password sitting in the middle of it.

Use unencrypted_regex instead if you can. Listing what stays readable fails safe; listing what gets encrypted fails silent.

Either way, put a scanner in front of git. A pre-commit hook running gitleaks catches the file you forgot to encrypt at all, which is the mistake this whole system exists to prevent and which SOPS itself does nothing about:

1
2
3
4
5
6

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.18.4
    hooks:
      - id: gitleaks

Rotation, and the thing rotation does not do

Two different operations get called rotation and conflating them is how people end up believing they are safe.

Rotating the data key re-encrypts the file with a fresh symmetric key. Cheap, fast, and worth doing when the recipient list changes:

1
2

sops updatekeys secrets/global/api.yaml    # apply .sops.yaml changes
sops -r -i secrets/global/api.yaml         # fresh data key, same recipients

Rotating the secret itself means the password changes at the service. That is real work and SOPS cannot help.

The trap: git remembers everything. If a laptop key is compromised, updatekeys removes that recipient from the current file — and every historical commit still contains the old ciphertext, encrypted to the old key, which the attacker holds. They cannot read the new file. They can git log -p and read yesterday’s, which contains the same password, because you rotated the key and left the value alone.

So: a compromised key means every secret it could read is compromised, forever, and must be changed at the source. Removing the recipient prevents future access and does nothing about the past. Plan for this by keeping the recipient list small and the per-host scoping tight, so the blast radius of one lost key is one host’s secrets.

Which is also why git history scrubbing is a poor plan. git filter-repo rewrites your history, and every clone, every fork, every backup of the repo still has the old objects. Change the password. It is the only thing that works.

Machines that need to decrypt

The homelab version of the CI problem is simple: a host boots, needs its secrets, and nobody is there to type a passphrase. The key must be on disk. Make peace with that and constrain it properly:

1
2

install -d -m 0700 -o root -g root /var/lib/sops
install -m 0400 -o root -g root age.key /var/lib/sops/keys.txt

Then a systemd unit that decrypts to a tmpfs at start and never writes plaintext to persistent storage:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

[Unit]
Description=Render secrets for myapp
Before=myapp.service

[Service]
Type=oneshot
RemainAfterExit=yes
Environment=SOPS_AGE_KEY_FILE=/var/lib/sops/keys.txt
RuntimeDirectory=myapp
RuntimeDirectoryMode=0750
ExecStart=/usr/local/bin/sops -d --output /run/myapp/env \
          /etc/homelab/secrets/hosts/media/myapp.yaml
ExecStartPost=/bin/chown root:myapp /run/myapp/env

[Install]
WantedBy=multi-user.target

RuntimeDirectory is the useful bit: systemd creates /run/myapp, which is tmpfs, and removes it when the service stops. The plaintext exists in RAM, for the lifetime of the service, readable by one group. A disk pulled from that machine yields ciphertext and a key file that only decrypts that host’s own secrets.

For real CI, put the key in the runner’s secret store and let it decrypt at job time:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# .gitea/workflows/deploy.yaml
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Decrypt and deploy
        env:
          SOPS_AGE_KEY: ${{ secrets.SOPS_AGE_KEY_CI }}
        run: |
          sops -d secrets/hosts/media/myapp.yaml > /run/env
          ./deploy.sh

Give CI its own recipient key, scoped by path_regex to exactly what it deploys. A runner that executes code from pull requests and holds a key for secrets/global/ is a hole with a build badge on it. On a cluster, the Kubernetes secrets question has the same shape and the same answer: the operator gets a key, the key is scoped, nothing else changes.

Adding a host without breaking the others

The operation you will perform most often is bringing a new machine into the scheme, and it is worth having the sequence written down because doing it out of order produces a file nobody can read.

On the new host, generate an identity and print only the public half:

1
2
3
4

age-keygen -o /var/lib/sops/keys.txt
chmod 0400 /var/lib/sops/keys.txt
age-keygen -y /var/lib/sops/keys.txt
# age1n3w4h0stpublickeygoeshere...

The public key is the recipient; it goes in .sops.yaml as a new anchor and gets referenced from exactly the creation rules that host should be able to read. Then — and this is the step people miss — the existing files have to be told:

1
2
3

# .sops.yaml is a rule for NEW files. Existing files carry their own recipient list.
sops updatekeys secrets/hosts/newbox/app.yaml
sops updatekeys secrets/global/api.yaml

updatekeys reads .sops.yaml, works out who should be able to decrypt each file, and rewrites the metadata block accordingly. It prompts for each change, which is tedious and correct — the prompt is the last thing standing between you and adding a recipient to a file you did not intend. Read what it says before you type y.

The commit that follows is worth reviewing carefully, because it is the only commit in your repository where a diff of pure metadata is security-relevant. You are looking at the recipient list changing. If a key appears on a file it has no business reading, this is where you catch it, and you catch it by reading rather than by tooling.

Removing a host is the same sequence in reverse, with the caveat from the rotation section attached: taking a recipient off a file does nothing about the ciphertext already in your git history. If the machine was decommissioned because it was compromised, the secrets it could read need changing at source. If it was decommissioned because it was old, updatekeys is enough.

Keys on hardware

An age key in a file is a file. age-plugin-yubikey moves the private key onto a YubiKey where it is generated on the token and cannot be extracted:

1
2

age-plugin-yubikey --generate --touch-policy cached
sops -d secrets/global/api.yaml     # taps the key, once per 15s

This is the right home for your admin key. It cannot be copied off a stolen laptop and every decryption is a physical act. The plugin needs to be installed wherever you decrypt, which rules it out for unattended hosts — those keep file-based keys, scoped narrowly, and that asymmetry is correct. The human key is hardware-backed; the machine keys are files that can only read one machine’s secrets.

Troubleshooting

no matching creation rules found. Your path_regex does not match. It is matched against the path as you typed it, so sops ./secrets/foo.yaml and sops secrets/foo.yaml behave differently against a regex anchored with ^. Do not anchor the front.

Error getting data key: 0 successful groups required, got 0. No key you hold is a recipient. Check sops -d --verbose and compare the recipient fingerprints in the file’s metadata against age-keygen -y /var/lib/sops/keys.txt. Ninety per cent of the time you edited .sops.yaml and forgot sops updatekeys, which is a manual step SOPS does not do for you.

Merge conflicts in encrypted files. Ugly and inevitable, since every value is a different ciphertext. Install the merge driver:

1
2

git config --global merge.sops.driver "sops merge %O %A %B"
echo "*.enc.yaml merge=sops diff=sops" >> .gitattributes

Diffs are unreadable. Add sops -d as a textconv and git diff shows plaintext locally while the repository holds ciphertext. Set it in .gitattributes alongside the merge driver.

A file silently contains plaintext. The sops: metadata block is missing entirely — someone edited it with a normal editor and saved. sops always adds that block; grep -L "^sops:" secrets/**/*.yaml finds the files that lost it, and it belongs in CI.

Rotation seems to work, the old key still decrypts. You rotated the data key and did not run updatekeys first, so the recipient list is unchanged. Order matters: updatekeys, then -r -i.

What SOPS is honestly bad at

Three things, and I would rather list them than have you find them.

It has no concept of who read what. The secret is in a file; the file is in a repo; the repo is cloned onto machines. Once a value is decrypted on a host, there is no record and no revocation. Compare that to a service handing out an hour-long lease and logging the request. For a single operator this is a shrug. For anything with a second person, it is the reason the file model eventually stops being enough.

Rotation is manual and therefore does not happen. Nothing prompts you. There is no expiry, no lease, no alert. A password encrypted in 2023 is still there in 2026, unchanged, because the system’s whole design goal is that it keeps working without attention. That property is a feature for availability and a liability for hygiene, and the only fix is a calendar entry you will resent.

Plaintext leaks sideways. sops -d file.yaml writes to stdout, which goes into your scrollback, which goes into your terminal’s log, and sops exec-env puts the values in a process environment where /proc/*/environ and any crash dump can see them. sops -d > /run/thing on a tmpfs is the disciplined pattern and it is one character away from > ./thing on a disk that gets backed up. The tool will not stop you.

None of these are reasons to avoid it. They are reasons to know what you have bought: a system that trades auditability and rotation pressure for the enormous benefit of having no dependencies at all. When your entire lab is down, the secrets are still readable from a laptop with a key on it, and that is worth a great deal at 2am.

The verdict

SOPS with age is the right default for homelab secrets and I have no plans to move. It costs nothing to run, it survives your infrastructure being down, the secrets restore with the repo, and the whole system is two static binaries.

Everything that goes wrong with it is decided early. Per-host keys, an offline recipient, YAML anchors, unencrypted_regex over encrypted_regex, a gitleaks hook, and the understanding that a leaked key means changing the passwords rather than editing the file. Twenty minutes at the start, and the difference between a tool you like in a year and a tool you are quietly afraid of.

The place it genuinely runs out is dynamic credentials — hour-long database leases that nothing file-based can produce. That is when a secrets service starts earning its containers. Until then, this is a solved problem and the solution fits in a repo.