SSH Certificates Beat Authorized Keys

The authorized_keys file is one of the great successes of pragmatic engineering. It is a text file, one key per line, no daemon, no database, no service to keep running. It has worked essentially unchanged for a quarter of a century, and for one person with one server it remains the correct answer.

It falls apart somewhere around the fifth machine, and it falls apart in a specific way that is worth naming precisely, because the failure is not dramatic. Nothing breaks. You just develop a slow, permanent, low-grade administrative burden that you stop noticing until the day it matters. Someone gets a new laptop and you edit eleven files. You spin up a VM and copy keys across. Your partner needs access to the media box and now their key is on three machines — or is it four? You have a distributed database with no schema, no index, no transactions, and manual replication performed by a tired human over SSH. That is what authorized_keys becomes at scale, and it is a filing system pretending to be a security control.

The specific thing certificates fix

An SSH certificate is a public key with a signed statement attached: who this key belongs to, which usernames it may assume, when it expires, and what it is allowed to do. The signature comes from a certificate authority — in the smallest useful case, an Ed25519 key on an encrypted USB stick in a drawer.

Servers are then configured to trust the CA’s public key rather than any individual user key. A machine that trusts your CA will accept any certificate the CA signed, without ever having heard of the person holding it. Add a new server: it needs the CA public key and nothing else, forever. Add a new user: sign them a certificate, touch zero servers. Remove a user: their certificate expires on its own.

That last property is the one that converts sceptics. With authorized_keys, revocation is a manual sweep across every machine, and a machine you forget is a machine that stays open. With certificates, you issue short-lived ones — I use sixteen hours, which comfortably covers a working day and expires before I have finished breakfast the next morning — and revocation becomes the absence of a renewal. Someone leaves, you stop signing for them, and by tomorrow they are out of everything. No sweep. No forgotten box.

The second thing certificates fix is one people rarely think about: host verification. Every time you connect to a new machine you get the “authenticity of host can’t be established” prompt, and every single one of us types yes without reading the fingerprint. That prompt is your only defence against a man-in-the-middle, and it has been trained out of us by sheer repetition. Host certificates remove the prompt by making it answerable: your client trusts the CA, the server presents a certificate signed by that CA, and the connection proceeds silently and correctly. The prompt disappears because the question has a real answer.

Building the CA

Two keys, because user and host CAs should be separate. If they are the same key, a compromised host key can sign user certificates for itself.

1
2
3

mkdir -p ~/ssh-ca && cd ~/ssh-ca
ssh-keygen -t ed25519 -f user_ca -C "user CA - mylab" 
ssh-keygen -t ed25519 -f host_ca -C "host CA - mylab"

Use a strong passphrase on both. These two files are now the crown jewels of your estate: anyone holding user_ca can mint themselves root on every machine that trusts it. They belong on encrypted removable media that lives in a drawer, plugged in only when you sign something. A CA private key sitting on your always-on laptop is a CA private key on the machine most likely to be compromised. If you want the key itself to be unstealable, generate it as an ed25519-sk key — FIDO2-backed keys work perfectly well as CA keys and mean the signing operation requires a physical tap.

Signing a user certificate

1
2
3
4
5
6

ssh-keygen -s user_ca \
  -I "smarc@thinkpad" \
  -n admin,deploy \
  -V +16h \
  -z 1001 \
  ~/.ssh/id_ed25519_sk.pub

This writes id_ed25519_sk-cert.pub next to the public key, and ssh picks it up automatically as long as the matching private key is available.

-I is the identity — a free-text label that lands in the server’s auth log on every connection. Make it descriptive; this is your audit trail. “smarc@thinkpad” tells you which human on which device; “key1” tells you nothing at 2am.

-n admin,deploy lists the principals: the usernames this certificate may log in as. This is the heart of the model. A certificate with -n admin can be admin@ on any machine trusting the CA. A certificate with -n backup can only be backup@. You express your entire access policy as principal strings, and you express it once at signing time rather than per-machine.

-V +16h sets validity. Short is the point. If you find yourself reaching for +52w you have rebuilt authorized_keys with extra ceremony and none of the benefit.

-z 1001 is the serial number. Keep a counter. You need it for revocation, and discovering that all your certificates have serial 0 at the moment you need to revoke one is a bad discovery.

Inspect what you made:

1

ssh-keygen -L -f ~/.ssh/id_ed25519_sk-cert.pub

It prints the principals, validity window, serial and the critical options. Read it once so the format stops being mysterious.

Configuring the servers

On every host, install the user CA’s public key and point sshd at it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# /etc/ssh/sshd_config.d/20-ca.conf

# Trust certificates signed by our user CA
TrustedUserCAKeys /etc/ssh/user_ca.pub

# Present our own host certificate to clients
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub

# Revocation list — checked on every auth
RevokedKeys /etc/ssh/revoked_keys

Sign the host key from a machine holding the host CA:

1
2
3
4

scp [email protected]:/etc/ssh/ssh_host_ed25519_key.pub .
ssh-keygen -s host_ca -I "web01" -h -n web01.mylab.local,192.168.1.20 -V +52w \
  ssh_host_ed25519_key.pub
scp ssh_host_ed25519_key-cert.pub [email protected]:/etc/ssh/

-h marks it as a host certificate; omitting it produces a user certificate and a confusing afternoon. The -n principals here are the names clients will type — if you connect by IP sometimes and hostname other times, both must be listed, or you get the host-key prompt back for the name you forgot. Host certificates can be long-lived; a year is reasonable, since rotating them means touching every server anyway.

Then, client side:

1
2

# ~/.ssh/known_hosts — one line, replaces every host entry
@cert-authority *.mylab.local,192.168.1.* ssh-ed25519 AAAAC3Nz... host CA - mylab

That single line replaces every accumulated known_hosts entry for your estate. Rebuild a server, and the prompt does not come back — you re-sign its new host key and clients accept it silently. This is genuinely delightful the first time it happens.

Roll it out with TrustedUserCAKeys alongside the existing authorized_keys files. Both work simultaneously. Prove certificates work on every machine, leave it a fortnight, then remove the old files. Any migration that requires a flag day across your whole estate is a migration you will abandon halfway. Whatever you use for configuration management — the same tooling that handles unattended security updates will do — the change is two files and a service reload per host.

Critical options: the part that makes certificates more than key management

Everything so far treats a certificate as a nicer way to distribute a key. The extensions are where it becomes a policy language.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

ssh-keygen -s user_ca \
  -I "backup-runner" \
  -n backup \
  -V +7d \
  -z 1002 \
  -O source-address=192.168.1.50/32 \
  -O force-command="/usr/local/bin/restic-wrapper" \
  -O no-agent-forwarding \
  -O no-port-forwarding \
  -O no-pty \
  backup_key.pub

That certificate can only be used from one address, can only run one command, cannot open a shell, and cannot forward anything. It expires in a week. Compare that to the equivalent in authorized_keys — the same options exist as a prefix on the key line, and they must be maintained identically on every host, by hand, forever. Here the policy travels with the credential and is signed, so a server cannot be tricked into applying a weaker version of it.

source-address is the underrated one. Certificates for automation get pinned to the host that runs the automation, which means a leaked automation key is useless from anywhere else. force-command turns a general-purpose credential into a single-purpose one — the wrapper script receives the original command in $SSH_ORIGINAL_COMMAND if it needs to inspect it, and can simply ignore it if it does not.

The default extensions on a user certificate are permissive: permit-pty, permit-agent-forwarding, permit-port-forwarding, permit-user-rc, permit-X11-forwarding. Signing with -O clear strips all of them, after which you add back only what is needed. For anything non-interactive, -O clear -O force-command=... is the shape to reach for.

Revocation, which you must set up before you need it

Certificates expire, which handles the ordinary case. For “this laptop was stolen and I need it dead in the next five minutes”, you need a Key Revocation List.

1
2
3
4
5

# Create the KRL, revoking serials 1001 and 1004
ssh-keygen -k -f revoked_keys -s user_ca -z 1 <<< $'serial: 1001\nserial: 1004'

# Verify a certificate against it
ssh-keygen -Q -f revoked_keys ~/.ssh/id_ed25519_sk-cert.pub

Distribute revoked_keys to /etc/ssh/revoked_keys on every host. A KRL is compact and appending to it is cheap, so push it on a schedule — I have a job that syncs it hourly, which means the worst case for a revocation is sixty minutes and the median is thirty.

Note the tension: if your certificates live sixteen hours, a KRL is mostly belt-and-braces. If you have talked yourself into month-long certificates, the KRL is the only revocation you have, and it is now a critical distribution path you must monitor. Short certificates make the whole system simpler. That is the trade, and it is why I keep insisting on it.

Troubleshooting

The certificate is ignored entirely and it falls back to the plain key. The -cert.pub file must sit beside the private key with exactly that name. ssh -v will say Offering public key without mentioning a certificate. This is the single most common problem and the fix is a rename.

key type [email protected] not in PubkeyAcceptedAlgorithms — you have hardened PubkeyAcceptedAlgorithms on the server and forgotten the certificate variants. Add [email protected] to the list.

Certificate invalid: name is not a listed principal — you are logging in as a username that the certificate’s -n list does not include. Read the certificate with ssh-keygen -L and compare with the user you asked for.

Certificate invalid: expired — your sixteen-hour certificate did what it said. This is the system working; get a fresh one.

Clock skew. Certificates are time-bound and a server whose clock has drifted twenty minutes will reject perfectly valid ones with the same “expired” or “not yet valid” error. Run NTP everywhere. This one costs people entire evenings because the error message points at the certificate rather than at the actual culprit.

Host certificate ignored, prompt still appears. The @cert-authority pattern in known_hosts does not match the name you typed, or a stale plain host entry for that name takes precedence. Delete the old entry.

The honest case against

You now run a PKI. A small one, but the CA key is a genuine single point of catastrophe. Lose it and you re-provision every server. Leak it and someone owns everything. authorized_keys has no such object, and that is not nothing.

Signing needs to be routine. Sixteen-hour certificates mean a daily signing ritual. If that means fetching a USB stick from a drawer each morning, you will drift towards longer validity, and then you have accepted the complexity while discarding the benefit. Something has to make signing trivial — a script, a shell function, or eventually a proper CA service with its own authentication in front of it.

Under five machines it is overkill. I mean that sincerely. Two servers and a laptop? Keep the file. The complexity is only worth it when the filing problem is genuinely costing you.

Emergency access needs a plan. When the CA is unreachable and a machine is misbehaving, you need a way in. Keep one break-glass key in authorized_keys on each host, generated offline, stored on paper or in a safe, and used approximately never. This pairs with having an out-of-band path at all — the two failure modes rhyme, and the answer to both is a route that does not depend on the thing that broke.

The verdict

Past roughly five machines, certificates are straightforwardly better and the migration is an afternoon. The revocation story alone justifies it: knowing that a stolen laptop is locked out by tomorrow morning without you touching a single server is worth real money in reduced anxiety.

Under five machines, authorized_keys is fine and I would not think less of you for it. The tell that you have crossed the line is simple: the next time you find yourself SSHing into machines one by one to paste a public key, count them. If the count is over five, you have your answer, and you have had it for a while.