FIDO2 SSH Keys: The Upgrade You Keep Postponing
The -sk key types have been in OpenSSH since 2020 and you still have a file
Contents
OpenSSH 8.2 shipped in February 2020 with support for FIDO2-backed key types. That was nearly six years ago. In that time I have set up dozens of machines, rotated keys after two laptop changes, written a fair amount about SSH hardening, and continued — every single time — to type ssh-keygen -t ed25519 out of muscle memory and move on. The upgrade sat there, one flag away, and I kept not doing it.
The setup takes about four minutes. The obstacle is that a software key already works, and “already works” beats “marginally better” every time you are actually trying to get something else done. So this piece is partly a technical walkthrough and partly an attempt to make the case sharply enough that you do it this evening rather than adding it to a list.
What the -sk suffix actually changes
A normal Ed25519 key is a pair of numbers in a file. The public half goes on servers; the private half sits in ~/.ssh/id_ed25519, encrypted with your passphrase if you were disciplined, in plaintext if you were not. Authentication is: ssh reads the file, decrypts it, signs the server’s challenge, sends the signature.
An ed25519-sk key moves the signing step off your computer. The private key is generated inside a hardware token and cannot be extracted — the device has no API that returns it, and that is the entire point. What lands in ~/.ssh/id_ed25519_sk is a key handle: an opaque blob that says “the credential for this is on a FIDO2 device, here is how to ask for it”. Authentication becomes: ssh passes the challenge to the token, the token signs it internally, ssh gets a signature back.
The consequence is worth being precise about, because people over- and under-sell it. A software key can be copied. Once copied, it works forever, from anywhere, silently. That is the threat: a build script with a postinstall hook, a compromised browser extension with filesystem access, a laptop stolen while suspended. Every one of those walks off with a file. Against an -sk key, all of them fail, because there is no file to walk off with.
What the -sk key does not protect against is malware asking the token to sign while the token is plugged in and you are sitting there tapping it. That attack exists. It is also enormously harder, requires persistence on your machine, and is bounded by your physical presence rather than being valid indefinitely. Converting “permanent silent compromise” into “temporary compromise requiring my finger” is a large win, and it is honest to describe it as exactly that rather than as immunity.
ecdsa-sk or ed25519-sk
Two -sk types exist. ecdsa-sk was there first, because early FIDO2 hardware only did NIST P-256. ed25519-sk needs a token with Ed25519 support in firmware, which every key sold in the last several years has.
Use ed25519-sk. If your token refuses it — some older or budget models — you will get a firm error at keygen time, and ecdsa-sk is a perfectly acceptable fallback. Both are backed by the same hardware guarantee; the curve choice is secondary to where the key lives.
Generating one properly
| |
Line by line, because each flag has a real consequence and the defaults are not what you want.
-O resident stores the credential on the token rather than deriving it from the handle file. This is the difference between “I can walk to any machine, plug in, and recover my keys” and “if I lose that stub file, the key is gone”. Resident credentials consume one of the device’s discoverable-credential slots — typically 25 on current hardware, which is plenty unless you are doing something unusual. Recovery on a new machine:
| |
-O verify-required demands the device PIN alongside the touch. Without it, a stolen token plus a stolen handle file is enough to authenticate, because the touch only proves that somebody is physically present. With it, you have genuine two-factor: something you have, something you know. The token locks after eight consecutive wrong PINs and needs a full reset, which is a feature. Servers must run OpenSSH 8.3+ to enforce the verification flag; 8.2 will accept the key but will not check that the PIN was entered.
-O application=ssh:homelab namespaces the credential. Resident credentials are listed by application string, so giving them meaningful names is the difference between ssh-keygen -K producing something you can identify and producing three files called id_ed25519_sk_rk. The string must start with ssh:.
-O no-touch-required exists and I am mentioning it only to say: think hard. It removes the presence check, which is the property that stops silent automated use. There is one defensible case — an unattended cron job on a machine with a token permanently inserted — and even then, an SSH certificate with a short lifetime is usually the better answer, which is a road I go down in SSH certificates beat authorized keys. The server must also permit it via PubkeyAuthOptions touch-required being absent, so it is not something you can quietly impose.
Living with resident credentials
Once you commit to -O resident, the token becomes a small inventory that needs occasional attention. It is worth knowing how to look inside it, because “how many credentials do I have on here and what are they for” is a question you will ask about eight months in, standing in front of two identical-looking tokens.
| |
ykman fido info reports the remaining discoverable-credential slots, which is why the -O application=ssh:homelab naming discipline pays off — a listing full of ssh: entries with no distinguishing suffix tells you nothing. I use one credential per class of access rather than one per host: ssh:homelab, ssh:vps, ssh:work. Three credentials, each deployed to the machines in that group, each revocable independently by pulling one public key from a handful of authorized_keys files.
That last point is the practical limit of this whole scheme. Revocation is still manual file surgery across every server. With four machines it is a two-minute job. With twenty it is the thing you postpone, get wrong, and later discover you missed three. The scaling answer is certificates, and the honest framing is that hardware keys fix the storage of the credential while leaving the distribution problem entirely untouched.
Deploying it without locking yourself out
The migration is mundane and the order matters:
| |
Keep the original session open during step 2. This is the single most common way people lock themselves out of a remote box, and it costs nothing to avoid.
Your client config wants to be explicit:
| |
IdentitiesOnly yes stops ssh offering every key in your agent before the right one — with several keys loaded you will trip MaxAuthTries and get “Too many authentication failures”, which reads like a hardware fault and is not. ControlPersist means one touch covers a working session rather than one touch per git push.
One more deployment detail that catches people: ssh-copy-id appends the public key, and for -sk types the public key line is long and contains the [email protected] algorithm name at the front. If you are pasting it by hand into an authorized_keys file, a wrapped line is a broken line. Use ssh-copy-id or cat, never a text editor with soft wrapping on.
If you manage authorized_keys with configuration management, the key type change is invisible to it — the file format is identical, only the algorithm string differs. Nothing in your existing tooling needs to know that the key is hardware-backed, which is a pleasant contrast to most security upgrades.
On the server side you can require hardware-backed keys outright:
| |
PubkeyAuthOptions enforces at the server what you asked for at keygen time, so a key generated without the flags gets rejected rather than quietly accepted. Test this from a second terminal before you restart sshd, and check sshd -t first. The same care applies as with SSH hardening generally — every one of these knobs is capable of removing your own access.
Agent forwarding, which surprises people
Forwarded agents work with -sk keys, and the touch happens on your local token when the remote machine requests a signature. That is precisely the behaviour you want: the remote box can ask, and you have to physically agree. If you have ever worried about a compromised jump host silently using your forwarded agent to hop onward, this closes that hole — the attacker needs you to tap at exactly the moment they ask.
It also means a jump chain gets chatty. Every hop is a tap. ProxyJump plus ControlPersist on the intermediate keeps it bearable.
For anything more than a couple of hops, hardware keys and forwarding stop being the right shape and a mesh overlay becomes the better answer — Tailscale SSH does away with the chain entirely.
Troubleshooting
Key enrollment failed: invalid format — OpenSSH older than 8.2, or libfido2 not installed. ssh -V and ldconfig -p | grep fido2 settle it in ten seconds.
device not found with the key visibly plugged in — udev permissions on Linux. Install libfido2 (which ships rules), confirm your user has access to the hidraw device, then log out and back in. Group changes do not apply to a running session, which is why “I added myself to the group and it still fails” is such a common follow-up.
sign_and_send_pubkey: signing failed: agent refused operation — the agent holds the handle but the token is absent or PIN-locked. ssh-add -l shows what the agent thinks it has.
Everything works locally and the server rejects it. Check the server’s OpenSSH version. Anything before 8.2 does not understand [email protected] at all and will log userauth_pubkey: unsupported public key algorithm. Debug from the client with ssh -vvv and read the line where the key is offered.
ssh-keygen -K finds nothing. No resident credential. The key was generated without -O resident and exists only as the handle file. If that file is gone, so is the key — regenerate and redeploy.
PIN prompt never appears, just the touch. You did not pass -O verify-required, or the token has no PIN set. Set one with ykman fido access change-pin or the equivalent for your device, then regenerate the key — the flag is baked in at creation.
The honest case against
Three real costs.
Old servers. Anything before OpenSSH 8.2 cannot use these keys. In practice that means distributions released before roughly 2020. If you have a legacy box you cannot upgrade, it needs a software key and you now maintain two schemes.
Automation. CI runners, backup jobs, Ansible runs across the estate — none of them can tap a token. You will keep software keys for those, which means your authorized_keys files still contain copyable credentials, and the hardware key protects only your interactive access. That is worth having, and it is less than the marketing implies. Automated Linux patching is exactly the workload that stays on a software key.
Friction. Every connection is a tap. Multiplexing absorbs most of it and the rest is a second. But if you ssh a hundred times a day across many hosts, you will feel it, and pretending otherwise is how people end up disabling the touch requirement and losing the whole benefit.
The verdict
Do it for your interactive keys. Do it tonight. It is four minutes of work — one ssh-keygen invocation, one ssh-copy-id, one line in your config — and the return is that the credential guarding your entire homelab stops being a file that any process running as you can read.
Buy two tokens and enrol both while you are in the mood, because a single hardware key is a single point of failure that you have deliberately created. Keep software keys for automation and for that one server you have not upgraded yet, and be clear-eyed that those remain your soft spots.
The reason this upgrade keeps getting postponed is that the existing thing works. That will be equally true next year, and the year after, right up until the evening you find out precisely which of your dependencies decided to have a look in ~/.ssh.
Related Content
