SSH Is the Quiet Hero of the Internet

Every server I have ever administered, from a decade-old ThinkCentre in a cupboard to production boxes at a proper job, has had exactly one door that matters more than any other: the SSH daemon. Everything else — the web app, the database, the monitoring stack — can be behind a reverse proxy, a firewall, an authentication layer of its own. SSH is the thing that lets you actually get in and fix what broke, and it has done that job, largely unchanged in its fundamentals, since Tatu Ylönen wrote the first version in 1995 after a password-sniffing attack hit his university network. Thirty years is an eternity in software. Almost nothing else you rely on daily has gone that long without a rewrite that broke compatibility with itself.

The problem SSH actually solved

Before SSH, remote administration meant telnet, rsh and rlogin, all of which sent credentials and session data in plaintext over the network. Anyone with a packet sniffer on the same segment — a shared university network, a compromised switch, an ISP with the wrong incentives — could read your root password off the wire the moment you typed it. The attack that motivated Ylönen was exactly this: a sniffer on the university’s network scooped up a large number of credentials in one sweep, and the response was to design a replacement protocol from first principles rather than bolt encryption onto the existing ones.

What SSH actually is, underneath the familiar ssh user@host invocation, is a layered protocol: a transport layer that negotiates encryption and verifies the server’s identity, a user authentication layer that proves who you are, and a connection layer that multiplexes the actual work — a shell, a port forward, a file transfer — over the single encrypted channel the first two layers established. That layering is why SSH has been able to swap out individual pieces (weak ciphers, deprecated key exchange algorithms) over three decades without touching the parts built on top, which is precisely the kind of incremental evolvability a monolithic design does not get for free.

Why the key exchange matters more than the encryption

People tend to focus on “SSH encrypts your session,” which is true but not the interesting part — plenty of protocols encrypt traffic. The harder problem SSH solves is agreeing on a shared secret over a channel an attacker might already be reading, without ever transmitting that secret. Diffie-Hellman key exchange (or its elliptic-curve variants in modern SSH) lets client and server each compute the same shared value from a combination of public and private numbers, where an eavesdropper who sees every public number exchanged still cannot derive the shared secret without solving a computationally infeasible discrete-logarithm problem. Everything downstream — the symmetric cipher encrypting your actual session — depends on that shared secret existing without ever having crossed the wire in a form an attacker could read.

The other half of the transport layer, host key verification, is the part people actively work around and shouldn’t. The first time you connect to a new host, SSH shows you a fingerprint and asks you to confirm it, then stores it in known_hosts. That prompt exists to defend against a machine-in-the-middle presenting itself as your server; skip verifying it once over a channel you trust (in person, or a value you already have from a provisioning script) and every future connection silently trusts whoever answered that IP address first.

Getting past passwords entirely

Password authentication over SSH is encrypted, which makes it categorically safer than telnet, but it is still the weakest authentication method SSH supports, vulnerable to brute-forcing against any host with a public IP and port 22 open. Public-key authentication removes the password from the equation entirely: your private key never leaves your machine, and the server only ever needs the corresponding public key to verify a signed challenge.

1
2

ssh-keygen -t ed25519 -C "homelab-admin"
ssh-copy-id -i ~/.ssh/id_ed25519.pub [email protected]

Ed25519 is the key type worth defaulting to now — smaller keys, faster signing, and no dependence on the same discrete-logarithm assumptions that make certain older RSA configurations a slower and larger alternative for no real security benefit at current key sizes. I have written in more detail about exactly how the asymmetric maths behind this works if you want the full mechanism.

Hardware-backed keys push this further. A YubiKey holding your SSH key means the private key material never exists on disk at all — signing happens on the token itself, so a compromised laptop still cannot exfiltrate a usable copy of your credential, only misuse it while the token is physically inserted.

Where SSH’s original design starts to strain

The bastion-host pattern that grew up around SSH — one hardened jump box with a public IP, everything else reachable only through it — is a workaround for IPv4 scarcity and firewall topology, not something SSH itself asked for. It works, but it concentrates risk onto a single machine and adds an operational hop every session has to survive. Mesh overlay networks solve the underlying routing problem SSH never tried to solve, which is why Tailscale SSH can retire the bastion pattern entirely for a lot of homelabs — every device gets a routable, authenticated address, so SSH’s own authentication becomes almost redundant at the network layer, though I would still keep key-based auth as a second layer rather than relying on network trust alone.

authorized_keys files sprawling across every host is the other place the original design shows its age at any scale beyond a handful of machines. Adding a new administrator means editing that file on every box individually, and revoking access means the same in reverse, reliably, everywhere, under pressure, which is exactly the condition under which someone forgets a host. SSH certificates solve this the way TLS solves the equivalent problem for websites: a certificate authority signs short-lived certificates for users and hosts, and every server trusts the CA’s public key once instead of maintaining an ever-growing list of individually authorized keys. I go through the full setup in a separate piece on SSH certificates, and it is the single change that has made the most difference to how I manage access once the number of machines passed “small enough to remember.”

The connection layer doing more than a shell

The part of SSH people use daily without thinking of it as a distinct feature is the connection layer’s ability to multiplex arbitrary channels over one encrypted transport. A single SSH session already carries your shell, but that same tunnel can carry a local port forward, a remote port forward, or a full SOCKS proxy, all authenticated once and encrypted together.

1
2
3

ssh -L 8080:localhost:80 [email protected]
ssh -D 1080 [email protected]
ssh -J bastion.example.com [email protected]

The first line reaches a web service on the far side that has no business being exposed to the internet directly, tunnelling it to your local machine as if it were running on localhost. The second turns your SSH client into a SOCKS proxy for an entire browser session, useful for reaching anything on a private network without standing up a full VPN. The third, ProxyJump, chains through an intermediate host in a single command rather than the older two-hop ssh -t bastion ssh internal-host — it configures the same jump-host pattern SSH has supported for years, just with cleaner syntax and, critically, without requiring your private key to be forwarded to the intermediate host at all, since the jump host only relays an encrypted stream it cannot itself decrypt.

Multiplexing connections is the other underused feature, and it fixes a genuinely annoying default: every new SSH connection to the same host renegotiates the transport and authentication from scratch, which is slow over a slower or higher-latency link. Setting ControlMaster auto and a ControlPath in your SSH config lets the second and subsequent connections to the same host reuse an already-authenticated transport, turning what would be a multi-second reconnect into something closer to instant.

1
2
3
4

Host *.mylab.local
  ControlMaster auto
  ControlPath ~/.ssh/sockets/%r@%h-%p
  ControlPersist 10m

ControlPersist keeps that shared connection alive for a period after the last session closes, so a burst of quick commands to the same host — a script running several ssh host command invocations in a loop, say — pays the authentication cost exactly once rather than once per invocation.

Troubleshooting the errors everyone hits

“Permission denied (publickey)” almost always means one of three things: the public key on the server does not match the private key you are offering (check both with ssh-keygen -lf, which prints a fingerprint you can compare), the authorized_keys file or .ssh directory has permissions loose enough that sshd refuses to trust it (it must be 700 for the directory and 600 for the file, owned by the user, or sshd silently ignores it rather than erroring clearly), or you are offering the wrong key because ssh-agent is holding several and trying them in an order you don’t control — ssh -v will show you exactly which key was offered and rejected, which is the first thing to check before assuming the server-side config is broken.

“Host key verification failed” after a server rebuild is SSH correctly doing its job rather than a bug: the host’s key genuinely changed, and the correct fix is confirming that the change is expected (you reinstalled the OS, you didn’t get MITM’d) before removing the stale line from known_hosts with ssh-keygen -R hostname, never by blanket-disabling StrictHostKeyChecking, which removes the exact protection the prompt exists to provide.

Agent forwarding failures or, worse, agent forwarding working when you didn’t intend to enable it, deserve a specific mention: forwarding your agent to a host you SSH through means anything with root on that intermediate host can request signatures from your key for as long as your session is open, effectively borrowing your identity. Use ForwardAgent no as a global default and enable it explicitly, per-host, only where you have actually thought about the trust boundary.

A connection that hangs indefinitely rather than failing cleanly, particularly over Wi-Fi or a flaky mobile link, is usually a dead TCP connection that neither end has noticed yet, because nothing sent a packet recently enough to trigger the OS’s own timeout. ServerAliveInterval 15 and ServerAliveCountMax 3 in your client config send a keepalive every fifteen seconds and give up after three misses, turning a session that would otherwise sit frozen for minutes into one that fails within forty-five seconds and lets you reconnect. The equivalent setting on the server side, ClientAliveInterval, is worth setting too, since it is what actually frees up the server’s own resources for a client that vanished without a clean disconnect.

Why it never needed a rewrite

Part of what makes SSH worth admiring rather than merely tolerating is how rarely its core abstractions have needed to change. The transport, authentication and connection layering from the original design is still the layering in OpenSSH today; what changed underneath it — ciphers, key exchange algorithms, the move from SSH-1 to SSH-2 after cryptographic weaknesses were found in the former — happened without the layer above needing to know or care. Compare that to how many times the web’s own authentication story has been reinvented in the same window, from cookies to OAuth to whatever the current session-token fashion is, and SSH’s stability starts to look less like luck and more like the payoff of having got the layering right the first time.

Is it worth keeping around

Completely. Nothing has come along that solves SSH’s actual problem — authenticated, encrypted remote access to a shell — more elegantly than SSH already does, and the layered design means the parts that needed modernising (ciphers, key exchange algorithms, certificate-based auth) have been added without breaking the parts that didn’t. Mesh VPNs and zero-trust proxies are genuinely useful additions to the picture, but they sit alongside SSH rather than replacing the need for it. Thirty years on, it remains the thing you reach for when everything else on the box has stopped working and you just need a shell.