mTLS Between Your Own Services, Demystified

Most of my services trusted each other for years for one embarrassing reason: they were on the same network. The metrics collector scraped an exporter because it could reach the port. The backup job pulled a database dump because nothing stopped it. The reverse proxy forwarded to an internal app over plain HTTP because “it’s only the LAN”. Every one of those links was authenticated by geography and nothing else, and geography is a terrible credential.

That works right up until it doesn’t. A single compromised container, a laptop that joined the wrong VLAN, a supply-chain nasty in some npm dependency — and suddenly the attacker is also on the same network, enjoying the same unearned trust every one of my services extended to any peer that could open a socket. Mutual TLS closes that gap. It makes every service prove who it is with a certificate before another service will talk to it, in both directions. This is a walk through what mTLS actually is, why the internal case matters more than people expect, and how to stand it up in a homelab without drowning in OpenSSL flags.

Why “TLS” and “mutual TLS” are different animals

Ordinary TLS — the padlock in your browser — authenticates one side. When you connect to https://shop.example.com, the server presents a certificate signed by a public certificate authority your browser already trusts, and your browser checks that the name matches and the signature chains up to a root it knows. You, the client, prove nothing. That is fine for the public web, where the server has no idea who you are until you log in.

Inside your own infrastructure the assumption flips. The database should care who is connecting. The metrics endpoint should refuse a scraper it doesn’t recognise. One-way TLS gives you an encrypted pipe to a server whose identity is verified, while the server accepts whatever turns up at the other end of that pipe. Mutual TLS adds the second handshake: the client also presents a certificate, the server validates it against a CA it trusts, and only then does the conversation continue. Both ends hold a private key, both present a certificate, both verify the other. Identity becomes something you carry, rather than something your IP address implies.

The threat this defends against is lateral movement. In a realistic homelab threat model the interesting question is rarely “can someone reach the front door” — it’s “what happens after they get a foothold on one box”. Flat internal trust means one foothold is a skeleton key. With mTLS, a compromised container can reach a peer’s port all it likes, but without a valid client certificate the handshake dies before any application data is exchanged. The attacker has to steal a key, not merely stand in the right place.

The one idea underneath all of it: your own CA

A certificate is a public key with a name attached, wrapped in a signature from someone the verifier trusts. For the public web that signer is Let’s Encrypt or DigiCert. For your internal services, you become the signer. You run a small private certificate authority, and you configure every service to trust exactly one root: yours. Public CAs are irrelevant here, and dragging them in only creates rate-limit headaches and leaks your internal names into public certificate-transparency logs.

A private CA is two files: a private key you guard with your life, and a self-signed root certificate you distribute everywhere. Every server and client certificate you issue is signed by that key, and because every service trusts the root, they transitively trust each other’s certificates. Guard the CA key, and the whole scheme holds; leak it, and anyone can mint a certificate your services will accept, so it is the one secret that genuinely matters.

You can build this with raw OpenSSL, and it is worth doing once so the moving parts stop being magic. Here is a minimal CA plus a server certificate for an internal service on mylab.local:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# 1. Create the CA private key and a long-lived self-signed root
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 \
  -subj "/CN=mylab.local Internal Root" -out ca.crt

# 2. Make a key + CSR for the service "vault.mylab.local"
openssl genrsa -out vault.key 2048
openssl req -new -key vault.key -subj "/CN=vault.mylab.local" -out vault.csr

# 3. Sign it, embedding the Subject Alternative Name (SAN) the peer will check
openssl x509 -req -in vault.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -days 365 -sha256 -out vault.crt \
  -extfile <(printf "subjectAltName=DNS:vault.mylab.local")

The Subject Alternative Name is the part everyone forgets, and its absence is the single most common reason an otherwise-correct setup refuses to connect. Modern TLS stacks ignore the old Common Name for hostname matching and check the SAN. No SAN, no match, no handshake — regardless of how perfect the rest of the certificate looks.

Issuing a client certificate

The client certificate is made the same way, with a name that identifies the caller rather than a hostname. If your metrics collector scrapes an exporter, the collector gets a certificate whose Common Name is something like prometheus-scraper, and the exporter is configured to accept only certificates signed by your CA. Reuse the CA from above:

1
2
3
4

openssl genrsa -out scraper.key 2048
openssl req -new -key scraper.key -subj "/CN=prometheus-scraper" -out scraper.csr
openssl x509 -req -in scraper.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -days 90 -sha256 -out scraper.crt

Note the shorter lifetime. Client certificates for automated callers should be short-lived precisely because renewing them can be automated. A 90-day cert that rotates itself is far safer than a 10-year cert you will never think about again, because a leaked short-lived credential expires on its own before you have finished writing the incident notes.

Doing this by hand teaches you the shapes, and then you should stop doing it by hand. For anything beyond a couple of services, run smallstep’s step-ca, which gives you an ACME endpoint and a step CLI so internal certificates renew the same automated way public ones do. The manual OpenSSL dance is the thing you graduate away from once you understand it.

Wiring it into a reverse proxy

Most homelabs already funnel traffic through a reverse proxy, which makes it the natural place to enforce client-certificate checks without touching every application. Nginx needs three directives on top of a normal TLS server block:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

server {
    listen 443 ssl;
    server_name app.mylab.local;

    # Server side of the handshake — proves the proxy's identity
    ssl_certificate     /etc/nginx/certs/app.crt;
    ssl_certificate_key /etc/nginx/certs/app.key;

    # Client side — demand and verify a cert signed by our CA
    ssl_client_certificate /etc/nginx/certs/ca.crt;
    ssl_verify_client on;
    ssl_verify_depth 1;

    location / {
        # Hand the verified client identity to the backend app
        proxy_set_header X-Client-CN $ssl_client_s_dn_cn;
        proxy_pass http://127.0.0.1:8080;
    }
}

ssl_verify_client on is the switch. With it, nginx completes the handshake only when the client presents a certificate that chains to ssl_client_certificate. The ssl_verify_depth 1 line says the client cert must be signed directly by your root with no intermediates, which matches the simple CA above; raise it if you add an intermediate CA later. Passing $ssl_client_s_dn_cn upstream lets the application know which verified client it is talking to, so you can layer authorisation on top of authentication — the exporter is allowed here, the scraper is not, even though both hold valid certificates.

Testing it from the command line is where the concept finally clicks:

1
2
3
4
5
6
7

# Fails: no client certificate offered
curl https://app.mylab.local/ --cacert ca.crt
# -> 400 No required SSL certificate was sent

# Succeeds: presents key + cert the server trusts
curl https://app.mylab.local/ --cacert ca.crt \
  --cert scraper.crt --key scraper.key

The first call proves the door is actually locked. If it succeeds, ssl_verify_client is off or misconfigured, and you have a padlock hanging on an open gate.

Rotation, revocation and not shooting yourself in the foot

Short-lived certificates are the whole game once you are past the toy stage. If a certificate lives 90 days and renews at 60, a stolen key is useful to an attacker for a bounded window, and you never have to maintain a revocation list — expiry does the revoking for you. This is why running step-ca (or a similar issuing service) pays for itself: certificate lifetimes measured in hours or days are only sane when a machine does the renewing.

If you insist on long-lived certificates, you also inherit revocation, which is genuinely awkward. Certificate Revocation Lists and OCSP both exist, both add moving parts, and both are the reason “just make everything a 10-year cert” feels tempting. Resist it. The reason people reach for decade-long certificates is to avoid an outage when one expires, and the correct fix for expiry outages is automated renewal with alerting, so you find out at 60 days rather than at 3am when the certificate finally lapses.

One more discipline: keep the CA key offline or as close to it as you can manage. On a small setup that might mean the key lives on one hardened host that does nothing else, issues certificates, and is firewalled away from everything. Every service trusts the root certificate, which is public and safe to copy anywhere; only the signing key is precious.

Troubleshooting the errors you will absolutely hit

SSL certificate problem: unable to get local issuer certificate. The client doesn’t trust the CA that signed the server’s certificate. You forgot to distribute ca.crt, or pointed --cacert at the wrong file. This is a trust-store problem on the client, and it appears before any mutual-auth logic runs.

tlsv1 alert unknown ca from the server side. The mirror image: the server doesn’t trust the CA that signed the client’s certificate. Check that ssl_client_certificate points at the same ca.crt that signed the client cert, and that you didn’t accidentally sign the client with a second, different CA key.

Connection works with curl -k but fails properly. -k disables verification, so all it proves is that TLS negotiates at all. If -k is your only success, you have a name-matching or trust problem you are papering over. A setup that only works with verification disabled is not working.

certificate verify failed: Hostname mismatch. The SAN doesn’t contain the name you connected to. You connected to 192.168.1.40 but the certificate’s SAN says vault.mylab.local, or vice versa. Add every name and IP callers will actually use to the SAN, because that is the only field checked.

Everything was fine, then broke overnight. Two usual suspects. Either a certificate expired (check with openssl x509 -enddate -noout -in vault.crt), or clock skew crept in — TLS validation checks the “not before” and “not after” timestamps, so a host whose clock has drifted will reject perfectly good certificates. Keep NTP running on every box; a wandering clock breaks TLS in ways that look like a mystery.

The backend app doesn’t know who called it. mTLS at the proxy authenticates the connection to the proxy, and the hop from proxy to app is a separate link. If the app needs the caller’s identity, pass it in a trusted header as shown above and make sure the app only trusts that header when it arrives from the proxy. Otherwise a client could set the header itself and impersonate anyone.

Is it worth it?

For a two-service homelab where everything runs on one box behind one firewall, honestly, no — mTLS is a solution to a problem you don’t have yet, and the certificate plumbing will cost you more evenings than it saves. Get SSH hardened and your reverse proxy sane first.

The moment it earns its keep is when you have services on different hosts, containers you don’t fully trust, or anything that would let a single compromise wander sideways through everything else. mTLS turns your flat internal network into one where every conversation is authenticated, and it does so without a login prompt, a shared secret in an environment variable, or a VPN wrapped around the lot. Once step-ca is issuing and rotating certificates on its own, the ongoing cost drops close to zero, and the day something does get a foothold, the blast radius is a single certificate rather than your entire network. Pair it with decent audit logging so that if a certificate is ever misused you can actually see it happen, and the two together are a genuine, grown-up defence rather than security theatre.