Let's Encrypt DNS-01 for Internal-Only Certificates

For years the homelab answer to “I want HTTPS on my internal services” was a self-signed certificate and a lifetime of clicking through browser warnings. Then people discovered you could run your own certificate authority, which solved the warnings and created a new job: distributing your root certificate to every device that would ever touch the network, including the ones that do not have a certificate store you can reach. Android has opinions. Your smart TV has different opinions. The Kindle has no opinions and no store.

The DNS-01 challenge sidesteps all of it. You get a certificate from Let’s Encrypt — publicly trusted, in every device’s store since the factory — for a hostname that resolves to 192.168.1.40 and is reachable from nowhere except your sofa. No inbound connection, no port 80, no public IP. Every device already trusts it because every device already trusts Let’s Encrypt.

I have been running this arrangement for a long time now and it is the single highest-value hour of homelab work I can recommend. Here is why it works, how to set it up without generating a support ticket for yourself, and the specific ways it breaks.

Why DNS-01 works when HTTP-01 cannot

Let’s Encrypt needs proof that you control a domain before it signs a certificate for it. The default proof — HTTP-01 — is: put a specific file at http://example.com/.well-known/acme-challenge/<token> and we will fetch it. That requires their validation servers to reach yours, which requires a public IP and an open port 80, which is exactly what an internal-only service does not have.

DNS-01 asks for a different proof: create a TXT record at _acme-challenge.app.example.com containing a value derived from your account key and the challenge token. Let’s Encrypt queries the record from their own resolvers and validates it. Your web server is never contacted. Your web server can be off. Your web server can be on a laptop in a bag.

The proof of control has moved from “you can serve HTTP on this name” to “you can write DNS records for this zone”, and that second capability lives at your registrar, which is on the internet whether or not you are. That is the whole trick, and it produces two properties you cannot get any other way:

Wildcards. Let’s Encrypt will only issue *.example.com via DNS-01, because a wildcard is a claim over an entire namespace and file-on-a-webserver is too weak a proof for that. One certificate, every internal hostname, renewed in one place. This alone is worth the setup.

Zero inbound surface. Nothing about this arrangement makes your network reachable. You are using a public CA as a signing service and telling it about your intentions through DNS. The certificate is public — more on that below — and the machine is not.

The registrar problem, and delegation

To automate DNS-01 you need an API to write TXT records, and your ACME client needs a credential for it. This is where the design gets uncomfortable, because the credential your registrar hands out is usually full control of the entire zone — every record, including MX and the A record for your actual website. You are about to put that credential on a container in a cupboard so it can write one TXT record every 60 days.

Think about that honestly. If that box is compromised, the attacker can repoint your mail. The blast radius of a homelab certificate renewer should not include your email.

There are three ways out, in increasing order of effort:

1. Use a provider with scoped tokens. Cloudflare’s API tokens can be restricted to Zone:DNS:Edit on a single zone, which is better than nothing and takes two minutes. Several other providers now offer something similar. It is still zone-wide DNS write, so an attacker can still repoint things, but they cannot touch billing or other zones.

2. Delegate the challenge with a CNAME. This is the elegant answer and almost nobody does it. Create a permanent CNAME at your real registrar:

   1	_acme-challenge.app.example.com. CNAME app.acme.example.net.

   Let’s Encrypt follows CNAMEs when looking for the challenge record. So your ACME client now needs write access to example.net — a throwaway zone with nothing in it — and no access at all to the zone that runs your life. The credential in the cupboard can do precisely one useless thing.

3. Run acme-dns. A tiny purpose-built DNS server that only serves TXT records for challenges, self-registers accounts, and gives each one a credential scoped to a single random subdomain. You point CNAMEs at it and delegate a subdomain to it with NS records. It is about 15MB of Go and it has been stable for years.

I use delegation. The extra work is a one-time CNAME per hostname, and the wildcard case needs only one CNAME for the whole zone. In exchange, the worst thing a compromised renewer can do is issue certificates for names it already had.

Actually doing it

Caddy makes this close to trivial, provided you use a build with the DNS provider module compiled in — the stock binary does not include them:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

{
	email [email protected]
}

*.example.com {
	tls {
		dns cloudflare {env.CF_API_TOKEN}
		resolvers 1.1.1.1
	}

	@jellyfin host media.example.com
	handle @jellyfin {
		reverse_proxy 192.168.1.40:8096
	}

	@gitea host git.example.com
	handle @gitea {
		reverse_proxy 192.168.1.41:3000
	}

	handle {
		respond "no such service" 404
	}
}

The resolvers line matters more than it looks. Caddy self-checks that its TXT record has propagated before telling Let’s Encrypt to validate, and if it asks your local resolver — which is very likely your own sinkhole with a stale cache — it will wait forever for a record it cannot see. Point the check at a public resolver and the problem evaporates.

For anything not fronted by Caddy, lego is the sharpest tool, and it renders the delegation story nicely:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

export CF_DNS_API_TOKEN="$(cat /run/secrets/cf_token)"

lego --email [email protected] \
     --dns cloudflare \
     --dns.resolvers 1.1.1.1:53 \
     --domains "*.example.com" \
     --domains "example.com" \
     --path /etc/lego \
     run

# Renewal, from a timer, 30 days out:
lego --email [email protected] --dns cloudflare \
     --domains "*.example.com" --path /etc/lego \
     renew --days 30 --renew-hook "systemctl reload nginx"

Keep the token out of your shell history and out of your repo. If your homelab lives in git — mine does — SOPS and age is the sane way to carry it. On Kubernetes, cert-manager does the same job with an Issuer and a solvers.dns01 block, and the delegation advice applies identically.

Making the name resolve inside

You now hold a certificate for media.example.com and your browser resolves that name to nothing, because the record does not exist publicly and should not.

The clean fix is split-horizon DNS: your internal resolver answers media.example.com with 192.168.1.40 while the rest of the world gets NXDOMAIN. Any of the usual sinkholes will do this with a local DNS entry, and it is a two-line change.

The grubby alternative is publishing internal A records in public DNS — media.example.com. A 192.168.1.40 for all the world to read. It works, and it tells anyone curious about your domain the shape of your internal network. I would rather they did not have that.

Note that split-horizon interacts badly with encrypted DNS on client devices. A phone that has decided to use DoH to a public resolver will never ask your resolver anything, and your internal names will fail on that device only, in a way that looks like magic. Check that before you rebuild your resolver at midnight.

Getting the certificate to things that are not the proxy

A wildcard behind one reverse proxy covers most of a homelab, because most of a homelab is HTTP and you were going to front it anyway. The awkward remainder is everything that terminates TLS itself: a mail server, a database with ssl = on, an MQTT broker, a printer. Those need the actual key and certificate files on disk, in a format they accept, reloaded when they change.

The pattern that has survived contact with reality here is one renewer and a distribution step. One host renews the wildcard; a hook copies the results where they need to go and restarts what needs restarting:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

#!/usr/bin/env bash
# /etc/lego/hooks/deploy.sh — run by --renew-hook
set -euo pipefail

CERT_DIR=/etc/lego/certificates
DOMAIN=_.example.com

install -o mosquitto -g mosquitto -m 0640   "$CERT_DIR/$DOMAIN.crt" /etc/mosquitto/certs/server.crt
install -o mosquitto -g mosquitto -m 0640   "$CERT_DIR/$DOMAIN.key" /etc/mosquitto/certs/server.key
systemctl reload mosquitto

# Postgres wants the key to be 0600 and owned by postgres, or it refuses to start.
install -o postgres -g postgres -m 0600   "$CERT_DIR/$DOMAIN.key" /var/lib/postgresql/server.key
systemctl reload postgresql

Two things bite here every time. The first is file permissions: several daemons refuse to start if the private key is group-readable, and they tell you so in a log line you will not see until you go looking for why the service did not come back. The second is that reload is a lie for some services — they parse the certificate once at startup and a reload does nothing, so the daemon happily serves an expired certificate for weeks. Test the reload path deliberately by renewing early with --days 90, and confirm the served certificate actually changed:

1

echo | openssl s_client -connect 192.168.1.45:8883 2>/dev/null   | openssl x509 -noout -dates

If the notAfter date did not move, you need restart rather than reload, and you have just learned something you would otherwise have learned in an outage.

Troubleshooting

“DNS problem: NXDOMAIN looking up TXT for _acme-challenge…” Either the record never got written — check the API credential — or Let’s Encrypt asked before propagation finished. Verify by hand against an authoritative server rather than your cache:

1

dig +short TXT _acme-challenge.app.example.com @1.1.1.1

If your zone has a long TTL on NXDOMAIN responses (the SOA minimum), a negative answer can be cached for hours. Drop the SOA minimum to 60 before you start, and re-raise it later if you care.

Validation succeeds by hand, fails from the client. The client’s propagation pre-check is querying a resolver that lies to it. Set --dns.resolvers or Caddy’s resolvers explicitly. This is the single most common DNS-01 failure and it is entirely self-inflicted.

CNAME delegation returns SERVFAIL. Something in the chain is DNSSEC-signed and the delegated zone is not, or is signed badly. Check with delv or a DNSSEC debugger; an unsigned island under a signed parent needs correct NS and DS records or validating resolvers will refuse the answer.

Renewals stop silently after months of working. The classic. Renewal runs from a timer nobody watches, the credential expired or the provider changed their API, and you find out when the certificate does. Point a dead-man’s-switch at the renewal hook so silence pages you, and alert on certificate expiry independently of the renewer’s own health.

Too many certificates already issued. Five duplicate certificates per week, and a wildcard plus its bare domain counts as one certificate you are re-requesting each time you experiment. Use --server https://acme-staging-v02.api.letsencrypt.org/directory while you iterate. There is a longer piece on rate limits if you have already hit the wall.

The cost nobody mentions: certificate transparency

Every certificate Let’s Encrypt issues is logged publicly, forever, in Certificate Transparency logs. Anyone can search those logs. Ask for media.example.com and vault.example.com and you have published, in a permanently indexed database, that those services exist and what they are called. There are bots that watch CT logs in real time and start probing new names within seconds of issuance.

For an internal-only service the probing goes nowhere, since the name resolves to a private address from outside. The information disclosure is real regardless. A wildcard fixes it completely — *.example.com appears in the log and tells the world nothing about which names exist behind it. That is a second, quieter reason to prefer wildcards, and it is why I stopped issuing per-service certificates for internal names.

If even the domain is sensitive, this is where you stop and run your own CA instead, accepting the root-distribution work as the price of a namespace nobody can enumerate.

The verdict

If your homelab has services you reach by hostname and you own a domain, do this. It takes an hour, it costs nothing, it removes the browser warnings permanently, and it works on the devices where installing a private root is impossible. A wildcard, a Caddy build with your provider’s module, a CNAME delegation to a throwaway zone, and a split-horizon record — that is the whole architecture and it renews itself.

Where it stops being right: if you need certificates for names you cannot publish to CT logs, if you want certificate lifetimes measured in hours for mTLS between services, or if you need to issue for machines that have no domain at all. Those are the cases a private CA exists for, and they are real cases. For the ordinary problem of “I want a green padlock on the thing in the cupboard”, DNS-01 is the answer, and the fact that it uses the public trust system for a machine the public cannot reach remains one of my favourite pieces of infrastructure design.