Nebula: Slack's Mesh VPN for the Rest of Us

Slack had a problem most of us would love to have: tens of thousands of hosts across several cloud providers, all needing to talk to each other securely, with no appetite for the operational archaeology of IPsec. What they built in response was Nebula, and in 2019 they open-sourced it. It has been quietly excellent ever since, and it remains the least discussed member of the mesh VPN family.

The reason it deserves attention in a homelab is a design decision that sets it apart from everything else in the category. Nebula has no control plane. There is no server holding your device list, no API deciding who may talk to whom, no account anywhere. Authorisation is baked into an X.509-ish certificate that you sign yourself with a CA that lives on your laptop, and once a node has its certificate, the mesh works with no ongoing dependency on any central service at all.

I have run Nebula alongside other overlays for a while now. It is the one I reach for when I care about the failure mode more than the convenience, and I want to explain exactly what that sentence means.

How it differs from the neighbours

Tailscale, Headscale, NetBird and Nebula all produce the same visible outcome: a flat overlay network where every machine has a stable address and talks directly to every other machine. The difference is in what has to be alive for that to keep being true.

A control-plane mesh has a coordination server that holds identity, distributes keys, publishes the node list and enforces the access policy. Tailscale runs that for you and it is superb; Headscale lets you run it yourself, which trades a dependency on a company for a dependency on a service you now maintain. Either way, something authoritative is running and something has to be online to enrol a new device or change a rule.

Nebula moves all of that into the certificate. A Nebula cert carries the node’s overlay address, its name, its group memberships and an expiry, all signed by your CA. When two nodes meet they exchange certificates, each verifies the other’s signature against the CA public key they both hold, and they read each other’s groups directly out of the signed document. No third party is consulted. The policy is evaluated locally by both ends using information the certificate itself carries.

Lighthouses are the part people mistake for a control plane. A lighthouse is an ordinary Nebula node that happens to have a public address and keeps a map of which node is currently at which real-world address. When node A wants node B, it asks a lighthouse where B is, then connects to B directly and punches through NAT. The lighthouse never sees the traffic, holds no keys and makes no decisions. It is a phone book. Take every lighthouse offline and existing tunnels keep flowing; only new connections between nodes that have not met struggle.

That is the whole pitch. The cost is that every authorisation decision is frozen at signing time, and revocation is genuinely awkward. More on that later, because it is the real objection.

Building the mesh

Set up the CA on a machine you trust and keep the key off the network:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

$ nebula-cert ca -name "mylab.local" -duration 26280h
# Produces ca.key and ca.crt. The .key signs every certificate you will
# ever issue. Treat it like an SSH private key you cannot rotate cheaply.

$ nebula-cert print -path ca.crt
NebulaCertificate {
        Details {
                Name: mylab.local
                Is CA: true
                Not before: 2025-08-15 09:02:11 +0000 UTC
                Not after: 2028-08-14 09:02:11 +0000 UTC
        }
        Fingerprint: 3f9a2b7c1d4e6f8a0b2c4d6e8f0a1b3c5d7e9f1a2b4c6d8e0f2a4b6c8d0e2f4a
}

Then sign a certificate per node, assigning its overlay address and groups as you go:

1
2
3
4
5

$ nebula-cert sign -name "lighthouse" -ip "10.42.0.1/24" -groups "infra"
$ nebula-cert sign -name "nas"        -ip "10.42.0.10/24" -groups "servers"
$ nebula-cert sign -name "docker01"   -ip "10.42.0.11/24" -groups "servers,web"
$ nebula-cert sign -name "laptop"     -ip "10.42.0.20/24" -groups "admin"
$ nebula-cert sign -name "phone"      -ip "10.42.0.21/24" -groups "clients"

The groups are the interesting part, and it is worth pausing on what just happened. docker01’s membership of servers and web is now a signed, tamper-proof claim inside its certificate. It cannot lie about it. No server needs to be asked. Every other node can verify that claim offline using only the CA cert, forever.

Each node gets three files — ca.crt, its own .crt, its own .key — plus a config. The lighthouse:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

# /etc/nebula/config.yml on the lighthouse (public address required)
pki:
  ca: /etc/nebula/ca.crt
  cert: /etc/nebula/lighthouse.crt
  key: /etc/nebula/lighthouse.key

static_host_map: {}

lighthouse:
  am_lighthouse: true
  interval: 60

listen:
  host: 0.0.0.0
  port: 4242

punchy:
  punch: true

firewall:
  outbound:
    - port: any
      proto: any
      host: any
  inbound:
    - port: any
      proto: icmp
      host: any

And an ordinary node:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

# /etc/nebula/config.yml on docker01
pki:
  ca: /etc/nebula/ca.crt
  cert: /etc/nebula/docker01.crt
  key: /etc/nebula/docker01.key

# Where to find the lighthouse in the real world.
static_host_map:
  "10.42.0.1": ["lighthouse.example.com:4242"]

lighthouse:
  am_lighthouse: false
  interval: 60
  hosts:
    - "10.42.0.1"

listen:
  host: 0.0.0.0
  port: 0            # 0 = random port, correct for NAT'd hosts

punchy:
  punch: true
  respond: true      # helps through symmetric NAT

tun:
  dev: nebula1
  mtu: 1300

firewall:
  conntrack:
    tcp_timeout: 12m
    default_timeout: 10m

  outbound:
    - port: any
      proto: any
      host: any

  inbound:
    # Anyone in the mesh may ping. Useful for debugging.
    - port: any
      proto: icmp
      host: any

    # Only the admin group gets SSH. Enforced by this node, locally.
    - port: 22
      proto: tcp
      groups:
        - admin

    # Web ports open to clients and admins.
    - port: 443
      proto: tcp
      groups:
        - clients
        - admin

That firewall block is where Nebula gets genuinely nice. It is host-based microsegmentation with no infrastructure behind it: docker01 itself decides that SSH is available to certificates bearing the admin group, and it makes that decision by reading a signature. There is no policy server to query, so there is no policy server to be down, be compromised, or be a company that changes its pricing.

Compare that with maintaining the same policy across a fleet by hand. Adding a new admin machine means signing one certificate with -groups "admin"; every node in the mesh immediately grants it SSH without any of them being reconfigured or even restarted. That property is worth a lot, and it falls straight out of the design.

Reaching things that cannot run Nebula

A mesh only covers hosts you can install software on, which leaves out the printer, the smart plugs, the managed switch and the IPMI interface on the old server — frequently the exact things you want to reach from a sofa.

Nebula’s answer is unsafe_routes, which lets one node advertise itself as the way into a subnet that lives outside the mesh:

1
2
3
4
5
6
7
8

# On the node acting as gateway into the physical LAN
tun:
  dev: nebula1
  mtu: 1300
  unsafe_routes:
    - route: 192.168.30.0/24
      via: 10.42.0.10
      mtu: 1300

Every node carrying that config learns to send 192.168.30.0/24 traffic to 10.42.0.10, which forwards it onto the real LAN. The gateway node needs ip_forward enabled and, in most cases, a masquerade rule so the return traffic finds its way back.

The name is a fair warning. Traffic to those addresses leaves the mesh’s encryption at the gateway and crosses the physical network in the clear, and the destination has no certificate, no group and no identity — so Nebula’s host firewall has nothing to evaluate and the microsegmentation story stops at the gateway’s edge. Keep the advertised range as narrow as the requirement genuinely is. Advertising an entire /16 because it saved typing turns one compromised mesh node into a router onto everything you own.

The revocation problem

Here is the honest weakness, stated plainly, because everything above sounds a bit too good.

A certificate is valid until it expires. If a laptop with an admin certificate is stolen, that certificate keeps working — it is a signed document asserting group membership, and the nodes evaluating it consult nothing external. Nebula’s answer is a blocklist of certificate fingerprints in the config:

1
2
3
4
5
6

pki:
  ca: /etc/nebula/ca.crt
  cert: /etc/nebula/nas.crt
  key: /etc/nebula/nas.key
  blocklist:
    - "9f4b2e8c6a1d3f5b7e9c0a2d4f6b8e0c2a4d6f8b0e2c4a6d8f0b2e4c6a8d0f2b"

Read that carefully. Revocation means editing a config file on every node in the mesh and reloading it. On five machines that is an Ansible play and a shrug. On fifty it is a real operational burden, and on a node that is currently offline the revocation does not apply until it comes back — which is exactly the node you are least sure about.

The mitigation is short certificate lifetimes. Sign laptops and phones for 30 days and automate the reissue; the window between compromise and natural expiry shrinks to something you can live with. That is the right answer, and it means building reissue automation, which is work that a control plane would have done for you. This is the trade in one paragraph: Nebula removes the control plane, and the control plane’s job moves onto your plate.

Do not sign anything for ten years. -duration defaults to one year and even that is generous for a mobile device.

Guarding the CA

The CA key is the whole security model in one file, and it deserves more thought than a mesh overlay usually gets.

Anyone holding ca.key can mint a certificate with any address, any name and any group. They can issue themselves admin and every node in your mesh will grant them SSH, verify the signature happily, and be entirely correct to do so. There is no second factor, no audit trail, and no revocation that does not involve touching every node. The key is the kingdom.

So it does not live on the mesh. Mine sits encrypted on removable media, and it comes out when I need to sign something, which is a handful of times a year. Signing happens on a machine that is not a mesh member. This sounds paranoid for a homelab until you notice that the alternative — the CA key sitting on a lighthouse, on a public VPS, because that was convenient — hands the entire network to anyone who gets a shell on the most exposed box you own.

The related decision is the CA’s own lifetime. nebula-cert ca takes a -duration, and when the CA certificate expires, every certificate it signed becomes worthless simultaneously and the mesh stops. Three years is a reasonable window. Nebula supports overlapping CAs precisely so you can roll one — you distribute a bundle containing both the old and new CA certificates, reissue node certificates against the new one, and retire the old once everything has moved.

Do that rotation once, deliberately, while nothing is broken. Discovering the mechanism on the day the CA expires and the mesh is down, with no way to reach the nodes you need to fix, is an experience worth paying to avoid.

Troubleshooting

“Nodes see the lighthouse and cannot see each other.” NAT punching failed. Confirm punchy: punch: true on both, and respond: true on at least one. Symmetric NAT — common on carrier-grade setups and some corporate networks — defeats punching entirely, and the fallback is relaying traffic through a node with a public address. Nebula’s relay support is explicit: name a relay node in the config with relay: use_relays: true. It works and it costs you the direct path.

“Handshake fails with a certificate error.” nebula-cert print -path node.crt and check the validity window and the CA fingerprint. Two causes dominate: the certificate expired, or the node has a ca.crt from a CA you regenerated at some point. Clock skew also breaks handshakes — a Pi with no RTC that boots to 1970 will fail every certificate check until NTP lands.

“Traffic flows and throughput is poor.” MTU. Nebula’s default of 1300 is conservative and usually right; if you have raised it, put it back. The same fragmentation symptoms apply as with any tunnel — small packets fine, large transfers stalling — and the same bisection with ping -M do finds the real number.

“The firewall block is being ignored.” Nebula’s inbound rules are evaluated on the receiving node, so a rule you added on your laptop does nothing about what your laptop may reach. Add it to the target. Also note the config is reloaded with SIGHUP — the firewall and blocklist sections reload live, and the PKI section requires a restart.

“Everything broke when the lighthouse rebooted.” Existing tunnels should survive. If they did not, your nodes have static_host_map entries pointing only at the lighthouse and no other way to find each other. Run two lighthouses; they are cheap, they hold no secrets beyond their own certificate, and the second one costs a few pounds a month on the smallest VPS available.

Verdict

Nebula is worth choosing when the failure mode is what you care about. If the question “what happens to my network when a company I depend on has a bad day” bothers you, Nebula answers it convincingly: nothing happens, because there is nobody to have a bad day. The mesh keeps running on certificates it already holds. For a lab that must work during an internet outage or an account suspension, that is a real property and nothing else in this category offers it as cleanly.

It is also the right answer if you want microsegmentation without buying microsegmentation. The group-based host firewall is genuinely good and it is enforced by every node independently, which is a stronger architecture than a central policy engine that can be bypassed by anything that gets past it.

Nebula is the wrong choice if you want to add a device by scanning a QR code. Every node needs a certificate signed on your CA machine and files copied into place, and there is no enrolment flow to make that pleasant. Onboarding a phone is a genuine faff compared to installing an app and signing in. For a household where other people’s devices need access, that friction is decisive and I would not fight it — I looked at where Nebula fits against Tailscale previously and the conclusion has held.

What I actually do: Nebula for the infrastructure mesh — servers, the offsite box, things that live in racks and get provisioned by Ansible anyway, where certificate distribution is one more task in a playbook that already exists. Something friendlier for the humans and their phones. And where the requirement is genuinely two networks joined rather than many hosts meeting, plain WireGuard site-to-site remains simpler than any mesh and I use that too. Three tools sounds indulgent; each one is doing a job the others do worse.