Container Networking Debugging: tcpdump, nsenter, and What Packets Are Actually Doing

I once spent the better part of an evening convinced a node in my home lab had a broken bridge. An app container kept logging connection refused against its database, the database was clearly up, and I’d restarted both more times than I’d like to admit. Three layers of NAT, a Docker bridge and a stack of iptables rules all insisted they were innocent, and I had no way to call their bluff — because the app image was a stripped-down distroless thing with no ping, no curl, no tcpdump, nothing you’d actually use to look. So I guessed. Eventually it “fixed itself” after a redeploy, which is the worst possible outcome: the problem was gone and I still didn’t know what it had been.

The fix to that whole class of misery is realising you don’t need any tooling inside the container at all. You need it inside the container’s network namespace, and you can point your host’s fully-equipped toolbox straight at it. Once that clicked, the same evening-long mysteries became sub-minute diagnoses. Here is how to stop guessing.

A container’s network is just a namespace

A container is not a tiny virtual machine. Its network is a Linux network namespace — an isolated copy of the interfaces, routing table, ARP cache and iptables rules, sharing the host kernel. The minimal image has no debugging tools, but the namespace it lives in is fully inspectable from the host by anyone with root. That is the entire trick. You find the container’s main PID, then enter its network namespace with nsenter and run your tools — the host’s tcpdump, ip, ss, getent — as if they were installed inside.

1
2
3
4
5
6
7
8

$ docker inspect -f '{{.State.Pid}}' myapp
30412

# Run the HOST's tcpdump inside the container's network namespace
$ sudo nsenter -t 30412 -n tcpdump -i eth0 -nn port 5432
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:22:01.337 IP 172.18.0.5.51022 > 172.18.0.3.5432: Flags [S], seq 81992...
14:22:01.337 IP 172.18.0.3.5432 > 172.18.0.5.51022: Flags [R.], seq 0, ack...

-t 30412 targets that PID, -n enters only the network namespace (not mount or PID, so your host binaries stay on the host filesystem), and everything after is an ordinary host command. The eth0 you capture on is the container’s eth0, not the host’s.

That second line is the whole story. Flags [R.] is a RST. The SYN reached the database container and the database actively refused it. So the network is fine — DNS resolved, the route worked, packets arrived — and the real fault is that Postgres isn’t listening on that interface, almost certainly bound to 127.0.0.1 instead of 0.0.0.0. Five seconds of tcpdump just saved an hour of blaming the bridge. This is also why understanding what costs you time in a lab matters as much as the hardware itself, something I dug into in the real cost of self-hosting — debugging time is the line item nobody budgets for.

The three things you ever see

Strip away the noise and packet captures on a failing connection show one of three patterns, and each points somewhere different:

• SYN out, then a RST back. The host is reachable, the port is closed. The app isn’t bound, or is bound to the wrong address. Fix the application, not the network.
• SYN out, then silence. The packet is being dropped somewhere. A firewall rule, a missing route, a Kubernetes NetworkPolicy, or the wrong network entirely. The reply never comes because something ate the request or the response.
• No SYN at all on the source. It never left. Name resolution failed, or the route is missing, so the kernel never put a packet on the wire.

That last one is where the most time gets wasted, because everyone assumes the name resolved. Check it from inside the namespace before you touch anything else:

1
2
3
4

$ sudo nsenter -t 30412 -n getent hosts db
$ sudo nsenter -t 30412 -n ip route
default via 172.18.0.1 dev eth0
172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.5

Empty getent output means DNS failed before a single packet was sent. No amount of tcpdump on the destination will show you anything, because nothing was transmitted. In Kubernetes this is the single most common failure I hit, and it’s almost always CoreDNS or a search-domain quirk rather than a “real” network problem — I went down that rabbit hole properly in CoreDNS and Kubernetes DNS, because “the pod can’t resolve a name” deserves its own investigation.

See who’s actually listening

When a RST comes back, confirm the bind before you go editing config. ss run inside the namespace shows exactly which address and port the service is on:

1
2
3

$ sudo nsenter -t 30412 -n ss -ltnp
State   Recv-Q  Send-Q  Local Address:Port   Peer Address:Port
LISTEN  0       128         127.0.0.1:5432        0.0.0.0:*     users:(("postgres",pid=42,fd=7))

There it is: 127.0.0.1:5432. The process is listening on loopback only, so anything arriving on eth0 gets a RST. The fix is in the application’s config — listen_addresses = '*' for Postgres, or 0.0.0.0 for whatever framework you’re running — not in any networking layer. This one mistake accounts for an embarrassing share of “the network is broken” tickets.

When it really is NAT and iptables

Docker’s bridge networking is, underneath, a stack of iptables DNAT rules. When a published port mysteriously doesn’t forward, the rules are where the truth lives:

1
2
3
4

$ sudo iptables -t nat -L DOCKER -n
Chain DOCKER (2 references)
target  prot opt source     destination
DNAT    tcp  --  0.0.0.0/0  0.0.0.0/0  tcp dpt:8080 to:172.18.0.5:80

If that DNAT line is missing, the port was never actually published — re-check your -p flag or the compose ports: block. If it’s present but traffic still doesn’t flow, capture in two places at once and compare: on the host’s bridge interface and inside the namespace.

1
2
3
4

# Terminal 1: on the host bridge
$ sudo tcpdump -i docker0 -nn port 8080
# Terminal 2: inside the container namespace
$ sudo nsenter -t 30412 -n tcpdump -i eth0 -nn port 80

Seeing the packet on docker0 but not inside the namespace localises the drop to the veth pair or a rule between them. Seeing it in neither means it never reached the host at all — look further upstream, at your firewall or the client. This “capture at both ends of the suspected hop” technique is the most reliable way to bisect a path. Don’t theorise about where the packet dies; watch it die.

On nftables-based hosts the chains live under nft list ruleset instead, and Docker writes its rules there on newer distributions. The logic is identical; only the inspection command changes. If you’re on a distro that’s switched to nftables and still reaching for iptables -L, you may be reading a compatibility shim that hides the real ruleset.

The same trick in Kubernetes

In Kubernetes the namespaces hang off the container runtime rather than Docker, but the method is unchanged. Get the PID via the runtime, then nsenter as before:

1
2
3
4
5

$ crictl ps --name myapp -q
3f9a1c2b...
$ crictl inspect 3f9a1c2b... | jq '.info.pid'
51234
$ sudo nsenter -t 51234 -n tcpdump -i eth0 -nn

Cleaner still, kubectl debug attaches an ephemeral container that shares the target pod’s network namespace, giving you a shell with real tools inside the pod’s network without rebuilding the image:

1
2

$ kubectl debug -it mypod --image=nicolaka/netshoot --target=myapp -- bash
# tcpdump, ss, dig, curl all available here, in the pod's netns

The --target flag is the part people miss — without it the ephemeral container gets its own namespace and you’re debugging the wrong network.

Troubleshooting: when nsenter itself misbehaves

A few traps that will waste your time if you don’t know them:

• nsenter: cannot open /proc/<pid>/ns/net: No such file or directory — the PID is wrong or the container already died and restarted. Re-run docker inspect or crictl inspect; the PID changes on every restart, so a value from thirty seconds ago may be stale.
• tcpdump: eth0: No such device exists — the interface inside the namespace might not be called eth0. Run sudo nsenter -t <pid> -n ip link first and capture on whatever’s actually there; some CNIs name it eth0 but others use generated names.
• You see SYN and SYN-ACK but the app still fails — the network completed the handshake; the problem is now above layer 4. TLS negotiation, an HTTP 5xx, or an auth rejection. Stop looking at packets and read the application log; tcpdump has already told you the network did its job.
• Capture is empty but you’re sure traffic flows — check your filter. port 5432 won’t show traffic on a remapped port, and a typo’d host filter silently matches nothing. Drop the filter, confirm you see anything, then narrow.
• Permission denied despite sudo — on locked-down hosts the namespace may live in a user namespace too; rootless Docker in particular puts containers in a user-namespaced tree. You may need to enter from the rootless daemon’s context rather than the host root.
• Distroless containers with no shell at all — that’s exactly the case nsenter and kubectl debug solve. You never needed a shell in the container; you needed tools in its namespace, supplied from outside.

A general rule: if a capture surprises you, capture one hop closer to the source before you form a theory. Almost every wrong diagnosis I’ve made came from inferring what a packet did instead of watching it.

Is it worth learning?

For anyone running containers in anger — and especially anyone self-hosting a stack they’re on the hook to keep alive — unambiguously yes. The payoff is turning “the network is broken and I don’t know why” into a precise, defensible diagnosis in under a minute, which is the single biggest force multiplier I’ve found for keeping a lab maintainable rather than mysterious.

The two ideas that carry all the weight are tiny: a container’s network is just a namespace, and nsenter lets you aim your full host toolkit at it. Everything after that is reading tcpdump output, which becomes second nature faster than you’d expect once you’ve internalised the three patterns — RST, silence, no-SYN. This isn’t arcane wizardry reserved for network engineers; it’s baseline competence that separates restarting things until they work from understanding your stack. Learn the three patterns and the two-point capture, and you’ll out-debug most of the people you’d otherwise escalate to.

Who’s it not for? If your entire deployment is one managed PaaS where you never see a host, you’ll rarely get the root access this needs, and the platform’s own tooling is your only window. But the moment you own the node — a VPS, a home server, a bare-metal cluster — this is the skill that pays for itself the first time you’d otherwise have spent an evening blaming an innocent bridge.