Container Networking Debugging: tcpdump, nsenter, and What Packets Are Actually Doing

Container networking is one of those things that works beautifully right up until it doesn’t, at which point you’re staring at connection refused while three different layers of NAT, bridge interfaces and iptables rules all insist they’re innocent. The frustrating part is that the container is usually a minimal image with no ping, no curl, no tcpdump — nothing you’d use to actually see what’s happening. So you guess, you restart things, and eventually it “fixes itself,” which is the worst possible outcome because now you don’t know what was wrong. Here’s how to stop guessing.

1 Containers are just network namespaces

The trick to all of this is realising a container’s network is a Linux network namespace — an isolated copy of the interfaces, routing table and iptables rules. You don’t need tooling inside the container; you need tooling inside the namespace, run from the host. That’s what nsenter is for. Find the container’s PID, then enter its network namespace with your host’s fully-equipped toolbox.

$ docker inspect -f '{{.State.Pid}}' myapp
30412

# Run host tcpdump *inside* the container's network namespace
$ sudo nsenter -t 30412 -n tcpdump -i eth0 -nn port 5432
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:22:01.337 IP 172.18.0.5.51022 > 172.18.0.3.5432: Flags [S], seq 81992...
14:22:01.337 IP 172.18.0.3.5432 > 172.18.0.5.51022: Flags [R.], seq 0, ack...

That second line is the whole story: Flags [R.] is a RST. The SYN reached the database container and the database actively refused it. So the network is fine — DNS resolved, packets arrived — and the problem is Postgres isn’t listening on that interface (almost certainly bound to 127.0.0.1 instead of 0.0.0.0). Five seconds of tcpdump just saved you an hour of blaming the bridge.

2 Reading the SYN, the silence, and the reset

There are really only three things you see, and each means something specific:

• SYN out, then a RST back — the host is reachable, the port is closed. App not bound, or bound to the wrong address. Fix the app.
• SYN out, then nothing — the packet is being dropped. A firewall, a missing route, a network policy, or the wrong network entirely. This is the classic Kubernetes NetworkPolicy symptom.
• No SYN at all on the destination — it never left, or never arrived. Check DNS resolution and the routing table first.

That last one is where people waste the most time, because they assume the name resolved. Check it from inside the namespace:

$ sudo nsenter -t 30412 -n getent hosts db
$ sudo nsenter -t 30412 -n ip route
default via 172.18.0.1 dev eth0
172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.5

Empty getent output means DNS failed before a single packet was ever sent. No amount of tcpdump on the destination will show you anything, because nothing was transmitted.

3 When it’s NAT and iptables

Docker’s bridge networking is a stack of iptables DNAT rules. When a published port mysteriously doesn’t forward, the rules are where the truth lives:

$ sudo iptables -t nat -L DOCKER -n
Chain DOCKER (2 references)
target  prot opt source     destination
DNAT    tcp  --  0.0.0.0/0  0.0.0.0/0  tcp dpt:8080 to:172.18.0.5:80

If that DNAT line is missing, the port was never actually published — re-check your -p flag or compose ports:. If it’s present but traffic still doesn’t flow, capture on the host’s docker0 (or the bridge) interface as well as inside the namespace, and compare. Seeing the packet on the bridge but not in the namespace localises the drop to the veth pair or a policy in between.

For Kubernetes the same logic applies, but the namespaces hang off the kubelet’s containers. crictl inspect <id> gives you the PID, and from there it’s identical nsenter work. Tools like kubectl debug with an ephemeral container that shares the target’s network namespace make this even tidier — you get a shell in the pod’s network without rebuilding the image.

4 Verdict

Is learning this worth it? For anyone who runs containers in production, unambiguously yes — it’s the difference between “the network is broken, I don’t know why” and a precise, defensible diagnosis in under a minute. The two ideas that matter are tiny: a container’s network is just a namespace, and nsenter lets you point your full host toolkit at it. Everything else is reading tcpdump output, which becomes second nature fast. This isn’t niche wizardry; it’s the baseline competence that separates restarting things until they work from actually understanding your stack. Learn the three packet patterns above and you’ll have already out-debugged most of the people you’ll ever escalate to.