Container-to-container: same host, cross host, and the overlay illusion
Containers on one user-defined bridge talk directly at L2; the default bridge lacks DNS. Across hosts, an overlay builds a virtual L2 over VXLAN with a distributed control plane — and the leaks (DNS scope, MTU overhead, hairpin NAT) are where multi-host networking actually fails.
It passed every test on a single node and fell over the moment they scaled to three. The app discovered its database peer by container name, and on one machine redis resolved instantly. Move the cache to a different host in the same swarm overlay and resolution still worked — but a slow trickle of requests timed out under load, never reproducibly. The team blamed the database, then the app, then the load balancer. The actual culprit was MTU. The overlay wrapped every packet in a VXLAN header, shrinking the usable payload from 1500 to 1450 bytes, but the application’s containers were still advertising a 1500-byte MSS. Small requests fit and succeeded; any response larger than 1450 bytes needed fragmentation that the path didn’t allow, so it silently stalled. The “intermittent database problem” was a 50-byte header tax on a virtual L2 network pretending to be a flat LAN. Container-to-container networking is easy on one host and full of leaky abstractions across hosts — and knowing where the abstraction leaks is the whole job.
Same host: bridge isolation is the default
Two containers on the same user-defined bridge network reach each other directly: their host-side veths share one bridge, so a frame from one is switched straight to the other at L2 — no routing, no NAT, no published port needed. You connect to http://other-container:port and the embedded DNS resolves the name to the peer’s bridge IP. This is the happy path, and it is genuinely simple.
The trap is the default bridge (docker0), which behaves differently from networks you create:
- On the default bridge, containers get no DNS-based service discovery (only legacy
--link, deprecated), and historically inter-container communication is governed by the daemon’sicc(inter-container communication) setting. The default bridge is for one-offdocker run, not for an application’s service mesh. - On a user-defined bridge (
docker network create app, or any Compose service network), you get embedded DNS, automatic name resolution, and — critically — network-scoped isolation: a container only reaches containers on a network it is attached to. Put your database on a back-end network the public-facing proxy can’t join, and the database is unreachable from the proxy at L3 — segmentation enforced by which bridge you attach to, not a firewall rule you might forget.
The senior pattern is one network per trust boundary: a front-end network for the proxy and API, a back-end network for the API and database, with the database never on the front-end network. The API straddles both (a container can attach to multiple networks, getting one veth and one IP per network); nothing else crosses the boundary. This is real segmentation built from the bridge primitive, and it is free.
You put 'db' on a back-end network and the API on both front-end and back-end. The public proxy is on front-end only. Why can't the proxy reach 'db' even though all three run on one host?
▸Why this works
Why prefer multiple user-defined networks over the simpler ‘everything on one network with firewall rules’? Because the bridge topology is the firewall — and it’s a firewall you can’t forget to apply. A container with no veth on the database’s network has no possible path to it; there is no rule to misconfigure, no ordering to get wrong, nothing to drift. Firewall rules are an allow/deny list layered on top of full connectivity; network segmentation removes the connectivity itself. The former fails open when a rule is missing; the latter fails closed by construction.
Across hosts: the overlay and its leaks
If same-host networking is the happy path, cross-host is where things get quietly wrong — and usually in ways that only surface under load.
A single bridge lives on one host; containers on different hosts can’t share it. The overlay driver builds a virtual L2 network spanning hosts by encapsulating container Ethernet frames in VXLAN (UDP, default port 4789) and tunneling them between host IPs. To the containers it looks like one flat LAN — they keep talking to peers by name and IP as if local. A distributed control plane (Swarm’s built-in store, or an external KV like etcd/Consul in plain Docker) shares which container IP lives on which host so the data plane knows where to send each tunneled frame.
The abstraction is convincing, and that is the danger — here is where it leaks:
- MTU overhead. The VXLAN header costs 50 bytes, so the overlay’s effective MTU is 1450, not 1500. If containers advertise 1500-byte MSS and the path can’t fragment, large packets stall — the Hook bug. Set the overlay MTU correctly and the tax becomes invisible; ignore it and you get the worst kind of failure, the intermittent one keyed to payload size.
- DNS scope is still per-network. The overlay does not flatten name resolution globally — a container only resolves names on overlay networks it shares. Cross-host does not mean cross-network.
- Underlay must allow VXLAN. UDP 4789 between host IPs has to be open; a security group or firewall that blocks it leaves the control plane healthy (containers appear attached, names resolve) while the data plane silently drops — connectivity that looks configured but never carries a packet.
- Encapsulation has a cost. Every cross-host packet is wrapped and unwrapped; without NIC offload this is measurable CPU and latency versus same-host L2, which is one reason latency-sensitive peers are worth co-locating.
Together, these four leaks define the failure surface of any overlay deployment: set the MTU, keep DNS scope in mind, verify the underlay transport, and avoid hairpin self-reference. Miss any one of them and you get the intermittent, load-keyed failure the Hook described — where the abstraction looks healthy and the problem is invisible until it isn’t.
Then there is hairpin NAT (NAT loopback): a container trying to reach itself via its own published host-IP:port — common when an app uses one externally-routable URL for everything. The packet must leave to the host, get DNATed, and come back to the origin container, and whether that round trip works depends on the userland-proxy and hairpin routing being in place. It’s the classic “every other container can reach my published port, but I can’t reach my own” puzzle, and the fix is to use the internal name/IP for self-reference, not the published host port.
On a Swarm overlay, container names resolve across hosts and 'docker network inspect' shows every container attached, but cross-host traffic never arrives. Same-host traffic is fine. What is the most likely cause?
- 01Contrast the default bridge, a user-defined bridge, and an overlay for container-to-container communication.
- 02List the four ways an overlay leaks and the diagnosis for each.
Container-to-container networking splits cleanly into same-host (simple, with one powerful trick) and cross-host (an illusion with well-known leaks). On a single host, two containers on the same user-defined bridge talk directly at Layer 2 — their host-side veths share a bridge, so frames switch peer-to-peer with no NAT and no published port, and the embedded DNS resolves names to bridge IPs. The default bridge docker0 is the exception: no DNS service discovery and meant for throwaway runs, not application architecture. The senior move on one host is network-scoped isolation: because a container only reaches peers on a network it’s attached to, you put the database on a back-end network the public proxy never joins, let the API straddle front-end and back-end with one veth and IP per network, and get firewall-grade segmentation from the topology itself — there is no rule to forget and no connectivity to misconfigure, because the unconnected container has no path at all. Across hosts the overlay driver fakes a flat L2 by encapsulating container frames in VXLAN tunneled over UDP 4789 between host IPs, with a distributed control plane tracking which container IP lives on which host. The illusion is convincing and that is the hazard, because it leaks at four predictable seams: the 50-byte VXLAN header cuts effective MTU to 1450, causing intermittent payload-size-keyed stalls when containers still advertise 1500-byte MSS; DNS scope stays per-network, so cross-host is not cross-network; the underlay must actually allow UDP 4789, or the control plane looks healthy while the data plane drops every frame; and hairpin self-connection through a published host port needs special routing that internal-name self-reference avoids entirely. Multi-host networking is not magic and not mysterious once you debug at the leak instead of the application. Now when you see an intermittent timeout that small requests pass and large ones fail, your first move is checking the overlay MTU — not opening a profiler.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.
Apply this
Put this lesson to work on a real build.