Networking & Protocols
IP packets: trace, decode, and fix a path-MTU black hole
Reading about TTL, longest-prefix match, and PMTUD black holes is not the same as watching them in a live capture. Build a small lab, decode the bytes IP actually puts on the wire, map a real path, then deliberately create the most common silent IP-layer failure — a path-MTU black hole — and fix it the way an on-call engineer would, with evidence at every step.
Turn the unit’s mental model into a reproducible diagnostic loop: capture and decode an IP header by hand, trace a path and read its TTL behaviour, reproduce a PMTUD black hole in a controlled tunnel, and prove the MSS-clamp fix with before/after captures.
Stand up a small networking lab (two hosts plus a low-MTU tunnel — WireGuard, a network namespace, or two containers), then use it to decode IP headers from a live capture, map a path with traceroute, reproduce a PMTUD black hole, and fix it without relying on ICMP — proving each finding from packet captures and command output, never from memory.
- A byte-annotated decode of both a real IPv4 and a real IPv6 header, each field tied to the captured bytes.
- Traceroute/mtr output with a written explanation of the TTL mechanism and a correct call on at least one asterisk/loss hop (ICMP withholding vs real data-plane loss).
- `ip route get` output for several destinations showing longest-prefix match selecting the most specific prefix and the default route as last resort.
- Before/after evidence of the PMTUD black hole: the failing large-transfer capture with repeated retransmits, then a passing capture after MSS clamping — with ICMP still blocked, proving the fix is ICMP-independent.
- Add a one-page on-call runbook: triage steps for 'small requests work, large ones hang', the diagnostic commands, and the MSS-clamp vs PLPMTUD vs unblock-ICMP decision.
- Inspect BGP and RPKI for a prefix you control or a public one: use a looking-glass to view its announcement from a remote AS and check its RPKI validity (e.g. via a public validator), then explain what an 'invalid' state would do to reachability.
- Demonstrate a NAT-traversal limitation: put one host behind a double NAT, show an unsolicited inbound connection fails, and that a relay (TURN-style) or IPv6 makes it work.
- Capture and explain a spoofing/amplification scenario in the lab (a small request producing a large response) and describe how BCP 38 egress filtering at the source network would stop it.
This is the loop you will run in every real IP-layer incident: capture first and decode the actual bytes, map the path with TTL-based traceroute and read its quirks correctly, resolve forwarding by longest-prefix match, and when ‘small works, large hangs’ appears, recognise the PMTUD black hole and fix it with MSS clamping rather than betting on ICMP surviving the path. Doing it once in a controlled lab turns the production version into muscle memory.