DEBUG JOURNAL

Packet-Level Troubleshooting

Five real production incidents, reconstructed from the tcpdump / Wireshark evidence — with the exact commands, screenshots and fixes our engineers used to close them out.

#01HighTCP
12 min read

The TCP RST mystery — when a load balancer kills long-lived sessions

Users saw random 502s from an internal API. tcpdump on the LB and backend told two different stories.

Step-by-step
  1. 1. Reproduce on a client and start a capture on both LB and backend.

  2. $ sudo tcpdump -i any -w /tmp/lb.pcap 'host 10.20.4.11 and tcp port 8080'

    Run on the LB; mirror on backend to /tmp/be.pcap.

  3. 2. Open in Wireshark, filter tcp.flags.reset == 1 — RSTs originate from LB, backend never sends FIN.

  4. 3. Compare Follow-TCP-Stream on both sides: gap of ~65 s before RST.

  5. $ ss -tinp | grep 8080

    Backend sockets are ESTABLISHED for 5+ min — connections are healthy.

  6. $ aws elbv2 describe-target-group-attributes --target-group-arn ...

    deregistration_delay.timeout_seconds = 60, idle_timeout = 60.

  7. 4. Confirm with a synthetic session that idle > 60 s always triggers RST from LB.

Root cause

Application used persistent keep-alive with a 300 s ping. The LB idle-timeout was 60 s, so it silently reaped the flow and RST'd on the next client packet.

Fix & prevention

Lower app keep-alive interval to 30 s AND raise LB idle-timeout to 350 s to match backend. Add CloudWatch alarm on TargetResetCount.

#02CriticalMTU
9 min read

The 1472-byte black hole — MTU mismatch across an IPsec tunnel

Small HTTP requests worked; large POSTs hung forever. Classic PMTUD failure across a site-to-site IPsec.

Step-by-step
  1. $ ping -M do -s 1472 10.99.0.10

    Fails with 'Message too long' — anything over 1418 bytes is dropped.

  2. $ traceroute --mtu 10.99.0.10

    MTU narrows from 1500 → 1438 at the VPN gateway.

  3. Capture on the WAN egress to see ICMP-Frag-Needed being dropped by an intermediate firewall.

  4. $ sudo tcpdump -i eth0 'icmp[icmptype] == 3'

    Zero packets — PMTUD is being black-holed upstream.

  5. $ iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

    Apply MSS clamp on both VPN endpoints.

Root cause

IPsec + GRE overhead reduced path MTU to 1418, and an ISP firewall stripped ICMP Type 3 Code 4. PMTUD never converged and large TCP segments were dropped silently.

Fix & prevention

Clamp TCP MSS to 1378 on both VPN peers; allow ICMP unreachable end-to-end; document effective MTU in the runbook.

#03CriticalBGP
14 min read

Chasing a 30-second BGP flap across a datacenter fabric

Two leaf switches lost eBGP session every 30 s. Not a link problem — the packet capture rewrote the theory.

Step-by-step
  1. $ show bgp neighbors 10.0.2.1 | i BFD|hold|state

    Session dropping with 'Hold Timer Expired', BFD not enabled.

  2. $ monitor capture LEAF-BGP interface Eth1/49 both match ipv4 host 10.0.2.1

    Start on-box capture on both leafs.

  3. Wireshark shows KEEPALIVE from Leaf-A leaves every 10 s, but only every 3rd frame arrives at Leaf-B — 66% loss.

  4. $ show interfaces Eth1/49 counters errors

    CRC errors incrementing at ~4/s — bad optic or fiber.

  5. $ show interfaces transceiver detail | i Rx|dBm

    Rx power -14.2 dBm; SFP threshold -12 dBm. Optic is failing.

Root cause

SFP-10G-LR on Leaf-A Eth1/49 had degraded Rx below sensitivity. Enough traffic got through for LLDP and interface up/down, but not enough for BGP keepalives to survive.

Fix & prevention

Swap the SFP, add BFD (300 ms × 3) on all fabric peers, alert on Rx-dBm < -11 in Prometheus.

#04HighTLS
10 min read

TLS handshake fails at ClientHello — a cipher-suite forensics case

A partner integration failed only from prod, worked from staging. The pcap surfaced the exact incompatibility.

Step-by-step
  1. $ openssl s_client -connect api.partner.com:443 -tls1_2

    'no cipher match' error.

  2. $ sudo tcpdump -i any -w tls.pcap 'host api.partner.com and port 443'

    Capture from a prod host during the failed call.

  3. Wireshark → Follow SSL Stream → inspect ClientHello: only ECDHE-ECDSA suites offered.

  4. $ openssl x509 -in server.pem -noout -text | grep 'Public Key Algorithm'

    Server cert is RSA — no ECDSA suite can ever match.

  5. $ nmap --script ssl-enum-ciphers -p 443 api.partner.com

    Server supports only ECDHE-RSA + AES-GCM.

Root cause

Prod hosts ran a hardened OpenSSL profile that removed all RSA key-exchange suites. Staging still had the default suite list, hiding the bug.

Fix & prevention

Reintroduce ECDHE-RSA-AES256-GCM-SHA384 in the hardened profile; pin OpenSSL config in configuration management; add pre-deploy tls-scan check.

#05MediumRouting
11 min read

Asymmetric routing killed a stateful firewall — packet capture told the truth

New DR site went live and 30% of sessions started timing out. Ping worked, TCP didn't.

Step-by-step
  1. $ traceroute -T -p 443 10.50.7.20

    Forward path goes via DC-A firewall.

  2. $ on target: sudo tcpdump -i any host <client> and port 443

    Return SYN-ACK leaves via DC-B firewall — different device, different state table.

  3. Firewall on DC-B logs: 'packet not part of any session, drop'.

  4. $ show ip route 10.50.7.0/24

    Two equal-cost paths from the DR site — ECMP picks DC-B for return.

  5. $ route-map ADJUST_METRIC permit 10; set metric 200

    Steer return traffic back through the same firewall.

Root cause

New OSPF adjacency made two paths equal-cost. Return traffic hashed to a firewall that had no session state, so it dropped every SYN-ACK.

Fix & prevention

Introduce metric offset so DR is only used on failover, enable session sync between the two firewalls, and add a synthetic canary that TCP-connects on port 443 every 30 s.