The TCP RST mystery — when a load balancer kills long-lived sessions
Users saw random 502s from an internal API. tcpdump on the LB and backend told two different stories.
1. Reproduce on a client and start a capture on both LB and backend.
$ sudo tcpdump -i any -w /tmp/lb.pcap 'host 10.20.4.11 and tcp port 8080'Run on the LB; mirror on backend to /tmp/be.pcap.
2. Open in Wireshark, filter tcp.flags.reset == 1 — RSTs originate from LB, backend never sends FIN.
3. Compare Follow-TCP-Stream on both sides: gap of ~65 s before RST.
$ ss -tinp | grep 8080Backend sockets are ESTABLISHED for 5+ min — connections are healthy.
$ aws elbv2 describe-target-group-attributes --target-group-arn ...deregistration_delay.timeout_seconds = 60, idle_timeout = 60.
4. Confirm with a synthetic session that idle > 60 s always triggers RST from LB.
Application used persistent keep-alive with a 300 s ping. The LB idle-timeout was 60 s, so it silently reaped the flow and RST'd on the next client packet.
Lower app keep-alive interval to 30 s AND raise LB idle-timeout to 350 s to match backend. Add CloudWatch alarm on TargetResetCount.
