Diagnose Linux TCP Retransmissions and Socket Queue Pressure

Admin

October 11, 2026

12 min read

Application timeouts do not automatically mean packet loss. A slow receiver, saturated listener, wrong route, overloaded NIC, or downstream dependency can create similar symptoms. This runbook uses Linux TCP state and packet evidence to locate the failing part of one client/server connection.

Identify the Flow and Its Route

Scope

Record the application’s error, request concurrency, peer address/port, and whether the failure is during connect, TLS negotiation, or data transfer. Inspect the route and source address in the same namespace as the application. For a container, the host route is not necessarily its route. Keep these variables in one Bash session, and choose the interface reported for the actual traffic path.

peer='192.0.2.50'   # Replace with the real peer
port='443'
ip route get "$peer"
ip -br address
ss -s
# For a known container/worker PID, inspect that network namespace:
# sudo nsenter -t REAL_PID -n ip route get "$peer"

Inspect Established Sockets and the Listener

Socket Evidence

Use ss to inspect the target sockets during a slow request. Internal TCP information can include RTT, retransmissions, congestion window, and delivery-rate estimates; fields vary with the kernel and iproute2 versions. For established sockets, Recv-Q represents unread received data and Send-Q data not yet acknowledged. For a listening socket, the queue columns describe pending connections and the configured backlog limit. These have different meanings, so do not compare them as bytes.

sudo ss -tinp dst "$peer"
# On the receiving server, replace the listening port if different:
sudo ss -lntp 'sport = :443'
ss -nt state syn-sent
ss -nt state syn-recv

❯ What to Verify

Look for changing retransmission/RTT values on the incident's socket,
unread receive data, unacknowledged send data, or a growing listener queue.
A high lifetime retransmission count alone cannot date the incident.

Sample Host TCP Counters Over the Same Interval

Counter Deltas

Take two absolute snapshots and compare their differences. TcpRetransSegs covers retransmitted TCP segments; TCPSynRetrans helps separate connection-establishment retries. ListenOverflows and ListenDrops can support an overloaded-listener hypothesis, although their relationship depends on the kernel’s failure path. These counters aggregate traffic in the current network namespace, so correlate them with the flow and other concurrent workloads.

for sample in 1 2; do
  date -u
  nstat -az TcpRetransSegs TcpOutSegs TcpExtTCPSynRetrans \
    TcpExtListenOverflows TcpExtListenDrops
  test "$sample" -eq 2 || sleep 10
done

Check NIC Drops and the Return-Path Design

Network Path

Read interface counters at both endpoints before and after the same request window. Check switch-port statistics, routing changes, and the firewall’s forward/return handling. Driver statistics are vendor-specific, and some drops can occur before a packet reaches tcpdump. Inspect reverse-path-filter settings when asymmetric routing is suspected, but do not disable them globally. Match source routes and security policy with the network owner.

iface='ens192' # Actual interface from Step 1
ip -s link show dev "$iface"
sudo ethtool -S "$iface"
sysctl net.ipv4.conf.all.rp_filter "net.ipv4.conf.$iface.rp_filter"
# Repeat on the peer with its own interface and route back to this client.

Capture One Bounded Reproduction

Packet Evidence

Capture only the approved peer/port, for 30 seconds or 5,000 packets, whichever limit is reached first. Use a fresh directory and a restricted umask. This example retains root with -Z root so tcpdump can write in the root-owned directory; use an approved capture account/path instead if your host policy requires privilege dropping. A 128-byte snapshot limits stored payload but does not guarantee that the capture contains no sensitive information. Repeat at the peer with aligned clocks when possible. A timeout exit status can be expected when the time limit ends the capture; inspect tcpdump’s captured/dropped counts.

sudo sh -c 'umask 077; mkdir /var/tmp/tcp-incident01' || exit 1
sudo timeout -s INT 30 tcpdump -Z root -i "$iface" -nn -s 128 -c 5000 \
  -w /var/tmp/tcp-incident01/flow.pcap \
  "host $peer and tcp port $port"
sudo tcpdump -nn -tttt -r /var/tmp/tcp-incident01/flow.pcap -c 20

Separate Handshake Loss, Data Loss, and Receiver Pressure

Interpretation

Repeated SYNs without a SYN/ACK suggest a connection-path or listener problem; check whether the peer receives the SYN and sends a response. Repeated data sequence ranges with duplicate ACKs/SACK information support loss or reordering after establishment. A zero advertised receive window points to receiver-side pressure rather than ordinary congestion loss. Compare both endpoint captures to locate the missing direction. Checksum warnings and oversized packets in a host capture may result from offloads, so confirm with another capture point before declaring wire corruption.

# Numeric addresses, timestamps, and TCP sequence information for offline review:
sudo tcpdump -nn -tttt -S -r /var/tmp/tcp-incident01/flow.pcap
# In Wireshark, inspect the single flow and packet sequence alongside both captures.
# Automated "retransmission" labels are hypotheses, not the final root cause.

Apply One Repair at the Identified Boundary

Controlled Change

For NIC or switch drops, repair the driver/fabric issue with its owner. For asymmetric returns, correct the route or stateful policy. For listener overflow, investigate the application’s accept rate, worker saturation, and documented listen backlog before raising kernel limits. For receiver pressure, measure application reads, locks, CPU, and downstream storage. Capture existing settings and a rollback before any canary adjustment; increasing a limit can move pressure elsewhere.

# Inspect existing limits; these commands do not tune them.
sysctl net.core.somaxconn net.ipv4.tcp_max_syn_backlog
sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem
# On the server, correlate with the actual service and incident time:
sudo journalctl -u app.service --since '30 minutes ago' --no-pager

Repeat the Workload and Validate the Whole Transaction

Closeout

Repeat the same concurrency and data set, checking errors, latency distribution, queue pressure, and counter deltas over the same interval. Use a second bounded capture only if needed to confirm the unresolved path. A successful connect or one fast request does not establish that sustained application transactions recovered. Retain flow identifiers and the comparison so later kernel or network changes can be evaluated consistently.

ss -tin dst "$peer"
nstat -az TcpRetransSegs TcpExtTCPSynRetrans \
  TcpExtListenOverflows TcpExtListenDrops
ip -s link show dev "$iface"
# Attach the matching application error/latency interval to these samples.