Listening TCP server ignores SYN or ACK for new connection handshake

Environment

• Red Hat Enterprise Linux
• TCP listening process

Issue

• Listening server ignores SYN for new TCP handshake
• Listening TCP server ignores ACK for new connection
• RHEL 5 and RHEL 6: The following stats are seen with the same value in netstat -s:

TcpExt:
    XXX times the listen queue of a socket overflowed
    XXX SYNs to LISTEN sockets ignored

• or nstat -a:

TcpExtListenOverflows
TcpExtListenDrops

• RHEL 7: The value of times the listen queue of a socket overflowed (TcpExtListenOverflows) may be larger than the value of SYNs to LISTEN sockets ignored (TcpExtListenDrops)

Resolution

Improve application accept() performance.

Increase the kernel maximum backlog (net.core.somaxconn) and application socket backlog as described in this article. Note that increasing this queue size will only help in the case of transient overloads. For continuous overload, higher connection latency will result, and connection requests will still be ignored.

Root Cause

Each listening TCP socket has a "listen backlog" which is the amount of un-handshaken connections which can sit waiting for the kernel to perform the TCP handshake.

The listen backlog is limited by the net.core.somaxconn kernel tunable, and the second parameter of the listen(int sockfd, int backlog) system call.

Each socket also has an "accept backlog" which is the amount of handshaken connections which can sit waiting for the application to call accept() and move the connection to a new socket file descriptor. If the application is busy and has not yet called accept() when the handshake completes then the connection may be queued if this backlog has not been reached.

There is no mechanism to set the size of the accept backlog, it is set to the same value as the listen backlog.

When a new connection request (i.e. a new SYN) comes in, the kernel checks to see if the accept backlog is full. If so, the new SYN is silently discarded.

On RHEL 5 and RHEL 6, the times the listen queue of a socket overflowed (TcpExtListenOverflows) and SYNs to LISTEN sockets ignored (TcpExtListenDrops) values are grown when the accept backlog overflows, but no counter is grown when a new SYN arrives and is silently discarded.

On RHEL 7, code review shows the times the listen queue of a socket overflowed (TcpExtListenOverflows) counter is grown when a SYN is discarded due to full accept backlog, so it could be expected for this counter to be higher on RHEL 7.

Diagnostic Steps

During this issue, inspect the socket backlog with ss -ntl.

The Send-Q column shows the socket backlog

The Recv-Q column shows the current number of connections in the backlog

If the Recv-Q value is approaching, equal to , or greater than the Send-Q value, this is confirmed to match the root cause.

The following example shows a process with a listen backlog of 5, and 6 connections waiting to be accepted, so the TCP server accept performance is too poor:

# ss -ntl '( sport = :9001 )'
State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              
LISTEN     6      5           :::9001                    :::*                  

Silent discard of SYN in RHEL 5.11 kernel source:

net/ipv4/ip_input.c

199 static inline int ip_local_deliver_finish(struct sk_buff *skb)
200 {

236                         ret = ipprot->handler(skb);

241                         IP_INC_STATS_BH(IPSTATS_MIB_INDELIVERS);

251                         kfree_skb(skb);

net/ipv4/tcp_ipv4.c

1631 int tcp_v4_rcv(struct sk_buff *skb)
1632 {

1122         if (!sock_owned_by_user(sk)) {

1128                         ret = tcp_v4_do_rcv(sk, skb);

1143         return ret;

1010 int tcp_v4_do_rcv(struct sock *sk, struct sk_buff *skb)
1011 {

1023         if (sk->sk_state == TCP_LISTEN) {
1024                 struct sock *nsk = tcp_v4_hnd_req(sk, skb);

1028                 if (nsk != sk) {
1029                         if (tcp_child_process(sk, nsk, skb))
1030                                 goto reset;
1031                         return 0;

650 int tcp_child_process(struct sock *parent, struct sock *child,
651                       struct sk_buff *skb)
652 {

656         if (!sock_owned_by_user(child)) {
657                 ret = tcp_rcv_state_process(child, skb, skb->h.th, skb->len);

672         return ret;
673 }                

net/ipv4/tcp_input.c

4422 int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb,
4423                           struct tcphdr *th, unsigned len)
4424 {

4442                 if(th->syn) {

4445                         if (icsk->icsk_af_ops->conn_request(sk, skb) < 0)
4446                                 return 1;

4465                         kfree_skb(skb);
4466                         return 0;
4467                 }

net/ipv4/tcp_ipv4.c

 739 int tcp_v4_conn_request(struct sock *sk, struct sk_buff *skb)
 740 {

 772         /* Accept backlog is full. If we have already queued enough
 773          * of warm entries in syn queue, drop request. It is better than
 774          * clogging syn queue with openreqs with exponentially increasing
 775          * timeout.    
 776          */
 777         if (sk_acceptq_is_full(sk) && inet_csk_reqsk_queue_young(sk) > 1)
 778                 goto drop;

 884 drop:
 885         return 0;
 886 }

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.