* TCP_USER_TIMEOUT, SYN-SENT and tcp_syn_retries @ 2019-09-25 8:46 Marek Majkowski 2019-09-26 15:05 ` Eric Dumazet 0 siblings, 1 reply; 5+ messages in thread From: Marek Majkowski @ 2019-09-25 8:46 UTC (permalink / raw) To: netdev Hello my favorite mailing list! Recently I've been looking into TCP_USER_TIMEOUT and noticed some strange behaviour on fresh sockets in SYN-SENT state. Full writeup: https://blog.cloudflare.com/when-tcp-sockets-refuse-to-die/ Here's a reproducer. It does a simple thing: sets TCP_USER_TIMEOUT and does connect() to a blackholed IP: $ wget https://gist.githubusercontent.com/majek/b4ad53c5795b226d62fad1fa4a87151a/raw/cbb928cb99cd6c5aa9f73ba2d3bc0aef22fbc2bf/user-timeout-and-syn.py $ sudo python3 user-timeout-and-syn.py 00:00.000000 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] 00:01.007053 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] 00:03.023051 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] 00:05.007096 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] 00:05.015037 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] 00:05.023020 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] 00:05.034983 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] The connect() times out with ETIMEDOUT after 5 seconds - as intended. But Linux (5.3.0-rc3) does something weird on the network - it sends remaining tcp_syn_retries packets aligned to the 5s mark. In other words: with TCP_USER_TIMEOUT we are sending spurious SYN packets on a timeout. For the record, the man page doesn't define what TCP_USER_TIMEOUT does on SYN-SENT state. Cheers, Marek ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: TCP_USER_TIMEOUT, SYN-SENT and tcp_syn_retries 2019-09-25 8:46 TCP_USER_TIMEOUT, SYN-SENT and tcp_syn_retries Marek Majkowski @ 2019-09-26 15:05 ` Eric Dumazet 2019-09-26 16:46 ` Eric Dumazet 0 siblings, 1 reply; 5+ messages in thread From: Eric Dumazet @ 2019-09-26 15:05 UTC (permalink / raw) To: Marek Majkowski, netdev On 9/25/19 1:46 AM, Marek Majkowski wrote: > Hello my favorite mailing list! > > Recently I've been looking into TCP_USER_TIMEOUT and noticed some > strange behaviour on fresh sockets in SYN-SENT state. Full writeup: > https://blog.cloudflare.com/when-tcp-sockets-refuse-to-die/ > > Here's a reproducer. It does a simple thing: sets TCP_USER_TIMEOUT and > does connect() to a blackholed IP: > > $ wget https://gist.githubusercontent.com/majek/b4ad53c5795b226d62fad1fa4a87151a/raw/cbb928cb99cd6c5aa9f73ba2d3bc0aef22fbc2bf/user-timeout-and-syn.py > > $ sudo python3 user-timeout-and-syn.py > 00:00.000000 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] > 00:01.007053 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] > 00:03.023051 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] > 00:05.007096 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] > 00:05.015037 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] > 00:05.023020 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] > 00:05.034983 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] > > The connect() times out with ETIMEDOUT after 5 seconds - as intended. > But Linux (5.3.0-rc3) does something weird on the network - it sends > remaining tcp_syn_retries packets aligned to the 5s mark. > > In other words: with TCP_USER_TIMEOUT we are sending spurious SYN > packets on a timeout. > > For the record, the man page doesn't define what TCP_USER_TIMEOUT does > on SYN-SENT state. > Exactly, so far this option has only be used on established flows. Feel free to send patches if you need to override the stack behavior for connection establishment (Same remark for passive side...) Thanks. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: TCP_USER_TIMEOUT, SYN-SENT and tcp_syn_retries 2019-09-26 15:05 ` Eric Dumazet @ 2019-09-26 16:46 ` Eric Dumazet 2019-09-26 16:57 ` Eric Dumazet 0 siblings, 1 reply; 5+ messages in thread From: Eric Dumazet @ 2019-09-26 16:46 UTC (permalink / raw) To: Marek Majkowski, netdev On 9/26/19 8:05 AM, Eric Dumazet wrote: > > > On 9/25/19 1:46 AM, Marek Majkowski wrote: >> Hello my favorite mailing list! >> >> Recently I've been looking into TCP_USER_TIMEOUT and noticed some >> strange behaviour on fresh sockets in SYN-SENT state. Full writeup: >> https://blog.cloudflare.com/when-tcp-sockets-refuse-to-die/ >> >> Here's a reproducer. It does a simple thing: sets TCP_USER_TIMEOUT and >> does connect() to a blackholed IP: >> >> $ wget https://gist.githubusercontent.com/majek/b4ad53c5795b226d62fad1fa4a87151a/raw/cbb928cb99cd6c5aa9f73ba2d3bc0aef22fbc2bf/user-timeout-and-syn.py >> >> $ sudo python3 user-timeout-and-syn.py >> 00:00.000000 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] >> 00:01.007053 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] >> 00:03.023051 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] >> 00:05.007096 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] >> 00:05.015037 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] >> 00:05.023020 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] >> 00:05.034983 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] >> >> The connect() times out with ETIMEDOUT after 5 seconds - as intended. >> But Linux (5.3.0-rc3) does something weird on the network - it sends >> remaining tcp_syn_retries packets aligned to the 5s mark. >> >> In other words: with TCP_USER_TIMEOUT we are sending spurious SYN >> packets on a timeout. >> >> For the record, the man page doesn't define what TCP_USER_TIMEOUT does >> on SYN-SENT state. >> > > Exactly, so far this option has only be used on established flows. > > Feel free to send patches if you need to override the stack behavior > for connection establishment (Same remark for passive side...) Also please take a look at TCP_SYNCNT, which predates TCP_USER_TIMEOUT ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: TCP_USER_TIMEOUT, SYN-SENT and tcp_syn_retries 2019-09-26 16:46 ` Eric Dumazet @ 2019-09-26 16:57 ` Eric Dumazet 2019-09-26 18:03 ` Eric Dumazet 0 siblings, 1 reply; 5+ messages in thread From: Eric Dumazet @ 2019-09-26 16:57 UTC (permalink / raw) To: Eric Dumazet, Marek Majkowski, netdev On 9/26/19 9:46 AM, Eric Dumazet wrote: > > > On 9/26/19 8:05 AM, Eric Dumazet wrote: >> >> >> On 9/25/19 1:46 AM, Marek Majkowski wrote: >>> Hello my favorite mailing list! >>> >>> Recently I've been looking into TCP_USER_TIMEOUT and noticed some >>> strange behaviour on fresh sockets in SYN-SENT state. Full writeup: >>> https://blog.cloudflare.com/when-tcp-sockets-refuse-to-die/ >>> >>> Here's a reproducer. It does a simple thing: sets TCP_USER_TIMEOUT and >>> does connect() to a blackholed IP: >>> >>> $ wget https://gist.githubusercontent.com/majek/b4ad53c5795b226d62fad1fa4a87151a/raw/cbb928cb99cd6c5aa9f73ba2d3bc0aef22fbc2bf/user-timeout-and-syn.py >>> >>> $ sudo python3 user-timeout-and-syn.py >>> 00:00.000000 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] >>> 00:01.007053 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] >>> 00:03.023051 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] >>> 00:05.007096 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] >>> 00:05.015037 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] >>> 00:05.023020 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] >>> 00:05.034983 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] >>> >>> The connect() times out with ETIMEDOUT after 5 seconds - as intended. >>> But Linux (5.3.0-rc3) does something weird on the network - it sends >>> remaining tcp_syn_retries packets aligned to the 5s mark. >>> >>> In other words: with TCP_USER_TIMEOUT we are sending spurious SYN >>> packets on a timeout. >>> >>> For the record, the man page doesn't define what TCP_USER_TIMEOUT does >>> on SYN-SENT state. >>> >> >> Exactly, so far this option has only be used on established flows. >> >> Feel free to send patches if you need to override the stack behavior >> for connection establishment (Same remark for passive side...) > > Also please take a look at TCP_SYNCNT, which predates TCP_USER_TIMEOUT > > I will test the following : diff --git a/net/ipv4/tcp_timer.c b/net/ipv4/tcp_timer.c index dbd9d2d0ee63aa46ad2dda417da6ec9409442b77..1182e51a6b794d75beb8c130354d7804fc83a307 100644 --- a/net/ipv4/tcp_timer.c +++ b/net/ipv4/tcp_timer.c @@ -220,7 +220,6 @@ static int tcp_write_timeout(struct sock *sk) sk_rethink_txhash(sk); } retry_until = icsk->icsk_syn_retries ? : net->ipv4.sysctl_tcp_syn_retries; - expired = icsk->icsk_retransmits >= retry_until; } else { if (retransmits_timed_out(sk, net->ipv4.sysctl_tcp_retries1, 0)) { /* Black hole detection */ @@ -242,9 +241,9 @@ static int tcp_write_timeout(struct sock *sk) if (tcp_out_of_resources(sk, do_reset)) return 1; } - expired = retransmits_timed_out(sk, retry_until, - icsk->icsk_user_timeout); } + expired = retransmits_timed_out(sk, retry_until, + icsk->icsk_user_timeout); tcp_fastopen_active_detect_blackhole(sk, expired); if (BPF_SOCK_OPS_TEST_FLAG(tp, BPF_SOCK_OPS_RTO_CB_FLAG)) ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: TCP_USER_TIMEOUT, SYN-SENT and tcp_syn_retries 2019-09-26 16:57 ` Eric Dumazet @ 2019-09-26 18:03 ` Eric Dumazet 0 siblings, 0 replies; 5+ messages in thread From: Eric Dumazet @ 2019-09-26 18:03 UTC (permalink / raw) To: Marek Majkowski, netdev On 9/26/19 9:57 AM, Eric Dumazet wrote: > > > On 9/26/19 9:46 AM, Eric Dumazet wrote: >> >> >> On 9/26/19 8:05 AM, Eric Dumazet wrote: >>> >>> >>> On 9/25/19 1:46 AM, Marek Majkowski wrote: >>>> Hello my favorite mailing list! >>>> >>>> Recently I've been looking into TCP_USER_TIMEOUT and noticed some >>>> strange behaviour on fresh sockets in SYN-SENT state. Full writeup: >>>> https://blog.cloudflare.com/when-tcp-sockets-refuse-to-die/ >>>> >>>> Here's a reproducer. It does a simple thing: sets TCP_USER_TIMEOUT and >>>> does connect() to a blackholed IP: >>>> >>>> $ wget https://gist.githubusercontent.com/majek/b4ad53c5795b226d62fad1fa4a87151a/raw/cbb928cb99cd6c5aa9f73ba2d3bc0aef22fbc2bf/user-timeout-and-syn.py >>>> >>>> $ sudo python3 user-timeout-and-syn.py >>>> 00:00.000000 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] >>>> 00:01.007053 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] >>>> 00:03.023051 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] >>>> 00:05.007096 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] >>>> 00:05.015037 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] >>>> 00:05.023020 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] >>>> 00:05.034983 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S] >>>> >>>> The connect() times out with ETIMEDOUT after 5 seconds - as intended. >>>> But Linux (5.3.0-rc3) does something weird on the network - it sends >>>> remaining tcp_syn_retries packets aligned to the 5s mark. >>>> >>>> In other words: with TCP_USER_TIMEOUT we are sending spurious SYN >>>> packets on a timeout. >>>> >>>> For the record, the man page doesn't define what TCP_USER_TIMEOUT does >>>> on SYN-SENT state. >>>> >>> >>> Exactly, so far this option has only be used on established flows. >>> >>> Feel free to send patches if you need to override the stack behavior >>> for connection establishment (Same remark for passive side...) >> >> Also please take a look at TCP_SYNCNT, which predates TCP_USER_TIMEOUT >> >> > > I will test the following : > > diff --git a/net/ipv4/tcp_timer.c b/net/ipv4/tcp_timer.c > index dbd9d2d0ee63aa46ad2dda417da6ec9409442b77..1182e51a6b794d75beb8c130354d7804fc83a307 100644 > --- a/net/ipv4/tcp_timer.c > +++ b/net/ipv4/tcp_timer.c > @@ -220,7 +220,6 @@ static int tcp_write_timeout(struct sock *sk) > sk_rethink_txhash(sk); > } > retry_until = icsk->icsk_syn_retries ? : net->ipv4.sysctl_tcp_syn_retries; > - expired = icsk->icsk_retransmits >= retry_until; > } else { > if (retransmits_timed_out(sk, net->ipv4.sysctl_tcp_retries1, 0)) { > /* Black hole detection */ > @@ -242,9 +241,9 @@ static int tcp_write_timeout(struct sock *sk) > if (tcp_out_of_resources(sk, do_reset)) > return 1; > } > - expired = retransmits_timed_out(sk, retry_until, > - icsk->icsk_user_timeout); > } > + expired = retransmits_timed_out(sk, retry_until, > + icsk->icsk_user_timeout); > tcp_fastopen_active_detect_blackhole(sk, expired); > > if (BPF_SOCK_OPS_TEST_FLAG(tp, BPF_SOCK_OPS_RTO_CB_FLAG)) > The patch works well, but reading again the man page, I see the existing behavior as been clearly documented. If we change the behavior, we might break applications that were setting TCP_USER_TIMEOUT on the listener, expecting the value to b inherited to children at accept() time but not expecting to change SYNACK rtx behavior. On the other hand, John Maxell patch (tcp: Add tcp_clamp_rto_to_user_timeout() helper to improve accuracy) has added this weird effect of sending remaining SYN every jiffie remaining = icsk->icsk_user_timeout - elapsed; if (remaining <= 0) return 1; /* user timeout has passed; fire ASAP */ So we probably just should extend TCP_USER_TIMEOUT to SYN_SENT/SYN_RECV states and change the man page accordingly. TCP_USER_TIMEOUT (since Linux 2.6.37) This option takes an unsigned int as an argument. When the value is greater than 0, it specifies the maximum amount of time in millisec‐ onds that transmitted data may remain unacknowledged before TCP will forcibly close the corresponding connection and return ETIMEDOUT to the application. If the option value is specified as 0, TCP will to use the system default. Increasing user timeouts allows a TCP connection to survive extended periods without end-to-end connectivity. Decreasing user timeouts allows applications to "fail fast", if so desired. Otherwise, fail‐ ure may take up to 20 minutes with the current system defaults in a normal WAN environment. This option can be set during any state of a TCP connection, but is effective only during the synchronized states of a connection (ESTAB‐ LISHED, FIN-WAIT-1, FIN-WAIT-2, CLOSE-WAIT, CLOSING, and LAST-ACK). Moreover, when used with the TCP keepalive (SO_KEEPALIVE) option, TCP_USER_TIMEOUT will override keepalive to determine when to close a connection due to keepalive failure. The option has no effect on when TCP retransmits a packet, nor when a keepalive probe is sent. This option, like many others, will be inherited by the socket re‐ turned by accept(2), if it was set on the listening socket. Further details on the user timeout feature can be found in RFC 793 and RFC 5482 ("TCP User Timeout Option"). ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2019-09-26 18:04 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-09-25 8:46 TCP_USER_TIMEOUT, SYN-SENT and tcp_syn_retries Marek Majkowski 2019-09-26 15:05 ` Eric Dumazet 2019-09-26 16:46 ` Eric Dumazet 2019-09-26 16:57 ` Eric Dumazet 2019-09-26 18:03 ` Eric Dumazet
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).