Re: TCP sender stuck in persist despite peer advertising non-zero window

From: Yuchung Cheng <ycheng@google.com>
To: Neal Cardwell <ncardwell@google.com>
Cc: Apollon Oikonomopoulos <apoikos@dmesg.gr>,
	Netdev <netdev@vger.kernel.org>,
	Eric Dumazet <edumazet@google.com>,
	Soheil Hassas Yeganeh <soheil@google.com>
Subject: Re: TCP sender stuck in persist despite peer advertising non-zero window
Date: Thu, 15 Oct 2020 14:39:24 -0700	[thread overview]
Message-ID: <CAK6E8=fCwjP47DvSj4YQQ6xn25bVBN_1mFtrBwOJPYU6jXVcgQ@mail.gmail.com> (raw)
In-Reply-To: <CADVnQym6OPVRcJ6PdR3hjN5Krcn0pugshdLZsrnzNQe1c52HXA@mail.gmail.com>

On Thu, Oct 15, 2020 at 1:22 PM Neal Cardwell <ncardwell@google.com> wrote:
>
> On Thu, Oct 15, 2020 at 2:31 PM Apollon Oikonomopoulos <apoikos@dmesg.gr> wrote:
> >
> > Hi,
> >
> > I'm trying to debug a (possible) TCP issue we have been encountering
> > sporadically during the past couple of years. Currently we're running
> > 4.9.144, but we've been observing this since at least 3.16.
> >
> > Tl;DR: I believe we are seeing a case where snd_wl1 fails to be properly
> > updated, leading to inability to recover from a TCP persist state and
> > would appreciate some help debugging this.
>
> Thanks for the detailed report and diagnosis. I think we may need a
> fix something like the following patch below.
>
> Eric/Yuchung/Soheil, what do you think?
wow hard to believe how old this bug can be. The patch looks good but
can Apollon verify this patch fix the issue?

>
> commit 42b37c72aa73aaabd0c01b8c05c2205236279021
> Author: Neal Cardwell <ncardwell@google.com>
> Date:   Thu Oct 15 16:06:11 2020 -0400
>
>     tcp: fix to update snd_wl1 in bulk receiver fast path
>
>     In the header prediction fast path for a bulk data receiver, if no
>     data is newly acknowledged then we do not call tcp_ack() and do not
>     call tcp_ack_update_window(). This means that a bulk receiver that
>     receives large amounts of data can have the incoming sequence numbers
>     wrap, so that the check in tcp_may_update_window fails:
>        after(ack_seq, tp->snd_wl1)
>
>     The fix is to update snd_wl1 in the header prediction fast path for a
>     bulk data receiver, so that it keeps up and does not see wrapping
>     problems.
>
>     Signed-off-by: Neal Cardwell <ncardwell@google.com>
>     Reported-By: Apollon Oikonomopoulos <apoikos@dmesg.gr>
>
> diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
> index b1ce2054291d..75be97f6a7da 100644
> --- a/net/ipv4/tcp_input.c
> +++ b/net/ipv4/tcp_input.c
> @@ -5766,6 +5766,8 @@ void tcp_rcv_established(struct sock *sk, struct
> sk_buff *skb)
>                                 tcp_data_snd_check(sk);
>                                 if (!inet_csk_ack_scheduled(sk))
>                                         goto no_ack;
> +                       } else {
> +                               tcp_update_wl(tp, TCP_SKB_CB(skb)->seq);
>                         }
>
>                         __tcp_ack_snd_check(sk, 0);
>
> neal