Re: TCP sender stuck in persist despite peer advertising non-zero window

From: Neal Cardwell <ncardwell@google.com>
To: Apollon Oikonomopoulos <apoikos@dmesg.gr>
Cc: Netdev <netdev@vger.kernel.org>,
	Eric Dumazet <edumazet@google.com>,
	Yuchung Cheng <ycheng@google.com>,
	Soheil Hassas Yeganeh <soheil@google.com>
Subject: Re: TCP sender stuck in persist despite peer advertising non-zero window
Date: Thu, 15 Oct 2020 16:22:21 -0400	[thread overview]
Message-ID: <CADVnQym6OPVRcJ6PdR3hjN5Krcn0pugshdLZsrnzNQe1c52HXA@mail.gmail.com> (raw)
In-Reply-To: <87eelz4abk.fsf@marvin.dmesg.gr>

On Thu, Oct 15, 2020 at 2:31 PM Apollon Oikonomopoulos <apoikos@dmesg.gr> wrote:
>
> Hi,
>
> I'm trying to debug a (possible) TCP issue we have been encountering
> sporadically during the past couple of years. Currently we're running
> 4.9.144, but we've been observing this since at least 3.16.
>
> Tl;DR: I believe we are seeing a case where snd_wl1 fails to be properly
> updated, leading to inability to recover from a TCP persist state and
> would appreciate some help debugging this.

Thanks for the detailed report and diagnosis. I think we may need a
fix something like the following patch below.

Eric/Yuchung/Soheil, what do you think?

commit 42b37c72aa73aaabd0c01b8c05c2205236279021
Author: Neal Cardwell <ncardwell@google.com>
Date:   Thu Oct 15 16:06:11 2020 -0400

    tcp: fix to update snd_wl1 in bulk receiver fast path

    In the header prediction fast path for a bulk data receiver, if no
    data is newly acknowledged then we do not call tcp_ack() and do not
    call tcp_ack_update_window(). This means that a bulk receiver that
    receives large amounts of data can have the incoming sequence numbers
    wrap, so that the check in tcp_may_update_window fails:
       after(ack_seq, tp->snd_wl1)

    The fix is to update snd_wl1 in the header prediction fast path for a
    bulk data receiver, so that it keeps up and does not see wrapping
    problems.

    Signed-off-by: Neal Cardwell <ncardwell@google.com>
    Reported-By: Apollon Oikonomopoulos <apoikos@dmesg.gr>

diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index b1ce2054291d..75be97f6a7da 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -5766,6 +5766,8 @@ void tcp_rcv_established(struct sock *sk, struct
sk_buff *skb)
                                tcp_data_snd_check(sk);
                                if (!inet_csk_ack_scheduled(sk))
                                        goto no_ack;
+                       } else {
+                               tcp_update_wl(tp, TCP_SKB_CB(skb)->seq);
                        }

                        __tcp_ack_snd_check(sk, 0);

neal
