All of lore.kernel.org
 help / color / mirror / Atom feed
From: Neal Cardwell <ncardwell@google.com>
To: Gil Pedersen <kanongil@gmail.com>
Cc: David Miller <davem@davemloft.net>,
	Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>,
	dsahern@kernel.org, Netdev <netdev@vger.kernel.org>,
	Yuchung Cheng <ycheng@google.com>,
	Eric Dumazet <edumazet@google.com>
Subject: Re: TCP stall issue
Date: Tue, 23 Feb 2021 10:41:31 -0500	[thread overview]
Message-ID: <CADVnQy=G=GU1USyEcGA_faJg5L-wLO6jS4EUocrVsjqkaGbvYw@mail.gmail.com> (raw)
In-Reply-To: <35A4DDAA-7E8D-43CB-A1F5-D1E46A4ED42E@gmail.com>

On Tue, Feb 23, 2021 at 5:13 AM Gil Pedersen <kanongil@gmail.com> wrote:
>
> Hi,
>
> I am investigating a TCP stall that can occur when sending to an Android device (kernel 4.9.148) from an Ubuntu server running kernel 5.11.0.
>
> The issue seems to be that RACK is not applied when a D-SACK (with SACK) is received on the server after an RTO re-transmission (CA_Loss state). Here the re-transmitted segment is considered to be already delivered and loss undo logic is applied. Then nothing is re-transmitted until the next RTO, where the next segment is sent and the same thing happens again. The causes the retransmitted segments to be delivered at a rate of ~1 per second, so a burst loss of eg. 20 segments cause a 20+ second stall. I would expect RACK to kick in long before this happens.
>
> Note the D-SACK should not be considered spurious, as the TSecr value matches the re-transmission TSval.
>
> Also, the Android receiver is definitely sending strange D-SACKs that does not properly advance the ACK number to include received segments. However, I can't control it and need to fix it on the server by quickly re-transmitting the segments. The connection itself is functional. If the client makes a request to the server in this state, it can respond and the client will receive any segments sent in reply.
>
> I can see from counters that TcpExtTCPLossUndo & TcpExtTCPSackFailures are incremented on the server when this happens.
> The issue appears both with F-RTO enabled and disabled. Also appears both with BBR and RENO.
>
> Any idea of why this happens, or suggestions on how to debug the issue further?
>
> /Gil

Thanks for the detailed report! It sounds like you have a trace. Can
you please attach (or post the URL of) a binary tcpdump .pcap trace
that illustrates the problem, to make sure we can understand and
reproduce the issue?

thanks,
neal

  reply	other threads:[~2021-02-23 15:42 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-02-23 10:09 TCP stall issue Gil Pedersen
2021-02-23 15:41 ` Neal Cardwell [this message]
2021-02-24 10:03   ` Gil Pedersen
2021-02-24 14:55     ` Neal Cardwell
2021-02-24 15:36       ` Gil Pedersen
2021-02-25 15:05         ` Neal Cardwell
2021-02-26 14:39           ` David Laight
2021-02-26 16:26             ` Gil Pedersen
2021-02-26 17:50               ` Neal Cardwell
2021-02-26 21:59                 ` Maciej Żenczykowski

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CADVnQy=G=GU1USyEcGA_faJg5L-wLO6jS4EUocrVsjqkaGbvYw@mail.gmail.com' \
    --to=ncardwell@google.com \
    --cc=davem@davemloft.net \
    --cc=dsahern@kernel.org \
    --cc=edumazet@google.com \
    --cc=kanongil@gmail.com \
    --cc=netdev@vger.kernel.org \
    --cc=ycheng@google.com \
    --cc=yoshfuji@linux-ipv6.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.