Re: Linux ECN Handling

From: Steve Ibanez <sibanez@stanford.edu>
To: Neal Cardwell <ncardwell@google.com>
Cc: Eric Dumazet <edumazet@google.com>,
	Yuchung Cheng <ycheng@google.com>,
	Daniel Borkmann <daniel@iogearbox.net>,
	Netdev <netdev@vger.kernel.org>, Florian Westphal <fw@strlen.de>,
	Mohammad Alizadeh <alizadeh@csail.mit.edu>,
	Lawrence Brakmo <brakmo@fb.com>
Subject: Re: Linux ECN Handling
Date: Mon, 4 Dec 2017 21:22:41 -0800	[thread overview]
Message-ID: <CACJspmKP4PKkZ2ywUeta9SuEk73RQ0cMLJWDfgJzQn9XkwjEbA@mail.gmail.com> (raw)
In-Reply-To: <CADVnQymoGNgVZP+gXmdKCPVihrWo+J8LQJV9_fmyj3f8mhMRtQ@mail.gmail.com>

[-- Attachment #1: Type: text/plain, Size: 2228 bytes --]

Hi Neal,

Happy to help out :) And thanks for the tip!

I was able to track down where the missing bytes that you pointed out
are being lost. It turns out the destination host seems to be
misbehaving. I performed a packet capture at the destination host
interface (a snapshot of the trace is attached). I see the following
sequence of events when a timeout occurs (note that I have NIC
offloading enabled so wireshark captures packets larger than the MTU):

1. The destination receives a data packet of length X with seqNo = Y
from the src with the CWR bit set and does not send back a
corresponding ACK.
2. The source times out and sends a retransmission packet of length Z
(where Z < X) with seqNo = Y
3. The destination sends back an ACK with AckNo = Y + X

So in other words, the packet which the destination host does not
initially ACK (causing the timeout) does not actually get lost because
after receiving the retransmission the AckNo moves forward all the way
past the bytes in the initial unACKed CWR packet. In the attached
screenshot, I've marked the unACKed CWR packet with a red box.

Have you seen this behavior before? And do you know what might be
causing the destination host not to ACK the CWR packet? In most cases
the CWR marked packets are ACKed properly, it's just occasionally they
are not.

Thanks!
-Steve

On Fri, Dec 1, 2017 at 8:35 AM, Neal Cardwell <ncardwell@google.com> wrote:
> On Mon, Nov 27, 2017 at 1:49 PM, Steve Ibanez <sibanez@stanford.edu> wrote:
>>
>> Hi Neal,
>>
>> I tried out your new suggested patches and indeed it looks like it is
>> working. The duration of the freezes looks like it has reduced from an
>> RTO to 10ms (tcp probe reports SRTT measurements of about 200us just
>> before the freeze). So the PTO value seems to be correctly set to
>> max(2*SRTT, 10ms).
>
> Great. Thank you for testing this!
>
> Our team will look into testing these patches more and sending some
> version of them upstream if things look good.
>
> Also, BTW, in newer kernels, with bb4d991a28cc ("tcp: adjust tail loss
> probe timeout") from July, the TLP timeout should be closer to 2*SRTT
> + 2 jiffies, so if your kernel has 1ms jiffies this should further
> improve things.
>
> Thanks,
> neal

[-- Attachment #2: han-5_timeout.png --]
[-- Type: image/png, Size: 151773 bytes --]