[PATCH V3 net-next] TLP: Don't reschedule PTO when there's one outstanding TLP retransmission

[PATCH V3 net-next] TLP: Don't reschedule PTO when there's one outstanding TLP retransmission

[Mao Wenan]

If there is one TLP probe went out(TLP use the write_queue_tail
packet as TLP probe, we assume this first TLP probe named A), and
this TLP probe was not acked by receive side.

Then the transmit side sent the next two packetes out(named B,C),
but unfortunately these two packets are also not acked by receive side.

And then there is one data packet with ack_seq A arrive at transmit
side, in tcp_ack() will call tcp_schedule_loss_probe() to rearm PTO,
the handler tcp_send_loss_probe() is to check
if(tp->tlp_high_seq) then go to rearm_timer(because there is one
outstanding TLP named A), so the new TLP probe can't be sent out and
it needs to rearm the RTO timer(timeout is relative to the transmit
time of the write queue head).

After that, there is another data packet with ack_seq A is received,
if the tlp_time_stamp is greater than rto_time_stamp, it will reset
the TLP timeout, which is before previous RTO timeout, so PTO is
rearm and previous RTO is cleared. Because there is no
retransmission packet was sent or no TLP sack receive,
tp->tlp_high_seq can't be reset to zero and the next TLP probe also
can't be sent out, so there is no way(or very long time)
to retransmit the lost packet.

This fix is to check(tp->tlp_high_seq) in tcp_schedule_loss_probe()
when TLP PTO is after RTO, It is not needed to reschedule PTO when
there is one outstanding TLP retransmission, so if the TLP A is lost
RTO can retransmit lost packet, then tp->tlp_high_seq will be set to
0, and TLP will go to the normal work process.

v1->v2
	refine some words of code and patch comments.
v2->v3
	delete senseless "{" and "}" in if clause.

Signed-off-by: Mao Wenan <maowenan@huawei.com>
---
 net/ipv4/tcp_output.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
index 886d874..b59975f 100644
--- a/net/ipv4/tcp_output.c
+++ b/net/ipv4/tcp_output.c
@@ -2377,6 +2377,7 @@ bool tcp_schedule_loss_probe(struct sock *sk)
 	struct inet_connection_sock *icsk = inet_csk(sk);
 	struct tcp_sock *tp = tcp_sk(sk);
 	u32 timeout, tlp_time_stamp, rto_time_stamp;
+	s32 delta;
 
 	/* No consecutive loss probes. */
 	if (WARN_ON(icsk->icsk_pending == ICSK_TIME_LOSS_PROBE)) {
@@ -2423,7 +2424,12 @@ bool tcp_schedule_loss_probe(struct sock *sk)
 	tlp_time_stamp = tcp_jiffies32 + timeout;
 	rto_time_stamp = (u32)inet_csk(sk)->icsk_timeout;
 	if ((s32)(tlp_time_stamp - rto_time_stamp) > 0) {
-		s32 delta = rto_time_stamp - tcp_jiffies32;
+		/* It is not needed to reschedule PTO when there 
+		 * is one outstanding TLP retransmission. 
+		 */
+		if (tp->tlp_high_seq)
+			return false;
+		delta = rto_time_stamp - tcp_jiffies32;
 		if (delta > 0)
 			timeout = delta;
 	}
-- 
2.5.0

Re: [PATCH V3 net-next] TLP: Don't reschedule PTO when there's one outstanding TLP retransmission

[Neal Cardwell]

On Thu, Jul 27, 2017 at 8:08 AM, Mao Wenan <maowenan@huawei.com> wrote:
> If there is one TLP probe went out(TLP use the write_queue_tail
> packet as TLP probe, we assume this first TLP probe named A), and
> this TLP probe was not acked by receive side.
>
> Then the transmit side sent the next two packetes out(named B,C),
> but unfortunately these two packets are also not acked by receive side.
>
> And then there is one data packet with ack_seq A arrive at transmit
> side, in tcp_ack() will call tcp_schedule_loss_probe() to rearm PTO,
> the handler tcp_send_loss_probe() is to check
> if(tp->tlp_high_seq) then go to rearm_timer(because there is one
> outstanding TLP named A), so the new TLP probe can't be sent out and
> it needs to rearm the RTO timer(timeout is relative to the transmit
> time of the write queue head).
>
> After that, there is another data packet with ack_seq A is received,
> if the tlp_time_stamp is greater than rto_time_stamp, it will reset
> the TLP timeout, which is before previous RTO timeout, so PTO is
> rearm and previous RTO is cleared. Because there is no
> retransmission packet was sent or no TLP sack receive,
> tp->tlp_high_seq can't be reset to zero and the next TLP probe also
> can't be sent out, so there is no way(or very long time)
> to retransmit the lost packet.
>
> This fix is to check(tp->tlp_high_seq) in tcp_schedule_loss_probe()
> when TLP PTO is after RTO, It is not needed to reschedule PTO when
> there is one outstanding TLP retransmission, so if the TLP A is lost
> RTO can retransmit lost packet, then tp->tlp_high_seq will be set to
> 0, and TLP will go to the normal work process.
>
> v1->v2
>         refine some words of code and patch comments.
> v2->v3
>         delete senseless "{" and "}" in if clause.
>
> Signed-off-by: Mao Wenan <maowenan@huawei.com>

Thanks for posting this patch with a detailed problem description, as
well as a trace in the thread for v1 of the patch. This was very
helpful!

Thinking about the problem you describe, and looking at the trace,
AFAICT I don't think this is the patch we want.

We can still have this problem of improperly/repeatedly rescheduling a
PTO even when the TLPs are new data. When the TLPs are new data
tp->tlp_high_seq is not set, and so the patch above will not help.

I think the broader problem is hinted at in this part of your commit
description:

> After that, there is another data packet with ack_seq A is received,
> if the tlp_time_stamp is greater than rto_time_stamp, it will reset
> the TLP timeout

The broader problem here is that an incoming data packet (with no new
ACK/SACK info) affected the TLP for our outbound data. That is a
problem because such incoming data can cause us to delay the TLP when
there is no reason to.

I think this is basically the same as the TLP issue from the "TCP fast
retransmit issues" thread on netdev from July 26. Our TCP team at
Google has a proposed fix for this more general issue that we have
tested and reviewed. I will post a quick summary of the proposed patch
in the "TCP fast retransmit issues" thread. Once the patch has
undergone a little more testing we will send it to the list, hopefully
next week.

Thanks!
neal

RE: [PATCH V3 net-next] TLP: Don't reschedule PTO when there's one outstanding TLP retransmission

[maowenan]

> -----Original Message-----
> From: Neal Cardwell [mailto:ncardwell@google.com]
> Sent: Saturday, July 29, 2017 6:48 AM
> To: maowenan
> Cc: Netdev; David Miller; Yuchung Cheng; Nandita Dukkipati; weiyongjun (A);
> Chenweilong; Wangkefeng (Kevin)
> Subject: Re: [PATCH V3 net-next] TLP: Don't reschedule PTO when there's one
> outstanding TLP retransmission
> 
> On Thu, Jul 27, 2017 at 8:08 AM, Mao Wenan <maowenan@huawei.com>
> wrote:
> > If there is one TLP probe went out(TLP use the write_queue_tail packet
> > as TLP probe, we assume this first TLP probe named A), and this TLP
> > probe was not acked by receive side.
> >
> > Then the transmit side sent the next two packetes out(named B,C), but
> > unfortunately these two packets are also not acked by receive side.
> >
> > And then there is one data packet with ack_seq A arrive at transmit
> > side, in tcp_ack() will call tcp_schedule_loss_probe() to rearm PTO,
> > the handler tcp_send_loss_probe() is to check
> > if(tp->tlp_high_seq) then go to rearm_timer(because there is one
> > outstanding TLP named A), so the new TLP probe can't be sent out and
> > it needs to rearm the RTO timer(timeout is relative to the transmit
> > time of the write queue head).
> >
> > After that, there is another data packet with ack_seq A is received,
> > if the tlp_time_stamp is greater than rto_time_stamp, it will reset
> > the TLP timeout, which is before previous RTO timeout, so PTO is rearm
> > and previous RTO is cleared. Because there is no retransmission packet
> > was sent or no TLP sack receive,
> > tp->tlp_high_seq can't be reset to zero and the next TLP probe also
> > can't be sent out, so there is no way(or very long time) to retransmit
> > the lost packet.
> >
> > This fix is to check(tp->tlp_high_seq) in tcp_schedule_loss_probe()
> > when TLP PTO is after RTO, It is not needed to reschedule PTO when
> > there is one outstanding TLP retransmission, so if the TLP A is lost
> > RTO can retransmit lost packet, then tp->tlp_high_seq will be set to
> > 0, and TLP will go to the normal work process.
> >
> > v1->v2
> >         refine some words of code and patch comments.
> > v2->v3
> >         delete senseless "{" and "}" in if clause.
> >
> > Signed-off-by: Mao Wenan <maowenan@huawei.com>
> 
> Thanks for posting this patch with a detailed problem description, as well as a
> trace in the thread for v1 of the patch. This was very helpful!
> 
> Thinking about the problem you describe, and looking at the trace, AFAICT I
> don't think this is the patch we want.
> 
> We can still have this problem of improperly/repeatedly rescheduling a PTO
> even when the TLPs are new data. When the TLPs are new data
> tp->tlp_high_seq is not set, and so the patch above will not help.
[Mao Wenan]ok, We have reproduced this issue with packetdrill yesterday, 
there is no the same issue when TLP send new data packet, RTO will be fired and 
retransmit packet.
> 
> I think the broader problem is hinted at in this part of your commit
> description:
> 
> > After that, there is another data packet with ack_seq A is received,
> > if the tlp_time_stamp is greater than rto_time_stamp, it will reset
> > the TLP timeout
> 
> The broader problem here is that an incoming data packet (with no new
> ACK/SACK info) affected the TLP for our outbound data. That is a problem
> because such incoming data can cause us to delay the TLP when there is no
> reason to.
> 
> I think this is basically the same as the TLP issue from the "TCP fast retransmit
> issues" thread on netdev from July 26. Our TCP team at Google has a proposed
> fix for this more general issue that we have tested and reviewed. I will post a
> quick summary of the proposed patch in the "TCP fast retransmit issues"
> thread. Once the patch has undergone a little more testing we will send it to
> the list, hopefully next week.
[Mao Wenan]OK.
> 
> Thanks!
> neal

Re: [PATCH V3 net-next] TLP: Don't reschedule PTO when there's one outstanding TLP retransmission

[Neal Cardwell]

On Fri, Jul 28, 2017 at 9:39 PM, maowenan <maowenan@huawei.com> wrote:
> [Mao Wenan]ok, We have reproduced this issue with packetdrill yesterday,
> there is no the same issue when TLP send new data packet, RTO will be fired and
> retransmit packet.

That's great to hear that you were able to reproduce this with
packetdrill. Would you be able to share the packetdrill scripts that
reproduce your issues? I would like to make sure our proposed patch
addresses your scenarios as well.

thanks,
neal

RE: [PATCH V3 net-next] TLP: Don't reschedule PTO when there's one outstanding TLP retransmission

[maowenan]

[-- Attachment #1: Type: text/plain, Size: 1446 bytes --]



> -----Original Message-----
> From: Neal Cardwell [mailto:ncardwell@google.com]
> Sent: Saturday, July 29, 2017 10:04 PM
> To: maowenan
> Cc: Netdev; David Miller; Yuchung Cheng; Nandita Dukkipati; weiyongjun (A);
> Chenweilong; Wangkefeng (Kevin)
> Subject: Re: [PATCH V3 net-next] TLP: Don't reschedule PTO when there's one
> outstanding TLP retransmission
> 
> On Fri, Jul 28, 2017 at 9:39 PM, maowenan <maowenan@huawei.com> wrote:
> > [Mao Wenan]ok, We have reproduced this issue with packetdrill
> > yesterday, there is no the same issue when TLP send new data packet,
> > RTO will be fired and retransmit packet.
> 
> That's great to hear that you were able to reproduce this with packetdrill.
> Would you be able to share the packetdrill scripts that reproduce your issues? I
> would like to make sure our proposed patch addresses your scenarios as well.
[Mao Wenan]please refer to the attachment, test.pkt is packetdrill script. 
In test.pcap, packet number 17 is the TLP probe, packet number 218 is the
retransmission packet because client don't send data packet to server. 
From the capture time, there are about 6 seconds the retransmission
packet can be sent, and this time can be added more as long as client 
send data packet continually.
I have reproduced this issue in Linux 4.13-rc3, 3.10, 4.1. Please check the timing 
When you use test.pkt to reproduce in your environment.
> 
> thanks,
> neal

[-- Attachment #2: test.pcap --]
[-- Type: application/octet-stream, Size: 135408 bytes --]

[-- Attachment #3: test.pkt --]
[-- Type: application/octet-stream, Size: 9892 bytes --]

0     socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0    setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0    setsockopt(3, SOL_SOCKET, SO_SNDBUF, [0x4000], 4) = 0
+0    setsockopt(3, SOL_SOCKET, SO_RCVBUF, [0x800000], 4) = 0
+0    bind(3, ..., ...) = 0
+0    listen(3, 1) = 0

+0    < S 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
+0    > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 2>
+.020 < . 1:1(0) ack 1 win 257
+0    accept(3, ..., ...) = 4

// Send 10 MSS.
+0    write(4, ..., 10000) = 10000
+0    > . 1:2001(2000) ack 1
+0    > . 2001:4001(2000) ack 1
+0    > . 4001:6001(2000) ack 1
+0    > . 6001:8001(2000) ack 1
+0    > P. 8001:10001(2000) ack 1

// Incoming data arrives
+.010 < . 1:1001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 1001

+0    read(4, ..., 1000) = 1000

+.010 < . 1001:2001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 2001

+0    read(4, ..., 1000) = 1000

+.010 < . 2001:3001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 3001

+0    read(4, ..., 1000) = 1000

+.010 < . 3001:4001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 4001

// At 2*RTT after the last transmit, send a TLP loss probe
*     > P. 9001:10001(1000) ack 4001

+0    read(4, ..., 1000) = 1000

+.05 < . 4001:5001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 5001


+.05 < . 5001:6001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 6001


+.05 < . 6001:7001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 7001


+.05 < . 7001:8001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 8001

+0    read(4, ..., 4000) = 4000

+.05 < . 8001:9001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 9001


+.05 < . 9001:10001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 10001


+.05 < . 10001:11001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 11001


+.05 < . 11001:12001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 12001

+0    read(4, ..., 4000) = 4000

+.05 < . 12001:13001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 13001


+.05 < . 13001:14001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 14001


+.05 < . 14001:15001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 15001


+.05 < . 15001:16001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 16001

+0    read(4, ..., 4000) = 4000

+.05 < . 16001:17001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 17001


+.05 < . 17001:18001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 18001


+.05 < . 18001:19001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 19001


+.05 < . 19001:20001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 20001

+0    read(4, ..., 4000) = 4000

+.05 < . 20001:21001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 21001


+.05 < . 21001:22001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 22001


+.05 < . 22001:23001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 23001


+.05 < . 23001:24001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 24001

+0    read(4, ..., 4000) = 4000

+.05 < . 24001:25001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 25001


+.05 < . 25001:26001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 26001


+.05 < . 26001:27001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 27001


+.05 < . 27001:28001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 28001

+0    read(4, ..., 4000) = 4000

+.05 < . 28001:29001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 29001


+.05 < . 29001:30001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 30001


+.05 < . 30001:31001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 31001


+.05 < . 31001:32001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 32001

+0    read(4, ..., 4000) = 4000

+.05 < . 32001:33001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 33001


+.05 < . 33001:34001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 34001


+.05 < . 34001:35001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 35001


+.05 < . 35001:36001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 36001

+0    read(4, ..., 4000) = 4000

+.05 < . 36001:37001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 37001


+.05 < . 37001:38001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 38001


+.05 < . 38001:39001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 39001


+.05 < . 39001:40001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 40001

+0    read(4, ..., 4000) = 4000

+.05 < . 40001:41001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 41001


+.05 < . 41001:42001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 42001


+.05 < . 42001:43001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 43001


+.05 < . 43001:44001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 44001

+0    read(4, ..., 4000) = 4000

+.05 < . 44001:45001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 45001


+.05 < . 45001:46001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 46001


+.05 < . 46001:47001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 47001


+.05 < . 47001:48001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 48001

+0    read(4, ..., 4000) = 4000

+.05 < . 48001:49001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 49001


+.05 < . 49001:50001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 50001


+.05 < . 50001:51001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 51001


+.05 < . 51001:52001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 52001

+0    read(4, ..., 4000) = 4000

+.05 < . 52001:53001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 53001


+.05 < . 53001:54001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 54001


+.05 < . 54001:55001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 55001


+.05 < . 55001:56001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 56001

+0    read(4, ..., 4000) = 4000

+.05 < . 56001:57001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 57001


+.05 < . 57001:58001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 58001


+.05 < . 58001:59001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 59001


+.05 < . 59001:60001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 60001

+0    read(4, ..., 4000) = 4000

+.05 < . 60001:61001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 61001


+.05 < . 61001:62001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 62001


+.05 < . 62001:63001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 63001


+.05 < . 63001:64001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 64001

+0    read(4, ..., 4000) = 4000

+.05 < . 64001:65001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 65001


+.05 < . 65001:66001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 66001


+.05 < . 66001:67001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 67001


+.05 < . 67001:68001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 68001

+0    read(4, ..., 4000) = 4000

+.05 < . 68001:69001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 69001


+.05 < . 69001:70001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 70001


+.05 < . 70001:71001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 71001


+.05 < . 71001:72001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 72001

+0    read(4, ..., 4000) = 4000

+.05 < . 72001:73001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 73001


+.05 < . 73001:74001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 74001


+.05 < . 74001:75001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 75001


+.05 < . 75001:76001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 76001

+0    read(4, ..., 4000) = 4000

+.05 < . 76001:77001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 77001


+.05 < . 77001:78001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 78001


+.05 < . 78001:79001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 79001


+.05 < . 79001:80001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 80001

+0    read(4, ..., 4000) = 4000

+.05 < . 80001:81001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 81001


+.05 < . 81001:82001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 82001


+.05 < . 82001:83001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 83001


+.05 < . 83001:84001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 84001

+0    read(4, ..., 4000) = 4000

+.05 < . 84001:85001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 85001


+.05 < . 85001:86001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 86001


+.05 < . 86001:87001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 87001


+.05 < . 87001:88001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 88001

+0    read(4, ..., 4000) = 4000

+.05 < . 88001:89001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 89001


+.05 < . 89001:90001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 90001


+.05 < . 90001:91001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 91001


+.05 < . 91001:92001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 92001

+0    read(4, ..., 4000) = 4000

+.05 < . 92001:93001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 93001


+.05 < . 93001:94001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 94001


+.05 < . 94001:95001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 95001


+.05 < . 95001:96001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 96001

+0    read(4, ..., 4000) = 4000

+.05 < . 96001:97001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 97001


+.05 < . 97001:98001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 98001


+.05 < . 98001:99001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 99001


+.05 < . 99001:100001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 100001

+0    read(4, ..., 4000) = 4000

+.05 < . 100001:101001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 101001


+.05 < . 101001:102001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 102001


+.05 < . 102001:103001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 103001


+.05 < . 103001:104001(1000) ack 1 win 257
*     > . 10001:10001(0) ack 104001

+0    read(4, ..., 4000) = 4000

//server retransmit one packet
* > . 1:1001(1000) ack 104001

Re: [PATCH V3 net-next] TLP: Don't reschedule PTO when there's one outstanding TLP retransmission

[Neal Cardwell]

On Sun, Jul 30, 2017 at 11:29 PM, maowenan <maowenan@huawei.com> wrote:
> [Mao Wenan]please refer to the attachment, test.pkt is packetdrill script.
> In test.pcap, packet number 17 is the TLP probe, packet number 218 is the
> retransmission packet because client don't send data packet to server.
> From the capture time, there are about 6 seconds the retransmission
> packet can be sent, and this time can be added more as long as client
> send data packet continually.
> I have reproduced this issue in Linux 4.13-rc3, 3.10, 4.1. Please check the timing
> When you use test.pkt to reproduce in your environment.

Thank you for your very nice packetdrill test case illustrating this
problem! And thanks for verifying that the problem shows up in those
kernel versions.

We are able to run the script in our environment and both verify that
the bug is the one we hypothesized, and verify our  proposed patch
fixes it (the RTO for the TLP happens 221ms after the TLP, instead of
~5 secs later). We will send out our proposed patches ASAP.

Thanks!
neal

Re: [PATCH V3 net-next] TLP: Don't reschedule PTO when there's one outstanding TLP retransmission

[Neal Cardwell]

On Mon, Jul 31, 2017 at 11:49 AM, Neal Cardwell <ncardwell@google.com> wrote:
> On Sun, Jul 30, 2017 at 11:29 PM, maowenan <maowenan@huawei.com> wrote:
>> [Mao Wenan]please refer to the attachment, test.pkt is packetdrill script.
>> In test.pcap, packet number 17 is the TLP probe, packet number 218 is the
>> retransmission packet because client don't send data packet to server.
>> From the capture time, there are about 6 seconds the retransmission
>> packet can be sent, and this time can be added more as long as client
>> send data packet continually.
>> I have reproduced this issue in Linux 4.13-rc3, 3.10, 4.1. Please check the timing
>> When you use test.pkt to reproduce in your environment.
>
> Thank you for your very nice packetdrill test case illustrating this
> problem! And thanks for verifying that the problem shows up in those
> kernel versions.
>
> We are able to run the script in our environment and both verify that
> the bug is the one we hypothesized, and verify our  proposed patch
> fixes it (the RTO for the TLP happens 221ms after the TLP, instead of
> ~5 secs later). We will send out our proposed patches ASAP.

The timer patches are upstream for review for the "net" branch:

  https://patchwork.ozlabs.org/patch/796057/
  https://patchwork.ozlabs.org/patch/796058/
  https://patchwork.ozlabs.org/patch/796059/

Again, thank you for reporting this and providing a packetdrill script
to reproduce this!

neal