TCP reaching to maximum throughput after a long time

TCP reaching to maximum throughput after a long time

[Machani, Yaniv]

Hi,
After updating from Kernel 3.14 to Kernel 4.4 we have seen a TCP performance degradation over Wi-Fi.
In 3.14 kernel, TCP got to its max throughout after less than a second, while in the 4.4  it is taking ~20-30 seconds.
UDP TX/RX and TCP RX performance is as expected.
We are using a Beagle Bone Black and a WiLink8 device.

Were there any related changes that might cause such behavior ?
Kernel configuration and sysctl values were compared, but no significant differences have been found.

See a log of the behavior below :
-----------------------------------------------------------
Client connecting to 10.2.46.5, TCP port 5001
TCP window size:  320 KByte (WARNING: requested  256 KByte)
------------------------------------------------------------
[  3] local 10.2.46.6 port 49282 connected with 10.2.46.5 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec  5.75 MBytes  48.2 Mbits/sec
[  3]  1.0- 2.0 sec  6.50 MBytes  54.5 Mbits/sec
[  3]  2.0- 3.0 sec  6.50 MBytes  54.5 Mbits/sec
[  3]  3.0- 4.0 sec  6.50 MBytes  54.5 Mbits/sec
[  3]  4.0- 5.0 sec  6.75 MBytes  56.6 Mbits/sec
[  3]  5.0- 6.0 sec  3.38 MBytes  28.3 Mbits/sec
[  3]  6.0- 7.0 sec  6.38 MBytes  53.5 Mbits/sec
[  3]  7.0- 8.0 sec  6.88 MBytes  57.7 Mbits/sec
[  3]  8.0- 9.0 sec  7.12 MBytes  59.8 Mbits/sec
[  3]  9.0-10.0 sec  7.12 MBytes  59.8 Mbits/sec
[  3] 10.0-11.0 sec  7.12 MBytes  59.8 Mbits/sec
[  3] 11.0-12.0 sec  7.25 MBytes  60.8 Mbits/sec
[  3] 12.0-13.0 sec  7.12 MBytes  59.8 Mbits/sec
[  3] 13.0-14.0 sec  7.25 MBytes  60.8 Mbits/sec
[  3] 14.0-15.0 sec  7.62 MBytes  64.0 Mbits/sec
[  3] 15.0-16.0 sec  7.88 MBytes  66.1 Mbits/sec
[  3] 16.0-17.0 sec  8.12 MBytes  68.2 Mbits/sec
[  3] 17.0-18.0 sec  8.25 MBytes  69.2 Mbits/sec
[  3] 18.0-19.0 sec  8.50 MBytes  71.3 Mbits/sec
[  3] 19.0-20.0 sec  8.88 MBytes  74.4 Mbits/sec
[  3] 20.0-21.0 sec  8.75 MBytes  73.4 Mbits/sec
[  3] 21.0-22.0 sec  8.62 MBytes  72.4 Mbits/sec
[  3] 22.0-23.0 sec  8.75 MBytes  73.4 Mbits/sec
[  3] 23.0-24.0 sec  8.50 MBytes  71.3 Mbits/sec
[  3] 24.0-25.0 sec  8.62 MBytes  72.4 Mbits/sec
[  3] 25.0-26.0 sec  8.62 MBytes  72.4 Mbits/sec
[  3] 26.0-27.0 sec  8.62 MBytes  72.4 Mbits/sec

--
Thanks,
Yaniv Machani

Re: TCP reaching to maximum throughput after a long time

[Eric Dumazet]

On Tue, 2016-04-12 at 12:17 +0000, Machani, Yaniv wrote:
> Hi,
> After updating from Kernel 3.14 to Kernel 4.4 we have seen a TCP performance degradation over Wi-Fi.
> In 3.14 kernel, TCP got to its max throughout after less than a second, while in the 4.4  it is taking ~20-30 seconds.
> UDP TX/RX and TCP RX performance is as expected.
> We are using a Beagle Bone Black and a WiLink8 device.
> 
> Were there any related changes that might cause such behavior ?
> Kernel configuration and sysctl values were compared, but no significant differences have been found.
> 
> See a log of the behavior below :
> -----------------------------------------------------------
> Client connecting to 10.2.46.5, TCP port 5001
> TCP window size:  320 KByte (WARNING: requested  256 KByte)
> ------------------------------------------------------------
> [  3] local 10.2.46.6 port 49282 connected with 10.2.46.5 port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0- 1.0 sec  5.75 MBytes  48.2 Mbits/sec
> [  3]  1.0- 2.0 sec  6.50 MBytes  54.5 Mbits/sec
> [  3]  2.0- 3.0 sec  6.50 MBytes  54.5 Mbits/sec
> [  3]  3.0- 4.0 sec  6.50 MBytes  54.5 Mbits/sec
> [  3]  4.0- 5.0 sec  6.75 MBytes  56.6 Mbits/sec
> [  3]  5.0- 6.0 sec  3.38 MBytes  28.3 Mbits/sec
> [  3]  6.0- 7.0 sec  6.38 MBytes  53.5 Mbits/sec
> [  3]  7.0- 8.0 sec  6.88 MBytes  57.7 Mbits/sec
> [  3]  8.0- 9.0 sec  7.12 MBytes  59.8 Mbits/sec
> [  3]  9.0-10.0 sec  7.12 MBytes  59.8 Mbits/sec
> [  3] 10.0-11.0 sec  7.12 MBytes  59.8 Mbits/sec
> [  3] 11.0-12.0 sec  7.25 MBytes  60.8 Mbits/sec
> [  3] 12.0-13.0 sec  7.12 MBytes  59.8 Mbits/sec
> [  3] 13.0-14.0 sec  7.25 MBytes  60.8 Mbits/sec
> [  3] 14.0-15.0 sec  7.62 MBytes  64.0 Mbits/sec
> [  3] 15.0-16.0 sec  7.88 MBytes  66.1 Mbits/sec
> [  3] 16.0-17.0 sec  8.12 MBytes  68.2 Mbits/sec
> [  3] 17.0-18.0 sec  8.25 MBytes  69.2 Mbits/sec
> [  3] 18.0-19.0 sec  8.50 MBytes  71.3 Mbits/sec
> [  3] 19.0-20.0 sec  8.88 MBytes  74.4 Mbits/sec
> [  3] 20.0-21.0 sec  8.75 MBytes  73.4 Mbits/sec
> [  3] 21.0-22.0 sec  8.62 MBytes  72.4 Mbits/sec
> [  3] 22.0-23.0 sec  8.75 MBytes  73.4 Mbits/sec
> [  3] 23.0-24.0 sec  8.50 MBytes  71.3 Mbits/sec
> [  3] 24.0-25.0 sec  8.62 MBytes  72.4 Mbits/sec
> [  3] 25.0-26.0 sec  8.62 MBytes  72.4 Mbits/sec
> [  3] 26.0-27.0 sec  8.62 MBytes  72.4 Mbits/sec
> 

CC netdev, where this is better discussed.

This could be a lot of different factors, and caused by a sender
problem, a receiver problem, ...

TCP behavior depends on the drivers, so maybe a change there can explain
this.

You could capture the first 5000 frames of the flow and post the pcap ?
(-s 128 to capture only the headers)

tcpdump -p -s 128 -i eth0 -c 5000 host 10.2.46.5 -w flow.pcap

 
Also, while test is running, you could fetch 
ss -temoi dst 10.2.46.5:5001

Re: TCP reaching to maximum throughput after a long time

[Ben Greear]

On 04/12/2016 07:52 AM, Eric Dumazet wrote:
> On Tue, 2016-04-12 at 12:17 +0000, Machani, Yaniv wrote:
>> Hi,
>> After updating from Kernel 3.14 to Kernel 4.4 we have seen a TCP performance degradation over Wi-Fi.
>> In 3.14 kernel, TCP got to its max throughout after less than a second, while in the 4.4  it is taking ~20-30 seconds.
>> UDP TX/RX and TCP RX performance is as expected.
>> We are using a Beagle Bone Black and a WiLink8 device.
>>
>> Were there any related changes that might cause such behavior ?
>> Kernel configuration and sysctl values were compared, but no significant differences have been found.

If you are using 'Cubic' TCP congestion control, then please try something different.
It was broken last I checked, at least when used with the ath10k driver.

https://marc.info/?l=linux-netdev&m=144405216005715&w=2

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: TCP reaching to maximum throughput after a long time

[Yuchung Cheng]

On Tue, Apr 12, 2016 at 7:52 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
>
> On Tue, 2016-04-12 at 12:17 +0000, Machani, Yaniv wrote:
> > Hi,
> > After updating from Kernel 3.14 to Kernel 4.4 we have seen a TCP performance degradation over Wi-Fi.
> > In 3.14 kernel, TCP got to its max throughout after less than a second, while in the 4.4  it is taking ~20-30 seconds.
> > UDP TX/RX and TCP RX performance is as expected.
> > We are using a Beagle Bone Black and a WiLink8 device.
> >
> > Were there any related changes that might cause such behavior ?
> > Kernel configuration and sysctl values were compared, but no significant differences have been found.
> >
> > See a log of the behavior below :
> > -----------------------------------------------------------
> > Client connecting to 10.2.46.5, TCP port 5001
> > TCP window size:  320 KByte (WARNING: requested  256 KByte)
> > ------------------------------------------------------------
> > [  3] local 10.2.46.6 port 49282 connected with 10.2.46.5 port 5001
> > [ ID] Interval       Transfer     Bandwidth
> > [  3]  0.0- 1.0 sec  5.75 MBytes  48.2 Mbits/sec
> > [  3]  1.0- 2.0 sec  6.50 MBytes  54.5 Mbits/sec
> > [  3]  2.0- 3.0 sec  6.50 MBytes  54.5 Mbits/sec
> > [  3]  3.0- 4.0 sec  6.50 MBytes  54.5 Mbits/sec
> > [  3]  4.0- 5.0 sec  6.75 MBytes  56.6 Mbits/sec
> > [  3]  5.0- 6.0 sec  3.38 MBytes  28.3 Mbits/sec
> > [  3]  6.0- 7.0 sec  6.38 MBytes  53.5 Mbits/sec
> > [  3]  7.0- 8.0 sec  6.88 MBytes  57.7 Mbits/sec
> > [  3]  8.0- 9.0 sec  7.12 MBytes  59.8 Mbits/sec
> > [  3]  9.0-10.0 sec  7.12 MBytes  59.8 Mbits/sec
> > [  3] 10.0-11.0 sec  7.12 MBytes  59.8 Mbits/sec
> > [  3] 11.0-12.0 sec  7.25 MBytes  60.8 Mbits/sec
> > [  3] 12.0-13.0 sec  7.12 MBytes  59.8 Mbits/sec
> > [  3] 13.0-14.0 sec  7.25 MBytes  60.8 Mbits/sec
> > [  3] 14.0-15.0 sec  7.62 MBytes  64.0 Mbits/sec
> > [  3] 15.0-16.0 sec  7.88 MBytes  66.1 Mbits/sec
> > [  3] 16.0-17.0 sec  8.12 MBytes  68.2 Mbits/sec
> > [  3] 17.0-18.0 sec  8.25 MBytes  69.2 Mbits/sec
> > [  3] 18.0-19.0 sec  8.50 MBytes  71.3 Mbits/sec
> > [  3] 19.0-20.0 sec  8.88 MBytes  74.4 Mbits/sec
> > [  3] 20.0-21.0 sec  8.75 MBytes  73.4 Mbits/sec
> > [  3] 21.0-22.0 sec  8.62 MBytes  72.4 Mbits/sec
> > [  3] 22.0-23.0 sec  8.75 MBytes  73.4 Mbits/sec
> > [  3] 23.0-24.0 sec  8.50 MBytes  71.3 Mbits/sec
> > [  3] 24.0-25.0 sec  8.62 MBytes  72.4 Mbits/sec
> > [  3] 25.0-26.0 sec  8.62 MBytes  72.4 Mbits/sec
> > [  3] 26.0-27.0 sec  8.62 MBytes  72.4 Mbits/sec
> >
>
> CC netdev, where this is better discussed.
>
> This could be a lot of different factors, and caused by a sender
> problem, a receiver problem, ...
>
> TCP behavior depends on the drivers, so maybe a change there can explain
> this.
>
> You could capture the first 5000 frames of the flow and post the pcap ?
> (-s 128 to capture only the headers)
pcap would be really helpful indeed. if possible please capture on
both 4.4 and 3.14 kernels.

>
> tcpdump -p -s 128 -i eth0 -c 5000 host 10.2.46.5 -w flow.pcap
>
>
> Also, while test is running, you could fetch
> ss -temoi dst 10.2.46.5:5001
>
>
>
>
>

RE: TCP reaching to maximum throughput after a long time

[Machani, Yaniv]

On Tue, Apr 12, 2016 at 18:04:52, Ben Greear wrote:
> On 04/12/2016 07:52 AM, Eric Dumazet wrote:
> > On Tue, 2016-04-12 at 12:17 +0000, Machani, Yaniv wrote:
>>>
> 
> If you are using 'Cubic' TCP congestion control, then please try 
> something different.
> It was broken last I checked, at least when used with the ath10k driver.
> 

Thanks Ben, this indeed seems to be the issue !
Switching to reno got me to max throughput instantly.

I'm still looking through the thread you have shared, but from what I understand there is no planned fix for it ?

--
Thanks,
Yaniv

Re: TCP reaching to maximum throughput after a long time

[Ben Greear]

On 04/12/2016 12:31 PM, Machani, Yaniv wrote:
> On Tue, Apr 12, 2016 at 18:04:52, Ben Greear wrote:
>> On 04/12/2016 07:52 AM, Eric Dumazet wrote:
>>> On Tue, 2016-04-12 at 12:17 +0000, Machani, Yaniv wrote:
>>>>
>>
>> If you are using 'Cubic' TCP congestion control, then please try
>> something different.
>> It was broken last I checked, at least when used with the ath10k driver.
>>
>
> Thanks Ben, this indeed seems to be the issue !
> Switching to reno got me to max throughput instantly.
>
> I'm still looking through the thread you have shared, but from what I understand there is no planned fix for it ?

I think at the time it was blamed on ath10k and no one cared to try to fix it.

Or, maybe no one really uses CUBIC anymore?

Either way, I have no plans to try to fix CUBIC, but maybe someone who knows
this code better could give it a try.

Thanks,
Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: TCP reaching to maximum throughput after a long time

[Eric Dumazet]

On Tue, 2016-04-12 at 13:11 -0700, Ben Greear wrote:
> On 04/12/2016 12:31 PM, Machani, Yaniv wrote:
> > On Tue, Apr 12, 2016 at 18:04:52, Ben Greear wrote:
> >> On 04/12/2016 07:52 AM, Eric Dumazet wrote:
> >>> On Tue, 2016-04-12 at 12:17 +0000, Machani, Yaniv wrote:
> >>>>
> >>
> >> If you are using 'Cubic' TCP congestion control, then please try
> >> something different.
> >> It was broken last I checked, at least when used with the ath10k driver.
> >>
> >
> > Thanks Ben, this indeed seems to be the issue !
> > Switching to reno got me to max throughput instantly.
> >
> > I'm still looking through the thread you have shared, but from what I understand there is no planned fix for it ?
> 
> I think at the time it was blamed on ath10k and no one cared to try to fix it.
> 
> Or, maybe no one really uses CUBIC anymore?
> 
> Either way, I have no plans to try to fix CUBIC, but maybe someone who knows
> this code better could give it a try.

Well, cubic seems to work in many cases, assuming they are not too many
drops.

Assuming one flow can get nominal speed in few RTT is kind a dream, and
so far nobody claimed a CC was able to do that, while still being fair
and resilient.

TCP CC are full of heuristics, and by definition heuristics that were
working 6 years ago might need to be refreshed.

We are still maintaining Cubic for sure.

Re: TCP reaching to maximum throughput after a long time

[Ben Greear]

On 04/12/2016 01:17 PM, Eric Dumazet wrote:
> On Tue, 2016-04-12 at 13:11 -0700, Ben Greear wrote:
>> On 04/12/2016 12:31 PM, Machani, Yaniv wrote:
>>> On Tue, Apr 12, 2016 at 18:04:52, Ben Greear wrote:
>>>> On 04/12/2016 07:52 AM, Eric Dumazet wrote:
>>>>> On Tue, 2016-04-12 at 12:17 +0000, Machani, Yaniv wrote:
>>>>>>
>>>>
>>>> If you are using 'Cubic' TCP congestion control, then please try
>>>> something different.
>>>> It was broken last I checked, at least when used with the ath10k driver.
>>>>
>>>
>>> Thanks Ben, this indeed seems to be the issue !
>>> Switching to reno got me to max throughput instantly.
>>>
>>> I'm still looking through the thread you have shared, but from what I understand there is no planned fix for it ?
>>
>> I think at the time it was blamed on ath10k and no one cared to try to fix it.
>>
>> Or, maybe no one really uses CUBIC anymore?
>>
>> Either way, I have no plans to try to fix CUBIC, but maybe someone who knows
>> this code better could give it a try.
>
> Well, cubic seems to work in many cases, assuming they are not too many
> drops.
>
> Assuming one flow can get nominal speed in few RTT is kind a dream, and
> so far nobody claimed a CC was able to do that, while still being fair
> and resilient.
>
> TCP CC are full of heuristics, and by definition heuristics that were
> working 6 years ago might need to be refreshed.
>
> We are still maintaining Cubic for sure.

It worked well enough for years that I didn't even know other algorithms were
available.  It was broken around 4.0 time, and I reported it to the list,
and no one seemed to really care enough to do anything about it.  I changed
to reno and ignored the problem as well.

It is trivially easy to see the regression when using ath10k NIC, and from this email
thread, I guess other NICs have similar issues.

Thanks,
Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: TCP reaching to maximum throughput after a long time

[Eric Dumazet]

On Tue, 2016-04-12 at 13:23 -0700, Ben Greear wrote:

> It worked well enough for years that I didn't even know other algorithms were
> available.  It was broken around 4.0 time, and I reported it to the list,
> and no one seemed to really care enough to do anything about it.  I changed
> to reno and ignored the problem as well.
> 
> It is trivially easy to see the regression when using ath10k NIC, and from this email
> thread, I guess other NICs have similar issues.

Since it is so trivial, why don't you start a bisection ?

I asked a capture, I did not say ' switch to Reno or whatever ', right ?

Guessing is nice, but investigating and fixing is better.

Do not assume that nothing can be done, please ?

Thanks.

Re: TCP reaching to maximum throughput after a long time

[Ben Greear]

On 04/12/2016 01:29 PM, Eric Dumazet wrote:
> On Tue, 2016-04-12 at 13:23 -0700, Ben Greear wrote:
>
>> It worked well enough for years that I didn't even know other algorithms were
>> available.  It was broken around 4.0 time, and I reported it to the list,
>> and no one seemed to really care enough to do anything about it.  I changed
>> to reno and ignored the problem as well.
>>
>> It is trivially easy to see the regression when using ath10k NIC, and from this email
>> thread, I guess other NICs have similar issues.
>
> Since it is so trivial, why don't you start a bisection ?

I vaguely remember doing a bisect, but I can't find any email about
that, so maybe I didn't.  At any rate, it is somewhere between 3.17 and 4.0.
 From memory, it was between 3.19 and 4.0, but I am not certain of that.

Neil's suggestion, from the thread below, is that it was likely:  "605ad7f tcp: refine TSO autosizing"

Here is previous email thread:

https://www.mail-archive.com/netdev@vger.kernel.org/msg80803.html

This one has a link to a pcap I made at the time:

https://www.mail-archive.com/netdev@vger.kernel.org/msg80890.html

>
> I asked a capture, I did not say ' switch to Reno or whatever ', right ?
>
> Guessing is nice, but investigating and fixing is better.
>
> Do not assume that nothing can be done, please ?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: TCP reaching to maximum throughput after a long time

[Yuchung Cheng]

On Tue, Apr 12, 2016 at 2:40 PM, Ben Greear <greearb@candelatech.com> wrote:
> On 04/12/2016 01:29 PM, Eric Dumazet wrote:
>>
>> On Tue, 2016-04-12 at 13:23 -0700, Ben Greear wrote:
>>
>>> It worked well enough for years that I didn't even know other algorithms
>>> were
>>> available.  It was broken around 4.0 time, and I reported it to the list,
>>> and no one seemed to really care enough to do anything about it.  I
>>> changed
>>> to reno and ignored the problem as well.
>>>
>>> It is trivially easy to see the regression when using ath10k NIC, and
>>> from this email
>>> thread, I guess other NICs have similar issues.
>>
>>
>> Since it is so trivial, why don't you start a bisection ?
>
>
> I vaguely remember doing a bisect, but I can't find any email about
> that, so maybe I didn't.  At any rate, it is somewhere between 3.17 and 4.0.
> From memory, it was between 3.19 and 4.0, but I am not certain of that.
>
> Neil's suggestion, from the thread below, is that it was likely:  "605ad7f
> tcp: refine TSO autosizing"
>
> Here is previous email thread:
>
> https://www.mail-archive.com/netdev@vger.kernel.org/msg80803.html
>
> This one has a link to a pcap I made at the time:
>
> https://www.mail-archive.com/netdev@vger.kernel.org/msg80890.html
based on the prev thread I propose we disable hystart ack-train. It is
brittle under various circumstances. We've disabled that at Google for
years.

>
>>
>> I asked a capture, I did not say ' switch to Reno or whatever ', right ?
>>
>> Guessing is nice, but investigating and fixing is better.
>>
>> Do not assume that nothing can be done, please ?
>
>
> Thanks,
> Ben
>
> --
> Ben Greear <greearb@candelatech.com>
> Candela Technologies Inc  http://www.candelatech.com
>

Re: TCP reaching to maximum throughput after a long time

[Eric Dumazet]

On Tue, 2016-04-12 at 20:08 -0700, Yuchung Cheng wrote:

> based on the prev thread I propose we disable hystart ack-train. It is
> brittle under various circumstances. We've disabled that at Google for
> years.

Right, but because we also use sch_fq packet scheduler and pacing ;)

RE: TCP reaching to maximum throughput after a long time

[Machani, Yaniv]

On Wed, Apr 13, 2016 at 17:32:29, Neal Cardwell wrote:
> Miller; Eric Dumazet; Nandita Dukkipati; open list; Kama, Meirav
> Subject: Re: TCP reaching to maximum throughput after a long time
> 
> I like the idea to disable hystart ack-train.
> 
> 
> Yaniv, can you please re-run your test with CUBIC in three different
> scenarios:
> 
> a) echo 0 > /sys/module/tcp_cubic/parameters/hystart_detect
This fixes the issues, got to max throughput immediately. 

> 
> b) echo 1 > /sys/module/tcp_cubic/parameters/hystart_detect
> 
This shows the same results as before, starting low and increasing slowly.

>
> c) echo 2 > /sys/module/tcp_cubic/parameters/hystart_detect
This gets us a bit higher at the beginning, but never (waited ~60 sec) goes to the max.


Appreciate your help on this.
Yaniv
> 
> 
> This should help us isolate whether the hystart ack-train algorithm is 
> causing problems in this particular case.
> 
> Thanks!
> neal
> 
> 
> 
> On Tue, Apr 12, 2016 at 11:32 PM, Eric Dumazet 
> <eric.dumazet@gmail.com>
> wrote:
> 
> 
> 	On Tue, 2016-04-12 at 20:08 -0700, Yuchung Cheng wrote:
> 
> 	> based on the prev thread I propose we disable hystart ack-train. It 
> is
> 	> brittle under various circumstances. We've disabled that at Google 
> for
> 	> years.
> 
> 	Right, but because we also use sch_fq packet scheduler and pacing ;)
> 
> 
> 
>