Variable download speed

Variable download speed

[sdrb]

Hi,

I've got a problem with network on one of my embedded boards.
I'm testing download speed of 256MB file from my PC to embedded board 
through 1Gbit ethernet link using ftp.

The problem is that sometimes I achieve 25MB/s and sometimes it is only 
14MB/s. There are also situations where the transfer speed starts at 
14MB/s and after a few seconds achieves 25MB/s.
I've caught the second case with tcpdump and I noticed that when the speed 
is 14MB/s - the tcp window size is 534368 bytes and when the speed 
achieved 25MB/s the tcp window size is 933888.

My question is: what causes such dynamic change in the window size (while 
transferring data)?  Is it some kernel parameter wrong set or something 
like this?
Do I have any influence on such dynamic change in tcp window size?

sdrb

Re: Variable download speed

[Neal Cardwell]

On Tue, Feb 23, 2016 at 6:19 AM, <sdrb@onet.eu> wrote:
> My question is: what causes such dynamic change in
> the window size (while transferring data)?
> Is it some kernel parameter wrong set or something like this?

That sounds like TCP receive buffer auto-tuning (also called "Dynamic
right-sizing (DRS)" (Fisk and Feng, 2001):

  http://permalink.lanl.gov/object/tr?what=info:lanl-repo/lareport/LA-UR-01-5460

The Linux TCP receiver will, by default, dynamically adjust the
receive window to a value that supports the rate at which the
application successfully reads data out of the socket.

> Do I have any influence on such dynamic change in tcp window size?

You might check on the receiver host with top/mpstat/strace/etc to see
whether the receiving application is limiting performance in some way.
This kind of behavior can show up if the receiver is sometimes
CPU-saturated, or limited by the throughput of the medium on to which
it is writing the data.

If you control the receiver software, you can use
setsockopt(SO_RCVBUF) to explicitly set the receive buffer size, and
see if that helps. If it doesn't, that would suggest that it is indeed
the receiving application that is limiting performance.

If you could provide a tcpdump trace (headers only, e.g.,  -s 96)
taken on the sender, we could check to see if we can see any problems
in the TCP sender or receiver behavior.

cheers,
neal

Re: Variable download speed

[sdrb]

On Tue, 23 Feb 2016, Neal Cardwell wrote:

> On Tue, Feb 23, 2016 at 6:19 AM, <sdrb@onet.eu> wrote:
>> My question is: what causes such dynamic change in
>> the window size (while transferring data)?
>> Is it some kernel parameter wrong set or something like this?
>
> That sounds like TCP receive buffer auto-tuning (also called "Dynamic
> right-sizing (DRS)" (Fisk and Feng, 2001):
>
>  http://permalink.lanl.gov/object/tr?what=info:lanl-repo/lareport/LA-UR-01-5460
>
> The Linux TCP receiver will, by default, dynamically adjust the
> receive window to a value that supports the rate at which the
> application successfully reads data out of the socket.
>
>> Do I have any influence on such dynamic change in tcp window size?
>
> You might check on the receiver host with top/mpstat/strace/etc to see
> whether the receiving application is limiting performance in some way.
> This kind of behavior can show up if the receiver is sometimes
> CPU-saturated, or limited by the throughput of the medium on to which
> it is writing the data.
>
> If you control the receiver software, you can use
> setsockopt(SO_RCVBUF) to explicitly set the receive buffer size, and
> see if that helps. If it doesn't, that would suggest that it is indeed
> the receiving application that is limiting performance.
>
> If you could provide a tcpdump trace (headers only, e.g.,  -s 96)
> taken on the sender, we could check to see if we can see any problems
> in the TCP sender or receiver behavior.
>

Hi,

I published example pcap file under following link:

https://www.dropbox.com/s/v8375ub16seyt1a/test7.pcap?dl=0

I hope it is possible to download it without creating dropbox account.

sdrb

Re: Variable download speed

[Neal Cardwell]

On Tue, Feb 23, 2016 at 8:57 AM,  <sdrb@onet.eu> wrote:
> I published example pcap file under following link:
>
> https://www.dropbox.com/s/v8375ub16seyt1a/test7.pcap?dl=0
>
> I hope it is possible to download it without creating dropbox account.

Thanks for the trace. It looks like for the first 0.8 seconds of the
trace some non-TCP component on the receiving machine (app, CPU, CPU
power-saving mechanisms, NIC, or NIC driver) is limiting throughput to
78Mbit/sec (e.g. the very first window of 10 packets is ACKed at that
rate). Then the throughput increases to over 200 Mbit/sec. AFAICT it
doesn't look like the TCP layer is doing anything wrong. I'd look for
issues with those other components. It might help to report the
receiver application, receiver CPU type, kernel version, NIC, and NIC
driver.

neal

Re: Variable download speed

[sdrb]

On Tue, 23 Feb 2016, Neal Cardwell wrote:

> On Tue, Feb 23, 2016 at 8:57 AM,  <sdrb@onet.eu> wrote:
>> I published example pcap file under following link:
>>
>> https://www.dropbox.com/s/v8375ub16seyt1a/test7.pcap?dl=0
>>
>> I hope it is possible to download it without creating dropbox account.
>
> Thanks for the trace. It looks like for the first 0.8 seconds of the
> trace some non-TCP component on the receiving machine (app, CPU, CPU
> power-saving mechanisms, NIC, or NIC driver) is limiting throughput to
> 78Mbit/sec (e.g. the very first window of 10 packets is ACKed at that
> rate). Then the throughput increases to over 200 Mbit/sec. AFAICT it
> doesn't look like the TCP layer is doing anything wrong. I'd look for
> issues with those other components. It might help to report the
> receiver application, receiver CPU type, kernel version, NIC, and NIC
> driver.

Hi,

The receiver application was "wget" using ftp protocol.
Linux kernel is in version 3.4.11-rt19.

Concerning hardware I have not much information about it.
Everything I've got is:

- CPU: ARMv7 Processor rev 1 (v7l)
- NIC: Broadcom PCI: 14e4:aa52
- RAM: 256MB

Unfortunately I've no much info about kernel. Seems like it is some kind 
of realtime version. Can this realtime extension cause such effect of 
increasing throughput?


I've made a graph with window size and throughput from published 
test7.pcap file:

https://www.dropbox.com/s/cwars3oaoax73fh/wget_test_test7pcap_20160224_1.png?dl=0


sdrb

Re: Variable download speed

[sdrb]

On Tue, 23 Feb 2016, Rick Jones wrote:

> On 02/23/2016 03:24 AM, sdrb@onet.eu wrote:
>>  Hi,
>>
>>  I've got a problem with network on one of my embedded boards.
>>  I'm testing download speed of 256MB file from my PC to embedded board
>>  through 1Gbit ethernet link using ftp.
>>
>>  The problem is that sometimes I achieve 25MB/s and sometimes it is only
>>  14MB/s. There are also situations where the transfer speed starts at
>>  14MB/s and after a few seconds achieves 25MB/s.
>>  I've caught the second case with tcpdump and I noticed that when the speed
>>  is 14MB/s - the tcp window size is 534368 bytes and when the speed
>>  achieved 25MB/s the tcp window size is 933888.
>>
>>  My question is: what causes such dynamic change in the window size (while
>>  transferring data)?  Is it some kernel parameter wrong set or something
>>  like this?
>>  Do I have any influence on such dynamic change in tcp window size?
>
>
> If an application using TCP does not make an explicit setsockopt() call to 
> set the SO_SNDBUF and/or SO_RCVBUF size, then the socket buffer and TCP 
> window size will "autotune" based on what the stack believes to be the 
> correct thing to do.  It will be bounded by the values in the tcp_rmem and 
> tcp_wmem sysctl settings:
>
>
> net.ipv4.tcp_rmem = 4096	87380	6291456
> net.ipv4.tcp_wmem = 4096	16384	4194304
>
> Those are min, initial, max, units of octets (bytes).
>
> If on the other hand an application makes an explicit setsockopt() call, 
> that will be the size of the socket buffer, though it will be "clipped" by 
> the values of:
>
> net.core.rmem_max = 4194304
> net.core.wmem_max = 4194304
>
> Those sysctls will default to different values based on how much memory is in 
> the system.  And I think in the case of those last two, I have tweaked them 
> myself away from their default values.
>
> You might also look at the CPU utilization of all the CPUs of your embedded 
> board, as well as the link-level statistics for your interface, and the 
> netstat statistics.  You would be looking for saturation, and "excessive" 
> drop rates.  I would also suggest testing network performance with something 
> other than FTP.  While one can try to craft things so there is no storage I/O 
> of note, it would still be better to use a network-specific tool such as 
> netperf or iperf.  Minimize the number of variables.
>
> happy benchmarking,

Hi,
To be honest I don't know if "wget" uses setsockopt() in this case.

As you suggested I observed the system information while using wget.

"top" shows that kernel thread which is used by ethernet driver uses 100% 
of first cpu and "wget" application uses at the same time 85% of second 
cpu.


More interesting are the information from vmstat:
procs -----------memory---------- ---swap-- -----io---- -system-- 
------cpu-----
  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id 
wa st
  0  0      0 119688      4  81096    0    0     0     0   55   24  0  6 93 
0  0
  0  0      0 119664      4  81132    0    0     0     0  632 1223  0  0 
100  0  0
  0  0      0 119664      4  81156    0    0     0     0  640 1300  0  0 
100  0  0
  0  0      0 119664      4  81156    0    0     0     0  632 1284  0  0 
100  0  0
  0  0      0 119664      4  81152    0    0     0     0  633 1283  0  0 
100  0  0
  0  0      0 119604      4  81148    0    0     0     0  637 1292  0  0 
100  0  0
  0  0      0 119604      4  81148    0    0     0     0  634 1289  0  0 
100  0  0
  0  0      0 119604      4  81144    0    0     0     0  637 1395  0  0 
100  0  0
  0  0      0 119604      4  81140    0    0     0     0  639 1296  0  0 
100  0  0
  0  0      0 119604      4  81132    0    0     0     0  633 1282  0  0 
100  0  0
  0  0      0 119604      4  81128    0    0     0     0  638 1298  0  0 
100  0  0
  0  0      0 119604      4  81128    0    0     0     0  635 1288  0  0 
100  0  0
  0  0      0 119604      4  81128    0    0     0     0  634 1283  0  0 
100  0  0
  0  0      0 119604      4  81128    0    0     0     0  626 1273  0  0 
100  0  0
  0  0      0 119604      4  81128    0    0     0     0  634 1287  0  0 
100  0  0
  0  0      0 119604      4  81128    0    0     0     0  633 1286  0  0 
100  0  0
  0  0      0 119604      4  81128    0    0     0     0  635 1399  0  0 
100  0  0
  0  0      0 119604      4  81128    0    0     0     0  638 1287  0  0 
100  0  0
  0  0      0 119604      4  81136    0    0     0     0  633 1286  0  0 
100  0  0
  0  0      0 119468      4  81168    0    0     0     0  669 1422  1  0 99 
0  0
start of wget at 14MB/s
  4  0      0 119444      4  81240    0    0     0     0 3541 6869  0 18 82 
0  0
  4  0      0 119444      4  81276    0    0     0     0 10200 20032  4 55 
40  0  0
  4  0      0 119444      4  81276    0    0     0     0 10175 19981  3 57 
39  0  0
  4  0      0 119444      4  81272    0    0     0     0 10170 19986  5 57 
37  0  0
  5  0      0 119444      4  81272    0    0     0     0 10158 19950  4 60 
36  0  0
  6  0      0 119412      4  81272    0    0     0     0 9711 19316  7 56 
37  0  0
  3  0      0 119460      4  81288    0    0     0     0 1828 3400  4 89  7 
0  0
speed 25MB/s
  4  0      0 119460      4  81288    0    0     0     0 1674 3044  9 89  2 
0  0
  4  0      0 119460      4  81288    0    0     0     0 1606 2929  4 93  2 
0  0
  4  0      0 119460      4  81288    0    0     0     0 1560 2832  2 93  4 
0  0
  5  0      0 119460      4  81284    0    0     0     0 1552 2806  3 94  3 
0  0
  4  0      0 119460      4  81280    0    0     0     0 1624 2945  2 95  3 
0  0
  5  0      0 119460      4  81276    0    0     0     0 1228 2165  6 93  1 
0  0
end of wget transmission
  0  0      0 119580      4  81276    0    0     0     0 1066 1935  2 26 72 
0  0
  0  0      0 119604      4  81280    0    0     0     0  608 1043  0  0 
100  0  0


Looks like when the wget transmission starts at 14MB/s - there are a lot 
of interrupts and context switchings (irqs: 10000 cs:20000), but when the 
speed increases to 25MB/s number of interrupts decreases to about 1500.
I suppose that in such a case the ethernet driver is suspected.


I've tested also as you suggested the throughput using iperf - and there 
is no such significant difference in download speed like in wget case.


sdrb

Re: Variable download speed

[Rick Jones]

On 02/23/2016 03:24 AM, sdrb@onet.eu wrote:
> Hi,
>
> I've got a problem with network on one of my embedded boards.
> I'm testing download speed of 256MB file from my PC to embedded board
> through 1Gbit ethernet link using ftp.
>
> The problem is that sometimes I achieve 25MB/s and sometimes it is only
> 14MB/s. There are also situations where the transfer speed starts at
> 14MB/s and after a few seconds achieves 25MB/s.
> I've caught the second case with tcpdump and I noticed that when the speed
> is 14MB/s - the tcp window size is 534368 bytes and when the speed
> achieved 25MB/s the tcp window size is 933888.
>
> My question is: what causes such dynamic change in the window size (while
> transferring data)?  Is it some kernel parameter wrong set or something
> like this?
> Do I have any influence on such dynamic change in tcp window size?


If an application using TCP does not make an explicit setsockopt() call 
to set the SO_SNDBUF and/or SO_RCVBUF size, then the socket buffer and 
TCP window size will "autotune" based on what the stack believes to be 
the correct thing to do.  It will be bounded by the values in the 
tcp_rmem and tcp_wmem sysctl settings:


net.ipv4.tcp_rmem = 4096	87380	6291456
net.ipv4.tcp_wmem = 4096	16384	4194304

Those are min, initial, max, units of octets (bytes).

If on the other hand an application makes an explicit setsockopt() call, 
  that will be the size of the socket buffer, though it will be 
"clipped" by the values of:

net.core.rmem_max = 4194304
net.core.wmem_max = 4194304

Those sysctls will default to different values based on how much memory 
is in the system.  And I think in the case of those last two, I have 
tweaked them myself away from their default values.

You might also look at the CPU utilization of all the CPUs of your 
embedded board, as well as the link-level statistics for your interface, 
and the netstat statistics.  You would be looking for saturation, and 
"excessive" drop rates.  I would also suggest testing network 
performance with something other than FTP.  While one can try to craft 
things so there is no storage I/O of note, it would still be better to 
use a network-specific tool such as netperf or iperf.  Minimize the 
number of variables.

happy benchmarking,

rick jones

Variable download speed

[sdrb]

Hi,

I've got a problem with network on one of my embedded boards.
I'm testing download speed of 256MB file from my PC to embedded board
through 1Gbit ethernet link using ftp.

The problem is that sometimes I achieve 25MB/s and sometimes it is only
14MB/s. There are also situations where the transfer speed starts at
14MB/s and after a few seconds achieves 25MB/s.
I've caught the second case with tcpdump and I noticed that when the speed
is 14MB/s - the tcp window size is 534368 bytes and when the speed
achieved 25MB/s the tcp window size is 933888.

My question is: what causes such dynamic change in the window size (while
transferring data)?  Is it some kernel parameter wrong set or something
like this?
Do I have any influence on such dynamic change in tcp window size?

sdrb