* network stream fairness
@ 2015-11-09 15:41 Niklas Cassel
2015-11-09 15:50 ` Eric Dumazet
0 siblings, 1 reply; 14+ messages in thread
From: Niklas Cassel @ 2015-11-09 15:41 UTC (permalink / raw)
To: edumazet; +Cc: netdev
I have a ethernet driver for a 100 Mbps NIC.
The NIC has dedicated hardware for offloading.
The driver has implemented TSO, GSO and BQL.
Since the CPU on the SoC is rather weak, I'd rather
not increase the CPU load by turning off offloading.
Since commit
605ad7f184b6 ("tcp: refine TSO autosizing")
the bandwidth is no longer fair between streams.
see output at the end of the mail, where I'm testing with 2 streams.
If I revert 605ad7f184b6 on 4.3, I get a stable 45 Mbps per stream.
I can also use vanilla 4.3 and do:
echo 3000 > /sys/class/net/eth0/queues/tx-0/byte_queue_limits/limit_max
to also get a stable 45 Mbps per stream.
My question is, am I supposed to set the BQL limit explicitly?
It is possible that I have missed something in my driver,
but my understanding is that the TCP stack sets and adjusts
the BQL limit automatically.
Perhaps the following info might help:
After running iperf3 on vanilla 4.3:
/sys/class/net/eth0/queues/tx-0/byte_queue_limits/
limit 89908
limit_max 1879048192
After running iperf3 on vanilla 4.3 + BQL explicitly set:
/sys/class/net/eth0/queues/tx-0/byte_queue_limits/
limit 3000
limit_max 3000
After running iperf3 on 4.3 + 605ad7f184b6 reverted:
/sys/class/net/eth0/queues/tx-0/byte_queue_limits/
limit 8886
limit_max 1879048192
Vanilla 4.3, no BQL limit explicitly set.
iperf3 -R -c 192.168.0.141 -t 240 -P 2
Connecting to host 192.168.0.141, port 5201
Reverse mode, remote host 192.168.0.141 is sending
[ 4] local 192.168.0.1 port 50902 connected to 192.168.0.141 port 5201
[ 6] local 192.168.0.1 port 50904 connected to 192.168.0.141 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 7.24 MBytes 60.7 Mbits/sec
[ 6] 0.00-1.00 sec 3.56 MBytes 29.9 Mbits/sec
[SUM] 0.00-1.00 sec 10.8 MBytes 90.6 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 1.00-2.00 sec 7.40 MBytes 62.1 Mbits/sec
[ 6] 1.00-2.00 sec 3.40 MBytes 28.5 Mbits/sec
[SUM] 1.00-2.00 sec 10.8 MBytes 90.6 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 2.00-3.00 sec 7.45 MBytes 62.5 Mbits/sec
[ 6] 2.00-3.00 sec 3.36 MBytes 28.2 Mbits/sec
[SUM] 2.00-3.00 sec 10.8 MBytes 90.7 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 3.00-4.00 sec 7.72 MBytes 64.8 Mbits/sec
[ 6] 3.00-4.00 sec 3.08 MBytes 25.9 Mbits/sec
[SUM] 3.00-4.00 sec 10.8 MBytes 90.7 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 4.00-5.00 sec 8.01 MBytes 67.2 Mbits/sec
[ 6] 4.00-5.00 sec 2.80 MBytes 23.5 Mbits/sec
[SUM] 4.00-5.00 sec 10.8 MBytes 90.7 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 5.00-6.00 sec 7.91 MBytes 66.3 Mbits/sec
[ 6] 5.00-6.00 sec 2.90 MBytes 24.3 Mbits/sec
[SUM] 5.00-6.00 sec 10.8 MBytes 90.7 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 6.00-7.00 sec 7.77 MBytes 65.2 Mbits/sec
[ 6] 6.00-7.00 sec 3.04 MBytes 25.5 Mbits/sec
[SUM] 6.00-7.00 sec 10.8 MBytes 90.7 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 7.00-8.00 sec 6.99 MBytes 58.6 Mbits/sec
[ 6] 7.00-8.00 sec 3.82 MBytes 32.0 Mbits/sec
[SUM] 7.00-8.00 sec 10.8 MBytes 90.7 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 8.00-9.00 sec 7.08 MBytes 59.4 Mbits/sec
[ 6] 8.00-9.00 sec 3.73 MBytes 31.3 Mbits/sec
[SUM] 8.00-9.00 sec 10.8 MBytes 90.7 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 9.00-10.00 sec 6.93 MBytes 58.1 Mbits/sec
[ 6] 9.00-10.00 sec 3.88 MBytes 32.6 Mbits/sec
[SUM] 9.00-10.00 sec 10.8 MBytes 90.7 Mbits/sec
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: network stream fairness
2015-11-09 15:41 network stream fairness Niklas Cassel
@ 2015-11-09 15:50 ` Eric Dumazet
2015-11-09 15:53 ` Niklas Cassel
0 siblings, 1 reply; 14+ messages in thread
From: Eric Dumazet @ 2015-11-09 15:50 UTC (permalink / raw)
To: Niklas Cassel; +Cc: edumazet, netdev
On Mon, 2015-11-09 at 16:41 +0100, Niklas Cassel wrote:
> I have a ethernet driver for a 100 Mbps NIC.
> The NIC has dedicated hardware for offloading.
> The driver has implemented TSO, GSO and BQL.
> Since the CPU on the SoC is rather weak, I'd rather
> not increase the CPU load by turning off offloading.
>
> Since commit
> 605ad7f184b6 ("tcp: refine TSO autosizing")
>
> the bandwidth is no longer fair between streams.
> see output at the end of the mail, where I'm testing with 2 streams.
>
>
> If I revert 605ad7f184b6 on 4.3, I get a stable 45 Mbps per stream.
>
> I can also use vanilla 4.3 and do:
> echo 3000 > /sys/class/net/eth0/queues/tx-0/byte_queue_limits/limit_max
> to also get a stable 45 Mbps per stream.
>
> My question is, am I supposed to set the BQL limit explicitly?
> It is possible that I have missed something in my driver,
> but my understanding is that the TCP stack sets and adjusts
> the BQL limit automatically.
>
>
> Perhaps the following info might help:
>
> After running iperf3 on vanilla 4.3:
> /sys/class/net/eth0/queues/tx-0/byte_queue_limits/
> limit 89908
> limit_max 1879048192
>
> After running iperf3 on vanilla 4.3 + BQL explicitly set:
> /sys/class/net/eth0/queues/tx-0/byte_queue_limits/
> limit 3000
> limit_max 3000
>
> After running iperf3 on 4.3 + 605ad7f184b6 reverted:
> /sys/class/net/eth0/queues/tx-0/byte_queue_limits/
> limit 8886
> limit_max 1879048192
>
There is absolutely nothing ensuring fairness among multiple TCP flows.
One TCP flow can very easily grab whole bandwidth for itself, there are
numerous descriptions of this phenomena in various TCP studies.
This is why we have packet schedulers ;)
tc qdisc replace dev eth0 root fq
This will probably help : No need to change BQL settings, so that you
keep minimal latencies for the interactive traffic (like ping)
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: network stream fairness
2015-11-09 15:50 ` Eric Dumazet
@ 2015-11-09 15:53 ` Niklas Cassel
2015-11-09 16:07 ` Eric Dumazet
0 siblings, 1 reply; 14+ messages in thread
From: Niklas Cassel @ 2015-11-09 15:53 UTC (permalink / raw)
To: Eric Dumazet; +Cc: edumazet, netdev
On 11/09/2015 04:50 PM, Eric Dumazet wrote:
> On Mon, 2015-11-09 at 16:41 +0100, Niklas Cassel wrote:
>> I have a ethernet driver for a 100 Mbps NIC.
>> The NIC has dedicated hardware for offloading.
>> The driver has implemented TSO, GSO and BQL.
>> Since the CPU on the SoC is rather weak, I'd rather
>> not increase the CPU load by turning off offloading.
>>
>> Since commit
>> 605ad7f184b6 ("tcp: refine TSO autosizing")
>>
>> the bandwidth is no longer fair between streams.
>> see output at the end of the mail, where I'm testing with 2 streams.
>>
>>
>> If I revert 605ad7f184b6 on 4.3, I get a stable 45 Mbps per stream.
>>
>> I can also use vanilla 4.3 and do:
>> echo 3000 > /sys/class/net/eth0/queues/tx-0/byte_queue_limits/limit_max
>> to also get a stable 45 Mbps per stream.
>>
>> My question is, am I supposed to set the BQL limit explicitly?
>> It is possible that I have missed something in my driver,
>> but my understanding is that the TCP stack sets and adjusts
>> the BQL limit automatically.
>>
>>
>> Perhaps the following info might help:
>>
>> After running iperf3 on vanilla 4.3:
>> /sys/class/net/eth0/queues/tx-0/byte_queue_limits/
>> limit 89908
>> limit_max 1879048192
>>
>> After running iperf3 on vanilla 4.3 + BQL explicitly set:
>> /sys/class/net/eth0/queues/tx-0/byte_queue_limits/
>> limit 3000
>> limit_max 3000
>>
>> After running iperf3 on 4.3 + 605ad7f184b6 reverted:
>> /sys/class/net/eth0/queues/tx-0/byte_queue_limits/
>> limit 8886
>> limit_max 1879048192
>>
>
> There is absolutely nothing ensuring fairness among multiple TCP flows.
>
> One TCP flow can very easily grab whole bandwidth for itself, there are
> numerous descriptions of this phenomena in various TCP studies.
>
> This is why we have packet schedulers ;)
Oh.. How stupid of me, I forgot to mention.. all of the measurements were
done with fq_codel.
>
> tc qdisc replace dev eth0 root fq
>
> This will probably help : No need to change BQL settings, so that you
> keep minimal latencies for the interactive traffic (like ping)
>
>
>
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: network stream fairness
2015-11-09 15:53 ` Niklas Cassel
@ 2015-11-09 16:07 ` Eric Dumazet
2015-11-09 16:24 ` Eric Dumazet
2015-11-20 15:33 ` Niklas Cassel
0 siblings, 2 replies; 14+ messages in thread
From: Eric Dumazet @ 2015-11-09 16:07 UTC (permalink / raw)
To: Niklas Cassel; +Cc: edumazet, netdev
On Mon, 2015-11-09 at 16:53 +0100, Niklas Cassel wrote:
> On 11/09/2015 04:50 PM, Eric Dumazet wrote:
> > On Mon, 2015-11-09 at 16:41 +0100, Niklas Cassel wrote:
> >> I have a ethernet driver for a 100 Mbps NIC.
> >> The NIC has dedicated hardware for offloading.
> >> The driver has implemented TSO, GSO and BQL.
> >> Since the CPU on the SoC is rather weak, I'd rather
> >> not increase the CPU load by turning off offloading.
> >>
> >> Since commit
> >> 605ad7f184b6 ("tcp: refine TSO autosizing")
> >>
> >> the bandwidth is no longer fair between streams.
> >> see output at the end of the mail, where I'm testing with 2 streams.
> >>
> >>
> >> If I revert 605ad7f184b6 on 4.3, I get a stable 45 Mbps per stream.
> >>
> >> I can also use vanilla 4.3 and do:
> >> echo 3000 > /sys/class/net/eth0/queues/tx-0/byte_queue_limits/limit_max
> >> to also get a stable 45 Mbps per stream.
> >>
> >> My question is, am I supposed to set the BQL limit explicitly?
> >> It is possible that I have missed something in my driver,
> >> but my understanding is that the TCP stack sets and adjusts
> >> the BQL limit automatically.
> >>
> >>
> >> Perhaps the following info might help:
> >>
> >> After running iperf3 on vanilla 4.3:
> >> /sys/class/net/eth0/queues/tx-0/byte_queue_limits/
> >> limit 89908
> >> limit_max 1879048192
> >>
> >> After running iperf3 on vanilla 4.3 + BQL explicitly set:
> >> /sys/class/net/eth0/queues/tx-0/byte_queue_limits/
> >> limit 3000
> >> limit_max 3000
> >>
> >> After running iperf3 on 4.3 + 605ad7f184b6 reverted:
> >> /sys/class/net/eth0/queues/tx-0/byte_queue_limits/
> >> limit 8886
> >> limit_max 1879048192
> >>
> >
> > There is absolutely nothing ensuring fairness among multiple TCP flows.
> >
> > One TCP flow can very easily grab whole bandwidth for itself, there are
> > numerous descriptions of this phenomena in various TCP studies.
> >
> > This is why we have packet schedulers ;)
>
> Oh.. How stupid of me, I forgot to mention.. all of the measurements were
> done with fq_codel.
Your numbers suggest a cwnd growth then, which might show a CC bug.
Please run the following when your iper3 runs on regular 4.3 kernel
for i in `seq 1 10`
do
ss -temoi dst 192.168.0.141
sleep 1
done
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: network stream fairness
2015-11-09 16:07 ` Eric Dumazet
@ 2015-11-09 16:24 ` Eric Dumazet
2015-11-09 16:50 ` Niklas Cassel
2015-11-20 15:33 ` Niklas Cassel
1 sibling, 1 reply; 14+ messages in thread
From: Eric Dumazet @ 2015-11-09 16:24 UTC (permalink / raw)
To: Niklas Cassel; +Cc: edumazet, netdev
On Mon, 2015-11-09 at 08:07 -0800, Eric Dumazet wrote:
> Your numbers suggest a cwnd growth then, which might show a CC bug.
>
> Please run the following when your iper3 runs on regular 4.3 kernel
>
> for i in `seq 1 10`
> do
> ss -temoi dst 192.168.0.141
> sleep 1
> done
>
Another thing to try is to change tcp_limit_output_bytes back to 131072
Of course, your driver be the problem as well, but I am guessing it is
not upstream ?
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: network stream fairness
2015-11-09 16:24 ` Eric Dumazet
@ 2015-11-09 16:50 ` Niklas Cassel
2015-11-09 17:23 ` Eric Dumazet
0 siblings, 1 reply; 14+ messages in thread
From: Niklas Cassel @ 2015-11-09 16:50 UTC (permalink / raw)
To: Eric Dumazet; +Cc: edumazet, netdev
On 11/09/2015 05:24 PM, Eric Dumazet wrote:
> On Mon, 2015-11-09 at 08:07 -0800, Eric Dumazet wrote:
>
>> Your numbers suggest a cwnd growth then, which might show a CC bug.
>>
>> Please run the following when your iper3 runs on regular 4.3 kernel
>>
>> for i in `seq 1 10`
>> do
>> ss -temoi dst 192.168.0.141
>> sleep 1
>> done
>>
See output at the end of the mail.
iperf3 output sample:
[ 4] 24.00-24.53 sec 1.92 MBytes 30.2 Mbits/sec
[ 6] 24.00-24.53 sec 3.84 MBytes 60.5 Mbits/sec
[SUM] 24.00-24.53 sec 5.76 MBytes 90.7 Mbits/sec
>
> Another thing to try is to change tcp_limit_output_bytes back to 131072
I tried both
sysctl -w net.ipv4.tcp_limit_output_bytes=131072
sysctl -w net.ipv4.tcp_limit_output_bytes=13107
Neither appears to solve the problem.
I don't know why but
echo 3000 > /sys/class/net/eth0/queues/tx-0/byte_queue_limits/limit_max
solves it. Feels weird that setting tcp_limit_output_bytes doesn't fix it.
>
> Of course, your driver be the problem as well, but I am guessing it is
> not upstream ?
It's not upstreamed, but it is GPL, so I could probably upload it somewhere
if that helps.
for i in `seq 1 20`; do ss -temoi dst 192.168.0.141; sleep 1; done
State Recv-Q Send-Q Local Address:Port Peer Address:Port
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 106 192.168.0.1:54574 192.168.0.141:5201 timer:(on,190ms,0) rto:0.2 ato:0.04 cwnd:10 qack:14 bidir uid:20283 ino:7902591 sk:ffff880223311800
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.0.1:54576 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901837 sk:ffff880143735800
ESTAB 0 0 192.168.0.1:54574 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 qack:10 uid:20283 ino:7902591 sk:ffff880223311800
ESTAB 0 0 192.168.0.1:54578 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901838 sk:ffff880143735040
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.0.1:54576 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901837 sk:ffff880143735800
ESTAB 0 0 192.168.0.1:54574 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 qack:10 uid:20283 ino:7902591 sk:ffff880223311800
ESTAB 0 0 192.168.0.1:54578 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901838 sk:ffff880143735040
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.0.1:54576 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901837 sk:ffff880143735800
ESTAB 0 0 192.168.0.1:54574 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 qack:10 uid:20283 ino:7902591 sk:ffff880223311800
ESTAB 0 0 192.168.0.1:54578 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901838 sk:ffff880143735040
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.0.1:54576 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901837 sk:ffff880143735800
ESTAB 0 0 192.168.0.1:54574 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 qack:10 uid:20283 ino:7902591 sk:ffff880223311800
ESTAB 0 0 192.168.0.1:54578 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901838 sk:ffff880143735040
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.0.1:54576 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901837 sk:ffff880143735800
ESTAB 0 0 192.168.0.1:54574 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 qack:10 uid:20283 ino:7902591 sk:ffff880223311800
ESTAB 0 0 192.168.0.1:54578 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901838 sk:ffff880143735040
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.0.1:54576 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901837 sk:ffff880143735800
ESTAB 0 0 192.168.0.1:54574 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 qack:10 uid:20283 ino:7902591 sk:ffff880223311800
ESTAB 0 0 192.168.0.1:54578 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901838 sk:ffff880143735040
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.0.1:54576 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901837 sk:ffff880143735800
ESTAB 0 0 192.168.0.1:54574 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 qack:10 uid:20283 ino:7902591 sk:ffff880223311800
ESTAB 0 0 192.168.0.1:54578 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901838 sk:ffff880143735040
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.0.1:54576 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901837 sk:ffff880143735800
ESTAB 0 0 192.168.0.1:54574 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 qack:10 uid:20283 ino:7902591 sk:ffff880223311800
ESTAB 0 0 192.168.0.1:54578 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901838 sk:ffff880143735040
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 1448 0 192.168.0.1:54576 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901837 sk:ffff880143735800
ESTAB 0 0 192.168.0.1:54574 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 qack:10 uid:20283 ino:7902591 sk:ffff880223311800
ESTAB 0 0 192.168.0.1:54578 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901838 sk:ffff880143735040
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.0.1:54576 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901837 sk:ffff880143735800
ESTAB 0 0 192.168.0.1:54574 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 qack:10 uid:20283 ino:7902591 sk:ffff880223311800
ESTAB 1448 0 192.168.0.1:54578 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901838 sk:ffff880143735040
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.0.1:54576 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901837 sk:ffff880143735800
ESTAB 0 0 192.168.0.1:54574 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 qack:10 uid:20283 ino:7902591 sk:ffff880223311800
ESTAB 0 0 192.168.0.1:54578 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901838 sk:ffff880143735040
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.0.1:54576 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901837 sk:ffff880143735800
ESTAB 0 0 192.168.0.1:54574 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 qack:10 uid:20283 ino:7902591 sk:ffff880223311800
ESTAB 0 0 192.168.0.1:54578 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901838 sk:ffff880143735040
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 1448 0 192.168.0.1:54576 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901837 sk:ffff880143735800
ESTAB 0 0 192.168.0.1:54574 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 qack:10 uid:20283 ino:7902591 sk:ffff880223311800
ESTAB 0 0 192.168.0.1:54578 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901838 sk:ffff880143735040
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.0.1:54576 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901837 sk:ffff880143735800
ESTAB 0 0 192.168.0.1:54574 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 qack:10 uid:20283 ino:7902591 sk:ffff880223311800
ESTAB 0 0 192.168.0.1:54578 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901838 sk:ffff880143735040
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.0.1:54576 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901837 sk:ffff880143735800
ESTAB 0 0 192.168.0.1:54574 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 qack:10 uid:20283 ino:7902591 sk:ffff880223311800
ESTAB 1448 0 192.168.0.1:54578 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901838 sk:ffff880143735040
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.0.1:54576 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901837 sk:ffff880143735800
ESTAB 0 0 192.168.0.1:54574 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 qack:10 uid:20283 ino:7902591 sk:ffff880223311800
ESTAB 0 0 192.168.0.1:54578 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901838 sk:ffff880143735040
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.0.1:54576 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901837 sk:ffff880143735800
ESTAB 0 0 192.168.0.1:54574 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 qack:10 uid:20283 ino:7902591 sk:ffff880223311800
ESTAB 0 0 192.168.0.1:54578 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901838 sk:ffff880143735040
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.0.1:54576 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901837 sk:ffff880143735800
ESTAB 0 0 192.168.0.1:54574 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 qack:10 uid:20283 ino:7902591 sk:ffff880223311800
ESTAB 0 0 192.168.0.1:54578 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901838 sk:ffff880143735040
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: network stream fairness
2015-11-09 16:50 ` Niklas Cassel
@ 2015-11-09 17:23 ` Eric Dumazet
2015-11-10 9:38 ` Niklas Cassel
0 siblings, 1 reply; 14+ messages in thread
From: Eric Dumazet @ 2015-11-09 17:23 UTC (permalink / raw)
To: Niklas Cassel; +Cc: edumazet, netdev
On Mon, 2015-11-09 at 17:50 +0100, Niklas Cassel wrote:
>
> for i in `seq 1 20`; do ss -temoi dst 192.168.0.141; sleep 1; done
...
> ESTAB 0 0 192.168.0.1:54578 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901838 sk:ffff880143735040
> State Recv-Q Send-Q Local Address:Port Peer Address:Port
> ESTAB 0 0 192.168.0.1:54576 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901837 sk:ffff880143735800
> ESTAB 0 0 192.168.0.1:54574 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 qack:10 uid:20283 ino:7902591 sk:ffff880223311800
> ESTAB 1448 0 192.168.0.1:54578 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901838 sk:ffff880143735040
This is the receiver, I would like to see the sender side, since the
changes we talk about are at his side, of course.
Thanks.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: network stream fairness
2015-11-09 17:23 ` Eric Dumazet
@ 2015-11-10 9:38 ` Niklas Cassel
0 siblings, 0 replies; 14+ messages in thread
From: Niklas Cassel @ 2015-11-10 9:38 UTC (permalink / raw)
To: Eric Dumazet; +Cc: edumazet, netdev
On 11/09/2015 06:23 PM, Eric Dumazet wrote:
> On Mon, 2015-11-09 at 17:50 +0100, Niklas Cassel wrote:
>
>>
>> for i in `seq 1 20`; do ss -temoi dst 192.168.0.141; sleep 1; done
> ...
>> ESTAB 0 0 192.168.0.1:54578 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901838 sk:ffff880143735040
>> State Recv-Q Send-Q Local Address:Port Peer Address:Port
>> ESTAB 0 0 192.168.0.1:54576 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901837 sk:ffff880143735800
>> ESTAB 0 0 192.168.0.1:54574 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 qack:10 uid:20283 ino:7902591 sk:ffff880223311800
>> ESTAB 1448 0 192.168.0.1:54578 192.168.0.141:5201 rto:0.2 ato:0.04 cwnd:10 uid:20283 ino:7901838 sk:ffff880143735040
>
> This is the receiver, I would like to see the sender side, since the
> changes we talk about are at his side, of course.
Sorry for that.
iperf3 sample. most samples were equal to this.
[ 4] 10.00-11.00 sec 3.75 MBytes 31.5 Mbits/sec
[ 6] 10.00-11.00 sec 7.05 MBytes 59.2 Mbits/sec
[SUM] 10.00-11.00 sec 10.8 MBytes 90.7 Mbits/sec
# for i in `seq 1 20`; do ./ss -temoi dst 192.168.0.1; sleep 1; done
State Recv-Q Send-Q Local Address:Port Peer Address:Port
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55404 ino:17349 sk:4 <->
skmem:(r0,rb328320,t0,tb44800,f20480,w0,o0,bl0) ts sack cubic wscale:7,5 rto:200 ato:40 mss:1448 cwnd:10 bytes_received:37 segs_out:1 segs_in:1 rcv_space:28960
ESTAB 0 1 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55402 timer:(on,210ms,0) ino:18856 sk:5 <->
skmem:(r0,rb328320,t0,tb44800,f1856,w2240,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:0.03/0.06 ato:40 mss:1448 cwnd:10 bytes_acked:1 bytes_received:143 segs_out:5 segs_in:4 send 3861.3Mbps lastack:40 pacing_rate 7690.6Mbps unacked:1 rcv_space:28960
ESTAB 37 0 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55406 ino:0 sk:6 <->
skmem:(r16576,rb328320,t0,tb44800,f3904,w0,o0,bl0) ts sack cubic wscale:7,5 rto:200 ato:40 mss:1448 cwnd:10 bytes_received:37 segs_out:1 segs_in:1 rcv_space:28960
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 159280 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55404 timer:(on,210ms,0) ino:17349 sk:4 <->
skmem:(r0,rb328320,t27472,tb192640,f120048,w162576,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:5.326/0.789 ato:40 mss:1448 cwnd:43 ssthresh:16 bytes_acked:4500384 bytes_received:37 segs_out:3126 segs_in:1586 send 93.5Mbps lastsnd:10 lastrcv:1040 pacing_rate 112.2Mbps unacked:17 rcv_space:28960
ESTAB 0 0 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55402 ino:18856 sk:5 <->
skmem:(r0,rb328320,t0,tb44800,f0,w0,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:3.738/6.898 ato:40 mss:1448 cwnd:10 bytes_acked:4 bytes_received:143 segs_out:7 segs_in:7 send 31.0Mbps lastsnd:940 lastrcv:1040 lastack:940 pacing_rate 62.0Mbps rcv_space:28960
ESTAB 0 273672 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55406 timer:(on,200ms,0) ino:17351 sk:6 <->
skmem:(r0,rb328320,t55576,tb277760,f44024,w279560,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:4.578/0.644 ato:40 mss:1448 cwnd:62 ssthresh:19 bytes_acked:6114904 bytes_received:37 segs_out:4261 segs_in:2147 send 156.9Mbps lastsnd:10 lastrcv:1040 lastack:10 pacing_rate 188.2Mbps unacked:37 rcv_space:28960
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 136112 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55404 timer:(on,210ms,0) ino:17349 sk:4 <->
skmem:(r0,rb328320,t15776,tb201600,f142512,w140112,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:6.518/1.397 ato:40 mss:1448 cwnd:45 ssthresh:16 bytes_acked:8954432 bytes_received:37 segs_out:6196 segs_in:3124 send 80.0Mbps lastsnd:10 lastrcv:2090 pacing_rate 96.0Mbps unacked:11 rcv_space:28960
ESTAB 0 0 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55402 ino:18856 sk:5 <->
skmem:(r0,rb328320,t0,tb44800,f0,w0,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:3.738/6.898 ato:40 mss:1448 cwnd:10 bytes_acked:4 bytes_received:143 segs_out:7 segs_in:7 send 31.0Mbps lastsnd:1990 lastrcv:2090 lastack:1990 pacing_rate 62.0Mbps rcv_space:28960
ESTAB 0 360552 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55406 timer:(on,210ms,0) ino:17351 sk:6 <->
skmem:(r0,rb328320,t31112,tb344960,f48776,w364920,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:5.294/0.545 ato:40 mss:1448 cwnd:77 ssthresh:19 bytes_acked:13586584 bytes_received:37 segs_out:9405 segs_in:4727 send 168.5Mbps lastsnd:10 lastrcv:2090 pacing_rate 202.2Mbps unacked:21 rcv_space:28960
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 141904 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55404 timer:(on,210ms,0) ino:17349 sk:4 <->
skmem:(r0,rb328320,t25168,tb201600,f136720,w145904,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:7.163/0.728 ato:40 mss:1448 cwnd:45 ssthresh:16 bytes_acked:12727920 bytes_received:37 segs_out:8806 segs_in:4427 send 72.8Mbps lastrcv:3130 pacing_rate 87.3Mbps unacked:15 rcv_space:28960
ESTAB 0 0 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55402 ino:18856 sk:5 <->
skmem:(r0,rb328320,t0,tb44800,f0,w0,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:3.738/6.898 ato:40 mss:1448 cwnd:10 bytes_acked:4 bytes_received:143 segs_out:7 segs_in:7 send 31.0Mbps lastsnd:3030 lastrcv:3130 lastack:3030 pacing_rate 62.0Mbps rcv_space:28960
ESTAB 0 295392 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55406 timer:(on,210ms,0) ino:17351 sk:6 <->
skmem:(r0,rb328320,t32560,tb362880,f146704,w299760,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:5.195/1.377 ato:40 mss:1448 cwnd:81 ssthresh:19 bytes_acked:21666424 bytes_received:37 segs_out:14987 segs_in:7517 send 180.6Mbps lastrcv:3130 pacing_rate 216.7Mbps unacked:23 rcv_space:28960
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 222992 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55404 timer:(on,210ms,0) ino:17349 sk:4 <->
skmem:(r0,rb328320,t20936,tb201600,f55040,w227584,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:7.367/2.105 ato:40 mss:1448 cwnd:45 ssthresh:16 bytes_acked:16295792 bytes_received:37 segs_out:11268 segs_in:5659 send 70.8Mbps lastrcv:4170 pacing_rate 84.9Mbps unacked:13 rcv_space:28960
ESTAB 0 0 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55402 ino:18856 sk:5 <->
skmem:(r0,rb328320,t0,tb44800,f0,w0,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:3.738/6.898 ato:40 mss:1448 cwnd:10 bytes_acked:4 bytes_received:143 segs_out:7 segs_in:7 send 31.0Mbps lastsnd:4070 lastrcv:4170 lastack:4070 pacing_rate 62.0Mbps rcv_space:28960
ESTAB 0 422816 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55406 timer:(on,210ms,0) ino:17351 sk:6 <->
skmem:(r0,rb328320,t55576,tb421120,f66432,w429184,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:6.213/1.45 ato:40 mss:1448 cwnd:94 ssthresh:19 bytes_acked:29879480 bytes_received:37 segs_out:20671 segs_in:10353 send 175.3Mbps lastrcv:4170 pacing_rate 210.3Mbps unacked:35 rcv_space:28960
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 156384 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55404 timer:(on,210ms,0) ino:17349 sk:4 <->
skmem:(r0,rb328320,t20936,tb201600,f122240,w160384,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:7.188/2.283 ato:40 mss:1448 cwnd:45 ssthresh:16 bytes_acked:19620400 bytes_received:37 segs_out:13564 segs_in:6807 send 72.5Mbps lastrcv:5230 pacing_rate 87.0Mbps unacked:13 rcv_space:28960
ESTAB 0 0 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55402 ino:18856 sk:5 <->
skmem:(r0,rb328320,t0,tb44800,f0,w0,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:3.738/6.898 ato:40 mss:1448 cwnd:10 bytes_acked:4 bytes_received:143 segs_out:7 segs_in:7 send 31.0Mbps lastsnd:5130 lastrcv:5230 lastack:5130 pacing_rate 62.0Mbps rcv_space:28960
ESTAB 0 403992 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55406 timer:(on,210ms,0) ino:17351 sk:6 <->
skmem:(r0,rb328320,t92480,tb443520,f105032,w411064,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:5.042/0.588 ato:40 mss:1448 cwnd:99 ssthresh:19 bytes_acked:38564584 bytes_received:37 segs_out:26688 segs_in:13352 send 227.5Mbps lastrcv:5230 pacing_rate 272.9Mbps unacked:54 rcv_space:28960
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 217200 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55404 timer:(on,210ms,0) ino:17349 sk:4 <->
skmem:(r0,rb328320,t20232,tb201600,f61536,w221088,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:6.221/0.969 ato:40 mss:1448 cwnd:45 ssthresh:16 bytes_acked:22947904 bytes_received:37 segs_out:15862 segs_in:7956 send 83.8Mbps lastrcv:6270 pacing_rate 100.5Mbps unacked:13 rcv_space:28960
ESTAB 0 0 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55402 ino:18856 sk:5 <->
skmem:(r0,rb328320,t0,tb44800,f0,w0,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:3.738/6.898 ato:40 mss:1448 cwnd:10 bytes_acked:4 bytes_received:143 segs_out:7 segs_in:7 send 31.0Mbps lastsnd:6170 lastrcv:6270 lastack:6170 pacing_rate 62.0Mbps rcv_space:28960
ESTAB 0 338832 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55406 timer:(on,210ms,0) ino:17351 sk:6 <->
skmem:(r0,rb328320,t70056,tb443520,f167968,w348128,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:6.561/1.376 ato:40 mss:1448 cwnd:99 ssthresh:19 bytes_acked:47035384 bytes_received:37 segs_out:32515 segs_in:16277 send 174.8Mbps lastrcv:6270 pacing_rate 209.7Mbps unacked:31 rcv_space:28960
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 165072 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55404 timer:(on,200ms,0) ino:17349 sk:4 <->
skmem:(r0,rb328320,t23088,tb201600,f112144,w170480,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:7.989/1.849 ato:40 mss:1448 cwnd:45 ssthresh:16 bytes_acked:26258032 bytes_received:37 segs_out:18149 segs_in:9099 send 65.2Mbps lastsnd:10 lastrcv:7320 lastack:10 pacing_rate 78.3Mbps unacked:14 rcv_space:28960
ESTAB 0 0 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55402 ino:18856 sk:5 <->
skmem:(r0,rb328320,t0,tb44800,f0,w0,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:3.738/6.898 ato:40 mss:1448 cwnd:10 bytes_acked:4 bytes_received:143 segs_out:7 segs_in:7 send 31.0Mbps lastsnd:7220 lastrcv:7320 lastack:7220 pacing_rate 62.0Mbps rcv_space:28960
ESTAB 0 375784 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55406 timer:(on,210ms,0) ino:17351 sk:6 <->
skmem:(r0,rb328320,t78744,tb443520,f133160,w382936,o0,bl198912) ts sack cubic wscale:7,5 rto:210 rtt:4.957/1.656 ato:40 mss:1448 cwnd:99 ssthresh:19 bytes_acked:55535144 bytes_received:37 segs_out:38415 segs_in:19222 send 231.4Mbps lastrcv:7320 pacing_rate 277.6Mbps unacked:61 rcv_space:28960
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 247608 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55404 timer:(on,210ms,0) ino:17349 sk:4 <->
skmem:(r0,rb328320,t30368,tb313600,f128136,w252792,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:7.656/0.514 ato:40 mss:1448 cwnd:70 ssthresh:16 bytes_acked:29498656 bytes_received:37 segs_out:20394 segs_in:10218 send 105.9Mbps lastrcv:8360 pacing_rate 127.1Mbps unacked:21 rcv_space:28960
ESTAB 0 0 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55402 ino:18856 sk:5 <->
skmem:(r0,rb328320,t0,tb44800,f0,w0,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:3.738/6.898 ato:40 mss:1448 cwnd:10 bytes_acked:4 bytes_received:143 segs_out:7 segs_in:7 send 31.0Mbps lastsnd:8260 lastrcv:8360 lastack:8260 pacing_rate 62.0Mbps rcv_space:28960
ESTAB 0 677664 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55406 timer:(on,210ms,0) ino:17351 sk:6 <->
skmem:(r0,rb328320,t81640,tb663040,f47488,w685696,o0,bl66304) ts sack cubic wscale:7,5 rto:210 rtt:6.851/1.229 ato:40 mss:1448 cwnd:148 ssthresh:19 bytes_acked:64159432 bytes_received:37 segs_out:44357 segs_in:22193 send 250.2Mbps lastrcv:8360 pacing_rate 310.5Mbps unacked:47 rcv_space:28960
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 249056 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55404 timer:(on,210ms,0) ino:17349 sk:4 <->
skmem:(r0,rb328320,t28176,tb313600,f125984,w254944,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:8.744/0.863 ato:40 mss:1448 cwnd:70 ssthresh:16 bytes_acked:33406808 bytes_received:37 segs_out:23091 segs_in:11567 send 92.7Mbps lastrcv:9400 pacing_rate 111.3Mbps unacked:19 rcv_space:28960
ESTAB 0 0 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55402 ino:18856 sk:5 <->
skmem:(r0,rb328320,t0,tb44800,f0,w0,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:3.738/6.898 ato:40 mss:1448 cwnd:10 bytes_acked:4 bytes_received:143 segs_out:7 segs_in:7 send 31.0Mbps lastsnd:9300 lastrcv:9400 lastack:9300 pacing_rate 62.0Mbps rcv_space:28960
ESTAB 0 464808 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55406 timer:(on,210ms,0) ino:17351 sk:6 <->
skmem:(r0,rb328320,t65008,tb663040,f262120,w471064,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:7.134/0.497 ato:40 mss:1448 cwnd:148 ssthresh:19 bytes_acked:72061168 bytes_received:37 segs_out:49795 segs_in:24945 send 240.3Mbps lastsnd:10 lastrcv:9410 pacing_rate 288.4Mbps unacked:28 rcv_space:28960
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 288152 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55404 timer:(on,210ms,0) ino:17349 sk:4 <->
skmem:(r0,rb328320,t15184,tb313600,f86888,w294040,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:9.285/1.307 ato:40 mss:1448 cwnd:70 ssthresh:16 bytes_acked:37212152 bytes_received:37 segs_out:25710 segs_in:12881 send 87.3Mbps lastrcv:10450 pacing_rate 104.8Mbps unacked:10 rcv_space:28960
ESTAB 0 0 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55402 ino:18856 sk:5 <->
skmem:(r0,rb328320,t0,tb44800,f0,w0,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:3.738/6.898 ato:40 mss:1448 cwnd:10 bytes_acked:4 bytes_received:143 segs_out:7 segs_in:7 send 31.0Mbps lastsnd:10350 lastrcv:10450 lastack:10350 pacing_rate 62.0Mbps rcv_space:28960
ESTAB 0 564720 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55406 timer:(on,210ms,0) ino:17351 sk:6 <->
skmem:(r0,rb328320,t51272,tb663040,f162432,w570752,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:8.322/1.752 ato:40 mss:1448 cwnd:148 ssthresh:19 bytes_acked:80106256 bytes_received:37 segs_out:55343 segs_in:27723 send 206.0Mbps lastsnd:10 lastrcv:10450 pacing_rate 247.2Mbps unacked:20 rcv_space:28960
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 267880 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55404 timer:(on,210ms,0) ino:17349 sk:4 <->
skmem:(r0,rb328320,t35304,tb313600,f107864,w273064,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:8.76/0.731 ato:40 mss:1448 cwnd:70 ssthresh:16 bytes_acked:41011704 bytes_received:37 segs_out:28347 segs_in:14193 send 92.6Mbps lastrcv:11490 pacing_rate 111.1Mbps unacked:23 rcv_space:28960
ESTAB 0 0 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55402 ino:18856 sk:5 <->
skmem:(r0,rb328320,t0,tb44800,f0,w0,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:3.738/6.898 ato:40 mss:1448 cwnd:10 bytes_acked:4 bytes_received:143 segs_out:7 segs_in:7 send 31.0Mbps lastsnd:11390 lastrcv:11490 lastack:11390 pacing_rate 62.0Mbps rcv_space:28960
ESTAB 0 486528 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55406 timer:(on,210ms,0) ino:17351 sk:6 <->
skmem:(r0,rb328320,t88880,tb663040,f240400,w492784,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:5.538/0.738 ato:40 mss:1448 cwnd:148 ssthresh:19 bytes_acked:88133968 bytes_received:37 segs_out:60921 segs_in:30495 send 309.6Mbps lastrcv:11490 pacing_rate 371.4Mbps unacked:54 rcv_space:28960
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 298288 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55404 timer:(on,210ms,0) ino:17349 sk:4 <->
skmem:(r0,rb328320,t33264,tb313600,f77456,w303472,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:5.306/0.724 ato:40 mss:1448 cwnd:70 ssthresh:16 bytes_acked:45347016 bytes_received:37 segs_out:31341 segs_in:15707 send 152.8Mbps lastrcv:12540 pacing_rate 183.4Mbps unacked:23 rcv_space:28960
ESTAB 0 0 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55402 ino:18856 sk:5 <->
skmem:(r0,rb328320,t0,tb44800,f0,w0,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:3.738/6.898 ato:40 mss:1448 cwnd:10 bytes_acked:4 bytes_received:143 segs_out:7 segs_in:7 send 31.0Mbps lastsnd:12440 lastrcv:12540 lastack:12440 pacing_rate 62.0Mbps rcv_space:28960
ESTAB 0 435848 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55406 timer:(on,210ms,0) ino:17351 sk:6 <->
skmem:(r0,rb328320,t65008,tb663040,f291784,w441400,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:5.295/0.793 ato:40 mss:1448 cwnd:148 ssthresh:19 bytes_acked:95678048 bytes_received:37 segs_out:66107 segs_in:33100 send 323.8Mbps lastrcv:12540 pacing_rate 388.5Mbps unacked:30 rcv_space:28960
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 272224 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55404 timer:(on,210ms,0) ino:17349 sk:4 <->
skmem:(r0,rb328320,t42544,tb313600,f103520,w277408,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:7.634/0.526 ato:40 mss:1448 cwnd:70 ssthresh:16 bytes_acked:49413000 bytes_received:37 segs_out:34154 segs_in:17111 send 106.2Mbps lastrcv:13590 pacing_rate 127.5Mbps unacked:28 rcv_space:28960
ESTAB 0 0 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55402 ino:18856 sk:5 <->
skmem:(r0,rb328320,t0,tb44800,f0,w0,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:3.738/6.898 ato:40 mss:1448 cwnd:10 bytes_acked:4 bytes_received:143 segs_out:7 segs_in:7 send 31.0Mbps lastsnd:13490 lastrcv:13590 lastack:13490 pacing_rate 62.0Mbps rcv_space:28960
ESTAB 0 637120 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55406 timer:(on,210ms,0) ino:17351 sk:6 <->
skmem:(r0,rb328320,t100352,tb663040,f87328,w645856,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:6.989/0.565 ato:40 mss:1448 cwnd:148 ssthresh:19 bytes_acked:103491456 bytes_received:37 segs_out:71531 segs_in:35798 send 245.3Mbps lastrcv:13590 pacing_rate 294.3Mbps unacked:58 rcv_space:28960
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 249056 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55404 timer:(on,210ms,0) ino:17349 sk:4 <->
skmem:(r0,rb328320,t47000,tb313600,f126688,w254240,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:5.486/1.174 ato:40 mss:1448 cwnd:70 ssthresh:16 bytes_acked:53345768 bytes_received:37 segs_out:36874 segs_in:18469 send 147.8Mbps lastrcv:14630 pacing_rate 177.3Mbps unacked:32 rcv_space:28960
ESTAB 0 0 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55402 ino:18856 sk:5 <->
skmem:(r0,rb328320,t0,tb44800,f0,w0,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:3.738/6.898 ato:40 mss:1448 cwnd:10 bytes_acked:4 bytes_received:143 segs_out:7 segs_in:7 send 31.0Mbps lastsnd:14530 lastrcv:14630 lastack:14530 pacing_rate 62.0Mbps rcv_space:28960
ESTAB 0 525624 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55406 timer:(on,210ms,0) ino:17351 sk:6 <->
skmem:(r0,rb328320,t70800,tb663040,f201416,w531768,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:5.716/0.694 ato:40 mss:1448 cwnd:148 ssthresh:19 bytes_acked:111356992 bytes_received:37 segs_out:76933 segs_in:38514 send 299.9Mbps lastrcv:14630 pacing_rate 359.9Mbps unacked:28 rcv_space:28960
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 267880 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55404 timer:(on,210ms,0) ino:17349 sk:4 <->
skmem:(r0,rb328320,t31664,tb313600,f106456,w274472,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:9.387/1.297 ato:40 mss:1448 cwnd:70 ssthresh:16 bytes_acked:57432024 bytes_received:37 segs_out:39685 segs_in:19880 send 86.4Mbps lastrcv:15670 pacing_rate 103.7Mbps unacked:21 rcv_space:28960
ESTAB 0 0 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55402 ino:18856 sk:5 <->
skmem:(r0,rb328320,t0,tb44800,f0,w0,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:3.738/6.898 ato:40 mss:1448 cwnd:10 bytes_acked:4 bytes_received:143 segs_out:7 segs_in:7 send 31.0Mbps lastsnd:15570 lastrcv:15670 lastack:15570 pacing_rate 62.0Mbps rcv_space:28960
ESTAB 0 553136 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55406 timer:(on,220ms,0) ino:17351 sk:6 <->
skmem:(r0,rb328320,t45480,tb663040,f174016,w559168,o0,bl0) ts sack cubic wscale:7,5 rto:220 rtt:10.243/1.677 ato:40 mss:1448 cwnd:148 ssthresh:19 bytes_acked:119083520 bytes_received:37 segs_out:82249 segs_in:41182 send 167.4Mbps lastsnd:10 lastrcv:15670 pacing_rate 200.8Mbps unacked:8 rcv_space:28960
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 330144 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55404 timer:(on,210ms,0) ino:17349 sk:4 <->
skmem:(r0,rb328320,t43992,tb461440,f171984,w335920,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:9.143/1.799 ato:40 mss:1448 cwnd:103 ssthresh:16 bytes_acked:61605160 bytes_received:37 segs_out:42575 segs_in:21321 send 130.5Mbps lastrcv:16730 pacing_rate 156.6Mbps unacked:29 rcv_space:28960
ESTAB 0 0 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55402 ino:18856 sk:5 <->
skmem:(r0,rb328320,t0,tb44800,f0,w0,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:3.738/6.898 ato:40 mss:1448 cwnd:10 bytes_acked:4 bytes_received:143 segs_out:7 segs_in:7 send 31.0Mbps lastsnd:16630 lastrcv:16730 lastack:16630 pacing_rate 62.0Mbps rcv_space:28960
ESTAB 0 641464 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55406 timer:(on,210ms,0) ino:17351 sk:6 <->
skmem:(r0,rb328320,t52720,tb663040,f85096,w648088,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:7.014/1.233 ato:40 mss:1448 cwnd:148 ssthresh:19 bytes_acked:126879552 bytes_received:37 segs_out:87654 segs_in:43875 send 244.4Mbps lastsnd:10 lastrcv:16730 pacing_rate 293.3Mbps unacked:29 rcv_space:28960
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 376480 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55404 timer:(on,200ms,0) ino:17349 sk:4 <->
skmem:(r0,rb328320,t34712,tb461440,f125760,w382144,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:7.648/0.774 ato:40 mss:1448 cwnd:103 ssthresh:16 bytes_acked:66510984 bytes_received:37 segs_out:45958 segs_in:23015 send 156.0Mbps lastsnd:10 lastrcv:17770 lastack:10 pacing_rate 187.2Mbps unacked:24 rcv_space:28960
ESTAB 0 0 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55402 ino:18856 sk:5 <->
skmem:(r0,rb328320,t0,tb44800,f0,w0,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:3.738/6.898 ato:40 mss:1448 cwnd:10 bytes_acked:4 bytes_received:143 segs_out:7 segs_in:7 send 31.0Mbps lastsnd:17670 lastrcv:17770 lastack:17670 pacing_rate 62.0Mbps rcv_space:28960
ESTAB 0 682008 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55406 timer:(on,220ms,0) ino:17351 sk:6 <->
skmem:(r0,rb328320,t45480,tb663040,f43256,w689928,o0,bl0) ts sack cubic wscale:7,5 rto:220 rtt:13.254/1.111 ato:40 mss:1448 cwnd:148 ssthresh:19 bytes_acked:133745968 bytes_received:37 segs_out:92374 segs_in:46245 send 129.4Mbps lastsnd:20 lastrcv:17770 pacing_rate 155.2Mbps unacked:7 rcv_space:28960
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 372136 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55404 timer:(on,200ms,0) ino:17349 sk:4 <->
skmem:(r0,rb328320,t26768,tb461440,f130104,w377800,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:7.694/0.761 ato:40 mss:1448 cwnd:103 ssthresh:16 bytes_acked:71402328 bytes_received:37 segs_out:49331 segs_in:24704 send 155.1Mbps lastsnd:10 lastrcv:18810 lastack:10 pacing_rate 186.1Mbps unacked:19 rcv_space:28960
ESTAB 0 0 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55402 ino:18856 sk:5 <->
skmem:(r0,rb328320,t0,tb44800,f0,w0,o0,bl0) ts sack cubic wscale:7,5 rto:210 rtt:3.738/6.898 ato:40 mss:1448 cwnd:10 bytes_acked:4 bytes_received:143 segs_out:7 segs_in:7 send 31.0Mbps lastsnd:18710 lastrcv:18810 lastack:18710 pacing_rate 62.0Mbps rcv_space:28960
ESTAB 0 556032 ::ffff:192.168.0.143:5201 ::ffff:192.168.0.1:55406 timer:(on,200ms,0) ino:17351 sk:6 <->
skmem:(r0,rb328320,t38944,tb663040,f168896,w564288,o0,bl149184) ts sack cubic wscale:7,5 rto:210 rtt:8.519/1.818 ato:40 mss:1448 cwnd:148 ssthresh:19 bytes_acked:140583424 bytes_received:37 segs_out:97134 segs_in:48615 send 201.2Mbps lastsnd:10 lastrcv:18810 lastack:10 pacing_rate 241.5Mbps unacked:45 rcv_space:28960
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: network stream fairness
2015-11-09 16:07 ` Eric Dumazet
2015-11-09 16:24 ` Eric Dumazet
@ 2015-11-20 15:33 ` Niklas Cassel
2015-11-20 18:16 ` Eric Dumazet
1 sibling, 1 reply; 14+ messages in thread
From: Niklas Cassel @ 2015-11-20 15:33 UTC (permalink / raw)
To: Eric Dumazet; +Cc: netdev
[-- Attachment #1: Type: text/plain, Size: 6814 bytes --]
On 11/09/2015 05:07 PM, Eric Dumazet wrote:
> On Mon, 2015-11-09 at 16:53 +0100, Niklas Cassel wrote:
>> On 11/09/2015 04:50 PM, Eric Dumazet wrote:
>>> On Mon, 2015-11-09 at 16:41 +0100, Niklas Cassel wrote:
>>>> I have a ethernet driver for a 100 Mbps NIC.
>>>> The NIC has dedicated hardware for offloading.
>>>> The driver has implemented TSO, GSO and BQL.
>>>> Since the CPU on the SoC is rather weak, I'd rather
>>>> not increase the CPU load by turning off offloading.
>>>>
>>>> Since commit
>>>> 605ad7f184b6 ("tcp: refine TSO autosizing")
>>>>
>>>> the bandwidth is no longer fair between streams.
>>>> see output at the end of the mail, where I'm testing with 2 streams.
>>>>
>>>>
>>>> If I revert 605ad7f184b6 on 4.3, I get a stable 45 Mbps per stream.
>>>>
>>>> I can also use vanilla 4.3 and do:
>>>> echo 3000 > /sys/class/net/eth0/queues/tx-0/byte_queue_limits/limit_max
>>>> to also get a stable 45 Mbps per stream.
>>>>
>>>> My question is, am I supposed to set the BQL limit explicitly?
>>>> It is possible that I have missed something in my driver,
>>>> but my understanding is that the TCP stack sets and adjusts
>>>> the BQL limit automatically.
>>>>
>>>>
>>>> Perhaps the following info might help:
>>>>
>>>> After running iperf3 on vanilla 4.3:
>>>> /sys/class/net/eth0/queues/tx-0/byte_queue_limits/
>>>> limit 89908
>>>> limit_max 1879048192
>>>>
>>>> After running iperf3 on vanilla 4.3 + BQL explicitly set:
>>>> /sys/class/net/eth0/queues/tx-0/byte_queue_limits/
>>>> limit 3000
>>>> limit_max 3000
>>>>
>>>> After running iperf3 on 4.3 + 605ad7f184b6 reverted:
>>>> /sys/class/net/eth0/queues/tx-0/byte_queue_limits/
>>>> limit 8886
>>>> limit_max 1879048192
>>>>
>>>
>>> There is absolutely nothing ensuring fairness among multiple TCP flows.
>>>
>>> One TCP flow can very easily grab whole bandwidth for itself, there are
>>> numerous descriptions of this phenomena in various TCP studies.
>>>
>>> This is why we have packet schedulers ;)
>>
>> Oh.. How stupid of me, I forgot to mention.. all of the measurements were
>> done with fq_codel.
>
> Your numbers suggest a cwnd growth then, which might show a CC bug.
>
> Please run the following when your iper3 runs on regular 4.3 kernel
>
> for i in `seq 1 10`
> do
> ss -temoi dst 192.168.0.141
> sleep 1
> done
>
>
I've been able to reproduce this on a ARMv7, single core, 100 Mbps NIC.
Kernel vanilla 4.3, driver has BQL implemented, but is unfortunately not upstreamed.
ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: off
tx-checksumming: on
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
ip addr show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:40:8c:18:58:c8 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.136/24 brd 192.168.0.255 scope global eth0
valid_lft forever preferred_lft forever
# before iperf3 run
tc -s -d qdisc
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
Sent 21001 bytes 45 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = cubic
# after iperf3 run
tc -s -d qdisc
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
Sent 5618224754 bytes 3710914 pkt (dropped 0, overlimits 0 requeues 1)
backlog 0b 0p requeues 1
maxpacket 1514 drop_overlimit 0 new_flow_count 2 ecn_mark 0
new_flows_len 0 old_flows_len 0
Note that it appears stable for 411 seconds before you can see the
congestion window growth. It appears that the amount of time you have
to wait before things go downhill varies a lot.
No switch was used between the server and client; they were connected directly.
For full iperf3 log and output from ss command, see attachment.
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 411.00-412.00 sec 5.09 MBytes 42.7 Mbits/sec 0 22.6 KBytes
[ 6] 411.00-412.00 sec 5.14 MBytes 43.1 Mbits/sec 0 22.6 KBytes
[SUM] 411.00-412.00 sec 10.2 MBytes 85.8 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 412.00-413.00 sec 5.12 MBytes 43.0 Mbits/sec 0 22.6 KBytes
[ 6] 412.00-413.00 sec 5.13 MBytes 43.0 Mbits/sec 0 22.6 KBytes
[SUM] 412.00-413.00 sec 10.3 MBytes 86.0 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 413.00-414.00 sec 5.17 MBytes 43.4 Mbits/sec 0 22.6 KBytes
[ 6] 413.00-414.00 sec 5.07 MBytes 42.6 Mbits/sec 0 22.6 KBytes
[SUM] 413.00-414.00 sec 10.2 MBytes 86.0 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 414.00-415.00 sec 5.11 MBytes 42.9 Mbits/sec 0 22.6 KBytes
[ 6] 414.00-415.00 sec 5.14 MBytes 43.1 Mbits/sec 0 22.6 KBytes
[SUM] 414.00-415.00 sec 10.3 MBytes 86.0 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 415.00-416.00 sec 5.11 MBytes 42.9 Mbits/sec 0 32.5 KBytes
[ 6] 415.00-416.00 sec 5.15 MBytes 43.2 Mbits/sec 0 22.6 KBytes
[SUM] 415.00-416.00 sec 10.3 MBytes 86.2 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 416.00-417.00 sec 6.18 MBytes 51.8 Mbits/sec 0 35.4 KBytes
[ 6] 416.00-417.00 sec 4.08 MBytes 34.3 Mbits/sec 0 22.6 KBytes
[SUM] 416.00-417.00 sec 10.3 MBytes 86.1 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 417.00-418.00 sec 6.24 MBytes 52.4 Mbits/sec 0 35.4 KBytes
[ 6] 417.00-418.00 sec 4.01 MBytes 33.6 Mbits/sec 0 22.6 KBytes
[SUM] 417.00-418.00 sec 10.3 MBytes 86.0 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 418.00-419.00 sec 6.28 MBytes 52.7 Mbits/sec 0 35.4 KBytes
[ 6] 418.00-419.00 sec 3.98 MBytes 33.4 Mbits/sec 0 22.6 KBytes
[SUM] 418.00-419.00 sec 10.3 MBytes 86.0 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 419.00-420.00 sec 6.30 MBytes 52.8 Mbits/sec 0 35.4 KBytes
[ 6] 419.00-420.00 sec 3.96 MBytes 33.2 Mbits/sec 0 22.6 KBytes
[SUM] 419.00-420.00 sec 10.3 MBytes 86.0 Mbits/sec 0
[-- Attachment #2: iperf3-ss-logs.tar.gz --]
[-- Type: application/gzip, Size: 62161 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: network stream fairness
2015-11-20 15:33 ` Niklas Cassel
@ 2015-11-20 18:16 ` Eric Dumazet
2015-11-25 12:47 ` Niklas Cassel
0 siblings, 1 reply; 14+ messages in thread
From: Eric Dumazet @ 2015-11-20 18:16 UTC (permalink / raw)
To: Niklas Cassel; +Cc: netdev
On Fri, 2015-11-20 at 16:33 +0100, Niklas Cassel wrote:
> I've been able to reproduce this on a ARMv7, single core, 100 Mbps NIC.
> Kernel vanilla 4.3, driver has BQL implemented, but is unfortunately not upstreamed.
>
> ethtool -k eth0
> Offload parameters for eth0:
> rx-checksumming: off
> tx-checksumming: on
> scatter-gather: off
> tcp segmentation offload: off
> udp fragmentation offload: off
> generic segmentation offload: off
>
> ip addr show dev eth0
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
> link/ether 00:40:8c:18:58:c8 brd ff:ff:ff:ff:ff:ff
> inet 192.168.0.136/24 brd 192.168.0.255 scope global eth0
> valid_lft forever preferred_lft forever
>
> # before iperf3 run
> tc -s -d qdisc
> qdisc noqueue 0: dev lo root refcnt 2
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
> Sent 21001 bytes 45 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
> new_flows_len 0 old_flows_len 0
>
> sysctl net.ipv4.tcp_congestion_control
> net.ipv4.tcp_congestion_control = cubic
>
> # after iperf3 run
> tc -s -d qdisc
> qdisc noqueue 0: dev lo root refcnt 2
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
> Sent 5618224754 bytes 3710914 pkt (dropped 0, overlimits 0 requeues 1)
> backlog 0b 0p requeues 1
> maxpacket 1514 drop_overlimit 0 new_flow_count 2 ecn_mark 0
> new_flows_len 0 old_flows_len 0
>
> Note that it appears stable for 411 seconds before you can see the
> congestion window growth. It appears that the amount of time you have
> to wait before things go downhill varies a lot.
> No switch was used between the server and client; they were connected directly.
Hi Niklas
Your results seem to show there is no special issue ;)
With TSO off and GSO off, there is no way a 'TSO autosizing' patch would
have any effect, since this code path is not taken.
You have to wait 400 seconds before getting into a mode where one of the
flow gets bigger cwnd (25 instead of 16), and then TCP cubic simply
shows typical unfairness ...
If you absolutely need to guarantee a given throughput per flow, you
might consider using fq packet scheduler and SO_MAX_PACING_RATE socket
option.
Thanks !
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: network stream fairness
2015-11-20 18:16 ` Eric Dumazet
@ 2015-11-25 12:47 ` Niklas Cassel
2015-11-25 13:49 ` Eric Dumazet
0 siblings, 1 reply; 14+ messages in thread
From: Niklas Cassel @ 2015-11-25 12:47 UTC (permalink / raw)
To: Eric Dumazet; +Cc: netdev
[-- Attachment #1: Type: text/plain, Size: 4239 bytes --]
On 11/20/2015 07:16 PM, Eric Dumazet wrote:
> On Fri, 2015-11-20 at 16:33 +0100, Niklas Cassel wrote:
>
>> I've been able to reproduce this on a ARMv7, single core, 100 Mbps NIC.
>> Kernel vanilla 4.3, driver has BQL implemented, but is unfortunately not upstreamed.
>>
>> ethtool -k eth0
>> Offload parameters for eth0:
>> rx-checksumming: off
>> tx-checksumming: on
>> scatter-gather: off
>> tcp segmentation offload: off
>> udp fragmentation offload: off
>> generic segmentation offload: off
>>
>> ip addr show dev eth0
>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
>> link/ether 00:40:8c:18:58:c8 brd ff:ff:ff:ff:ff:ff
>> inet 192.168.0.136/24 brd 192.168.0.255 scope global eth0
>> valid_lft forever preferred_lft forever
>>
>> # before iperf3 run
>> tc -s -d qdisc
>> qdisc noqueue 0: dev lo root refcnt 2
>> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>> backlog 0b 0p requeues 0
>> qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
>> Sent 21001 bytes 45 pkt (dropped 0, overlimits 0 requeues 0)
>> backlog 0b 0p requeues 0
>> maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
>> new_flows_len 0 old_flows_len 0
>>
>> sysctl net.ipv4.tcp_congestion_control
>> net.ipv4.tcp_congestion_control = cubic
>>
>> # after iperf3 run
>> tc -s -d qdisc
>> qdisc noqueue 0: dev lo root refcnt 2
>> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>> backlog 0b 0p requeues 0
>> qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
>> Sent 5618224754 bytes 3710914 pkt (dropped 0, overlimits 0 requeues 1)
>> backlog 0b 0p requeues 1
>> maxpacket 1514 drop_overlimit 0 new_flow_count 2 ecn_mark 0
>> new_flows_len 0 old_flows_len 0
>>
>> Note that it appears stable for 411 seconds before you can see the
>> congestion window growth. It appears that the amount of time you have
>> to wait before things go downhill varies a lot.
>> No switch was used between the server and client; they were connected directly.
>
> Hi Niklas
>
> Your results seem to show there is no special issue ;)
>
> With TSO off and GSO off, there is no way a 'TSO autosizing' patch would
> have any effect, since this code path is not taken.
You are right of course.
The arm unit uses a completely different hardware and driver (without TSO),
and was be unfair even before your patch,
so those measurements should be ignored.
The mips unit (with TSO), started being unfair after the patch in question.
With TSO off and GSO off, iperf streams are fair within 2 seconds,
and stay being fair for as long as I have tested.
With TSO on and GSO on, iperf streams never converge,
usually one stream around 60 Mbps and one at 30 Mbps.
With TSO on and GSO on, and also calling netif_set_gso_max_size(netdev, 16384)
in the driver, things appear to be working slightly better.
The problem remaining now is that it takes about 40 seconds before the streams
are fair in iperf. Once they are fair, they appear to stay fair forever.
Looking at the logs, it appears that we start out with 2 cwnds about the same size,
one of the windows grows to about twice the size of the other window,
then at ~40 seconds, the smaller window has grown to the same size as the
other window. (See attached logs.)
All tests were done with sch_fq.
It might be that our traffic is now shaped in a way that TCP cubic does not
handle that well, and therefore takes so long to converge.
Is there anything we can do (except turning off TSO) to make it converge faster?
E.g., is there any way to can tune TCP cubic to be more aggressive?
(SO_MAX_PACING_RATE would require modifying a lot of different user-space
programs, so that is not really an option :( )
Thanks for all your help!
>
> You have to wait 400 seconds before getting into a mode where one of the
> flow gets bigger cwnd (25 instead of 16), and then TCP cubic simply
> shows typical unfairness ...
>
> If you absolutely need to guarantee a given throughput per flow, you
> might consider using fq packet scheduler and SO_MAX_PACING_RATE socket
> option.
>
> Thanks !
>
>
[-- Attachment #2: iperf_ss_logs.tar.gz --]
[-- Type: application/gzip, Size: 14851 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: network stream fairness
2015-11-25 12:47 ` Niklas Cassel
@ 2015-11-25 13:49 ` Eric Dumazet
2015-11-29 2:41 ` Niklas Cassel
0 siblings, 1 reply; 14+ messages in thread
From: Eric Dumazet @ 2015-11-25 13:49 UTC (permalink / raw)
To: Niklas Cassel; +Cc: netdev
On Wed, 2015-11-25 at 13:47 +0100, Niklas Cassel wrote:
> (SO_MAX_PACING_RATE would require modifying a lot of different user-space
> programs, so that is not really an option :( )
You can set a max rate on the packet scheduler itself, for all flows.
tc qdisc replace dev eth0 root fq maxrate 30Mbit
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: network stream fairness
2015-11-25 13:49 ` Eric Dumazet
@ 2015-11-29 2:41 ` Niklas Cassel
2015-11-29 17:40 ` Eric Dumazet
0 siblings, 1 reply; 14+ messages in thread
From: Niklas Cassel @ 2015-11-29 2:41 UTC (permalink / raw)
To: Eric Dumazet; +Cc: netdev
Greetings fellow hackers :)
I have managed to reproduce my streaming unfairness with the intel e1000 driver.
I believe that commit
605ad7f184b6 ("tcp: refine TSO autosizing")
changed the behavior for 100 Mbps NIC:s with TSO enabled.
I have 2 e1000 cards connected in the same computer, with a network cable
connecting the NIC:s directly.
Please note that I used a intel e1000 card/driver,
since the e1000e driver turns off TSO when setting the speed to 100.
I also added netif_set_gso_max_size(netdev, 16384) to the e1000 driver,
but that did not affect the end result.
See iperf results with and without 605ad7f184b6 at the end of this mail.
When talking to Eric I've learned that TCP is not fair by design,
so I guess that one could argue that we previously, by coincidence,
happened to be more fair, and that we now just happen to be not as fair.
If I didn't know that though, my expectations would be that the
Fair Queue packet scheduler would behave just like its name says, fair. :P
Working within normal parameters or not, at least now we know that the patch
actually did cause a change in behavior for 100 Mbps NIC:s with TSO enabled.
My setup script, to not use loopback, etc., looks like this (thx Steve Kehlet):
tc qdisc replace dev eth0 root fq
tc qdisc replace dev eth2 root fq
ethtool -s eth2 speed 100 duplex full autoneg on
ethtool -s eth0 speed 100 duplex full autoneg on
ip addr add 10.50.0.1/24 dev eth2
ip addr add 10.50.1.1/24 dev eth0
iptables -t nat -A POSTROUTING -s 10.50.0.1 -d 10.60.1.1 -j SNAT --to-source 10.60.0.1
iptables -t nat -A PREROUTING -d 10.60.0.1 -j DNAT --to-destination 10.50.0.1
iptables -t nat -A POSTROUTING -s 10.50.1.1 -d 10.60.0.1 -j SNAT --to-source 10.60.1.1
iptables -t nat -A PREROUTING -d 10.60.1.1 -j DNAT --to-destination 10.50.1.1
ip route add 10.60.1.1 dev eth2
arp -i eth2 -s 10.60.1.1 00:11:22:33:44:55 # MAC address of eth0
ip route add 10.60.0.1 dev eth0
arp -i eth0 -s 10.60.0.1 55:44:33:22:11:00 # MAC address of eth2
Running iperf locked to a certain NIC:
iperf3 -B 10.50.0.1 -s
iperf3 -B 10.50.1.1 -c 10.60.0.1 -t 240 -P 2
Vanilla 4.3, sch_fq, e1000 driver, 100 Mbps speed, TSO on:
Connecting to host 10.60.0.1, port 5201
[ 4] local 10.50.1.1 port 40698 connected to 10.60.0.1 port 5201
[ 6] local 10.50.1.1 port 39755 connected to 10.60.0.1 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 5.78 MBytes 48.4 Mbits/sec 0 46.7 KBytes
[ 6] 0.00-1.00 sec 5.70 MBytes 47.8 Mbits/sec 0 48.1 KBytes
[SUM] 0.00-1.00 sec 11.5 MBytes 96.2 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 1.00-2.00 sec 5.53 MBytes 46.4 Mbits/sec 0 49.5 KBytes
[ 6] 1.00-2.00 sec 5.65 MBytes 47.4 Mbits/sec 0 48.1 KBytes
[SUM] 1.00-2.00 sec 11.2 MBytes 93.8 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 2.00-3.00 sec 5.65 MBytes 47.4 Mbits/sec 0 49.5 KBytes
[ 6] 2.00-3.00 sec 5.59 MBytes 46.9 Mbits/sec 0 48.1 KBytes
[SUM] 2.00-3.00 sec 11.2 MBytes 94.3 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 3.00-4.00 sec 4.86 MBytes 40.8 Mbits/sec 0 49.5 KBytes
[ 6] 3.00-4.00 sec 6.44 MBytes 54.0 Mbits/sec 0 66.5 KBytes
[SUM] 3.00-4.00 sec 11.3 MBytes 94.8 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 4.00-5.00 sec 4.01 MBytes 33.6 Mbits/sec 0 49.5 KBytes
[ 6] 4.00-5.00 sec 7.22 MBytes 60.5 Mbits/sec 0 66.5 KBytes
[SUM] 4.00-5.00 sec 11.2 MBytes 94.2 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 5.00-6.00 sec 4.31 MBytes 36.2 Mbits/sec 0 49.5 KBytes
[ 6] 5.00-6.00 sec 6.84 MBytes 57.3 Mbits/sec 0 66.5 KBytes
[SUM] 5.00-6.00 sec 11.1 MBytes 93.5 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 6.00-7.00 sec 4.44 MBytes 37.2 Mbits/sec 0 49.5 KBytes
[ 6] 6.00-7.00 sec 6.91 MBytes 58.0 Mbits/sec 0 66.5 KBytes
[SUM] 6.00-7.00 sec 11.3 MBytes 95.2 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 7.00-8.00 sec 4.31 MBytes 36.2 Mbits/sec 0 49.5 KBytes
[ 6] 7.00-8.00 sec 6.84 MBytes 57.3 Mbits/sec 0 66.5 KBytes
[SUM] 7.00-8.00 sec 11.1 MBytes 93.5 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 8.00-9.00 sec 4.50 MBytes 37.7 Mbits/sec 0 49.5 KBytes
[ 6] 8.00-9.00 sec 6.76 MBytes 56.7 Mbits/sec 0 66.5 KBytes
[SUM] 8.00-9.00 sec 11.3 MBytes 94.4 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 9.00-10.00 sec 4.19 MBytes 35.2 Mbits/sec 0 49.5 KBytes
[ 6] 9.00-10.00 sec 6.99 MBytes 58.6 Mbits/sec 0 66.5 KBytes
[SUM] 9.00-10.00 sec 11.2 MBytes 93.8 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 10.00-11.00 sec 4.25 MBytes 35.7 Mbits/sec 0 49.5 KBytes
[ 6] 10.00-11.00 sec 6.99 MBytes 58.6 Mbits/sec 0 66.5 KBytes
[SUM] 10.00-11.00 sec 11.2 MBytes 94.3 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 11.00-12.00 sec 4.19 MBytes 35.2 Mbits/sec 0 49.5 KBytes
[ 6] 11.00-12.00 sec 7.06 MBytes 59.3 Mbits/sec 0 66.5 KBytes
[SUM] 11.00-12.00 sec 11.3 MBytes 94.4 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 12.00-13.00 sec 4.50 MBytes 37.7 Mbits/sec 0 49.5 KBytes
[ 6] 12.00-13.00 sec 6.68 MBytes 56.1 Mbits/sec 0 66.5 KBytes
[SUM] 12.00-13.00 sec 11.2 MBytes 93.8 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 13.00-14.00 sec 4.01 MBytes 33.6 Mbits/sec 0 49.5 KBytes
[ 6] 13.00-14.00 sec 7.22 MBytes 60.5 Mbits/sec 0 66.5 KBytes
[SUM] 13.00-14.00 sec 11.2 MBytes 94.2 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-14.00 sec 64.5 MBytes 38.7 Mbits/sec 0 sender
[ 4] 0.00-14.00 sec 64.5 MBytes 38.6 Mbits/sec receiver
[ 6] 0.00-14.00 sec 92.9 MBytes 55.6 Mbits/sec 0 sender
[ 6] 0.00-14.00 sec 92.7 MBytes 55.6 Mbits/sec receiver
[SUM] 0.00-14.00 sec 157 MBytes 94.3 Mbits/sec 0 sender
[SUM] 0.00-14.00 sec 157 MBytes 94.2 Mbits/sec receiver
iperf Done.
Kernel 4.3, 605ad7f184b6 reverted, sch_fq, e1000 driver, 100 Mbps speed, TSO on:
Connecting to host 10.60.0.1, port 5201
[ 4] local 10.50.1.1 port 40949 connected to 10.60.0.1 port 5201
[ 6] local 10.50.1.1 port 39238 connected to 10.60.0.1 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 6.09 MBytes 51.1 Mbits/sec 0 219 KBytes
[ 6] 0.00-1.00 sec 6.05 MBytes 50.8 Mbits/sec 0 219 KBytes
[SUM] 0.00-1.00 sec 12.1 MBytes 102 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 1.00-2.00 sec 5.65 MBytes 47.4 Mbits/sec 0 219 KBytes
[ 6] 1.00-2.00 sec 5.65 MBytes 47.4 Mbits/sec 0 219 KBytes
[SUM] 1.00-2.00 sec 11.3 MBytes 94.8 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 2.00-3.00 sec 5.47 MBytes 45.9 Mbits/sec 0 219 KBytes
[ 6] 2.00-3.00 sec 5.65 MBytes 47.4 Mbits/sec 0 219 KBytes
[SUM] 2.00-3.00 sec 11.1 MBytes 93.3 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 3.00-4.00 sec 5.65 MBytes 47.4 Mbits/sec 0 219 KBytes
[ 6] 3.00-4.00 sec 5.47 MBytes 45.9 Mbits/sec 0 219 KBytes
[SUM] 3.00-4.00 sec 11.1 MBytes 93.3 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 4.00-5.00 sec 5.65 MBytes 47.4 Mbits/sec 0 219 KBytes
[ 6] 4.00-5.00 sec 5.65 MBytes 47.4 Mbits/sec 0 219 KBytes
[SUM] 4.00-5.00 sec 11.3 MBytes 94.8 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 5.00-6.00 sec 5.65 MBytes 47.4 Mbits/sec 0 219 KBytes
[ 6] 5.00-6.00 sec 5.65 MBytes 47.4 Mbits/sec 0 219 KBytes
[SUM] 5.00-6.00 sec 11.3 MBytes 94.8 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 6.00-7.00 sec 5.65 MBytes 47.4 Mbits/sec 0 219 KBytes
[ 6] 6.00-7.00 sec 5.65 MBytes 47.4 Mbits/sec 0 219 KBytes
[SUM] 6.00-7.00 sec 11.3 MBytes 94.8 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 7.00-8.00 sec 5.47 MBytes 45.9 Mbits/sec 0 219 KBytes
[ 6] 7.00-8.00 sec 5.47 MBytes 45.9 Mbits/sec 0 219 KBytes
[SUM] 7.00-8.00 sec 10.9 MBytes 91.7 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 8.00-9.00 sec 5.65 MBytes 47.4 Mbits/sec 0 219 KBytes
[ 6] 8.00-9.00 sec 5.65 MBytes 47.4 Mbits/sec 0 219 KBytes
[SUM] 8.00-9.00 sec 11.3 MBytes 94.8 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 9.00-10.00 sec 5.65 MBytes 47.4 Mbits/sec 0 219 KBytes
[ 6] 9.00-10.00 sec 5.65 MBytes 47.4 Mbits/sec 0 219 KBytes
[SUM] 9.00-10.00 sec 11.3 MBytes 94.8 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 10.00-11.00 sec 5.65 MBytes 47.4 Mbits/sec 0 219 KBytes
[ 6] 10.00-11.00 sec 5.65 MBytes 47.4 Mbits/sec 0 219 KBytes
[SUM] 10.00-11.00 sec 11.3 MBytes 94.8 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 11.00-12.00 sec 5.47 MBytes 45.9 Mbits/sec 0 219 KBytes
[ 6] 11.00-12.00 sec 5.65 MBytes 47.4 Mbits/sec 0 219 KBytes
[SUM] 11.00-12.00 sec 11.1 MBytes 93.3 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 12.00-13.00 sec 5.65 MBytes 47.4 Mbits/sec 0 219 KBytes
[ 6] 12.00-13.00 sec 5.47 MBytes 45.9 Mbits/sec 0 219 KBytes
[SUM] 12.00-13.00 sec 11.1 MBytes 93.3 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 13.00-14.00 sec 5.65 MBytes 47.4 Mbits/sec 0 219 KBytes
[ 6] 13.00-14.00 sec 5.65 MBytes 47.4 Mbits/sec 0 219 KBytes
[SUM] 13.00-14.00 sec 11.3 MBytes 94.8 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-14.00 sec 79.0 MBytes 47.3 Mbits/sec 0 sender
[ 4] 0.00-14.00 sec 78.6 MBytes 47.1 Mbits/sec receiver
[ 6] 0.00-14.00 sec 79.0 MBytes 47.3 Mbits/sec 0 sender
[ 6] 0.00-14.00 sec 78.6 MBytes 47.1 Mbits/sec receiver
[SUM] 0.00-14.00 sec 158 MBytes 94.7 Mbits/sec 0 sender
[SUM] 0.00-14.00 sec 157 MBytes 94.2 Mbits/sec receiver
iperf Done.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: network stream fairness
2015-11-29 2:41 ` Niklas Cassel
@ 2015-11-29 17:40 ` Eric Dumazet
0 siblings, 0 replies; 14+ messages in thread
From: Eric Dumazet @ 2015-11-29 17:40 UTC (permalink / raw)
To: Niklas Cassel; +Cc: netdev
> If I didn't know that though, my expectations would be that the
> Fair Queue packet scheduler would behave just like its name says, fair. :P
>
Only if the flows have same characteristics. This cannot be ensured for
hours long flows. This should be obvious.
If one flow is not trying to push enough, it will loose its round in the
Round Robin scheduler, because no packet is sitting in the packet
scheduler for this flow.
A simple slowdown, cause by a packet drop, or scheduler artifact on the
sender or receiver can make one flow steal whole bandwidth, even when FQ
is in the picture.
I have no idea why you are expecting multiple TCP flows being fair.
There is absolutely nothing in TCP protocol to ensure this.
In short, fq_codel or fq are not doing what you think they do.
If you want fair flows, you MUST pace them say at 40Mbits each.
The fact that one 'patch' slightly changes the condition for unfairness
to start is absolutely irrelevant.
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2015-11-29 17:40 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-11-09 15:41 network stream fairness Niklas Cassel
2015-11-09 15:50 ` Eric Dumazet
2015-11-09 15:53 ` Niklas Cassel
2015-11-09 16:07 ` Eric Dumazet
2015-11-09 16:24 ` Eric Dumazet
2015-11-09 16:50 ` Niklas Cassel
2015-11-09 17:23 ` Eric Dumazet
2015-11-10 9:38 ` Niklas Cassel
2015-11-20 15:33 ` Niklas Cassel
2015-11-20 18:16 ` Eric Dumazet
2015-11-25 12:47 ` Niklas Cassel
2015-11-25 13:49 ` Eric Dumazet
2015-11-29 2:41 ` Niklas Cassel
2015-11-29 17:40 ` Eric Dumazet
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.