Interaction between GSO and TSQ for single TCP stream

Interaction between GSO and TSQ for single TCP stream

[Tom Herbert]

Eric,

I am looking at why we are unable to achieve line rate throughput with
a single connection using UDP encapsulation that supports GSO/GRO. I
discovered that even for a non-encpasulated TCP connection I'm not
able to get line rate using GSO/GRO (but do with TSO/LRO). There seems
to be some interaction between TSQ and GSO. Setting
tcp_limit_output_bytes to 1000000 gets single connection up to line
rate for both native and encapsulated TCP.

Some data:

- netperf TCP_STREAM for a single connection no encapsulation
  Using TSO/LRO
    9412.44 Gbps
  Using GSO/GRO (ethtool -K eth0 gso on gro on tso off lro off)
    9286.08 Gbps
  Using GSO/GRO with larger limit (echo 1000000 >
/proc/sys/net/ipv4/tcp_limit_output_bytes)
    9412.75

- netperf TCP_STREAM for a single connection for GUE-IPIP encapsulation with RCO
  Using GSO/GRO
    8842.89 Gbps
  Using GSO/GRO with larger limit (echo 1000000 >
/proc/sys/net/ipv4/tcp_limit_output_bytes)
    9143.99 Gbps

Any ideas on how to address this?

Thanks,
Tom

Re: Interaction between GSO and TSQ for single TCP stream

[Eric Dumazet]

On Tue, Jan 27, 2015 at 11:57 AM, Tom Herbert <therbert@google.com> wrote:
> Eric,
>
> I am looking at why we are unable to achieve line rate throughput with
> a single connection using UDP encapsulation that supports GSO/GRO. I
> discovered that even for a non-encpasulated TCP connection I'm not
> able to get line rate using GSO/GRO (but do with TSO/LRO). There seems
> to be some interaction between TSQ and GSO. Setting
> tcp_limit_output_bytes to 1000000 gets single connection up to line
> rate for both native and encapsulated TCP.
>
> Some data:
>
> - netperf TCP_STREAM for a single connection no encapsulation
>   Using TSO/LRO
>     9412.44 Gbps
>   Using GSO/GRO (ethtool -K eth0 gso on gro on tso off lro off)
>     9286.08 Gbps
>   Using GSO/GRO with larger limit (echo 1000000 >
> /proc/sys/net/ipv4/tcp_limit_output_bytes)
>     9412.75
>
> - netperf TCP_STREAM for a single connection for GUE-IPIP encapsulation with RCO
>   Using GSO/GRO
>     8842.89 Gbps
>   Using GSO/GRO with larger limit (echo 1000000 >
> /proc/sys/net/ipv4/tcp_limit_output_bytes)
>     9143.99 Gbps
>
> Any ideas on how to address this?
>
> Thanks,
> Tom

Hi Tom.

Yes, Eyal Perry  reported the issue a while back, we plan to send
upstream patches to fix the issue today.

Make sure you tweaked coalescing parameters if you use mlx4 :

ethtool -C eth0 tx-usecs 4 tx-frames 4

Otherwise, the NIC accumulates too many packets before sending an
interrupt for TX completion.

Then you can either wait the patches, or try the one I sent :
http://permalink.gmane.org/gmane.linux.network/347023

Re: Interaction between GSO and TSQ for single TCP stream

[Tom Herbert]

On Tue, Jan 27, 2015 at 12:02 PM, Eric Dumazet <edumazet@google.com> wrote:
> On Tue, Jan 27, 2015 at 11:57 AM, Tom Herbert <therbert@google.com> wrote:
>> Eric,
>>
>> I am looking at why we are unable to achieve line rate throughput with
>> a single connection using UDP encapsulation that supports GSO/GRO. I
>> discovered that even for a non-encpasulated TCP connection I'm not
>> able to get line rate using GSO/GRO (but do with TSO/LRO). There seems
>> to be some interaction between TSQ and GSO. Setting
>> tcp_limit_output_bytes to 1000000 gets single connection up to line
>> rate for both native and encapsulated TCP.
>>
>> Some data:
>>
>> - netperf TCP_STREAM for a single connection no encapsulation
>>   Using TSO/LRO
>>     9412.44 Gbps
>>   Using GSO/GRO (ethtool -K eth0 gso on gro on tso off lro off)
>>     9286.08 Gbps
>>   Using GSO/GRO with larger limit (echo 1000000 >
>> /proc/sys/net/ipv4/tcp_limit_output_bytes)
>>     9412.75
>>
>> - netperf TCP_STREAM for a single connection for GUE-IPIP encapsulation with RCO
>>   Using GSO/GRO
>>     8842.89 Gbps
>>   Using GSO/GRO with larger limit (echo 1000000 >
>> /proc/sys/net/ipv4/tcp_limit_output_bytes)
>>     9143.99 Gbps
>>
>> Any ideas on how to address this?
>>
>> Thanks,
>> Tom
>
> Hi Tom.
>
> Yes, Eyal Perry  reported the issue a while back, we plan to send
> upstream patches to fix the issue today.
>
Awesome, looking forward to seeing them!

> Make sure you tweaked coalescing parameters if you use mlx4 :
>
> ethtool -C eth0 tx-usecs 4 tx-frames 4
>
It's bnx2x.

> Otherwise, the NIC accumulates too many packets before sending an
> interrupt for TX completion.
>
> Then you can either wait the patches, or try the one I sent :
> http://permalink.gmane.org/gmane.linux.network/347023