A performance regression of gretap

A performance regression of gretap

[Cong Wang]

Hi, Pravin

I just noticed the performance over gretap is very bad,
which is probably caused by your GRE refactor patches.

See below:

# netperf -4 -H 192.168.2.1
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.2.1 () port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  16384    14.01       0.02   

Could you please take a look? And, gre tunnel is fine, this
problem only exists for gretap. I reviewed the gretap code,
but can't find any bug.

It is very easy to reproduce, just setup a gretap device
on both ends, and run netperf over it.

Thanks!

Re: A performance regression of gretap

[Rick Jones]

On 07/10/2013 03:01 AM, Cong Wang wrote:
> Hi, Pravin
>
> I just noticed the performance over gretap is very bad,
> which is probably caused by your GRE refactor patches.
>
> See below:
>
> # netperf -4 -H 192.168.2.1
> MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.2.1 () port 0 AF_INET
> Recv   Send    Send
> Socket Socket  Message  Elapsed
> Size   Size    Size     Time     Throughput
> bytes  bytes   bytes    secs.    10^6bits/sec
>
>   87380  16384  16384    14.01       0.02
>
> Could you please take a look? And, gre tunnel is fine, this
> problem only exists for gretap. I reviewed the gretap code,
> but can't find any bug.
>
> It is very easy to reproduce, just setup a gretap device
> on both ends, and run netperf over it.

One way to provide more information from the netperf level in future 
endeavours would be to extend the command line a bit:

netperf -4 -H 192.168.2.1 -c -C -- -o 
throughput,local_sd,remote_ds,local_transport_retrans

and then again sans gretap.  The -c/-C and [local|remote]_sd would be to 
show the difference in path length, local_transport_retrans to see what 
sort of difference there is in retransmissions.

happy benchmarking,

rick jones

Re: A performance regression of gretap

[Cong Wang]

On Wed, 2013-07-10 at 06:01 -0400, Cong Wang wrote:
> 
> Could you please take a look? And, gre tunnel is fine, this
> problem only exists for gretap. I reviewed the gretap code,
> but can't find any bug.

After digging it a little bit, I found some of the packets on RX side
have incorrect TCP checksum after going through gretap, while the
packets captured on eth0 are all correct.

Thanks.

Re: A performance regression of gretap

[Pravin Shelar]

On Wed, Jul 10, 2013 at 7:15 PM, Cong Wang <amwang@redhat.com> wrote:
> On Wed, 2013-07-10 at 06:01 -0400, Cong Wang wrote:
>>
>> Could you please take a look? And, gre tunnel is fine, this
>> problem only exists for gretap. I reviewed the gretap code,
>> but can't find any bug.
>
> After digging it a little bit, I found some of the packets on RX side
> have incorrect TCP checksum after going through gretap, while the
> packets captured on eth0 are all correct.
>
> Thanks.
>
I do not see it on my end.
Can you tell me NIC type and features set for eth device?

Re: A performance regression of gretap

[Cong Wang]

On Wed, 2013-07-10 at 19:28 -0700, Pravin Shelar wrote:
> On Wed, Jul 10, 2013 at 7:15 PM, Cong Wang <amwang@redhat.com> wrote:
> > On Wed, 2013-07-10 at 06:01 -0400, Cong Wang wrote:
> >>
> >> Could you please take a look? And, gre tunnel is fine, this
> >> problem only exists for gretap. I reviewed the gretap code,
> >> but can't find any bug.
> >
> > After digging it a little bit, I found some of the packets on RX side
> > have incorrect TCP checksum after going through gretap, while the
> > packets captured on eth0 are all correct.
> >
> > Thanks.
> >
> I do not see it on my end.
> Can you tell me NIC type and features set for eth device?

Sure, it is virtio_net which is not able to do rx checksum, therefore
->ip_summed is CHECKSUM_NONE.

# ethtool -i eth0
driver: virtio_net
version: 1.0.0
firmware-version: 
bus-info: 0000:00:08.0
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: off
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off
receive-hashing: off

Re: A performance regression of gretap

[Pravin Shelar]

On Wed, Jul 10, 2013 at 7:31 PM, Cong Wang <amwang@redhat.com> wrote:
> On Wed, 2013-07-10 at 19:28 -0700, Pravin Shelar wrote:
>> On Wed, Jul 10, 2013 at 7:15 PM, Cong Wang <amwang@redhat.com> wrote:
>> > On Wed, 2013-07-10 at 06:01 -0400, Cong Wang wrote:
>> >>
>> >> Could you please take a look? And, gre tunnel is fine, this
>> >> problem only exists for gretap. I reviewed the gretap code,
>> >> but can't find any bug.
>> >
>> > After digging it a little bit, I found some of the packets on RX side
>> > have incorrect TCP checksum after going through gretap, while the
>> > packets captured on eth0 are all correct.
>> >
>> > Thanks.
>> >
>> I do not see it on my end.
>> Can you tell me NIC type and features set for eth device?
>
> Sure, it is virtio_net which is not able to do rx checksum, therefore
> ->ip_summed is CHECKSUM_NONE.
>
> # ethtool -i eth0
> driver: virtio_net
> version: 1.0.0
> firmware-version:
> bus-info: 0000:00:08.0
> supports-statistics: no
> supports-test: no
> supports-eeprom-access: no
> supports-register-dump: no
> supports-priv-flags: no
>
> # ethtool -k eth0
> Offload parameters for eth0:
> rx-checksumming: off
> tx-checksumming: on
> scatter-gather: on
> tcp-segmentation-offload: on
> udp-fragmentation-offload: on
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off
> rx-vlan-offload: off
> tx-vlan-offload: off
> ntuple-filters: off
> receive-hashing: off
>
>

I could not reproduce the issue on NIC with similar feat, I will try
virtio tomorrow.