macvtap performance regression (bisected) between 3.13 and 3.14-rc1

macvtap performance regression (bisected) between 3.13 and 3.14-rc1

[Christian Borntraeger]

Vlad,

commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
    macvtap: Add support of packet capture on macvtap device.

causes a performance regression for iperf traffic between two KVM guests
on my s390 system. Both guests are connected via two macvtaps on the same OSA
network card.
Before that patch I get ~20 Gbit/sec between two guests, afterwards I get
~4Gbit/sec

Latency seems to be unchanges (uperf 1byte ping pong).

According to ifconfig in the guest, I have ~ 1500 bytes per packet with this
patch and ~  40000 bytes without. So for some reason this patch causes the
network stack to do segmentation. (the guest kernel stays the same, only host 
kernel is changed).

Any ideas?


Christian

Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1

[Vlad Yasevich]

On 02/27/2014 03:52 PM, Christian Borntraeger wrote:
> Vlad,
> 
> commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>     macvtap: Add support of packet capture on macvtap device.
> 
> causes a performance regression for iperf traffic between two KVM guests
> on my s390 system. Both guests are connected via two macvtaps on the same OSA
> network card.
> Before that patch I get ~20 Gbit/sec between two guests, afterwards I get
> ~4Gbit/sec
> 
> Latency seems to be unchanges (uperf 1byte ping pong).
> 
> According to ifconfig in the guest, I have ~ 1500 bytes per packet with this
> patch and ~  40000 bytes without. So for some reason this patch causes the
> network stack to do segmentation. (the guest kernel stays the same, only host 
> kernel is changed).
> 
> Any ideas?

I am looking.  It shouldn't cause addition segmentations and when I ran
netperf on the code I didn't see any difference in the throughput.

-vlad

> 
> 
> Christian
> 

Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1

[Christian Borntraeger]

On 28/02/14 23:14, Vlad Yasevich wrote:
> On 02/27/2014 03:52 PM, Christian Borntraeger wrote:
>> Vlad,
>>
>> commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>>     macvtap: Add support of packet capture on macvtap device.
>>
>> causes a performance regression for iperf traffic between two KVM guests
>> on my s390 system. Both guests are connected via two macvtaps on the same OSA
>> network card.
>> Before that patch I get ~20 Gbit/sec between two guests, afterwards I get
>> ~4Gbit/sec
>>
>> Latency seems to be unchanges (uperf 1byte ping pong).
>>
>> According to ifconfig in the guest, I have ~ 1500 bytes per packet with this
>> patch and ~  40000 bytes without. So for some reason this patch causes the
>> network stack to do segmentation. (the guest kernel stays the same, only host 
>> kernel is changed).
>>
>> Any ideas?
> 
> I am looking.  It shouldn't cause addition segmentations and when I ran
> netperf on the code I didn't see any difference in the throughput.

Dont know if the different bytes/packets ratio is really the reason or
just a side effect. As a hint: the underlying network device does not support
segmentation, but this should not matter for traffic between to guests.

Maybe you remember, we had a similar situation with commit 3e4f8b787370978733ca6cae452720a4f0c296b8
(macvtap: Perform GSO on forwarding path), the setup is basically the same.


Christian

Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1

[Vlad Yasevich]

On 03/01/2014 06:15 AM, Christian Borntraeger wrote:
> On 28/02/14 23:14, Vlad Yasevich wrote:
>> On 02/27/2014 03:52 PM, Christian Borntraeger wrote:
>>> Vlad,
>>>
>>> commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>>>     macvtap: Add support of packet capture on macvtap device.
>>>
>>> causes a performance regression for iperf traffic between two KVM guests
>>> on my s390 system. Both guests are connected via two macvtaps on the same OSA
>>> network card.
>>> Before that patch I get ~20 Gbit/sec between two guests, afterwards I get
>>> ~4Gbit/sec
>>>
>>> Latency seems to be unchanges (uperf 1byte ping pong).
>>>
>>> According to ifconfig in the guest, I have ~ 1500 bytes per packet with this
>>> patch and ~  40000 bytes without. So for some reason this patch causes the
>>> network stack to do segmentation. (the guest kernel stays the same, only host 
>>> kernel is changed).
>>>
>>> Any ideas?
>>
>> I am looking.  It shouldn't cause addition segmentations and when I ran
>> netperf on the code I didn't see any difference in the throughput.
> 
> Dont know if the different bytes/packets ratio is really the reason or
> just a side effect. As a hint: the underlying network device does not support
> segmentation, but this should not matter for traffic between to guests.

Could you post 'ethtool -k' output for both lower-level device and the
macvtap device?

Thanks
-vlad

> 
> Maybe you remember, we had a similar situation with commit 3e4f8b787370978733ca6cae452720a4f0c296b8
> (macvtap: Perform GSO on forwarding path), the setup is basically the same.
> 
> 
> Christian
> 
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1

[Vlad Yasevich]

On 03/01/2014 02:27 PM, Vlad Yasevich wrote:
> On 03/01/2014 06:15 AM, Christian Borntraeger wrote:
>> On 28/02/14 23:14, Vlad Yasevich wrote:
>>> On 02/27/2014 03:52 PM, Christian Borntraeger wrote:
>>>> Vlad,
>>>>
>>>> commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>>>>     macvtap: Add support of packet capture on macvtap device.
>>>>
>>>> causes a performance regression for iperf traffic between two KVM guests
>>>> on my s390 system. Both guests are connected via two macvtaps on the same OSA
>>>> network card.
>>>> Before that patch I get ~20 Gbit/sec between two guests, afterwards I get
>>>> ~4Gbit/sec
>>>>
>>>> Latency seems to be unchanges (uperf 1byte ping pong).
>>>>
>>>> According to ifconfig in the guest, I have ~ 1500 bytes per packet with this
>>>> patch and ~  40000 bytes without. So for some reason this patch causes the
>>>> network stack to do segmentation. (the guest kernel stays the same, only host 
>>>> kernel is changed).
>>>>
>>>> Any ideas?
>>>
>>> I am looking.  It shouldn't cause addition segmentations and when I ran
>>> netperf on the code I didn't see any difference in the throughput.
>>
>> Dont know if the different bytes/packets ratio is really the reason or
>> just a side effect. As a hint: the underlying network device does not support
>> segmentation, but this should not matter for traffic between to guests.
> 
> Could you post 'ethtool -k' output for both lower-level device and the
> macvtap device?
> 
> Thanks
> -vlad
> 

Ok.  I think I see what's happening.  Since you turn off offloads on
lower device, that's propagated to macvlan device.  As a result, when
when we call dev_queue_xmit on the vlan->dev, we end up segmenting since
lower level says it does support segmentation.

One way to fix this is to never disable offloads on macvlan.  macvlan
will always try to use __dev_queue_xmit() with it's lower device, so any
segmentation can happen there.

-vlad

>>
>> Maybe you remember, we had a similar situation with commit 3e4f8b787370978733ca6cae452720a4f0c296b8
>> (macvtap: Perform GSO on forwarding path), the setup is basically the same.
>>
>>
>> Christian
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> 

Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1

[Christian Borntraeger]

On 01/03/14 20:27, Vlad Yasevich wrote:
> On 03/01/2014 06:15 AM, Christian Borntraeger wrote:
>> On 28/02/14 23:14, Vlad Yasevich wrote:
>>> On 02/27/2014 03:52 PM, Christian Borntraeger wrote:
>>>> Vlad,
>>>>
>>>> commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>>>>     macvtap: Add support of packet capture on macvtap device.
>>>>
>>>> causes a performance regression for iperf traffic between two KVM guests
>>>> on my s390 system. Both guests are connected via two macvtaps on the same OSA
>>>> network card.
>>>> Before that patch I get ~20 Gbit/sec between two guests, afterwards I get
>>>> ~4Gbit/sec
>>>>
>>>> Latency seems to be unchanges (uperf 1byte ping pong).
>>>>
>>>> According to ifconfig in the guest, I have ~ 1500 bytes per packet with this
>>>> patch and ~  40000 bytes without. So for some reason this patch causes the
>>>> network stack to do segmentation. (the guest kernel stays the same, only host 
>>>> kernel is changed).
>>>>
>>>> Any ideas?
>>>
>>> I am looking.  It shouldn't cause addition segmentations and when I ran
>>> netperf on the code I didn't see any difference in the throughput.
>>
>> Dont know if the different bytes/packets ratio is really the reason or
>> just a side effect. As a hint: the underlying network device does not support
>> segmentation, but this should not matter for traffic between to guests.
> 
> Could you post 'ethtool -k' output for both lower-level device and the
> macvtap device?




Features for eth0:
rx-checksumming: off [fixed]
tx-checksumming: off
	tx-checksum-ipv4: off [fixed]
	tx-checksum-ip-generic: off [fixed]
	tx-checksum-ipv6: off [fixed]
	tx-checksum-fcoe-crc: off [fixed]
	tx-checksum-sctp: off [fixed]
scatter-gather: off
	tx-scatter-gather: off [fixed]
	tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
	tx-tcp-segmentation: off [fixed]
	tx-tcp-ecn-segmentation: off [fixed]
	tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]


Features for macvtap1:
rx-checksumming: off [fixed]
tx-checksumming: off
	tx-checksum-ipv4: off [fixed]
	tx-checksum-ip-generic: off [fixed]
	tx-checksum-ipv6: off [fixed]
	tx-checksum-fcoe-crc: off [fixed]
	tx-checksum-sctp: off [fixed]
scatter-gather: off
	tx-scatter-gather: off [fixed]
	tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
	tx-tcp-segmentation: off [fixed]
	tx-tcp-ecn-segmentation: off [fixed]
	tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]

Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1

[Christian Borntraeger]

On 02/03/14 02:21, Vlad Yasevich wrote:
> On 03/01/2014 02:27 PM, Vlad Yasevich wrote:
>> On 03/01/2014 06:15 AM, Christian Borntraeger wrote:
>>> On 28/02/14 23:14, Vlad Yasevich wrote:
>>>> On 02/27/2014 03:52 PM, Christian Borntraeger wrote:
>>>>> Vlad,
>>>>>
>>>>> commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>>>>>     macvtap: Add support of packet capture on macvtap device.
>>>>>
>>>>> causes a performance regression for iperf traffic between two KVM guests
>>>>> on my s390 system. Both guests are connected via two macvtaps on the same OSA
>>>>> network card.
>>>>> Before that patch I get ~20 Gbit/sec between two guests, afterwards I get
>>>>> ~4Gbit/sec
>>>>>
>>>>> Latency seems to be unchanges (uperf 1byte ping pong).
>>>>>
>>>>> According to ifconfig in the guest, I have ~ 1500 bytes per packet with this
>>>>> patch and ~  40000 bytes without. So for some reason this patch causes the
>>>>> network stack to do segmentation. (the guest kernel stays the same, only host 
>>>>> kernel is changed).
>>>>>
>>>>> Any ideas?
>>>>
>>>> I am looking.  It shouldn't cause addition segmentations and when I ran
>>>> netperf on the code I didn't see any difference in the throughput.
>>>
>>> Dont know if the different bytes/packets ratio is really the reason or
>>> just a side effect. As a hint: the underlying network device does not support
>>> segmentation, but this should not matter for traffic between to guests.
>>
>> Could you post 'ethtool -k' output for both lower-level device and the
>> macvtap device?
>>
>> Thanks
>> -vlad
>>
> 
> Ok.  I think I see what's happening.  Since you turn off offloads on
> lower device, that's propagated to macvlan device.  As a result, when
> when we call dev_queue_xmit on the vlan->dev, we end up segmenting since
> lower level says it does support segmentation.
> 
> One way to fix this is to never disable offloads on macvlan.  macvlan
> will always try to use __dev_queue_xmit() with it's lower device, so any
> segmentation can happen there.

If you have anything that I should test, let me know.

Christian

Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1

[Vlad Yasevich]

On 03/03/2014 04:13 AM, Christian Borntraeger wrote:
> On 02/03/14 02:21, Vlad Yasevich wrote:
>> On 03/01/2014 02:27 PM, Vlad Yasevich wrote:
>>> On 03/01/2014 06:15 AM, Christian Borntraeger wrote:
>>>> On 28/02/14 23:14, Vlad Yasevich wrote:
>>>>> On 02/27/2014 03:52 PM, Christian Borntraeger wrote:
>>>>>> Vlad,
>>>>>>
>>>>>> commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>>>>>>     macvtap: Add support of packet capture on macvtap device.
>>>>>>
>>>>>> causes a performance regression for iperf traffic between two KVM guests
>>>>>> on my s390 system. Both guests are connected via two macvtaps on the same OSA
>>>>>> network card.
>>>>>> Before that patch I get ~20 Gbit/sec between two guests, afterwards I get
>>>>>> ~4Gbit/sec
>>>>>>
>>>>>> Latency seems to be unchanges (uperf 1byte ping pong).
>>>>>>
>>>>>> According to ifconfig in the guest, I have ~ 1500 bytes per packet with this
>>>>>> patch and ~  40000 bytes without. So for some reason this patch causes the
>>>>>> network stack to do segmentation. (the guest kernel stays the same, only host 
>>>>>> kernel is changed).
>>>>>>
>>>>>> Any ideas?
>>>>>
>>>>> I am looking.  It shouldn't cause addition segmentations and when I ran
>>>>> netperf on the code I didn't see any difference in the throughput.
>>>>
>>>> Dont know if the different bytes/packets ratio is really the reason or
>>>> just a side effect. As a hint: the underlying network device does not support
>>>> segmentation, but this should not matter for traffic between to guests.
>>>
>>> Could you post 'ethtool -k' output for both lower-level device and the
>>> macvtap device?
>>>
>>> Thanks
>>> -vlad
>>>
>>
>> Ok.  I think I see what's happening.  Since you turn off offloads on
>> lower device, that's propagated to macvlan device.  As a result, when
>> when we call dev_queue_xmit on the vlan->dev, we end up segmenting since
>> lower level says it does support segmentation.
>>
>> One way to fix this is to never disable offloads on macvlan.  macvlan
>> will always try to use __dev_queue_xmit() with it's lower device, so any
>> segmentation can happen there.
> 
> If you have anything that I should test, let me know.

Hi Christian

Just sent out a patch to fix this.  I tried it with namespaces and
kvm guests and it seems to restore performance for me.

Please give it a try.

Thanks
-vlad
> 
> Christian
>