* [Qemu-devel] GRO not happening in VM with VxLAN
@ 2015-06-29 5:46 Santosh R
2015-06-29 15:34 ` [Qemu-devel] [Qemu-discuss] " Vlad Yasevich
[not found] ` <CAOoYtEs5PPUQiE5rUgKZ2LjXK05YBGmZoG5cqKyubsn6G90-Zw@mail.gmail.com>
0 siblings, 2 replies; 9+ messages in thread
From: Santosh R @ 2015-06-29 5:46 UTC (permalink / raw)
To: qemu-devel, qemu-discuss
[-- Attachment #1: Type: text/plain, Size: 5002 bytes --]
All,
I am testing VxLAN performance in VM. For this I am using below command
to bring up the VM.
# qemu-system-x86_64 -m 4096 -smp 4 -boot c -device
virtio-net-pci,netdev=tap0,mac=00:11:22:33:44:55 -netdev
tap,id=tap0,script=no,vhost=on -drive file=/root/vdisk_rhel65.img &
This is resulting in lower throughput in VM. On looking further I found
that GRO is not happing on the tap interface.
Using tcpdump I could see GRO happeing in physcial nic, vxlan and bridge
interface. but _not_ on tap interface.
So, I thought to use veth pair instead of the tap interface in VM. What is
the command to do this?
Also,
1. Why is rx-checksumming off for tap interface?
2. Why is GRO not happening with tap interface?
Pls copy me in all the replies as I am not on the list.
# brctl show
bridge name bridge id STP enabled interfaces
br-vx 8000.323a9146b2d2 no tap0
vxlan0
# ethtool -k eth35
Features for eth35:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: on
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: on
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: on
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: on [fixed]
# ethtool -k vxlan0
Features for vxlan0:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp6-segmentation: on
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]
# ethtool -k tap0
Features for tap0:
rx-checksumming: off [fixed]
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: on
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]
Thanks
-Santosh
[-- Attachment #2: Type: text/html, Size: 5966 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [Qemu-discuss] GRO not happening in VM with VxLAN
2015-06-29 5:46 [Qemu-devel] GRO not happening in VM with VxLAN Santosh R
@ 2015-06-29 15:34 ` Vlad Yasevich
2015-06-29 15:54 ` Santosh R
[not found] ` <CAOoYtEs5PPUQiE5rUgKZ2LjXK05YBGmZoG5cqKyubsn6G90-Zw@mail.gmail.com>
1 sibling, 1 reply; 9+ messages in thread
From: Vlad Yasevich @ 2015-06-29 15:34 UTC (permalink / raw)
To: qemu-discuss; +Cc: skrastapur, qemu-devel
On 06/29/2015 01:46 AM, Santosh R wrote:
> All,
>
> I am testing VxLAN performance in VM. For this I am using below command
> to bring up the VM.
> # qemu-system-x86_64 -m 4096 -smp 4 -boot c -device
> virtio-net-pci,netdev=tap0,mac=00:11:22:33:44:55 -netdev
> tap,id=tap0,script=no,vhost=on -drive file=/root/vdisk_rhel65.img &
>
> This is resulting in lower throughput in VM. On looking further I found
> that GRO is not happing on the tap interface.
> Using tcpdump I could see GRO happeing in physcial nic, vxlan and bridge
> interface. but _not_ on tap interface.
I don't think this is GRO. GRO only happens at the lowest possible layer.
For packets coming from external sources, that happens just above the nic
when the packet enters the host for the first time. From then on, the features
of other devices determine if the large aggregated packet will be segmented
again or not.
>
> So, I thought to use veth pair instead of the tap interface in VM. What is
> the command to do this?
veth pair will not help with this. you need a tap or a macvtap to have
VM talk to host.
> Also,
> 1. Why is rx-checksumming off for tap interface?
because tap doesn't do checksum validation. it allows lower level devices or
VM to do so.
> 2. Why is GRO not happening with tap interface?
>
Again. I don't think it's GRO. Try capturing traffic at tap and see if you get
large packets (more the 1 mtu long) on the tap. If not, then someone else is fragmenting
packets and you need to find who.
-vlad
> Pls copy me in all the replies as I am not on the list.
>
> # brctl show
> bridge name bridge id STP enabled interfaces
> br-vx 8000.323a9146b2d2 no tap0
> vxlan0
> # ethtool -k eth35
> Features for eth35:
> rx-checksumming: on
> tx-checksumming: on
> tx-checksum-ipv4: on
> tx-checksum-ip-generic: off [fixed]
> tx-checksum-ipv6: on
> tx-checksum-fcoe-crc: off [fixed]
> tx-checksum-sctp: off [fixed]
> scatter-gather: on
> tx-scatter-gather: on
> tx-scatter-gather-fraglist: off [fixed]
> tcp-segmentation-offload: on
> tx-tcp-segmentation: on
> tx-tcp-ecn-segmentation: on
> tx-tcp6-segmentation: on
> udp-fragmentation-offload: off [fixed]
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off [fixed]
> rx-vlan-offload: on
> tx-vlan-offload: on
> ntuple-filters: off [fixed]
> receive-hashing: off [fixed]
> highdma: on
> rx-vlan-filter: off [fixed]
> vlan-challenged: off [fixed]
> tx-lockless: off [fixed]
> netns-local: off [fixed]
> tx-gso-robust: off [fixed]
> tx-fcoe-segmentation: off [fixed]
> tx-gre-segmentation: off [fixed]
> tx-ipip-segmentation: off [fixed]
> tx-sit-segmentation: off [fixed]
> tx-udp_tnl-segmentation: on
> tx-mpls-segmentation: off [fixed]
> fcoe-mtu: off [fixed]
> tx-nocache-copy: off
> loopback: off [fixed]
> rx-fcs: off [fixed]
> rx-all: off [fixed]
> tx-vlan-stag-hw-insert: off [fixed]
> rx-vlan-stag-hw-parse: off [fixed]
> rx-vlan-stag-filter: off [fixed]
> l2-fwd-offload: off [fixed]
> busy-poll: on [fixed]
>
> # ethtool -k vxlan0
> Features for vxlan0:
> rx-checksumming: on
> tx-checksumming: on
> tx-checksum-ipv4: off [fixed]
> tx-checksum-ip-generic: on
> tx-checksum-ipv6: off [fixed]
> tx-checksum-fcoe-crc: off [fixed]
> tx-checksum-sctp: off [fixed]
> scatter-gather: on
> tx-scatter-gather: on
> tx-scatter-gather-fraglist: off [fixed]
> tcp-segmentation-offload: on
> tx-tcp-segmentation: on
> tx-tcp-ecn-segmentation: on
> tx-tcp6-segmentation: on
> udp-fragmentation-offload: on
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off [fixed]
> rx-vlan-offload: off [fixed]
> tx-vlan-offload: on
> ntuple-filters: off [fixed]
> receive-hashing: off [fixed]
> highdma: off [fixed]
> rx-vlan-filter: off [fixed]
> vlan-challenged: off [fixed]
> tx-lockless: on [fixed]
> netns-local: off [fixed]
> tx-gso-robust: off [fixed]
> tx-fcoe-segmentation: off [fixed]
> tx-gre-segmentation: off [fixed]
> tx-ipip-segmentation: off [fixed]
> tx-sit-segmentation: off [fixed]
> tx-udp_tnl-segmentation: off [fixed]
> tx-mpls-segmentation: off [fixed]
> fcoe-mtu: off [fixed]
> tx-nocache-copy: off
> loopback: off [fixed]
> rx-fcs: off [fixed]
> rx-all: off [fixed]
> tx-vlan-stag-hw-insert: on
> rx-vlan-stag-hw-parse: off [fixed]
> rx-vlan-stag-filter: off [fixed]
> l2-fwd-offload: off [fixed]
> busy-poll: off [fixed]
>
> # ethtool -k tap0
> Features for tap0:
> rx-checksumming: off [fixed]
> tx-checksumming: on
> tx-checksum-ipv4: off [fixed]
> tx-checksum-ip-generic: on
> tx-checksum-ipv6: off [fixed]
> tx-checksum-fcoe-crc: off [fixed]
> tx-checksum-sctp: off [fixed]
> scatter-gather: on
> tx-scatter-gather: on
> tx-scatter-gather-fraglist: on
> tcp-segmentation-offload: on
> tx-tcp-segmentation: on
> tx-tcp-ecn-segmentation: on
> tx-tcp6-segmentation: on
> udp-fragmentation-offload: off [fixed]
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off [fixed]
> rx-vlan-offload: off [fixed]
> tx-vlan-offload: on
> ntuple-filters: off [fixed]
> receive-hashing: off [fixed]
> highdma: off [fixed]
> rx-vlan-filter: off [fixed]
> vlan-challenged: off [fixed]
> tx-lockless: off [fixed]
> netns-local: off [fixed]
> tx-gso-robust: off [fixed]
> tx-fcoe-segmentation: off [fixed]
> tx-gre-segmentation: off [fixed]
> tx-ipip-segmentation: off [fixed]
> tx-sit-segmentation: off [fixed]
> tx-udp_tnl-segmentation: off [fixed]
> tx-mpls-segmentation: off [fixed]
> fcoe-mtu: off [fixed]
> tx-nocache-copy: off
> loopback: off [fixed]
> rx-fcs: off [fixed]
> rx-all: off [fixed]
> tx-vlan-stag-hw-insert: on
> rx-vlan-stag-hw-parse: off [fixed]
> rx-vlan-stag-filter: off [fixed]
> l2-fwd-offload: off [fixed]
> busy-poll: off [fixed]
> Thanks
> -Santosh
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [Qemu-discuss] GRO not happening in VM with VxLAN
2015-06-29 15:34 ` [Qemu-devel] [Qemu-discuss] " Vlad Yasevich
@ 2015-06-29 15:54 ` Santosh R
2015-06-30 6:54 ` Santosh R
0 siblings, 1 reply; 9+ messages in thread
From: Santosh R @ 2015-06-29 15:54 UTC (permalink / raw)
To: vyasevic; +Cc: qemu-devel, qemu-discuss
[-- Attachment #1: Type: text/plain, Size: 6922 bytes --]
On Mon, Jun 29, 2015 at 9:04 PM, Vlad Yasevich <vyasevic@redhat.com> wrote:
> On 06/29/2015 01:46 AM, Santosh R wrote:
> > All,
> >
> > I am testing VxLAN performance in VM. For this I am using below
> command
> > to bring up the VM.
> > # qemu-system-x86_64 -m 4096 -smp 4 -boot c -device
> > virtio-net-pci,netdev=tap0,mac=00:11:22:33:44:55 -netdev
> > tap,id=tap0,script=no,vhost=on -drive file=/root/vdisk_rhel65.img &
> >
> > This is resulting in lower throughput in VM. On looking further I found
> > that GRO is not happing on the tap interface.
> > Using tcpdump I could see GRO happeing in physcial nic, vxlan and bridge
> > interface. but _not_ on tap interface.
>
> I don't think this is GRO. GRO only happens at the lowest possible layer.
> For packets coming from external sources, that happens just above the nic
> when the packet enters the host for the first time. From then on, the
> features
> of other devices determine if the large aggregated packet will be segmented
> again or not.
>
>
> >
> > So, I thought to use veth pair instead of the tap interface in VM. What
> is
> > the command to do this?
>
> veth pair will not help with this. you need a tap or a macvtap to have
> VM talk to host.
>
> > Also,
> > 1. Why is rx-checksumming off for tap interface?
>
> because tap doesn't do checksum validation. it allows lower level devices
> or
> VM to do so.
>
> > 2. Why is GRO not happening with tap interface?
> >
>
> Again. I don't think it's GRO. Try capturing traffic at tap and see if
> you get
> large packets (more the 1 mtu long) on the tap. If not, then someone else
> is fragmenting
> packets and you need to find who.
>
[Santosh]: Thanks for the reply.
With VxLAN, aggregated packets are seen on physical, VxLAN and bridge
interface but _not_ on the tap interface.
Without VxLAN, I do see aggregated (GRO) packets all the way upto VM (i.e.
physical, bridge and tap interface).
As you rightly pointed, looks like VxLAN packets are getting fragmented.
Any pointers to look for?
>
> -vlad
>
> > Pls copy me in all the replies as I am not on the list.
> >
> > # brctl show
> > bridge name bridge id STP enabled interfaces
> > br-vx 8000.323a9146b2d2 no tap0
> > vxlan0
> > # ethtool -k eth35
> > Features for eth35:
> > rx-checksumming: on
> > tx-checksumming: on
> > tx-checksum-ipv4: on
> > tx-checksum-ip-generic: off [fixed]
> > tx-checksum-ipv6: on
> > tx-checksum-fcoe-crc: off [fixed]
> > tx-checksum-sctp: off [fixed]
> > scatter-gather: on
> > tx-scatter-gather: on
> > tx-scatter-gather-fraglist: off [fixed]
> > tcp-segmentation-offload: on
> > tx-tcp-segmentation: on
> > tx-tcp-ecn-segmentation: on
> > tx-tcp6-segmentation: on
> > udp-fragmentation-offload: off [fixed]
> > generic-segmentation-offload: on
> > generic-receive-offload: on
> > large-receive-offload: off [fixed]
> > rx-vlan-offload: on
> > tx-vlan-offload: on
> > ntuple-filters: off [fixed]
> > receive-hashing: off [fixed]
> > highdma: on
> > rx-vlan-filter: off [fixed]
> > vlan-challenged: off [fixed]
> > tx-lockless: off [fixed]
> > netns-local: off [fixed]
> > tx-gso-robust: off [fixed]
> > tx-fcoe-segmentation: off [fixed]
> > tx-gre-segmentation: off [fixed]
> > tx-ipip-segmentation: off [fixed]
> > tx-sit-segmentation: off [fixed]
> > tx-udp_tnl-segmentation: on
> > tx-mpls-segmentation: off [fixed]
> > fcoe-mtu: off [fixed]
> > tx-nocache-copy: off
> > loopback: off [fixed]
> > rx-fcs: off [fixed]
> > rx-all: off [fixed]
> > tx-vlan-stag-hw-insert: off [fixed]
> > rx-vlan-stag-hw-parse: off [fixed]
> > rx-vlan-stag-filter: off [fixed]
> > l2-fwd-offload: off [fixed]
> > busy-poll: on [fixed]
> >
> > # ethtool -k vxlan0
> > Features for vxlan0:
> > rx-checksumming: on
> > tx-checksumming: on
> > tx-checksum-ipv4: off [fixed]
> > tx-checksum-ip-generic: on
> > tx-checksum-ipv6: off [fixed]
> > tx-checksum-fcoe-crc: off [fixed]
> > tx-checksum-sctp: off [fixed]
> > scatter-gather: on
> > tx-scatter-gather: on
> > tx-scatter-gather-fraglist: off [fixed]
> > tcp-segmentation-offload: on
> > tx-tcp-segmentation: on
> > tx-tcp-ecn-segmentation: on
> > tx-tcp6-segmentation: on
> > udp-fragmentation-offload: on
> > generic-segmentation-offload: on
> > generic-receive-offload: on
> > large-receive-offload: off [fixed]
> > rx-vlan-offload: off [fixed]
> > tx-vlan-offload: on
> > ntuple-filters: off [fixed]
> > receive-hashing: off [fixed]
> > highdma: off [fixed]
> > rx-vlan-filter: off [fixed]
> > vlan-challenged: off [fixed]
> > tx-lockless: on [fixed]
> > netns-local: off [fixed]
> > tx-gso-robust: off [fixed]
> > tx-fcoe-segmentation: off [fixed]
> > tx-gre-segmentation: off [fixed]
> > tx-ipip-segmentation: off [fixed]
> > tx-sit-segmentation: off [fixed]
> > tx-udp_tnl-segmentation: off [fixed]
> > tx-mpls-segmentation: off [fixed]
> > fcoe-mtu: off [fixed]
> > tx-nocache-copy: off
> > loopback: off [fixed]
> > rx-fcs: off [fixed]
> > rx-all: off [fixed]
> > tx-vlan-stag-hw-insert: on
> > rx-vlan-stag-hw-parse: off [fixed]
> > rx-vlan-stag-filter: off [fixed]
> > l2-fwd-offload: off [fixed]
> > busy-poll: off [fixed]
> >
> > # ethtool -k tap0
> > Features for tap0:
> > rx-checksumming: off [fixed]
> > tx-checksumming: on
> > tx-checksum-ipv4: off [fixed]
> > tx-checksum-ip-generic: on
> > tx-checksum-ipv6: off [fixed]
> > tx-checksum-fcoe-crc: off [fixed]
> > tx-checksum-sctp: off [fixed]
> > scatter-gather: on
> > tx-scatter-gather: on
> > tx-scatter-gather-fraglist: on
> > tcp-segmentation-offload: on
> > tx-tcp-segmentation: on
> > tx-tcp-ecn-segmentation: on
> > tx-tcp6-segmentation: on
> > udp-fragmentation-offload: off [fixed]
> > generic-segmentation-offload: on
> > generic-receive-offload: on
> > large-receive-offload: off [fixed]
> > rx-vlan-offload: off [fixed]
> > tx-vlan-offload: on
> > ntuple-filters: off [fixed]
> > receive-hashing: off [fixed]
> > highdma: off [fixed]
> > rx-vlan-filter: off [fixed]
> > vlan-challenged: off [fixed]
> > tx-lockless: off [fixed]
> > netns-local: off [fixed]
> > tx-gso-robust: off [fixed]
> > tx-fcoe-segmentation: off [fixed]
> > tx-gre-segmentation: off [fixed]
> > tx-ipip-segmentation: off [fixed]
> > tx-sit-segmentation: off [fixed]
> > tx-udp_tnl-segmentation: off [fixed]
> > tx-mpls-segmentation: off [fixed]
> > fcoe-mtu: off [fixed]
> > tx-nocache-copy: off
> > loopback: off [fixed]
> > rx-fcs: off [fixed]
> > rx-all: off [fixed]
> > tx-vlan-stag-hw-insert: on
> > rx-vlan-stag-hw-parse: off [fixed]
> > rx-vlan-stag-filter: off [fixed]
> > l2-fwd-offload: off [fixed]
> > busy-poll: off [fixed]
> > Thanks
> > -Santosh
> >
>
>
[-- Attachment #2: Type: text/html, Size: 8948 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [Qemu-discuss] GRO not happening in VM with VxLAN
2015-06-29 15:54 ` Santosh R
@ 2015-06-30 6:54 ` Santosh R
2015-07-01 18:50 ` Santosh R
0 siblings, 1 reply; 9+ messages in thread
From: Santosh R @ 2015-06-30 6:54 UTC (permalink / raw)
To: vyasevic; +Cc: qemu-devel, qemu-discuss
[-- Attachment #1: Type: text/plain, Size: 2419 bytes --]
On Mon, Jun 29, 2015 at 9:24 PM, Santosh R <skrastapur@gmail.com> wrote:
>
> On Mon, Jun 29, 2015 at 9:04 PM, Vlad Yasevich <vyasevic@redhat.com>
> wrote:
>
>> On 06/29/2015 01:46 AM, Santosh R wrote:
>> > All,
>> >
>> > I am testing VxLAN performance in VM. For this I am using below
>> command
>> > to bring up the VM.
>> > # qemu-system-x86_64 -m 4096 -smp 4 -boot c -device
>> > virtio-net-pci,netdev=tap0,mac=00:11:22:33:44:55 -netdev
>> > tap,id=tap0,script=no,vhost=on -drive file=/root/vdisk_rhel65.img &
>> >
>> > This is resulting in lower throughput in VM. On looking further I found
>> > that GRO is not happing on the tap interface.
>> > Using tcpdump I could see GRO happeing in physcial nic, vxlan and bridge
>> > interface. but _not_ on tap interface.
>>
>> I don't think this is GRO. GRO only happens at the lowest possible
>> layer.
>> For packets coming from external sources, that happens just above the nic
>> when the packet enters the host for the first time. From then on, the
>> features
>> of other devices determine if the large aggregated packet will be
>> segmented
>> again or not.
>>
>>
>> >
>> > So, I thought to use veth pair instead of the tap interface in VM. What
>> is
>> > the command to do this?
>>
>> veth pair will not help with this. you need a tap or a macvtap to have
>> VM talk to host.
>>
>> > Also,
>> > 1. Why is rx-checksumming off for tap interface?
>>
>> because tap doesn't do checksum validation. it allows lower level
>> devices or
>> VM to do so.
>>
>> > 2. Why is GRO not happening with tap interface?
>> >
>>
>> Again. I don't think it's GRO. Try capturing traffic at tap and see if
>> you get
>> large packets (more the 1 mtu long) on the tap. If not, then someone
>> else is fragmenting
>> packets and you need to find who.
>>
> [Santosh]: Thanks for the reply.
> With VxLAN, aggregated packets are seen on physical, VxLAN and bridge
> interface but _not_ on the tap interface.
> Without VxLAN, I do see aggregated (GRO) packets all the way upto VM (i.e.
> physical, bridge and tap interface).
> As you rightly pointed, looks like VxLAN packets are getting fragmented.
> Any pointers to look for?
>
>
By the way, I was referring below link for VxLAN testing and ran into the
GRO issue.
This mentions "The veth0 is suppose to be connected to the VM, while the
veth1 is connected to the Linux bridge".
https://community.mellanox.com/docs/DOC-1473
[-- Attachment #2: Type: text/html, Size: 3435 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [Qemu-discuss] GRO not happening in VM with VxLAN
2015-06-30 6:54 ` Santosh R
@ 2015-07-01 18:50 ` Santosh R
2015-07-01 19:07 ` Vlad Yasevich
0 siblings, 1 reply; 9+ messages in thread
From: Santosh R @ 2015-07-01 18:50 UTC (permalink / raw)
To: vyasevic; +Cc: qemu-devel, qemu-discuss
[-- Attachment #1: Type: text/plain, Size: 3257 bytes --]
Since the vxlan UDP header checksum is 0, udp_tunnel_gro_complete (called
via vxlan_gro_complete) is setting SKB_GSO_UDP_TUNNEL in
skb_shinfo(skb)->gso_type.
Later when bridge interface tries to forward this packet to tap interface
(br_dev_queue_push_xmit -> __dev_queue_xmit -> validate_xmit_skb)
netif_needs_gso will succeed (skb_gso_ok returns 0)
resulting in packet segmentation (which was earlier aggregated) as tap
interface doesn't have this feature. Is there a way to enable
tx-udp_tnl-segmentation for tap interface?
# ethtool -K tap0 tx-udp_tnl-segmentation on
Could not change any device features
# ethtool -k tap0 | grep -i tnl
tx-udp_tnl-segmentation: off [fixed]
On Tue, Jun 30, 2015 at 12:24 PM, Santosh R <skrastapur@gmail.com> wrote:
>
> On Mon, Jun 29, 2015 at 9:24 PM, Santosh R <skrastapur@gmail.com> wrote:
>
>>
>> On Mon, Jun 29, 2015 at 9:04 PM, Vlad Yasevich <vyasevic@redhat.com>
>> wrote:
>>
>>> On 06/29/2015 01:46 AM, Santosh R wrote:
>>> > All,
>>> >
>>> > I am testing VxLAN performance in VM. For this I am using below
>>> command
>>> > to bring up the VM.
>>> > # qemu-system-x86_64 -m 4096 -smp 4 -boot c -device
>>> > virtio-net-pci,netdev=tap0,mac=00:11:22:33:44:55 -netdev
>>> > tap,id=tap0,script=no,vhost=on -drive file=/root/vdisk_rhel65.img &
>>> >
>>> > This is resulting in lower throughput in VM. On looking further I found
>>> > that GRO is not happing on the tap interface.
>>> > Using tcpdump I could see GRO happeing in physcial nic, vxlan and
>>> bridge
>>> > interface. but _not_ on tap interface.
>>>
>>> I don't think this is GRO. GRO only happens at the lowest possible
>>> layer.
>>> For packets coming from external sources, that happens just above the nic
>>> when the packet enters the host for the first time. From then on, the
>>> features
>>> of other devices determine if the large aggregated packet will be
>>> segmented
>>> again or not.
>>>
>>>
>>> >
>>> > So, I thought to use veth pair instead of the tap interface in VM.
>>> What is
>>> > the command to do this?
>>>
>>> veth pair will not help with this. you need a tap or a macvtap to have
>>> VM talk to host.
>>>
>>> > Also,
>>> > 1. Why is rx-checksumming off for tap interface?
>>>
>>> because tap doesn't do checksum validation. it allows lower level
>>> devices or
>>> VM to do so.
>>>
>>> > 2. Why is GRO not happening with tap interface?
>>> >
>>>
>>> Again. I don't think it's GRO. Try capturing traffic at tap and see if
>>> you get
>>> large packets (more the 1 mtu long) on the tap. If not, then someone
>>> else is fragmenting
>>> packets and you need to find who.
>>>
>> [Santosh]: Thanks for the reply.
>> With VxLAN, aggregated packets are seen on physical, VxLAN and bridge
>> interface but _not_ on the tap interface.
>> Without VxLAN, I do see aggregated (GRO) packets all the way upto VM
>> (i.e. physical, bridge and tap interface).
>> As you rightly pointed, looks like VxLAN packets are getting fragmented.
>> Any pointers to look for?
>>
>>
> By the way, I was referring below link for VxLAN testing and ran into the
> GRO issue.
> This mentions "The veth0 is suppose to be connected to the VM, while the
> veth1 is connected to the Linux bridge".
> https://community.mellanox.com/docs/DOC-1473
>
[-- Attachment #2: Type: text/html, Size: 4587 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [Qemu-discuss] GRO not happening in VM with VxLAN
2015-07-01 18:50 ` Santosh R
@ 2015-07-01 19:07 ` Vlad Yasevich
2015-07-01 19:14 ` Santosh R
0 siblings, 1 reply; 9+ messages in thread
From: Vlad Yasevich @ 2015-07-01 19:07 UTC (permalink / raw)
To: Santosh R; +Cc: qemu-devel, qemu-discuss
On 07/01/2015 02:50 PM, Santosh R wrote:
> Since the vxlan UDP header checksum is 0, udp_tunnel_gro_complete (called
> via vxlan_gro_complete) is setting SKB_GSO_UDP_TUNNEL in
> skb_shinfo(skb)->gso_type.
> Later when bridge interface tries to forward this packet to tap interface
> (br_dev_queue_push_xmit -> __dev_queue_xmit -> validate_xmit_skb)
> netif_needs_gso will succeed (skb_gso_ok returns 0)
> resulting in packet segmentation (which was earlier aggregated) as tap
> interface doesn't have this feature. Is there a way to enable
> tx-udp_tnl-segmentation for tap interface?
No. Those features are set by qemu when the guest is created and
mirror any features that are supported by the guest. So, if the guest
virtio doesn't have that feature, neither will the tap device.
Which kernel version are you using on the host?
-vlad
>
> # ethtool -K tap0 tx-udp_tnl-segmentation on
> Could not change any device features
>
> # ethtool -k tap0 | grep -i tnl
> tx-udp_tnl-segmentation: off [fixed]
>
> On Tue, Jun 30, 2015 at 12:24 PM, Santosh R <skrastapur@gmail.com> wrote:
>
>>
>> On Mon, Jun 29, 2015 at 9:24 PM, Santosh R <skrastapur@gmail.com> wrote:
>>
>>>
>>> On Mon, Jun 29, 2015 at 9:04 PM, Vlad Yasevich <vyasevic@redhat.com>
>>> wrote:
>>>
>>>> On 06/29/2015 01:46 AM, Santosh R wrote:
>>>>> All,
>>>>>
>>>>> I am testing VxLAN performance in VM. For this I am using below
>>>> command
>>>>> to bring up the VM.
>>>>> # qemu-system-x86_64 -m 4096 -smp 4 -boot c -device
>>>>> virtio-net-pci,netdev=tap0,mac=00:11:22:33:44:55 -netdev
>>>>> tap,id=tap0,script=no,vhost=on -drive file=/root/vdisk_rhel65.img &
>>>>>
>>>>> This is resulting in lower throughput in VM. On looking further I found
>>>>> that GRO is not happing on the tap interface.
>>>>> Using tcpdump I could see GRO happeing in physcial nic, vxlan and
>>>> bridge
>>>>> interface. but _not_ on tap interface.
>>>>
>>>> I don't think this is GRO. GRO only happens at the lowest possible
>>>> layer.
>>>> For packets coming from external sources, that happens just above the nic
>>>> when the packet enters the host for the first time. From then on, the
>>>> features
>>>> of other devices determine if the large aggregated packet will be
>>>> segmented
>>>> again or not.
>>>>
>>>>
>>>>>
>>>>> So, I thought to use veth pair instead of the tap interface in VM.
>>>> What is
>>>>> the command to do this?
>>>>
>>>> veth pair will not help with this. you need a tap or a macvtap to have
>>>> VM talk to host.
>>>>
>>>>> Also,
>>>>> 1. Why is rx-checksumming off for tap interface?
>>>>
>>>> because tap doesn't do checksum validation. it allows lower level
>>>> devices or
>>>> VM to do so.
>>>>
>>>>> 2. Why is GRO not happening with tap interface?
>>>>>
>>>>
>>>> Again. I don't think it's GRO. Try capturing traffic at tap and see if
>>>> you get
>>>> large packets (more the 1 mtu long) on the tap. If not, then someone
>>>> else is fragmenting
>>>> packets and you need to find who.
>>>>
>>> [Santosh]: Thanks for the reply.
>>> With VxLAN, aggregated packets are seen on physical, VxLAN and bridge
>>> interface but _not_ on the tap interface.
>>> Without VxLAN, I do see aggregated (GRO) packets all the way upto VM
>>> (i.e. physical, bridge and tap interface).
>>> As you rightly pointed, looks like VxLAN packets are getting fragmented.
>>> Any pointers to look for?
>>>
>>>
>> By the way, I was referring below link for VxLAN testing and ran into the
>> GRO issue.
>> This mentions "The veth0 is suppose to be connected to the VM, while the
>> veth1 is connected to the Linux bridge".
>> https://community.mellanox.com/docs/DOC-1473
>>
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [Qemu-discuss] GRO not happening in VM with VxLAN
2015-07-01 19:07 ` Vlad Yasevich
@ 2015-07-01 19:14 ` Santosh R
0 siblings, 0 replies; 9+ messages in thread
From: Santosh R @ 2015-07-01 19:14 UTC (permalink / raw)
To: vyasevic; +Cc: qemu-devel, qemu-discuss
[-- Attachment #1: Type: text/plain, Size: 4532 bytes --]
I am using 3.18 kernel.
#uname -a
Linux Indra.asicdesigners.com 3.18.0 #11 SMP Wed Jul 1 21:49:44 IST 2015
x86_64 x86_64 x86_64 GNU/Linux
#cat /etc/issue
Red Hat Enterprise Linux Server release 6.5 (Santiago)
Kernel \r on an \m
As mentioned in my initial post, this is how I am starting the VM. How do
we enable/pass tx-udp_tnl-segmentation feature in the below command? Is
this an option while creating the VM?
*# qemu-system-x86_64 -m 4096 -smp 4 -boot c -device
virtio-net-pci,netdev=tap0,mac=00:11:22:33:44:55 -netdev
tap,id=tap0,script=no,vhost=on -drive file=/root/vdisk_rhel65.img &*
On Thu, Jul 2, 2015 at 12:37 AM, Vlad Yasevich <vyasevic@redhat.com> wrote:
> On 07/01/2015 02:50 PM, Santosh R wrote:
> > Since the vxlan UDP header checksum is 0, udp_tunnel_gro_complete (called
> > via vxlan_gro_complete) is setting SKB_GSO_UDP_TUNNEL in
> > skb_shinfo(skb)->gso_type.
> > Later when bridge interface tries to forward this packet to tap interface
> > (br_dev_queue_push_xmit -> __dev_queue_xmit -> validate_xmit_skb)
> > netif_needs_gso will succeed (skb_gso_ok returns 0)
> > resulting in packet segmentation (which was earlier aggregated) as tap
> > interface doesn't have this feature. Is there a way to enable
> > tx-udp_tnl-segmentation for tap interface?
>
> No. Those features are set by qemu when the guest is created and
> mirror any features that are supported by the guest. So, if the guest
> virtio doesn't have that feature, neither will the tap device.
>
> Which kernel version are you using on the host?
>
> -vlad
>
> >
> > # ethtool -K tap0 tx-udp_tnl-segmentation on
> > Could not change any device features
> >
> > # ethtool -k tap0 | grep -i tnl
> > tx-udp_tnl-segmentation: off [fixed]
> >
> > On Tue, Jun 30, 2015 at 12:24 PM, Santosh R <skrastapur@gmail.com>
> wrote:
> >
> >>
> >> On Mon, Jun 29, 2015 at 9:24 PM, Santosh R <skrastapur@gmail.com>
> wrote:
> >>
> >>>
> >>> On Mon, Jun 29, 2015 at 9:04 PM, Vlad Yasevich <vyasevic@redhat.com>
> >>> wrote:
> >>>
> >>>> On 06/29/2015 01:46 AM, Santosh R wrote:
> >>>>> All,
> >>>>>
> >>>>> I am testing VxLAN performance in VM. For this I am using below
> >>>> command
> >>>>> to bring up the VM.
> >>>>> # qemu-system-x86_64 -m 4096 -smp 4 -boot c -device
> >>>>> virtio-net-pci,netdev=tap0,mac=00:11:22:33:44:55 -netdev
> >>>>> tap,id=tap0,script=no,vhost=on -drive file=/root/vdisk_rhel65.img &
> >>>>>
> >>>>> This is resulting in lower throughput in VM. On looking further I
> found
> >>>>> that GRO is not happing on the tap interface.
> >>>>> Using tcpdump I could see GRO happeing in physcial nic, vxlan and
> >>>> bridge
> >>>>> interface. but _not_ on tap interface.
> >>>>
> >>>> I don't think this is GRO. GRO only happens at the lowest possible
> >>>> layer.
> >>>> For packets coming from external sources, that happens just above the
> nic
> >>>> when the packet enters the host for the first time. From then on, the
> >>>> features
> >>>> of other devices determine if the large aggregated packet will be
> >>>> segmented
> >>>> again or not.
> >>>>
> >>>>
> >>>>>
> >>>>> So, I thought to use veth pair instead of the tap interface in VM.
> >>>> What is
> >>>>> the command to do this?
> >>>>
> >>>> veth pair will not help with this. you need a tap or a macvtap to
> have
> >>>> VM talk to host.
> >>>>
> >>>>> Also,
> >>>>> 1. Why is rx-checksumming off for tap interface?
> >>>>
> >>>> because tap doesn't do checksum validation. it allows lower level
> >>>> devices or
> >>>> VM to do so.
> >>>>
> >>>>> 2. Why is GRO not happening with tap interface?
> >>>>>
> >>>>
> >>>> Again. I don't think it's GRO. Try capturing traffic at tap and see
> if
> >>>> you get
> >>>> large packets (more the 1 mtu long) on the tap. If not, then someone
> >>>> else is fragmenting
> >>>> packets and you need to find who.
> >>>>
> >>> [Santosh]: Thanks for the reply.
> >>> With VxLAN, aggregated packets are seen on physical, VxLAN and bridge
> >>> interface but _not_ on the tap interface.
> >>> Without VxLAN, I do see aggregated (GRO) packets all the way upto VM
> >>> (i.e. physical, bridge and tap interface).
> >>> As you rightly pointed, looks like VxLAN packets are getting
> fragmented.
> >>> Any pointers to look for?
> >>>
> >>>
> >> By the way, I was referring below link for VxLAN testing and ran into
> the
> >> GRO issue.
> >> This mentions "The veth0 is suppose to be connected to the VM, while the
> >> veth1 is connected to the Linux bridge".
> >> https://community.mellanox.com/docs/DOC-1473
> >>
> >
>
>
[-- Attachment #2: Type: text/html, Size: 6559 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* GRO not happening in VM with VxLAN
[not found] ` <CAOoYtEs5PPUQiE5rUgKZ2LjXK05YBGmZoG5cqKyubsn6G90-Zw@mail.gmail.com>
@ 2015-07-03 6:47 ` Santosh R
2015-07-03 18:27 ` Tom Herbert
0 siblings, 1 reply; 9+ messages in thread
From: Santosh R @ 2015-07-03 6:47 UTC (permalink / raw)
To: netdev
Hi,
I am testing VxLAN performance between VMs using 3.18 host kernel.
For this I am using below command to bring up the VM.
# qemu-system-x86_64 -m 4096 -smp 4 -boot c -device
virtio-net-pci,netdev=tap0,mac=00:11:22:33:44:55 -netdev
tap,id=tap0,script=no,vhost=on -drive file=/root/vdisk_rhel65.img &
Using the below setup, I am not getting good performance between VMs.
VM1--tap1--bridge1--vxlan1--10G Nic1 <---> 10G Nic2--vxlan2--bridge2--tap2--VM2
The reason being, when VM1 sends traffic to VM2, the received packets
are correctly aggregated at Nic2-vxlan2…bridge2 in host2.
But they get segmented at tap2 interface as it doesn’t support VxLAN
segmentation offload (tx-udp_tnl-segmentation: off [fixed]).
When bridge interface tries to forward this packet to tap interface
(br_dev_queue_push_xmit -> __dev_queue_xmit -> validate_xmit_skb),
netif_needs_gso will succeed (skb_gso_ok returns 0) resulting in
packet segmentation (which was earlier aggregated).
Is this an expected behaviour? Does tap interface support
tx-udp_tnl-segmentation?
I had tried to discuss this issue with qemu-devel forum. Here is the
link http://lists.nongnu.org/archive/html/qemu-devel/2015-06/msg07159.html
Driver is updating csum_level of sk_buff to 1 and calls napi_gro_frags
for VxLAN packets.
# brctl show
bridge name bridge id STP enabled interfaces
br-vx 8000.323a9146b2d2 no tap0
vxlan0
# ethtool -k eth35
Features for eth35:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: on
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: on
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: on
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: on [fixed]
# ethtool -k vxlan0
Features for vxlan0:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp6-segmentation: on
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]
# ethtool -k tap0
Features for tap0:
rx-checksumming: off [fixed]
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: on
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]
Thanks
-Santosh
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: GRO not happening in VM with VxLAN
2015-07-03 6:47 ` Santosh R
@ 2015-07-03 18:27 ` Tom Herbert
0 siblings, 0 replies; 9+ messages in thread
From: Tom Herbert @ 2015-07-03 18:27 UTC (permalink / raw)
To: Santosh R; +Cc: Linux Kernel Network Developers
On Thu, Jul 2, 2015 at 11:47 PM, Santosh R <skrastapur@gmail.com> wrote:
> Hi,
>
> I am testing VxLAN performance between VMs using 3.18 host kernel.
> For this I am using below command to bring up the VM.
> # qemu-system-x86_64 -m 4096 -smp 4 -boot c -device
> virtio-net-pci,netdev=tap0,mac=00:11:22:33:44:55 -netdev
> tap,id=tap0,script=no,vhost=on -drive file=/root/vdisk_rhel65.img &
>
> Using the below setup, I am not getting good performance between VMs.
>
> VM1--tap1--bridge1--vxlan1--10G Nic1 <---> 10G Nic2--vxlan2--bridge2--tap2--VM2
>
This has already been reported. If zero UDP checksums are in use and
the device does not provide checksum unnecessary for the encapsulation
checksum or checksum complete, then GRO is not done at the device.
Also, GRO was not implemented at the vxlan device. Please see the
patches I posted to address the latter.
Thanks,
Tom
> The reason being, when VM1 sends traffic to VM2, the received packets
> are correctly aggregated at Nic2-vxlan2…bridge2 in host2.
> But they get segmented at tap2 interface as it doesn’t support VxLAN
> segmentation offload (tx-udp_tnl-segmentation: off [fixed]).
>
> When bridge interface tries to forward this packet to tap interface
> (br_dev_queue_push_xmit -> __dev_queue_xmit -> validate_xmit_skb),
> netif_needs_gso will succeed (skb_gso_ok returns 0) resulting in
> packet segmentation (which was earlier aggregated).
>
> Is this an expected behaviour? Does tap interface support
> tx-udp_tnl-segmentation?
>
> I had tried to discuss this issue with qemu-devel forum. Here is the
> link http://lists.nongnu.org/archive/html/qemu-devel/2015-06/msg07159.html
>
> Driver is updating csum_level of sk_buff to 1 and calls napi_gro_frags
> for VxLAN packets.
>
> # brctl show
> bridge name bridge id STP enabled interfaces
> br-vx 8000.323a9146b2d2 no tap0
> vxlan0
> # ethtool -k eth35
> Features for eth35:
> rx-checksumming: on
> tx-checksumming: on
> tx-checksum-ipv4: on
> tx-checksum-ip-generic: off [fixed]
> tx-checksum-ipv6: on
> tx-checksum-fcoe-crc: off [fixed]
> tx-checksum-sctp: off [fixed]
> scatter-gather: on
> tx-scatter-gather: on
> tx-scatter-gather-fraglist: off [fixed]
> tcp-segmentation-offload: on
> tx-tcp-segmentation: on
> tx-tcp-ecn-segmentation: on
> tx-tcp6-segmentation: on
> udp-fragmentation-offload: off [fixed]
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off [fixed]
> rx-vlan-offload: on
> tx-vlan-offload: on
> ntuple-filters: off [fixed]
> receive-hashing: off [fixed]
> highdma: on
> rx-vlan-filter: off [fixed]
> vlan-challenged: off [fixed]
> tx-lockless: off [fixed]
> netns-local: off [fixed]
> tx-gso-robust: off [fixed]
> tx-fcoe-segmentation: off [fixed]
> tx-gre-segmentation: off [fixed]
> tx-ipip-segmentation: off [fixed]
> tx-sit-segmentation: off [fixed]
> tx-udp_tnl-segmentation: on
> tx-mpls-segmentation: off [fixed]
> fcoe-mtu: off [fixed]
> tx-nocache-copy: off
> loopback: off [fixed]
> rx-fcs: off [fixed]
> rx-all: off [fixed]
> tx-vlan-stag-hw-insert: off [fixed]
> rx-vlan-stag-hw-parse: off [fixed]
> rx-vlan-stag-filter: off [fixed]
> l2-fwd-offload: off [fixed]
> busy-poll: on [fixed]
>
> # ethtool -k vxlan0
> Features for vxlan0:
> rx-checksumming: on
> tx-checksumming: on
> tx-checksum-ipv4: off [fixed]
> tx-checksum-ip-generic: on
> tx-checksum-ipv6: off [fixed]
> tx-checksum-fcoe-crc: off [fixed]
> tx-checksum-sctp: off [fixed]
> scatter-gather: on
> tx-scatter-gather: on
> tx-scatter-gather-fraglist: off [fixed]
> tcp-segmentation-offload: on
> tx-tcp-segmentation: on
> tx-tcp-ecn-segmentation: on
> tx-tcp6-segmentation: on
> udp-fragmentation-offload: on
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off [fixed]
> rx-vlan-offload: off [fixed]
> tx-vlan-offload: on
> ntuple-filters: off [fixed]
> receive-hashing: off [fixed]
> highdma: off [fixed]
> rx-vlan-filter: off [fixed]
> vlan-challenged: off [fixed]
> tx-lockless: on [fixed]
> netns-local: off [fixed]
> tx-gso-robust: off [fixed]
> tx-fcoe-segmentation: off [fixed]
> tx-gre-segmentation: off [fixed]
> tx-ipip-segmentation: off [fixed]
> tx-sit-segmentation: off [fixed]
> tx-udp_tnl-segmentation: off [fixed]
> tx-mpls-segmentation: off [fixed]
> fcoe-mtu: off [fixed]
> tx-nocache-copy: off
> loopback: off [fixed]
> rx-fcs: off [fixed]
> rx-all: off [fixed]
> tx-vlan-stag-hw-insert: on
> rx-vlan-stag-hw-parse: off [fixed]
> rx-vlan-stag-filter: off [fixed]
> l2-fwd-offload: off [fixed]
> busy-poll: off [fixed]
>
> # ethtool -k tap0
> Features for tap0:
> rx-checksumming: off [fixed]
> tx-checksumming: on
> tx-checksum-ipv4: off [fixed]
> tx-checksum-ip-generic: on
> tx-checksum-ipv6: off [fixed]
> tx-checksum-fcoe-crc: off [fixed]
> tx-checksum-sctp: off [fixed]
> scatter-gather: on
> tx-scatter-gather: on
> tx-scatter-gather-fraglist: on
> tcp-segmentation-offload: on
> tx-tcp-segmentation: on
> tx-tcp-ecn-segmentation: on
> tx-tcp6-segmentation: on
> udp-fragmentation-offload: off [fixed]
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off [fixed]
> rx-vlan-offload: off [fixed]
> tx-vlan-offload: on
> ntuple-filters: off [fixed]
> receive-hashing: off [fixed]
> highdma: off [fixed]
> rx-vlan-filter: off [fixed]
> vlan-challenged: off [fixed]
> tx-lockless: off [fixed]
> netns-local: off [fixed]
> tx-gso-robust: off [fixed]
> tx-fcoe-segmentation: off [fixed]
> tx-gre-segmentation: off [fixed]
> tx-ipip-segmentation: off [fixed]
> tx-sit-segmentation: off [fixed]
> tx-udp_tnl-segmentation: off [fixed]
> tx-mpls-segmentation: off [fixed]
> fcoe-mtu: off [fixed]
> tx-nocache-copy: off
> loopback: off [fixed]
> rx-fcs: off [fixed]
> rx-all: off [fixed]
> tx-vlan-stag-hw-insert: on
> rx-vlan-stag-hw-parse: off [fixed]
> rx-vlan-stag-filter: off [fixed]
> l2-fwd-offload: off [fixed]
> busy-poll: off [fixed]
>
> Thanks
> -Santosh
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2015-07-03 18:27 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-06-29 5:46 [Qemu-devel] GRO not happening in VM with VxLAN Santosh R
2015-06-29 15:34 ` [Qemu-devel] [Qemu-discuss] " Vlad Yasevich
2015-06-29 15:54 ` Santosh R
2015-06-30 6:54 ` Santosh R
2015-07-01 18:50 ` Santosh R
2015-07-01 19:07 ` Vlad Yasevich
2015-07-01 19:14 ` Santosh R
[not found] ` <CAOoYtEs5PPUQiE5rUgKZ2LjXK05YBGmZoG5cqKyubsn6G90-Zw@mail.gmail.com>
2015-07-03 6:47 ` Santosh R
2015-07-03 18:27 ` Tom Herbert
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.