PROBLEM: xen-netfront: ethtool settings changed in 4.4.134 causing packet loss

PROBLEM: xen-netfront: ethtool settings changed in 4.4.134 causing packet loss

[Liam Shepherd]

When upgrading from 4.4.132 to 4.4.135 I started observing about 3%
packet loss on PV on HVM domU interfaces. Specifically TX dropped
packets (As reported by ifconfig).

It seems that the default ethtool settings have changed.

Previously (4.4.132):
tx-checksumming: on
    tx-checksum-ipv4: on [fixed]
    tx-checksum-ip-generic: off [fixed]
    tx-checksum-ipv6: on
    tx-checksum-fcoe-crc: off [fixed]
    tx-checksum-sctp: off [fixed]
scatter-gather: on
    tx-scatter-gather: on
    tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
    tx-tcp-segmentation: on
    tx-tcp-ecn-segmentation: off [fixed]
    tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on

Now (4.4.135):
tx-checksumming: on
    tx-checksum-ipv4: on [fixed]
    tx-checksum-ip-generic: off [fixed]
    tx-checksum-ipv6: off
    tx-checksum-fcoe-crc: off [fixed]
    tx-checksum-sctp: off [fixed]
scatter-gather: off
    tx-scatter-gather: off
    tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
    tx-tcp-segmentation: off
    tx-tcp-ecn-segmentation: off [fixed]
    tx-tcp6-segmentation: off
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off [requested on]

Manually running 'ethtool -K INT tso on sg on' on all interfaces stops
the packet loss again. They're then set like this:
tx-checksumming: on
    tx-checksum-ipv4: on [fixed]
    tx-checksum-ip-generic: off [fixed]
    tx-checksum-ipv6: off
    tx-checksum-fcoe-crc: off [fixed]
    tx-checksum-sctp: off [fixed]
scatter-gather: on
    tx-scatter-gather: on
    tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
    tx-tcp-segmentation: on
    tx-tcp-ecn-segmentation: off [fixed]
    tx-tcp6-segmentation: off [requested on]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on

I've traced it to a commit in 4.4.134:
6be4fe832954db3b359b72107a6c60e34d939b26
xen-netfront: Fix race between device setup and open
[ Upstream commit f599c64fdf7d9c108e8717fb04bc41c680120da4 ]
https://patchwork.kernel.org/patch/10328969/

Once that is reverted on 4.4.135, the ethtool settings are as 4.4.132
and there is no packet loss.

The VMs and hypervisor are all running CentOS 7.5. The hypervisor is
on the centos-virt provided kernel, and the VMs are using kernel-lt
from elrepo.

# xl info
host                   : blah
release                : 4.9.86-30.el7.x86_64
version                : #1 SMP Mon Mar 5 16:58:23 UTC 2018
machine                : x86_64
nr_cpus                : 40
max_cpu_id             : 191
nr_nodes               : 2
cores_per_socket       : 10
threads_per_core       : 2
cpu_mhz                : 2197
hw_caps                :
b7ebfbff:77fef3ff:2c100800:00000121:00000001:001cbfbb:00000000:00000100
virt_caps              : hvm hvm_directio
total_memory           : 81826
free_memory            : 13237
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 8
xen_extra              : .3-5.el7
xen_version            : 4.8.3-5.el7
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : Thu Apr 26 23:28:44 2018 -0700 git:10d7690-dirty
xen_commandline        : placeholder dom0_mem=4096M,max:4096M cpuinfo
com1=115200,8n1 console=com1,tty loglvl=all guest_loglvl=all
gnttab_max_frames=1024 cpufreq=none max_cstate=0
cc_compiler            : gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)
cc_compile_by          : mockbuild
cc_compile_domain      : centos.org
cc_compile_date        : Tue May  8 17:17:55 UTC 2018
build_id               : 484fda280c1d4a21346da47907cd1a8e9aba99df
xend_config_format     : 4

Please let me know anything else I can provide.

Thanks

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PROBLEM: xen-netfront: ethtool settings changed in 4.4.134 causing packet loss

[Ross Lagerwall]

On 06/05/2018 04:43 PM, Liam Shepherd wrote:
> When upgrading from 4.4.132 to 4.4.135 I started observing about 3%
> packet loss on PV on HVM domU interfaces. Specifically TX dropped
> packets (As reported by ifconfig).
> 
> It seems that the default ethtool settings have changed.
> 
> Previously (4.4.132):
> tx-checksumming: on
>      tx-checksum-ipv4: on [fixed]
>      tx-checksum-ip-generic: off [fixed]
>      tx-checksum-ipv6: on
>      tx-checksum-fcoe-crc: off [fixed]
>      tx-checksum-sctp: off [fixed]
> scatter-gather: on
>      tx-scatter-gather: on
>      tx-scatter-gather-fraglist: off [fixed]
> tcp-segmentation-offload: on
>      tx-tcp-segmentation: on
>      tx-tcp-ecn-segmentation: off [fixed]
>      tx-tcp6-segmentation: on
> udp-fragmentation-offload: off [fixed]
> generic-segmentation-offload: on
> 
> Now (4.4.135):
> tx-checksumming: on
>      tx-checksum-ipv4: on [fixed]
>      tx-checksum-ip-generic: off [fixed]
>      tx-checksum-ipv6: off
>      tx-checksum-fcoe-crc: off [fixed]
>      tx-checksum-sctp: off [fixed]
> scatter-gather: off
>      tx-scatter-gather: off
>      tx-scatter-gather-fraglist: off [fixed]
> tcp-segmentation-offload: off
>      tx-tcp-segmentation: off
>      tx-tcp-ecn-segmentation: off [fixed]
>      tx-tcp6-segmentation: off
> udp-fragmentation-offload: off [fixed]
> generic-segmentation-offload: off [requested on]
> 
> Manually running 'ethtool -K INT tso on sg on' on all interfaces stops
> the packet loss again. They're then set like this:
> tx-checksumming: on
>      tx-checksum-ipv4: on [fixed]
>      tx-checksum-ip-generic: off [fixed]
>      tx-checksum-ipv6: off
>      tx-checksum-fcoe-crc: off [fixed]
>      tx-checksum-sctp: off [fixed]
> scatter-gather: on
>      tx-scatter-gather: on
>      tx-scatter-gather-fraglist: off [fixed]
> tcp-segmentation-offload: on
>      tx-tcp-segmentation: on
>      tx-tcp-ecn-segmentation: off [fixed]
>      tx-tcp6-segmentation: off [requested on]
> udp-fragmentation-offload: off [fixed]
> generic-segmentation-offload: on
> 
> I've traced it to a commit in 4.4.134:
> 6be4fe832954db3b359b72107a6c60e34d939b26
> xen-netfront: Fix race between device setup and open
> [ Upstream commit f599c64fdf7d9c108e8717fb04bc41c680120da4 ]
> https://patchwork.kernel.org/patch/10328969/
> 
> Once that is reverted on 4.4.135, the ethtool settings are as 4.4.132
> and there is no packet loss.
> 
Thanks for the bug report. Changing the ethtool settings was certainly 
not intended with that change. I'll take a look to see what has gone wrong.

-- 
Ross Lagerwall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PROBLEM: xen-netfront: ethtool settings changed in 4.4.134 causing packet loss

[Liam Shepherd]

[-- Attachment #1.1: Type: text/plain, Size: 2807 bytes --]

Thank you.

As a side point David Vrabel is listed in the maintainers file for xen
hypervisor interface in 4.4.135 but his email is no longer active it seems.

On 7 Jun 2018 09:35, "Ross Lagerwall" <ross.lagerwall@citrix.com> wrote:

On 06/05/2018 04:43 PM, Liam Shepherd wrote:
> When upgrading from 4.4.132 to 4.4.135 I started observing about 3%
> packet loss on PV on HVM domU interfaces. Specifically TX dropped
> packets (As reported by ifconfig).
>
> It seems that the default ethtool settings have changed.
>
> Previously (4.4.132):
> tx-checksumming: on
>      tx-checksum-ipv4: on [fixed]
>      tx-checksum-ip-generic: off [fixed]
>      tx-checksum-ipv6: on
>      tx-checksum-fcoe-crc: off [fixed]
>      tx-checksum-sctp: off [fixed]
> scatter-gather: on
>      tx-scatter-gather: on
>      tx-scatter-gather-fraglist: off [fixed]
> tcp-segmentation-offload: on
>      tx-tcp-segmentation: on
>      tx-tcp-ecn-segmentation: off [fixed]
>      tx-tcp6-segmentation: on
> udp-fragmentation-offload: off [fixed]
> generic-segmentation-offload: on
>
> Now (4.4.135):
> tx-checksumming: on
>      tx-checksum-ipv4: on [fixed]
>      tx-checksum-ip-generic: off [fixed]
>      tx-checksum-ipv6: off
>      tx-checksum-fcoe-crc: off [fixed]
>      tx-checksum-sctp: off [fixed]
> scatter-gather: off
>      tx-scatter-gather: off
>      tx-scatter-gather-fraglist: off [fixed]
> tcp-segmentation-offload: off
>      tx-tcp-segmentation: off
>      tx-tcp-ecn-segmentation: off [fixed]
>      tx-tcp6-segmentation: off
> udp-fragmentation-offload: off [fixed]
> generic-segmentation-offload: off [requested on]
>
> Manually running 'ethtool -K INT tso on sg on' on all interfaces stops
> the packet loss again. They're then set like this:
> tx-checksumming: on
>      tx-checksum-ipv4: on [fixed]
>      tx-checksum-ip-generic: off [fixed]
>      tx-checksum-ipv6: off
>      tx-checksum-fcoe-crc: off [fixed]
>      tx-checksum-sctp: off [fixed]
> scatter-gather: on
>      tx-scatter-gather: on
>      tx-scatter-gather-fraglist: off [fixed]
> tcp-segmentation-offload: on
>      tx-tcp-segmentation: on
>      tx-tcp-ecn-segmentation: off [fixed]
>      tx-tcp6-segmentation: off [requested on]
> udp-fragmentation-offload: off [fixed]
> generic-segmentation-offload: on
>
> I've traced it to a commit in 4.4.134:
> 6be4fe832954db3b359b72107a6c60e34d939b26
> xen-netfront: Fix race between device setup and open
> [ Upstream commit f599c64fdf7d9c108e8717fb04bc41c680120da4 ]
> https://patchwork.kernel.org/patch/10328969/
>
> Once that is reverted on 4.4.135, the ethtool settings are as 4.4.132
> and there is no packet loss.
>
Thanks for the bug report. Changing the ethtool settings was certainly
not intended with that change. I'll take a look to see what has gone wrong.


-- 
Ross Lagerwall

[-- Attachment #1.2: Type: text/html, Size: 3989 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PROBLEM: xen-netfront: ethtool settings changed in 4.4.134 causing packet loss

[Ross Lagerwall]

On 06/07/2018 09:33 PM, Liam Shepherd wrote:
> Thank you.
> 
> As a side point David Vrabel is listed in the maintainers file for xen 
> hypervisor interface in 4.4.135 but his email is no longer active it seems.
> 

Yeah, he has left Citrix. The MAINTAINERS file is not usually updated in 
stable releases. You can consult the master branch MAINTAINERS file to 
see the up-to-date maintainers.

-- 
Ross Lagerwall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel