High pings when using SR-IOV w\ macvtap & virtio

High pings when using SR-IOV w\ macvtap & virtio

[kvmuser99]

KVM Team,

I'm going to the mailing lists because I'm at a loss.  I have thousands of Linux (CentOS 7) VMs deployed and a very small percent of them get into a state when there's some amount of network IO the guests will constantly clock high ping times (> 100 ms local network) and performance will suffer.  Some hosts will always clock the high ping times, other times running a CLI speedtest will cause the condition to surface.

Rebooting the guest does not solve the issue but a reboot of the host does for awhile.  Running something such as the following helps in most cases.  For reference eth0 is the physical interface from which all the VFs come from.

===
  126  ethtool -L eth0 combined 16 <-- Arbitrary number  
128  ethtool -L eth0 combined 4 <--- Number of CPU reserved for the host (it as previously set to this)
===

In most of the scenarios pings even to a 'bridge' interface which just handles guest\host communication also shoot high even though there is low network IO on that particular interface.


Config is as follows

-   Queues = number of CPU assigned to guest
-   Affinity is enabled.
    

===
    <interface type='direct' trustGuestRxFilters='yes'>      <mac address='MACHERE'/>
      <source dev='eth26' mode='passthrough'/>
      <model type='virtio'/>
      <driver name='vhost' queues='5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
=====


===
lshw -c network -businfo
pci@0000:03:02.5  eth12           network        Ethernet Virtual Function 700 Series

ip link show | grep eth12
26: eth12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000110: macvtap30@eth12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 5000

====

Info about the interface

===
[root@HOST~]# ethtool -k eth12Features for eth12:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: on
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: on
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: on
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: on
        tx-tcp6-segmentation: on
        tx-tcp-mangleid-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: off
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-ipip-segmentation: on
tx-sit-segmentation: on
tx-udp_tnl-segmentation: on
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
busy-poll: off [fixed]
tx-gre-csum-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-sctp-segmentation: off [fixed]
rx-gro-hw: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: on
rx-udp_tunnel-port-offload: off [fixed]
====


Versions
===
cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)



modinfo i40efilename:       /lib/modules/3.10.0-957.el7.x86_64/kernel/drivers/net/ethernet/intel/i40e/i40e.ko.xz
version:        2.3.2-k
license:        GPL
description:    Intel(R) Ethernet Connection XL710 Network Driver
author:         Intel Corporation, <e1000-devel@lists.sourceforge.net>
retpoline:      Y
rhelversion:    7.6

uname -r3.10.0-957.el7.x86_64

/usr/libexec/qemu-kvm -version
QEMU emulator version 5.1.0

=====


-kvmuser99