* ping latency issue on passthrough guest
@ 2018-05-11 7:20 ` Zhu Yijun
0 siblings, 0 replies; 10+ messages in thread
From: Zhu Yijun @ 2018-05-11 7:20 UTC (permalink / raw)
To: kvm, Alex Williamson, Paolo Bonzini, qemu-devel
Hi all,
I booted two sr-iov guests using KVM-VFIO and pinged each other with
no-load one night. I found that most of the latency was little than 0.1ms,
but several icmp_seq greater than 10ms, even up to 1000ms;
root@test-ping01:~# grep "time=[0-9][0-9]" outputfile
Mon May 7 23:05:12 201864 bytes from 192.168.0.200: icmp_seq=16703 ttl=64
time=123 ms
Mon May 7 23:26:28 201864 bytes from 192.168.0.200: icmp_seq=17979 ttl=64
time=123 ms
Mon May 7 23:45:08 201864 bytes from 192.168.0.200: icmp_seq=19099 ttl=64
time=127 ms
Tue May 8 00:19:21 201864 bytes from 192.168.0.200: icmp_seq=21152 ttl=64
time=132 ms
Tue May 8 00:20:16 201864 bytes from 192.168.0.200: icmp_seq=21207 ttl=64
time=131 ms
Tue May 8 00:20:58 201864 bytes from 192.168.0.200: icmp_seq=21249 ttl=64
time=131 ms
Tue May 8 00:24:36 201864 bytes from 192.168.0.200: icmp_seq=21467 ttl=64
time=131 ms
Tue May 8 03:17:53 201864 bytes from 192.168.0.200: icmp_seq=31864 ttl=64
time=102 ms
Tue May 8 04:51:35 201864 bytes from 192.168.0.200: icmp_seq=37486 ttl=64
time=134 ms
Tue May 8 04:51:39 201864 bytes from 192.168.0.200: icmp_seq=37490 ttl=64
time=134 ms
Tue May 8 04:55:31 201864 bytes from 192.168.0.200: icmp_seq=37722 ttl=64
time=138 ms
Tue May 8 04:59:23 201864 bytes from 192.168.0.200: icmp_seq=37954 ttl=64
time=138 ms
Tue May 8 05:15:05 201864 bytes from 192.168.0.200: icmp_seq=38896 ttl=64
time=76.4 ms
Tue May 8 05:19:05 201864 bytes from 192.168.0.200: icmp_seq=39136 ttl=64
time=146 ms
Tue May 8 07:07:40 201864 bytes from 192.168.0.200: icmp_seq=45651 ttl=64
time=10.7 ms
Tue May 8 07:12:42 201864 bytes from 192.168.0.200: icmp_seq=45953 ttl=64
time=30.7 ms
Tue May 8 07:17:40 201864 bytes from 192.168.0.200: icmp_seq=46251 ttl=64
time=166 ms
Tue May 8 08:03:33 201864 bytes from 192.168.0.200: icmp_seq=49004 ttl=64
time=14.6 ms
Tue May 8 08:13:34 201864 bytes from 192.168.0.200: icmp_seq=49605 ttl=64
time=174 ms
Tue May 8 08:13:36 201864 bytes from 192.168.0.200: icmp_seq=49607 ttl=64
time=174 ms
Tue May 8 08:42:38 201864 bytes from 192.168.0.200: icmp_seq=51349 ttl=64
time=84.0 ms
Tue May 8 08:51:50 201864 bytes from 192.168.0.200: icmp_seq=51901 ttl=64
time=92.0 ms
Tue May 8 09:34:06 201864 bytes from 192.168.0.200: icmp_seq=54437 ttl=64
time=283 ms
VF used by these two guest from same port(eth0) of intel X710 Ethernet
controller. By contrast, I selected another two VFs and set them to
separate network namespace, this issue did not exits, all the latency
litter than 0.2ms.
Advised by other guys, I disabled the BIOS C-State, set cpu power to
"performance", add kernel parameter "idle=poll, pcie_aspm=off", but it
makes no sense.
I think it may be not the hardware issue, but may relate to KVM
hypervisor or guest kernel. As a result, I reported here, any advice and
suggestions will be greatly appreciated.
The test enviroment as below:
CPU model: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
HostOS: debian_9.3, kernel_version: 4.9.0-3
QEMU 2.8.1
libvirt 2.4.0
NIC: intel X710
GuestOS: debian_9.1
xml:
<domain type='kvm'>
<name>ping_test01</name>
<uuid>98132486-7f49-45bf-a6d7-6ceaf71ff4be</uuid>
<maxMemory slots='64' unit='KiB'>134217728</maxMemory>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<blkiotune>
<device>
<path>/dev/sdb</path>
<read_iops_sec>1000</read_iops_sec>
<write_iops_sec>1000</write_iops_sec>
<read_bytes_sec>104857600</read_bytes_sec>
<write_bytes_sec>104857600</write_bytes_sec>
</device>
</blkiotune>
<vcpu placement='static' cpuset='5-31,33-55' current='2'>64</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='9'/>
<vcpupin vcpu='1' cpuset='37'/>
</cputune>
<resource>
<partition>/machine</partition>
</resource>
<sysinfo type='smbios'>
<system>
<entry name='manufacturer'>OpenStack Foundation</entry>
<entry name='product'>OpenStack Nova</entry>
<entry name='version'>2013.2.2</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='pc-i440fx-2.8'>hvm</type>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='custom' match='exact'>
<model fallback='forbid'>Broadwell</model>
<vendor>Intel</vendor>
<feature policy='require' name='vme'/>
<feature policy='require' name='ds'/>
<feature policy='require' name='acpi'/>
<feature policy='require' name='ss'/>
<feature policy='require' name='ht'/>
<feature policy='require' name='tm'/>
<feature policy='require' name='pbe'/>
<feature policy='require' name='dtes64'/>
<feature policy='require' name='ds_cpl'/>
<feature policy='require' name='vmx'/>
<feature policy='require' name='smx'/>
<feature policy='require' name='est'/>
<feature policy='require' name='tm2'/>
<feature policy='require' name='xtpr'/>
<feature policy='require' name='pdcm'/>
<feature policy='require' name='dca'/>
<feature policy='require' name='osxsave'/>
<feature policy='require' name='f16c'/>
<feature policy='require' name='rdrand'/>
<feature policy='require' name='arat'/>
<feature policy='require' name='tsc_adjust'/>
<feature policy='require' name='xsaveopt'/>
<feature policy='require' name='pdpe1gb'/>
<feature policy='require' name='abm'/>
<numa>
<cell id='0' cpus='0-63' memory='4194304' unit='KiB'/>
</numa>
</cpu>
<clock offset='utc'>
<timer name='pit' tickpolicy='delay'/>
<timer name='rtc' tickpolicy='catchup'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source file='/images/debian_8.6_pub_dhcp_sr_i4_0428155812.dsk'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
</disk>
<controller type='usb' index='0' model='piix3-uhci'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
</controller>
<interface type='hostdev' managed='yes'>
<mac address='52:54:00:0f:3b:b3'/>
<driver name='vfio'/>
<source>
<address type='pci' domain='0x0000' bus='0x06' slot='0x03'
function='0x4'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x0'/>
</interface>
<serial type='file'>
<source path='/tmp/vm_console.log'/>
<target port='0'/>
</serial>
<serial type='pty'>
<target port='1'/>
</serial>
<console type='file'>
<source path='/tmp/zyj_console.log'/>
<target type='serial' port='0'/>
</console>
<input type='tablet' bus='usb'>
<address type='usb' bus='0' port='1'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes' listen='10.194.8.178'
keymap='en-us'>
<listen type='address' address='10.194.8.178'/>
</graphics>
<video>
<model type='cirrus' vram='16384' heads='1' primary='yes'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
</video>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0'/>
</memballoon>
</devices>
<seclabel type='none' model='none'/>
<seclabel type='dynamic' model='dac' relabel='yes'/>
</domain>
Thanks!
Zhu Yijun
^ permalink raw reply [flat|nested] 10+ messages in thread
* [Qemu-devel] ping latency issue on passthrough guest
@ 2018-05-11 7:20 ` Zhu Yijun
0 siblings, 0 replies; 10+ messages in thread
From: Zhu Yijun @ 2018-05-11 7:20 UTC (permalink / raw)
To: kvm, Alex Williamson, Paolo Bonzini, qemu-devel
Hi all,
I booted two sr-iov guests using KVM-VFIO and pinged each other with
no-load one night. I found that most of the latency was little than 0.1ms,
but several icmp_seq greater than 10ms, even up to 1000ms;
root@test-ping01:~# grep "time=[0-9][0-9]" outputfile
Mon May 7 23:05:12 201864 bytes from 192.168.0.200: icmp_seq=16703 ttl=64
time=123 ms
Mon May 7 23:26:28 201864 bytes from 192.168.0.200: icmp_seq=17979 ttl=64
time=123 ms
Mon May 7 23:45:08 201864 bytes from 192.168.0.200: icmp_seq=19099 ttl=64
time=127 ms
Tue May 8 00:19:21 201864 bytes from 192.168.0.200: icmp_seq=21152 ttl=64
time=132 ms
Tue May 8 00:20:16 201864 bytes from 192.168.0.200: icmp_seq=21207 ttl=64
time=131 ms
Tue May 8 00:20:58 201864 bytes from 192.168.0.200: icmp_seq=21249 ttl=64
time=131 ms
Tue May 8 00:24:36 201864 bytes from 192.168.0.200: icmp_seq=21467 ttl=64
time=131 ms
Tue May 8 03:17:53 201864 bytes from 192.168.0.200: icmp_seq=31864 ttl=64
time=102 ms
Tue May 8 04:51:35 201864 bytes from 192.168.0.200: icmp_seq=37486 ttl=64
time=134 ms
Tue May 8 04:51:39 201864 bytes from 192.168.0.200: icmp_seq=37490 ttl=64
time=134 ms
Tue May 8 04:55:31 201864 bytes from 192.168.0.200: icmp_seq=37722 ttl=64
time=138 ms
Tue May 8 04:59:23 201864 bytes from 192.168.0.200: icmp_seq=37954 ttl=64
time=138 ms
Tue May 8 05:15:05 201864 bytes from 192.168.0.200: icmp_seq=38896 ttl=64
time=76.4 ms
Tue May 8 05:19:05 201864 bytes from 192.168.0.200: icmp_seq=39136 ttl=64
time=146 ms
Tue May 8 07:07:40 201864 bytes from 192.168.0.200: icmp_seq=45651 ttl=64
time=10.7 ms
Tue May 8 07:12:42 201864 bytes from 192.168.0.200: icmp_seq=45953 ttl=64
time=30.7 ms
Tue May 8 07:17:40 201864 bytes from 192.168.0.200: icmp_seq=46251 ttl=64
time=166 ms
Tue May 8 08:03:33 201864 bytes from 192.168.0.200: icmp_seq=49004 ttl=64
time=14.6 ms
Tue May 8 08:13:34 201864 bytes from 192.168.0.200: icmp_seq=49605 ttl=64
time=174 ms
Tue May 8 08:13:36 201864 bytes from 192.168.0.200: icmp_seq=49607 ttl=64
time=174 ms
Tue May 8 08:42:38 201864 bytes from 192.168.0.200: icmp_seq=51349 ttl=64
time=84.0 ms
Tue May 8 08:51:50 201864 bytes from 192.168.0.200: icmp_seq=51901 ttl=64
time=92.0 ms
Tue May 8 09:34:06 201864 bytes from 192.168.0.200: icmp_seq=54437 ttl=64
time=283 ms
VF used by these two guest from same port(eth0) of intel X710 Ethernet
controller. By contrast, I selected another two VFs and set them to
separate network namespace, this issue did not exits, all the latency
litter than 0.2ms.
Advised by other guys, I disabled the BIOS C-State, set cpu power to
"performance", add kernel parameter "idle=poll, pcie_aspm=off", but it
makes no sense.
I think it may be not the hardware issue, but may relate to KVM
hypervisor or guest kernel. As a result, I reported here, any advice and
suggestions will be greatly appreciated.
The test enviroment as below:
CPU model: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
HostOS: debian_9.3, kernel_version: 4.9.0-3
QEMU 2.8.1
libvirt 2.4.0
NIC: intel X710
GuestOS: debian_9.1
xml:
<domain type='kvm'>
<name>ping_test01</name>
<uuid>98132486-7f49-45bf-a6d7-6ceaf71ff4be</uuid>
<maxMemory slots='64' unit='KiB'>134217728</maxMemory>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<blkiotune>
<device>
<path>/dev/sdb</path>
<read_iops_sec>1000</read_iops_sec>
<write_iops_sec>1000</write_iops_sec>
<read_bytes_sec>104857600</read_bytes_sec>
<write_bytes_sec>104857600</write_bytes_sec>
</device>
</blkiotune>
<vcpu placement='static' cpuset='5-31,33-55' current='2'>64</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='9'/>
<vcpupin vcpu='1' cpuset='37'/>
</cputune>
<resource>
<partition>/machine</partition>
</resource>
<sysinfo type='smbios'>
<system>
<entry name='manufacturer'>OpenStack Foundation</entry>
<entry name='product'>OpenStack Nova</entry>
<entry name='version'>2013.2.2</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='pc-i440fx-2.8'>hvm</type>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='custom' match='exact'>
<model fallback='forbid'>Broadwell</model>
<vendor>Intel</vendor>
<feature policy='require' name='vme'/>
<feature policy='require' name='ds'/>
<feature policy='require' name='acpi'/>
<feature policy='require' name='ss'/>
<feature policy='require' name='ht'/>
<feature policy='require' name='tm'/>
<feature policy='require' name='pbe'/>
<feature policy='require' name='dtes64'/>
<feature policy='require' name='ds_cpl'/>
<feature policy='require' name='vmx'/>
<feature policy='require' name='smx'/>
<feature policy='require' name='est'/>
<feature policy='require' name='tm2'/>
<feature policy='require' name='xtpr'/>
<feature policy='require' name='pdcm'/>
<feature policy='require' name='dca'/>
<feature policy='require' name='osxsave'/>
<feature policy='require' name='f16c'/>
<feature policy='require' name='rdrand'/>
<feature policy='require' name='arat'/>
<feature policy='require' name='tsc_adjust'/>
<feature policy='require' name='xsaveopt'/>
<feature policy='require' name='pdpe1gb'/>
<feature policy='require' name='abm'/>
<numa>
<cell id='0' cpus='0-63' memory='4194304' unit='KiB'/>
</numa>
</cpu>
<clock offset='utc'>
<timer name='pit' tickpolicy='delay'/>
<timer name='rtc' tickpolicy='catchup'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source file='/images/debian_8.6_pub_dhcp_sr_i4_0428155812.dsk'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
</disk>
<controller type='usb' index='0' model='piix3-uhci'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
</controller>
<interface type='hostdev' managed='yes'>
<mac address='52:54:00:0f:3b:b3'/>
<driver name='vfio'/>
<source>
<address type='pci' domain='0x0000' bus='0x06' slot='0x03'
function='0x4'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x0'/>
</interface>
<serial type='file'>
<source path='/tmp/vm_console.log'/>
<target port='0'/>
</serial>
<serial type='pty'>
<target port='1'/>
</serial>
<console type='file'>
<source path='/tmp/zyj_console.log'/>
<target type='serial' port='0'/>
</console>
<input type='tablet' bus='usb'>
<address type='usb' bus='0' port='1'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes' listen='10.194.8.178'
keymap='en-us'>
<listen type='address' address='10.194.8.178'/>
</graphics>
<video>
<model type='cirrus' vram='16384' heads='1' primary='yes'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
</video>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0'/>
</memballoon>
</devices>
<seclabel type='none' model='none'/>
<seclabel type='dynamic' model='dac' relabel='yes'/>
</domain>
Thanks!
Zhu Yijun
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: ping latency issue on passthrough guest
2018-05-11 7:20 ` [Qemu-devel] " Zhu Yijun
@ 2018-05-11 14:26 ` Alex Williamson
-1 siblings, 0 replies; 10+ messages in thread
From: Alex Williamson @ 2018-05-11 14:26 UTC (permalink / raw)
To: Zhu Yijun; +Cc: Paolo Bonzini, qemu-devel, kvm
On Fri, 11 May 2018 15:20:05 +0800
Zhu Yijun <lovemrd@gmail.com> wrote:
> Hi all,
>
> I booted two sr-iov guests using KVM-VFIO and pinged each other with
> no-load one night. I found that most of the latency was little than 0.1ms,
> but several icmp_seq greater than 10ms, even up to 1000ms;
>
> root@test-ping01:~# grep "time=[0-9][0-9]" outputfile
> Mon May 7 23:05:12 201864 bytes from 192.168.0.200: icmp_seq=16703 ttl=64
> time=123 ms
...
>
> VF used by these two guest from same port(eth0) of intel X710 Ethernet
> controller. By contrast, I selected another two VFs and set them to
> separate network namespace, this issue did not exits, all the latency
> litter than 0.2ms.
I think you've proved by this experiment that it's not likely to be a
device assignment issue. Device assignment knows nothing about the
device above the base PCI level operation, so if changing the
networking configuration resolves the issue, the problem is perhaps
more likely in how the PF, or maybe the guest itself, is routing VF
packets. Thanks,
Alex
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Qemu-devel] ping latency issue on passthrough guest
@ 2018-05-11 14:26 ` Alex Williamson
0 siblings, 0 replies; 10+ messages in thread
From: Alex Williamson @ 2018-05-11 14:26 UTC (permalink / raw)
To: Zhu Yijun; +Cc: kvm, Paolo Bonzini, qemu-devel
On Fri, 11 May 2018 15:20:05 +0800
Zhu Yijun <lovemrd@gmail.com> wrote:
> Hi all,
>
> I booted two sr-iov guests using KVM-VFIO and pinged each other with
> no-load one night. I found that most of the latency was little than 0.1ms,
> but several icmp_seq greater than 10ms, even up to 1000ms;
>
> root@test-ping01:~# grep "time=[0-9][0-9]" outputfile
> Mon May 7 23:05:12 201864 bytes from 192.168.0.200: icmp_seq=16703 ttl=64
> time=123 ms
...
>
> VF used by these two guest from same port(eth0) of intel X710 Ethernet
> controller. By contrast, I selected another two VFs and set them to
> separate network namespace, this issue did not exits, all the latency
> litter than 0.2ms.
I think you've proved by this experiment that it's not likely to be a
device assignment issue. Device assignment knows nothing about the
device above the base PCI level operation, so if changing the
networking configuration resolves the issue, the problem is perhaps
more likely in how the PF, or maybe the guest itself, is routing VF
packets. Thanks,
Alex
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: ping latency issue on passthrough guest
2018-05-11 7:20 ` [Qemu-devel] " Zhu Yijun
@ 2018-05-14 6:13 ` Alexey Kardashevskiy
-1 siblings, 0 replies; 10+ messages in thread
From: Alexey Kardashevskiy @ 2018-05-14 6:13 UTC (permalink / raw)
To: Zhu Yijun, kvm, Alex Williamson, Paolo Bonzini, qemu-devel
On 11/5/18 5:20 pm, Zhu Yijun wrote:
> Hi all,
>
> I booted two sr-iov guests using KVM-VFIO and pinged each other with
> no-load one night. I found that most of the latency was little than 0.1ms,
> but several icmp_seq greater than 10ms, even up to 1000ms;
>
[...]
>
> VF used by these two guest from same port(eth0) of intel X710 Ethernet
> controller. By contrast, I selected another two VFs and set them to
> separate network namespace, this issue did not exits, all the latency
> litter than 0.2ms.
>
> Advised by other guys, I disabled the BIOS C-State, set cpu power to
> "performance", add kernel parameter "idle=poll, pcie_aspm=off", but it
> makes no sense.
>
> I think it may be not the hardware issue, but may relate to KVM
> hypervisor or guest kernel. As a result, I reported here, any advice and
> suggestions will be greatly appreciated.
I'd suggest trying the exact same VFs on the host and make sure:
1) trafic is actually going through the physical device and not routed by
the host kernel (because the ip addresses on VFs are on the same network,
etc, ifconfig's statistic should tell). I use a script like this to do the
setup:
===
# !/bin/bash
function cfg_run() {
echo cfg and run: $*
IP0=$1
IP1=$2
ETH0=$3
ETH1=$4
BASENUM=$5
ip addr add $IP0/24 dev $ETH0
ip addr add $IP1/24 dev $ETH1
ip link set $ETH0 up
ip link set $ETH1 up
ip link set $ETH0 mtu 9000
ip link set $ETH1 mtu 9000
ip rule add priority $(expr $BASENUM + 10 ) from all to $IP0 lookup
$(expr $BASENUM + 10 )
ip rule add priority $(expr $BASENUM + 20 ) from all to $IP1 lookup
$(expr $BASENUM + 20 )
ip route add table $(expr $BASENUM + 10 ) default dev $ETH0 scope link
ip route add table $(expr $BASENUM + 20 ) default dev $ETH1 scope link
ip rule add priority $BASENUM from all iif $ETH0 lookup local
ip rule add priority $BASENUM from all iif $ETH1 lookup local
ip rule add priority $(expr 20000 + $BASENUM ) from all lookup local
ip rule del priority 0 from all lookup local
ip addr
}
cfg_run 172.12.0.1 172.12.1.1 enp1s0f0 enp1s0f1 100
===
2) distro for the host host and the guest.
Also you can kill the irqbalance daemon (I doubt it will make such a
difference though).
Is the ethernet cable needed for the test? How do you know if the outer
network is not the problem?
What is your guest doing other than pinging? Any chance it might start
swapping sometime? 4GB of RAM is not extraordinary huge amount for a
gui-enabled linux system.
--
Alexey
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Qemu-devel] ping latency issue on passthrough guest
@ 2018-05-14 6:13 ` Alexey Kardashevskiy
0 siblings, 0 replies; 10+ messages in thread
From: Alexey Kardashevskiy @ 2018-05-14 6:13 UTC (permalink / raw)
To: Zhu Yijun, kvm, Alex Williamson, Paolo Bonzini, qemu-devel
On 11/5/18 5:20 pm, Zhu Yijun wrote:
> Hi all,
>
> I booted two sr-iov guests using KVM-VFIO and pinged each other with
> no-load one night. I found that most of the latency was little than 0.1ms,
> but several icmp_seq greater than 10ms, even up to 1000ms;
>
[...]
>
> VF used by these two guest from same port(eth0) of intel X710 Ethernet
> controller. By contrast, I selected another two VFs and set them to
> separate network namespace, this issue did not exits, all the latency
> litter than 0.2ms.
>
> Advised by other guys, I disabled the BIOS C-State, set cpu power to
> "performance", add kernel parameter "idle=poll, pcie_aspm=off", but it
> makes no sense.
>
> I think it may be not the hardware issue, but may relate to KVM
> hypervisor or guest kernel. As a result, I reported here, any advice and
> suggestions will be greatly appreciated.
I'd suggest trying the exact same VFs on the host and make sure:
1) trafic is actually going through the physical device and not routed by
the host kernel (because the ip addresses on VFs are on the same network,
etc, ifconfig's statistic should tell). I use a script like this to do the
setup:
===
# !/bin/bash
function cfg_run() {
echo cfg and run: $*
IP0=$1
IP1=$2
ETH0=$3
ETH1=$4
BASENUM=$5
ip addr add $IP0/24 dev $ETH0
ip addr add $IP1/24 dev $ETH1
ip link set $ETH0 up
ip link set $ETH1 up
ip link set $ETH0 mtu 9000
ip link set $ETH1 mtu 9000
ip rule add priority $(expr $BASENUM + 10 ) from all to $IP0 lookup
$(expr $BASENUM + 10 )
ip rule add priority $(expr $BASENUM + 20 ) from all to $IP1 lookup
$(expr $BASENUM + 20 )
ip route add table $(expr $BASENUM + 10 ) default dev $ETH0 scope link
ip route add table $(expr $BASENUM + 20 ) default dev $ETH1 scope link
ip rule add priority $BASENUM from all iif $ETH0 lookup local
ip rule add priority $BASENUM from all iif $ETH1 lookup local
ip rule add priority $(expr 20000 + $BASENUM ) from all lookup local
ip rule del priority 0 from all lookup local
ip addr
}
cfg_run 172.12.0.1 172.12.1.1 enp1s0f0 enp1s0f1 100
===
2) distro for the host host and the guest.
Also you can kill the irqbalance daemon (I doubt it will make such a
difference though).
Is the ethernet cable needed for the test? How do you know if the outer
network is not the problem?
What is your guest doing other than pinging? Any chance it might start
swapping sometime? 4GB of RAM is not extraordinary huge amount for a
gui-enabled linux system.
--
Alexey
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: ping latency issue on passthrough guest
2018-05-11 14:26 ` [Qemu-devel] " Alex Williamson
@ 2018-05-14 9:00 ` Zhu Yijun
-1 siblings, 0 replies; 10+ messages in thread
From: Zhu Yijun @ 2018-05-14 9:00 UTC (permalink / raw)
To: Alex Williamson; +Cc: Paolo Bonzini, qemu-devel, kvm
Hi Alex,
Thanks for your reply, I will try to assign PF to VM and test.
2018-05-11 22:26 GMT+08:00 Alex Williamson <alex.williamson@redhat.com>:
> On Fri, 11 May 2018 15:20:05 +0800
> Zhu Yijun <lovemrd@gmail.com> wrote:
>
>> Hi all,
>>
>> I booted two sr-iov guests using KVM-VFIO and pinged each other with
>> no-load one night. I found that most of the latency was little than 0.1ms,
>> but several icmp_seq greater than 10ms, even up to 1000ms;
>>
>> root@test-ping01:~# grep "time=[0-9][0-9]" outputfile
>> Mon May 7 23:05:12 201864 bytes from 192.168.0.200: icmp_seq=16703 ttl=64
>> time=123 ms
> ...
>>
>> VF used by these two guest from same port(eth0) of intel X710 Ethernet
>> controller. By contrast, I selected another two VFs and set them to
>> separate network namespace, this issue did not exits, all the latency
>> litter than 0.2ms.
>
> I think you've proved by this experiment that it's not likely to be a
> device assignment issue. Device assignment knows nothing about the
> device above the base PCI level operation, so if changing the
> networking configuration resolves the issue, the problem is perhaps
> more likely in how the PF, or maybe the guest itself, is routing VF
> packets. Thanks,
>
> Alex
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Qemu-devel] ping latency issue on passthrough guest
@ 2018-05-14 9:00 ` Zhu Yijun
0 siblings, 0 replies; 10+ messages in thread
From: Zhu Yijun @ 2018-05-14 9:00 UTC (permalink / raw)
To: Alex Williamson; +Cc: kvm, Paolo Bonzini, qemu-devel
Hi Alex,
Thanks for your reply, I will try to assign PF to VM and test.
2018-05-11 22:26 GMT+08:00 Alex Williamson <alex.williamson@redhat.com>:
> On Fri, 11 May 2018 15:20:05 +0800
> Zhu Yijun <lovemrd@gmail.com> wrote:
>
>> Hi all,
>>
>> I booted two sr-iov guests using KVM-VFIO and pinged each other with
>> no-load one night. I found that most of the latency was little than 0.1ms,
>> but several icmp_seq greater than 10ms, even up to 1000ms;
>>
>> root@test-ping01:~# grep "time=[0-9][0-9]" outputfile
>> Mon May 7 23:05:12 201864 bytes from 192.168.0.200: icmp_seq=16703 ttl=64
>> time=123 ms
> ...
>>
>> VF used by these two guest from same port(eth0) of intel X710 Ethernet
>> controller. By contrast, I selected another two VFs and set them to
>> separate network namespace, this issue did not exits, all the latency
>> litter than 0.2ms.
>
> I think you've proved by this experiment that it's not likely to be a
> device assignment issue. Device assignment knows nothing about the
> device above the base PCI level operation, so if changing the
> networking configuration resolves the issue, the problem is perhaps
> more likely in how the PF, or maybe the guest itself, is routing VF
> packets. Thanks,
>
> Alex
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: ping latency issue on passthrough guest
2018-05-14 6:13 ` [Qemu-devel] " Alexey Kardashevskiy
@ 2018-05-14 9:02 ` Zhu Yijun
-1 siblings, 0 replies; 10+ messages in thread
From: Zhu Yijun @ 2018-05-14 9:02 UTC (permalink / raw)
To: Alexey Kardashevskiy; +Cc: Paolo Bonzini, Alex Williamson, qemu-devel, kvm
Hi Alexey, thanks for your advise.
2018-05-14 14:13 GMT+08:00 Alexey Kardashevskiy <aik@ozlabs.ru>:
> On 11/5/18 5:20 pm, Zhu Yijun wrote:
>> Hi all,
>>
>> I booted two sr-iov guests using KVM-VFIO and pinged each other with
>> no-load one night. I found that most of the latency was little than 0.1ms,
>> but several icmp_seq greater than 10ms, even up to 1000ms;
>>
> [...]
>>
>> VF used by these two guest from same port(eth0) of intel X710 Ethernet
>> controller. By contrast, I selected another two VFs and set them to
>> separate network namespace, this issue did not exits, all the latency
>> litter than 0.2ms.
>>
>> Advised by other guys, I disabled the BIOS C-State, set cpu power to
>> "performance", add kernel parameter "idle=poll, pcie_aspm=off", but it
>> makes no sense.
>>
>> I think it may be not the hardware issue, but may relate to KVM
>> hypervisor or guest kernel. As a result, I reported here, any advice and
>> suggestions will be greatly appreciated.
>
>
> I'd suggest trying the exact same VFs on the host and make sure:
>
> 1) trafic is actually going through the physical device and not routed by
> the host kernel (because the ip addresses on VFs are on the same network,
> etc, ifconfig's statistic should tell). I use a script like this to do the
> setup:
>
> ===
> # !/bin/bash
>
> function cfg_run() {
> echo cfg and run: $*
> IP0=$1
> IP1=$2
> ETH0=$3
> ETH1=$4
> BASENUM=$5
> ip addr add $IP0/24 dev $ETH0
> ip addr add $IP1/24 dev $ETH1
> ip link set $ETH0 up
> ip link set $ETH1 up
> ip link set $ETH0 mtu 9000
> ip link set $ETH1 mtu 9000
> ip rule add priority $(expr $BASENUM + 10 ) from all to $IP0 lookup
> $(expr $BASENUM + 10 )
> ip rule add priority $(expr $BASENUM + 20 ) from all to $IP1 lookup
> $(expr $BASENUM + 20 )
> ip route add table $(expr $BASENUM + 10 ) default dev $ETH0 scope link
> ip route add table $(expr $BASENUM + 20 ) default dev $ETH1 scope link
> ip rule add priority $BASENUM from all iif $ETH0 lookup local
> ip rule add priority $BASENUM from all iif $ETH1 lookup local
> ip rule add priority $(expr 20000 + $BASENUM ) from all lookup local
> ip rule del priority 0 from all lookup local
> ip addr
> }
> cfg_run 172.12.0.1 172.12.1.1 enp1s0f0 enp1s0f1 100
> ===
>
Thanks, will try.
> 2) distro for the host host and the guest.
>
> Also you can kill the irqbalance daemon (I doubt it will make such a
> difference though).
>
host & guest did not load irqbalance daemon.
> Is the ethernet cable needed for the test? How do you know if the outer
> network is not the problem?
>
1) I test ping in VMs, 2)running ping test in another two VFs which in
separate network namespace, 3) run ping test on PF between two hosts
at the same time;
1) have latency issue, but others did not have. So it may not the
outer network problem.
> What is your guest doing other than pinging? Any chance it might start
> swapping sometime? 4GB of RAM is not extraordinary huge amount for a
> gui-enabled linux system.
>
ping is the only thing I do once VM starts, but I will check the daemon process.
VM with 100Gb RAM also has this issue, so it not relate to swap or mem size.
>
>
> --
> Alexey
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Qemu-devel] ping latency issue on passthrough guest
@ 2018-05-14 9:02 ` Zhu Yijun
0 siblings, 0 replies; 10+ messages in thread
From: Zhu Yijun @ 2018-05-14 9:02 UTC (permalink / raw)
To: Alexey Kardashevskiy; +Cc: kvm, Alex Williamson, Paolo Bonzini, qemu-devel
Hi Alexey, thanks for your advise.
2018-05-14 14:13 GMT+08:00 Alexey Kardashevskiy <aik@ozlabs.ru>:
> On 11/5/18 5:20 pm, Zhu Yijun wrote:
>> Hi all,
>>
>> I booted two sr-iov guests using KVM-VFIO and pinged each other with
>> no-load one night. I found that most of the latency was little than 0.1ms,
>> but several icmp_seq greater than 10ms, even up to 1000ms;
>>
> [...]
>>
>> VF used by these two guest from same port(eth0) of intel X710 Ethernet
>> controller. By contrast, I selected another two VFs and set them to
>> separate network namespace, this issue did not exits, all the latency
>> litter than 0.2ms.
>>
>> Advised by other guys, I disabled the BIOS C-State, set cpu power to
>> "performance", add kernel parameter "idle=poll, pcie_aspm=off", but it
>> makes no sense.
>>
>> I think it may be not the hardware issue, but may relate to KVM
>> hypervisor or guest kernel. As a result, I reported here, any advice and
>> suggestions will be greatly appreciated.
>
>
> I'd suggest trying the exact same VFs on the host and make sure:
>
> 1) trafic is actually going through the physical device and not routed by
> the host kernel (because the ip addresses on VFs are on the same network,
> etc, ifconfig's statistic should tell). I use a script like this to do the
> setup:
>
> ===
> # !/bin/bash
>
> function cfg_run() {
> echo cfg and run: $*
> IP0=$1
> IP1=$2
> ETH0=$3
> ETH1=$4
> BASENUM=$5
> ip addr add $IP0/24 dev $ETH0
> ip addr add $IP1/24 dev $ETH1
> ip link set $ETH0 up
> ip link set $ETH1 up
> ip link set $ETH0 mtu 9000
> ip link set $ETH1 mtu 9000
> ip rule add priority $(expr $BASENUM + 10 ) from all to $IP0 lookup
> $(expr $BASENUM + 10 )
> ip rule add priority $(expr $BASENUM + 20 ) from all to $IP1 lookup
> $(expr $BASENUM + 20 )
> ip route add table $(expr $BASENUM + 10 ) default dev $ETH0 scope link
> ip route add table $(expr $BASENUM + 20 ) default dev $ETH1 scope link
> ip rule add priority $BASENUM from all iif $ETH0 lookup local
> ip rule add priority $BASENUM from all iif $ETH1 lookup local
> ip rule add priority $(expr 20000 + $BASENUM ) from all lookup local
> ip rule del priority 0 from all lookup local
> ip addr
> }
> cfg_run 172.12.0.1 172.12.1.1 enp1s0f0 enp1s0f1 100
> ===
>
Thanks, will try.
> 2) distro for the host host and the guest.
>
> Also you can kill the irqbalance daemon (I doubt it will make such a
> difference though).
>
host & guest did not load irqbalance daemon.
> Is the ethernet cable needed for the test? How do you know if the outer
> network is not the problem?
>
1) I test ping in VMs, 2)running ping test in another two VFs which in
separate network namespace, 3) run ping test on PF between two hosts
at the same time;
1) have latency issue, but others did not have. So it may not the
outer network problem.
> What is your guest doing other than pinging? Any chance it might start
> swapping sometime? 4GB of RAM is not extraordinary huge amount for a
> gui-enabled linux system.
>
ping is the only thing I do once VM starts, but I will check the daemon process.
VM with 100Gb RAM also has this issue, so it not relate to swap or mem size.
>
>
> --
> Alexey
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2018-05-14 9:02 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-05-11 7:20 ping latency issue on passthrough guest Zhu Yijun
2018-05-11 7:20 ` [Qemu-devel] " Zhu Yijun
2018-05-11 14:26 ` Alex Williamson
2018-05-11 14:26 ` [Qemu-devel] " Alex Williamson
2018-05-14 9:00 ` Zhu Yijun
2018-05-14 9:00 ` [Qemu-devel] " Zhu Yijun
2018-05-14 6:13 ` Alexey Kardashevskiy
2018-05-14 6:13 ` [Qemu-devel] " Alexey Kardashevskiy
2018-05-14 9:02 ` Zhu Yijun
2018-05-14 9:02 ` [Qemu-devel] " Zhu Yijun
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.