* [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host
[not found] ` <AM5PR0501MB2499CB4F2D01BCDA47C0ACB6C95A0@AM5PR0501MB2499.eurprd05.prod.outlook.com>
@ 2018-07-12 7:33 ` Elijah Shakkour
2018-07-26 14:52 ` Stefan Hajnoczi
0 siblings, 1 reply; 17+ messages in thread
From: Elijah Shakkour @ 2018-07-12 7:33 UTC (permalink / raw)
To: qemu-devel; +Cc: Tal Attaly
Hey,
Our team is adding a NIC functional emulation to QEMU.
One of the features we are adding to this NIC is SRIOV.
Here is the error message I get when checking SRIOV support of our emulated NIC on Win2016 server (the hyper-v VM).
"
SR-IOV cannot be used on this system as the PCI Express hardware does not support Access Control Services (ACS) at any root port. Contact your system vendor for further information.
"
Could you please advise about what could be the issue here?
BTW: I use same configuration (VM XML file attached) when running linux VM (RH7.2) image (instead of Win Hyper-V) over the same host and SRIOV is working for me there.
Here the XML file I use to define the VM (our emulated NIC is added at the end of XML):
"
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>nst105</name>
<uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
<memory unit='KiB'>8388608</memory>
<currentMemory unit='KiB'>8388608</currentMemory>
<vcpu placement='static'>8</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
</os>
<features>
<acpi/>
<apic/>
<hyperv>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>
</hyperv>
</features>
<cpu mode='custom' match='exact' check='none'>
<model fallback='forbid'>SandyBridge</model>
<feature policy='require' name='hypervisor'/>
<feature policy='require' name='vmx'/>
</cpu>
<clock offset='localtime'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
<timer name='hypervclock' present='yes'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/images/nst105.qcow2'/>
<target dev='sda' bus='sata'/>
<boot order='2'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<controller type='usb' index='0' model='ich9-ehci1'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
</controller>
<controller type='sata' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pcie-root'/>
<controller type='pci' index='1' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='1' port='0x10'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='2' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='2' port='0x11'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='3' port='0x12'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
</controller>
<controller type='pci' index='4' model='dmi-to-pci-bridge'>
<model name='i82801b11-bridge'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
</controller>
<controller type='pci' index='5' model='pci-bridge'>
<model name='pci-bridge'/>
<target chassisNr='5'/>
<address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='00:50:56:1b:dd:6f'/>
<source bridge='br0'/>
<model type='virtio'/>
<boot order='1'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<input type='tablet' bus='usb'>
<address type='usb' bus='0' port='1'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1' keymap='en-us'>
<listen type='address' address='127.0.0.1'/>
</graphics>
<video>
<model type='vga' vram='16384' heads='1' primary='yes'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
</video>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
</memballoon>
</devices>
<seclabel type='none' model='none'/>
<seclabel type='dynamic' model='dac' relabel='yes'/>
<qemu:commandline>
<qemu:arg value='-device'/>
<qemu:arg value='pcie-root-port,pref64-reserve=500M,slot=0,id=pcie_port.1'/>
<qemu:arg value='-netdev'/>
<qemu:arg value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
<qemu:arg value='-device'/>
<qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
</qemu:commandline>
</domain>
"
______________________
General info:
Host OS: RH7.0 (Kernel: 4.14.13)
QEMU version: 2.11
libvirt version: 3.2.0
Running the following on the host shows that both nested and IOMMU are
enabled:
~]#: cat /sys/module/kvm_intel/parameters/nested
Y
~]# dmesg | grep -e DMAR -e IOMMU
[ 0.000000] DMAR: IOMMU enabled
Thanks,
Elijah
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host
2018-07-12 7:33 ` [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host Elijah Shakkour
@ 2018-07-26 14:52 ` Stefan Hajnoczi
2018-07-26 15:51 ` Marcel Apfelbaum
0 siblings, 1 reply; 17+ messages in thread
From: Stefan Hajnoczi @ 2018-07-26 14:52 UTC (permalink / raw)
To: Elijah Shakkour
Cc: qemu-devel, Tal Attaly, Michael S. Tsirkin, Marcel Apfelbaum
[-- Attachment #1: Type: text/plain, Size: 6871 bytes --]
On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
> Hey,
>
> Our team is adding a NIC functional emulation to QEMU.
> One of the features we are adding to this NIC is SRIOV.
>
> Here is the error message I get when checking SRIOV support of our emulated NIC on Win2016 server (the hyper-v VM).
> "
> SR-IOV cannot be used on this system as the PCI Express hardware does not support Access Control Services (ACS) at any root port. Contact your system vendor for further information.
> "
I'm not sure what the status of emulated SR-IOV is so I have CCed
Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.
>
> Could you please advise about what could be the issue here?
>
> BTW: I use same configuration (VM XML file attached) when running linux VM (RH7.2) image (instead of Win Hyper-V) over the same host and SRIOV is working for me there.
>
> Here the XML file I use to define the VM (our emulated NIC is added at the end of XML):
> "
> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
> <name>nst105</name>
> <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
> <memory unit='KiB'>8388608</memory>
> <currentMemory unit='KiB'>8388608</currentMemory>
> <vcpu placement='static'>8</vcpu>
> <resource>
> <partition>/machine</partition>
> </resource>
> <os>
> <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
> </os>
> <features>
> <acpi/>
> <apic/>
> <hyperv>
> <relaxed state='on'/>
> <vapic state='on'/>
> <spinlocks state='on' retries='8191'/>
> </hyperv>
> </features>
> <cpu mode='custom' match='exact' check='none'>
> <model fallback='forbid'>SandyBridge</model>
> <feature policy='require' name='hypervisor'/>
> <feature policy='require' name='vmx'/>
> </cpu>
> <clock offset='localtime'>
> <timer name='rtc' tickpolicy='catchup'/>
> <timer name='pit' tickpolicy='delay'/>
> <timer name='hpet' present='no'/>
> <timer name='hypervclock' present='yes'/>
> </clock>
> <on_poweroff>destroy</on_poweroff>
> <on_reboot>restart</on_reboot>
> <on_crash>destroy</on_crash>
> <pm>
> <suspend-to-mem enabled='no'/>
> <suspend-to-disk enabled='no'/>
> </pm>
> <devices>
> <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
> <disk type='file' device='disk'>
> <driver name='qemu' type='qcow2'/>
> <source file='/images/nst105.qcow2'/>
> <target dev='sda' bus='sata'/>
> <boot order='2'/>
> <address type='drive' controller='0' bus='0' target='0' unit='0'/>
> </disk>
> <controller type='usb' index='0' model='ich9-ehci1'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
> </controller>
> <controller type='usb' index='0' model='ich9-uhci1'>
> <master startport='0'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
> </controller>
> <controller type='usb' index='0' model='ich9-uhci2'>
> <master startport='2'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
> </controller>
> <controller type='usb' index='0' model='ich9-uhci3'>
> <master startport='4'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
> </controller>
> <controller type='sata' index='0'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
> </controller>
> <controller type='pci' index='0' model='pcie-root'/>
> <controller type='pci' index='1' model='pcie-root-port'>
> <model name='pcie-root-port'/>
> <target chassis='1' port='0x10'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
> </controller>
> <controller type='pci' index='2' model='pcie-root-port'>
> <model name='pcie-root-port'/>
> <target chassis='2' port='0x11'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
> </controller>
> <controller type='pci' index='3' model='pcie-root-port'>
> <model name='pcie-root-port'/>
> <target chassis='3' port='0x12'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
> </controller>
> <controller type='pci' index='4' model='dmi-to-pci-bridge'>
> <model name='i82801b11-bridge'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
> </controller>
> <controller type='pci' index='5' model='pci-bridge'>
> <model name='pci-bridge'/>
> <target chassisNr='5'/>
> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
> </controller>
> <interface type='bridge'>
> <mac address='00:50:56:1b:dd:6f'/>
> <source bridge='br0'/>
> <model type='virtio'/>
> <boot order='1'/>
> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
> </interface>
> <serial type='pty'>
> <target port='0'/>
> </serial>
> <console type='pty'>
> <target type='serial' port='0'/>
> </console>
> <input type='tablet' bus='usb'>
> <address type='usb' bus='0' port='1'/>
> </input>
> <input type='mouse' bus='ps2'/>
> <input type='keyboard' bus='ps2'/>
> <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1' keymap='en-us'>
> <listen type='address' address='127.0.0.1'/>
> </graphics>
> <video>
> <model type='vga' vram='16384' heads='1' primary='yes'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
> </video>
> <memballoon model='virtio'>
> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
> </memballoon>
> </devices>
> <seclabel type='none' model='none'/>
> <seclabel type='dynamic' model='dac' relabel='yes'/>
> <qemu:commandline>
> <qemu:arg value='-device'/>
> <qemu:arg value='pcie-root-port,pref64-reserve=500M,slot=0,id=pcie_port.1'/>
> <qemu:arg value='-netdev'/>
> <qemu:arg value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
> <qemu:arg value='-device'/>
> <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
> </qemu:commandline>
> </domain>
> "
> ______________________
> General info:
> Host OS: RH7.0 (Kernel: 4.14.13)
> QEMU version: 2.11
> libvirt version: 3.2.0
> Running the following on the host shows that both nested and IOMMU are
> enabled:
> ~]#: cat /sys/module/kvm_intel/parameters/nested
> Y
> ~]# dmesg | grep -e DMAR -e IOMMU
> [ 0.000000] DMAR: IOMMU enabled
>
> Thanks,
> Elijah
>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host
2018-07-26 14:52 ` Stefan Hajnoczi
@ 2018-07-26 15:51 ` Marcel Apfelbaum
2018-07-26 16:14 ` Michael S. Tsirkin
2018-07-26 16:38 ` Knut Omang
0 siblings, 2 replies; 17+ messages in thread
From: Marcel Apfelbaum @ 2018-07-26 15:51 UTC (permalink / raw)
To: Stefan Hajnoczi, Elijah Shakkour, Knut Omang
Cc: qemu-devel, Tal Attaly, Michael S. Tsirkin
Hi
On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
> On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
>> Hey,
>>
>> Our team is adding a NIC functional emulation to QEMU.
>> One of the features we are adding to this NIC is SRIOV.
>>
>> Here is the error message I get when checking SRIOV support of our emulated NIC on Win2016 server (the hyper-v VM).
>> "
>> SR-IOV cannot be used on this system as the PCI Express hardware does not support Access Control Services (ACS) at any root port.
QEMU's emulated PCI Express Root Ports do not support ACS yet, however I
am not sure ACS is a prerequisite
for SR-IOV. We would need ARI support for allowing more than 8VFs, but
QEMU doesn't support that either (yet).
Knut Omag has some working patches, he successfully implemented SR-IOV
with QEMU, see:
https://github.com/knuto/qemu/tree/sriov_patches_v7
The code was not merged since we need at least a device with SR-IOV
support to justify the addition.
>> Contact your system vendor for further information.
>> "
> I'm not sure what the status of emulated SR-IOV is so I have CCed
> Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.
Thanks,
Marcel
>> Could you please advise about what could be the issue here?
>>
>> BTW: I use same configuration (VM XML file attached) when running linux VM (RH7.2) image (instead of Win Hyper-V) over the same host and SRIOV is working for me there.
>>
>> Here the XML file I use to define the VM (our emulated NIC is added at the end of XML):
>> "
>> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
>> <name>nst105</name>
>> <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
>> <memory unit='KiB'>8388608</memory>
>> <currentMemory unit='KiB'>8388608</currentMemory>
>> <vcpu placement='static'>8</vcpu>
>> <resource>
>> <partition>/machine</partition>
>> </resource>
>> <os>
>> <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
>> </os>
>> <features>
>> <acpi/>
>> <apic/>
>> <hyperv>
>> <relaxed state='on'/>
>> <vapic state='on'/>
>> <spinlocks state='on' retries='8191'/>
>> </hyperv>
>> </features>
>> <cpu mode='custom' match='exact' check='none'>
>> <model fallback='forbid'>SandyBridge</model>
>> <feature policy='require' name='hypervisor'/>
>> <feature policy='require' name='vmx'/>
>> </cpu>
>> <clock offset='localtime'>
>> <timer name='rtc' tickpolicy='catchup'/>
>> <timer name='pit' tickpolicy='delay'/>
>> <timer name='hpet' present='no'/>
>> <timer name='hypervclock' present='yes'/>
>> </clock>
>> <on_poweroff>destroy</on_poweroff>
>> <on_reboot>restart</on_reboot>
>> <on_crash>destroy</on_crash>
>> <pm>
>> <suspend-to-mem enabled='no'/>
>> <suspend-to-disk enabled='no'/>
>> </pm>
>> <devices>
>> <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
>> <disk type='file' device='disk'>
>> <driver name='qemu' type='qcow2'/>
>> <source file='/images/nst105.qcow2'/>
>> <target dev='sda' bus='sata'/>
>> <boot order='2'/>
>> <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>> </disk>
>> <controller type='usb' index='0' model='ich9-ehci1'>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
>> </controller>
>> <controller type='usb' index='0' model='ich9-uhci1'>
>> <master startport='0'/>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
>> </controller>
>> <controller type='usb' index='0' model='ich9-uhci2'>
>> <master startport='2'/>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
>> </controller>
>> <controller type='usb' index='0' model='ich9-uhci3'>
>> <master startport='4'/>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
>> </controller>
>> <controller type='sata' index='0'>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
>> </controller>
>> <controller type='pci' index='0' model='pcie-root'/>
>> <controller type='pci' index='1' model='pcie-root-port'>
>> <model name='pcie-root-port'/>
>> <target chassis='1' port='0x10'/>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
>> </controller>
>> <controller type='pci' index='2' model='pcie-root-port'>
>> <model name='pcie-root-port'/>
>> <target chassis='2' port='0x11'/>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
>> </controller>
>> <controller type='pci' index='3' model='pcie-root-port'>
>> <model name='pcie-root-port'/>
>> <target chassis='3' port='0x12'/>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
>> </controller>
>> <controller type='pci' index='4' model='dmi-to-pci-bridge'>
>> <model name='i82801b11-bridge'/>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
>> </controller>
>> <controller type='pci' index='5' model='pci-bridge'>
>> <model name='pci-bridge'/>
>> <target chassisNr='5'/>
>> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
>> </controller>
>> <interface type='bridge'>
>> <mac address='00:50:56:1b:dd:6f'/>
>> <source bridge='br0'/>
>> <model type='virtio'/>
>> <boot order='1'/>
>> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
>> </interface>
>> <serial type='pty'>
>> <target port='0'/>
>> </serial>
>> <console type='pty'>
>> <target type='serial' port='0'/>
>> </console>
>> <input type='tablet' bus='usb'>
>> <address type='usb' bus='0' port='1'/>
>> </input>
>> <input type='mouse' bus='ps2'/>
>> <input type='keyboard' bus='ps2'/>
>> <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1' keymap='en-us'>
>> <listen type='address' address='127.0.0.1'/>
>> </graphics>
>> <video>
>> <model type='vga' vram='16384' heads='1' primary='yes'/>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
>> </video>
>> <memballoon model='virtio'>
>> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
>> </memballoon>
>> </devices>
>> <seclabel type='none' model='none'/>
>> <seclabel type='dynamic' model='dac' relabel='yes'/>
>> <qemu:commandline>
>> <qemu:arg value='-device'/>
>> <qemu:arg value='pcie-root-port,pref64-reserve=500M,slot=0,id=pcie_port.1'/>
>> <qemu:arg value='-netdev'/>
>> <qemu:arg value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
>> <qemu:arg value='-device'/>
>> <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
>> </qemu:commandline>
>> </domain>
>> "
>> ______________________
>> General info:
>> Host OS: RH7.0 (Kernel: 4.14.13)
>> QEMU version: 2.11
>> libvirt version: 3.2.0
>> Running the following on the host shows that both nested and IOMMU are
>> enabled:
>> ~]#: cat /sys/module/kvm_intel/parameters/nested
>> Y
>> ~]# dmesg | grep -e DMAR -e IOMMU
>> [ 0.000000] DMAR: IOMMU enabled
>>
>> Thanks,
>> Elijah
>>
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host
2018-07-26 15:51 ` Marcel Apfelbaum
@ 2018-07-26 16:14 ` Michael S. Tsirkin
2018-07-26 16:34 ` Marcel Apfelbaum
2018-07-26 16:42 ` Alex Williamson
2018-07-26 16:38 ` Knut Omang
1 sibling, 2 replies; 17+ messages in thread
From: Michael S. Tsirkin @ 2018-07-26 16:14 UTC (permalink / raw)
To: Marcel Apfelbaum
Cc: Stefan Hajnoczi, Elijah Shakkour, Knut Omang, qemu-devel, Tal Attaly
On Thu, Jul 26, 2018 at 06:51:13PM +0300, Marcel Apfelbaum wrote:
> Hi
>
> On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
> > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
> > > Hey,
> > >
> > > Our team is adding a NIC functional emulation to QEMU.
> > > One of the features we are adding to this NIC is SRIOV.
> > >
> > > Here is the error message I get when checking SRIOV support of our emulated NIC on Win2016 server (the hyper-v VM).
> > > "
> > > SR-IOV cannot be used on this system as the PCI Express hardware does not support Access Control Services (ACS) at any root port.
>
> QEMU's emulated PCI Express Root Ports do not support ACS yet, however I am
> not sure ACS is a prerequisite
> for SR-IOV.
Looks like windows blocks dev assignment in nested VMs without it.
Thinking about it, doesn't vfio do the same by default? I think vfio has
a flag to override this though.
> We would need ARI support for allowing more than 8VFs, but QEMU
> doesn't support that either (yet).
>
> Knut Omag has some working patches, he successfully implemented SR-IOV with
> QEMU, see:
> https://github.com/knuto/qemu/tree/sriov_patches_v7
>
> The code was not merged since we need at least a device with SR-IOV support
> to justify the addition.
>
> > > Contact your system vendor for further information.
> > > "
> > I'm not sure what the status of emulated SR-IOV is so I have CCed
> > Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.
>
> Thanks,
> Marcel
>
> > > Could you please advise about what could be the issue here?
> > >
> > > BTW: I use same configuration (VM XML file attached) when running linux VM (RH7.2) image (instead of Win Hyper-V) over the same host and SRIOV is working for me there.
> > >
> > > Here the XML file I use to define the VM (our emulated NIC is added at the end of XML):
> > > "
> > > <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
> > > <name>nst105</name>
> > > <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
> > > <memory unit='KiB'>8388608</memory>
> > > <currentMemory unit='KiB'>8388608</currentMemory>
> > > <vcpu placement='static'>8</vcpu>
> > > <resource>
> > > <partition>/machine</partition>
> > > </resource>
> > > <os>
> > > <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
> > > </os>
> > > <features>
> > > <acpi/>
> > > <apic/>
> > > <hyperv>
> > > <relaxed state='on'/>
> > > <vapic state='on'/>
> > > <spinlocks state='on' retries='8191'/>
> > > </hyperv>
> > > </features>
> > > <cpu mode='custom' match='exact' check='none'>
> > > <model fallback='forbid'>SandyBridge</model>
> > > <feature policy='require' name='hypervisor'/>
> > > <feature policy='require' name='vmx'/>
> > > </cpu>
> > > <clock offset='localtime'>
> > > <timer name='rtc' tickpolicy='catchup'/>
> > > <timer name='pit' tickpolicy='delay'/>
> > > <timer name='hpet' present='no'/>
> > > <timer name='hypervclock' present='yes'/>
> > > </clock>
> > > <on_poweroff>destroy</on_poweroff>
> > > <on_reboot>restart</on_reboot>
> > > <on_crash>destroy</on_crash>
> > > <pm>
> > > <suspend-to-mem enabled='no'/>
> > > <suspend-to-disk enabled='no'/>
> > > </pm>
> > > <devices>
> > > <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
> > > <disk type='file' device='disk'>
> > > <driver name='qemu' type='qcow2'/>
> > > <source file='/images/nst105.qcow2'/>
> > > <target dev='sda' bus='sata'/>
> > > <boot order='2'/>
> > > <address type='drive' controller='0' bus='0' target='0' unit='0'/>
> > > </disk>
> > > <controller type='usb' index='0' model='ich9-ehci1'>
> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
> > > </controller>
> > > <controller type='usb' index='0' model='ich9-uhci1'>
> > > <master startport='0'/>
> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
> > > </controller>
> > > <controller type='usb' index='0' model='ich9-uhci2'>
> > > <master startport='2'/>
> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
> > > </controller>
> > > <controller type='usb' index='0' model='ich9-uhci3'>
> > > <master startport='4'/>
> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
> > > </controller>
> > > <controller type='sata' index='0'>
> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
> > > </controller>
> > > <controller type='pci' index='0' model='pcie-root'/>
> > > <controller type='pci' index='1' model='pcie-root-port'>
> > > <model name='pcie-root-port'/>
> > > <target chassis='1' port='0x10'/>
> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
> > > </controller>
> > > <controller type='pci' index='2' model='pcie-root-port'>
> > > <model name='pcie-root-port'/>
> > > <target chassis='2' port='0x11'/>
> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
> > > </controller>
> > > <controller type='pci' index='3' model='pcie-root-port'>
> > > <model name='pcie-root-port'/>
> > > <target chassis='3' port='0x12'/>
> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
> > > </controller>
> > > <controller type='pci' index='4' model='dmi-to-pci-bridge'>
> > > <model name='i82801b11-bridge'/>
> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
> > > </controller>
> > > <controller type='pci' index='5' model='pci-bridge'>
> > > <model name='pci-bridge'/>
> > > <target chassisNr='5'/>
> > > <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
> > > </controller>
> > > <interface type='bridge'>
> > > <mac address='00:50:56:1b:dd:6f'/>
> > > <source bridge='br0'/>
> > > <model type='virtio'/>
> > > <boot order='1'/>
> > > <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
> > > </interface>
> > > <serial type='pty'>
> > > <target port='0'/>
> > > </serial>
> > > <console type='pty'>
> > > <target type='serial' port='0'/>
> > > </console>
> > > <input type='tablet' bus='usb'>
> > > <address type='usb' bus='0' port='1'/>
> > > </input>
> > > <input type='mouse' bus='ps2'/>
> > > <input type='keyboard' bus='ps2'/>
> > > <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1' keymap='en-us'>
> > > <listen type='address' address='127.0.0.1'/>
> > > </graphics>
> > > <video>
> > > <model type='vga' vram='16384' heads='1' primary='yes'/>
> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
> > > </video>
> > > <memballoon model='virtio'>
> > > <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
> > > </memballoon>
> > > </devices>
> > > <seclabel type='none' model='none'/>
> > > <seclabel type='dynamic' model='dac' relabel='yes'/>
> > > <qemu:commandline>
> > > <qemu:arg value='-device'/>
> > > <qemu:arg value='pcie-root-port,pref64-reserve=500M,slot=0,id=pcie_port.1'/>
> > > <qemu:arg value='-netdev'/>
> > > <qemu:arg value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
> > > <qemu:arg value='-device'/>
> > > <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
> > > </qemu:commandline>
> > > </domain>
> > > "
> > > ______________________
> > > General info:
> > > Host OS: RH7.0 (Kernel: 4.14.13)
> > > QEMU version: 2.11
> > > libvirt version: 3.2.0
> > > Running the following on the host shows that both nested and IOMMU are
> > > enabled:
> > > ~]#: cat /sys/module/kvm_intel/parameters/nested
> > > Y
> > > ~]# dmesg | grep -e DMAR -e IOMMU
> > > [ 0.000000] DMAR: IOMMU enabled
> > >
> > > Thanks,
> > > Elijah
> > >
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host
2018-07-26 16:14 ` Michael S. Tsirkin
@ 2018-07-26 16:34 ` Marcel Apfelbaum
2018-07-26 16:38 ` Michael S. Tsirkin
2018-07-26 16:42 ` Alex Williamson
1 sibling, 1 reply; 17+ messages in thread
From: Marcel Apfelbaum @ 2018-07-26 16:34 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Stefan Hajnoczi, Elijah Shakkour, Knut Omang, qemu-devel, Tal Attaly
On 07/26/2018 07:14 PM, Michael S. Tsirkin wrote:
> On Thu, Jul 26, 2018 at 06:51:13PM +0300, Marcel Apfelbaum wrote:
>> Hi
>>
>> On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
>>> On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
>>>> Hey,
>>>>
>>>> Our team is adding a NIC functional emulation to QEMU.
>>>> One of the features we are adding to this NIC is SRIOV.
>>>>
>>>> Here is the error message I get when checking SRIOV support of our emulated NIC on Win2016 server (the hyper-v VM).
>>>> "
>>>> SR-IOV cannot be used on this system as the PCI Express hardware does not support Access Control Services (ACS) at any root port.
>> QEMU's emulated PCI Express Root Ports do not support ACS yet, however I am
>> not sure ACS is a prerequisite
>> for SR-IOV.
> Looks like windows blocks dev assignment in nested VMs without it.
> Thinking about it, doesn't vfio do the same by default?
vfio requires vIOMMU even for nested, as far as I know. They do have a
"no-iommu"
option, but I don't know how it can help in this case.
Elijah, what about implementing ACS for QEMU's generic PCIe Root Ports?
Maybe is not too complicated and it seems like a handy feature to have
anyway.
Patches would be welcomed :)
Thanks,
Marcel
> I think vfio has
> a flag to override this though.
>
>> We would need ARI support for allowing more than 8VFs, but QEMU
>> doesn't support that either (yet).
>>
>> Knut Omag has some working patches, he successfully implemented SR-IOV with
>> QEMU, see:
>> https://github.com/knuto/qemu/tree/sriov_patches_v7
>>
>> The code was not merged since we need at least a device with SR-IOV support
>> to justify the addition.
>>
>>>> Contact your system vendor for further information.
>>>> "
>>> I'm not sure what the status of emulated SR-IOV is so I have CCed
>>> Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.
>> Thanks,
>> Marcel
>>
>>>> Could you please advise about what could be the issue here?
>>>>
>>>> BTW: I use same configuration (VM XML file attached) when running linux VM (RH7.2) image (instead of Win Hyper-V) over the same host and SRIOV is working for me there.
>>>>
>>>> Here the XML file I use to define the VM (our emulated NIC is added at the end of XML):
>>>> "
>>>> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
>>>> <name>nst105</name>
>>>> <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
>>>> <memory unit='KiB'>8388608</memory>
>>>> <currentMemory unit='KiB'>8388608</currentMemory>
>>>> <vcpu placement='static'>8</vcpu>
>>>> <resource>
>>>> <partition>/machine</partition>
>>>> </resource>
>>>> <os>
>>>> <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
>>>> </os>
>>>> <features>
>>>> <acpi/>
>>>> <apic/>
>>>> <hyperv>
>>>> <relaxed state='on'/>
>>>> <vapic state='on'/>
>>>> <spinlocks state='on' retries='8191'/>
>>>> </hyperv>
>>>> </features>
>>>> <cpu mode='custom' match='exact' check='none'>
>>>> <model fallback='forbid'>SandyBridge</model>
>>>> <feature policy='require' name='hypervisor'/>
>>>> <feature policy='require' name='vmx'/>
>>>> </cpu>
>>>> <clock offset='localtime'>
>>>> <timer name='rtc' tickpolicy='catchup'/>
>>>> <timer name='pit' tickpolicy='delay'/>
>>>> <timer name='hpet' present='no'/>
>>>> <timer name='hypervclock' present='yes'/>
>>>> </clock>
>>>> <on_poweroff>destroy</on_poweroff>
>>>> <on_reboot>restart</on_reboot>
>>>> <on_crash>destroy</on_crash>
>>>> <pm>
>>>> <suspend-to-mem enabled='no'/>
>>>> <suspend-to-disk enabled='no'/>
>>>> </pm>
>>>> <devices>
>>>> <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
>>>> <disk type='file' device='disk'>
>>>> <driver name='qemu' type='qcow2'/>
>>>> <source file='/images/nst105.qcow2'/>
>>>> <target dev='sda' bus='sata'/>
>>>> <boot order='2'/>
>>>> <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>>> </disk>
>>>> <controller type='usb' index='0' model='ich9-ehci1'>
>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
>>>> </controller>
>>>> <controller type='usb' index='0' model='ich9-uhci1'>
>>>> <master startport='0'/>
>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
>>>> </controller>
>>>> <controller type='usb' index='0' model='ich9-uhci2'>
>>>> <master startport='2'/>
>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
>>>> </controller>
>>>> <controller type='usb' index='0' model='ich9-uhci3'>
>>>> <master startport='4'/>
>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
>>>> </controller>
>>>> <controller type='sata' index='0'>
>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
>>>> </controller>
>>>> <controller type='pci' index='0' model='pcie-root'/>
>>>> <controller type='pci' index='1' model='pcie-root-port'>
>>>> <model name='pcie-root-port'/>
>>>> <target chassis='1' port='0x10'/>
>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
>>>> </controller>
>>>> <controller type='pci' index='2' model='pcie-root-port'>
>>>> <model name='pcie-root-port'/>
>>>> <target chassis='2' port='0x11'/>
>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
>>>> </controller>
>>>> <controller type='pci' index='3' model='pcie-root-port'>
>>>> <model name='pcie-root-port'/>
>>>> <target chassis='3' port='0x12'/>
>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
>>>> </controller>
>>>> <controller type='pci' index='4' model='dmi-to-pci-bridge'>
>>>> <model name='i82801b11-bridge'/>
>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
>>>> </controller>
>>>> <controller type='pci' index='5' model='pci-bridge'>
>>>> <model name='pci-bridge'/>
>>>> <target chassisNr='5'/>
>>>> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
>>>> </controller>
>>>> <interface type='bridge'>
>>>> <mac address='00:50:56:1b:dd:6f'/>
>>>> <source bridge='br0'/>
>>>> <model type='virtio'/>
>>>> <boot order='1'/>
>>>> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
>>>> </interface>
>>>> <serial type='pty'>
>>>> <target port='0'/>
>>>> </serial>
>>>> <console type='pty'>
>>>> <target type='serial' port='0'/>
>>>> </console>
>>>> <input type='tablet' bus='usb'>
>>>> <address type='usb' bus='0' port='1'/>
>>>> </input>
>>>> <input type='mouse' bus='ps2'/>
>>>> <input type='keyboard' bus='ps2'/>
>>>> <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1' keymap='en-us'>
>>>> <listen type='address' address='127.0.0.1'/>
>>>> </graphics>
>>>> <video>
>>>> <model type='vga' vram='16384' heads='1' primary='yes'/>
>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
>>>> </video>
>>>> <memballoon model='virtio'>
>>>> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
>>>> </memballoon>
>>>> </devices>
>>>> <seclabel type='none' model='none'/>
>>>> <seclabel type='dynamic' model='dac' relabel='yes'/>
>>>> <qemu:commandline>
>>>> <qemu:arg value='-device'/>
>>>> <qemu:arg value='pcie-root-port,pref64-reserve=500M,slot=0,id=pcie_port.1'/>
>>>> <qemu:arg value='-netdev'/>
>>>> <qemu:arg value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
>>>> <qemu:arg value='-device'/>
>>>> <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
>>>> </qemu:commandline>
>>>> </domain>
>>>> "
>>>> ______________________
>>>> General info:
>>>> Host OS: RH7.0 (Kernel: 4.14.13)
>>>> QEMU version: 2.11
>>>> libvirt version: 3.2.0
>>>> Running the following on the host shows that both nested and IOMMU are
>>>> enabled:
>>>> ~]#: cat /sys/module/kvm_intel/parameters/nested
>>>> Y
>>>> ~]# dmesg | grep -e DMAR -e IOMMU
>>>> [ 0.000000] DMAR: IOMMU enabled
>>>>
>>>> Thanks,
>>>> Elijah
>>>>
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host
2018-07-26 16:34 ` Marcel Apfelbaum
@ 2018-07-26 16:38 ` Michael S. Tsirkin
0 siblings, 0 replies; 17+ messages in thread
From: Michael S. Tsirkin @ 2018-07-26 16:38 UTC (permalink / raw)
To: Marcel Apfelbaum
Cc: Stefan Hajnoczi, Elijah Shakkour, Knut Omang, qemu-devel, Tal Attaly
On Thu, Jul 26, 2018 at 07:34:41PM +0300, Marcel Apfelbaum wrote:
>
>
> On 07/26/2018 07:14 PM, Michael S. Tsirkin wrote:
> > On Thu, Jul 26, 2018 at 06:51:13PM +0300, Marcel Apfelbaum wrote:
> > > Hi
> > >
> > > On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
> > > > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
> > > > > Hey,
> > > > >
> > > > > Our team is adding a NIC functional emulation to QEMU.
> > > > > One of the features we are adding to this NIC is SRIOV.
> > > > >
> > > > > Here is the error message I get when checking SRIOV support of our emulated NIC on Win2016 server (the hyper-v VM).
> > > > > "
> > > > > SR-IOV cannot be used on this system as the PCI Express hardware does not support Access Control Services (ACS) at any root port.
> > > QEMU's emulated PCI Express Root Ports do not support ACS yet, however I am
> > > not sure ACS is a prerequisite
> > > for SR-IOV.
> > Looks like windows blocks dev assignment in nested VMs without it.
> > Thinking about it, doesn't vfio do the same by default?
> vfio requires vIOMMU even for nested, as far as I know.
Absolutely.
> They do have a
> "no-iommu"
> option, but I don't know how it can help in this case.
>
> Elijah, what about implementing ACS for QEMU's generic PCIe Root Ports?
> Maybe is not too complicated and it seems like a handy feature to have
> anyway.
> Patches would be welcomed :)
>
> Thanks,
> Marcel
>
> > I think vfio has
> > a flag to override this though.
> >
> > > We would need ARI support for allowing more than 8VFs, but QEMU
> > > doesn't support that either (yet).
> > >
> > > Knut Omag has some working patches, he successfully implemented SR-IOV with
> > > QEMU, see:
> > > https://github.com/knuto/qemu/tree/sriov_patches_v7
> > >
> > > The code was not merged since we need at least a device with SR-IOV support
> > > to justify the addition.
> > >
> > > > > Contact your system vendor for further information.
> > > > > "
> > > > I'm not sure what the status of emulated SR-IOV is so I have CCed
> > > > Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.
> > > Thanks,
> > > Marcel
> > >
> > > > > Could you please advise about what could be the issue here?
> > > > >
> > > > > BTW: I use same configuration (VM XML file attached) when running linux VM (RH7.2) image (instead of Win Hyper-V) over the same host and SRIOV is working for me there.
> > > > >
> > > > > Here the XML file I use to define the VM (our emulated NIC is added at the end of XML):
> > > > > "
> > > > > <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
> > > > > <name>nst105</name>
> > > > > <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
> > > > > <memory unit='KiB'>8388608</memory>
> > > > > <currentMemory unit='KiB'>8388608</currentMemory>
> > > > > <vcpu placement='static'>8</vcpu>
> > > > > <resource>
> > > > > <partition>/machine</partition>
> > > > > </resource>
> > > > > <os>
> > > > > <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
> > > > > </os>
> > > > > <features>
> > > > > <acpi/>
> > > > > <apic/>
> > > > > <hyperv>
> > > > > <relaxed state='on'/>
> > > > > <vapic state='on'/>
> > > > > <spinlocks state='on' retries='8191'/>
> > > > > </hyperv>
> > > > > </features>
> > > > > <cpu mode='custom' match='exact' check='none'>
> > > > > <model fallback='forbid'>SandyBridge</model>
> > > > > <feature policy='require' name='hypervisor'/>
> > > > > <feature policy='require' name='vmx'/>
> > > > > </cpu>
> > > > > <clock offset='localtime'>
> > > > > <timer name='rtc' tickpolicy='catchup'/>
> > > > > <timer name='pit' tickpolicy='delay'/>
> > > > > <timer name='hpet' present='no'/>
> > > > > <timer name='hypervclock' present='yes'/>
> > > > > </clock>
> > > > > <on_poweroff>destroy</on_poweroff>
> > > > > <on_reboot>restart</on_reboot>
> > > > > <on_crash>destroy</on_crash>
> > > > > <pm>
> > > > > <suspend-to-mem enabled='no'/>
> > > > > <suspend-to-disk enabled='no'/>
> > > > > </pm>
> > > > > <devices>
> > > > > <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
> > > > > <disk type='file' device='disk'>
> > > > > <driver name='qemu' type='qcow2'/>
> > > > > <source file='/images/nst105.qcow2'/>
> > > > > <target dev='sda' bus='sata'/>
> > > > > <boot order='2'/>
> > > > > <address type='drive' controller='0' bus='0' target='0' unit='0'/>
> > > > > </disk>
> > > > > <controller type='usb' index='0' model='ich9-ehci1'>
> > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
> > > > > </controller>
> > > > > <controller type='usb' index='0' model='ich9-uhci1'>
> > > > > <master startport='0'/>
> > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
> > > > > </controller>
> > > > > <controller type='usb' index='0' model='ich9-uhci2'>
> > > > > <master startport='2'/>
> > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
> > > > > </controller>
> > > > > <controller type='usb' index='0' model='ich9-uhci3'>
> > > > > <master startport='4'/>
> > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
> > > > > </controller>
> > > > > <controller type='sata' index='0'>
> > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
> > > > > </controller>
> > > > > <controller type='pci' index='0' model='pcie-root'/>
> > > > > <controller type='pci' index='1' model='pcie-root-port'>
> > > > > <model name='pcie-root-port'/>
> > > > > <target chassis='1' port='0x10'/>
> > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
> > > > > </controller>
> > > > > <controller type='pci' index='2' model='pcie-root-port'>
> > > > > <model name='pcie-root-port'/>
> > > > > <target chassis='2' port='0x11'/>
> > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
> > > > > </controller>
> > > > > <controller type='pci' index='3' model='pcie-root-port'>
> > > > > <model name='pcie-root-port'/>
> > > > > <target chassis='3' port='0x12'/>
> > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
> > > > > </controller>
> > > > > <controller type='pci' index='4' model='dmi-to-pci-bridge'>
> > > > > <model name='i82801b11-bridge'/>
> > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
> > > > > </controller>
> > > > > <controller type='pci' index='5' model='pci-bridge'>
> > > > > <model name='pci-bridge'/>
> > > > > <target chassisNr='5'/>
> > > > > <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
> > > > > </controller>
> > > > > <interface type='bridge'>
> > > > > <mac address='00:50:56:1b:dd:6f'/>
> > > > > <source bridge='br0'/>
> > > > > <model type='virtio'/>
> > > > > <boot order='1'/>
> > > > > <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
> > > > > </interface>
> > > > > <serial type='pty'>
> > > > > <target port='0'/>
> > > > > </serial>
> > > > > <console type='pty'>
> > > > > <target type='serial' port='0'/>
> > > > > </console>
> > > > > <input type='tablet' bus='usb'>
> > > > > <address type='usb' bus='0' port='1'/>
> > > > > </input>
> > > > > <input type='mouse' bus='ps2'/>
> > > > > <input type='keyboard' bus='ps2'/>
> > > > > <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1' keymap='en-us'>
> > > > > <listen type='address' address='127.0.0.1'/>
> > > > > </graphics>
> > > > > <video>
> > > > > <model type='vga' vram='16384' heads='1' primary='yes'/>
> > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
> > > > > </video>
> > > > > <memballoon model='virtio'>
> > > > > <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
> > > > > </memballoon>
> > > > > </devices>
> > > > > <seclabel type='none' model='none'/>
> > > > > <seclabel type='dynamic' model='dac' relabel='yes'/>
> > > > > <qemu:commandline>
> > > > > <qemu:arg value='-device'/>
> > > > > <qemu:arg value='pcie-root-port,pref64-reserve=500M,slot=0,id=pcie_port.1'/>
> > > > > <qemu:arg value='-netdev'/>
> > > > > <qemu:arg value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
> > > > > <qemu:arg value='-device'/>
> > > > > <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
> > > > > </qemu:commandline>
> > > > > </domain>
> > > > > "
> > > > > ______________________
> > > > > General info:
> > > > > Host OS: RH7.0 (Kernel: 4.14.13)
> > > > > QEMU version: 2.11
> > > > > libvirt version: 3.2.0
> > > > > Running the following on the host shows that both nested and IOMMU are
> > > > > enabled:
> > > > > ~]#: cat /sys/module/kvm_intel/parameters/nested
> > > > > Y
> > > > > ~]# dmesg | grep -e DMAR -e IOMMU
> > > > > [ 0.000000] DMAR: IOMMU enabled
> > > > >
> > > > > Thanks,
> > > > > Elijah
> > > > >
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host
2018-07-26 15:51 ` Marcel Apfelbaum
2018-07-26 16:14 ` Michael S. Tsirkin
@ 2018-07-26 16:38 ` Knut Omang
2018-07-26 16:41 ` Michael S. Tsirkin
1 sibling, 1 reply; 17+ messages in thread
From: Knut Omang @ 2018-07-26 16:38 UTC (permalink / raw)
To: Marcel Apfelbaum, Stefan Hajnoczi, Elijah Shakkour
Cc: qemu-devel, Tal Attaly, Michael S. Tsirkin
On Thu, 2018-07-26 at 18:51 +0300, Marcel Apfelbaum wrote:
> Hi
>
> On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
> > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
> > > Hey,
> > >
> > > Our team is adding a NIC functional emulation to QEMU.
> > > One of the features we are adding to this NIC is SRIOV.
> > >
> > > Here is the error message I get when checking SRIOV support of our
> > > emulated NIC on Win2016 server (the hyper-v VM).
> > > "
> > > SR-IOV cannot be used on this system as the PCI Express hardware does not
> > > support Access Control Services (ACS) at any root port.
>
> QEMU's emulated PCI Express Root Ports do not support ACS yet, however I
> am not sure ACS is a prerequisite
> for SR-IOV. We would need ARI support for allowing more than 8VFs, but
> QEMU doesn't support that either (yet).
>
> Knut Omag has some working patches, he successfully implemented SR-IOV
> with QEMU, see:
> https://github.com/knuto/qemu/tree/sriov_patches_v7
>
> The code was not merged since we need at least a device with SR-IOV
> support to justify the addition.
FYI, I recently rebased these to latest master but just didn't get to push them
out. I did just now - they are available here:
https://github.com/knuto/qemu/tree/sriov_patches_v8
As far as I know ARI support with my patch set works just fine - I have tested
it with lots of VFs.
One of the patches in the series (pci: Make use of the devfn property when
registering new devices) is necessary to make ARI work as it should with SR/IOV.
For the hardware model I developed the SR/IOV patches for, I also added enough
ACS support in the root port (PCIe capability helper patch + usage in
ioh3420) to make VFIO "happy". I haven't submitted them because they
are "questionable" since they likely do not reflect the actual features of the
ioh3420. I can make those available if interesting.
Thanks,
Knut
> > > Contact your system vendor for further information.
> > > "
> >
> > I'm not sure what the status of emulated SR-IOV is so I have CCed
> > Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.
>
> Thanks,
> Marcel
>
> > > Could you please advise about what could be the issue here?
> > >
> > > BTW: I use same configuration (VM XML file attached) when running linux VM
> > > (RH7.2) image (instead of Win Hyper-V) over the same host and SRIOV is
> > > working for me there.
> > >
> > > Here the XML file I use to define the VM (our emulated NIC is added at the
> > > end of XML):
> > > "
> > > <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'
> > > >;
> > > <name>nst105</name>
> > > <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
> > > <memory unit='KiB'>8388608</memory>
> > > <currentMemory unit='KiB'>8388608</currentMemory>
> > > <vcpu placement='static'>8</vcpu>
> > > <resource>
> > > <partition>/machine</partition>
> > > </resource>
> > > <os>
> > > <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
> > > </os>
> > > <features>
> > > <acpi/>
> > > <apic/>
> > > <hyperv>
> > > <relaxed state='on'/>
> > > <vapic state='on'/>
> > > <spinlocks state='on' retries='8191'/>
> > > </hyperv>
> > > </features>
> > > <cpu mode='custom' match='exact' check='none'>
> > > <model fallback='forbid'>SandyBridge</model>
> > > <feature policy='require' name='hypervisor'/>
> > > <feature policy='require' name='vmx'/>
> > > </cpu>
> > > <clock offset='localtime'>
> > > <timer name='rtc' tickpolicy='catchup'/>
> > > <timer name='pit' tickpolicy='delay'/>
> > > <timer name='hpet' present='no'/>
> > > <timer name='hypervclock' present='yes'/>
> > > </clock>
> > > <on_poweroff>destroy</on_poweroff>
> > > <on_reboot>restart</on_reboot>
> > > <on_crash>destroy</on_crash>
> > > <pm>
> > > <suspend-to-mem enabled='no'/>
> > > <suspend-to-disk enabled='no'/>
> > > </pm>
> > > <devices>
> > > <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
> > > <disk type='file' device='disk'>
> > > <driver name='qemu' type='qcow2'/>
> > > <source file='/images/nst105.qcow2'/>
> > > <target dev='sda' bus='sata'/>
> > > <boot order='2'/>
> > > <address type='drive' controller='0' bus='0' target='0' unit='0'/>
> > > </disk>
> > > <controller type='usb' index='0' model='ich9-ehci1'>
> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > function='0x7'/>
> > > </controller>
> > > <controller type='usb' index='0' model='ich9-uhci1'>
> > > <master startport='0'/>
> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > function='0x0' multifunction='on'/>
> > > </controller>
> > > <controller type='usb' index='0' model='ich9-uhci2'>
> > > <master startport='2'/>
> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > function='0x1'/>
> > > </controller>
> > > <controller type='usb' index='0' model='ich9-uhci3'>
> > > <master startport='4'/>
> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > function='0x2'/>
> > > </controller>
> > > <controller type='sata' index='0'>
> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1f'
> > > function='0x2'/>
> > > </controller>
> > > <controller type='pci' index='0' model='pcie-root'/>
> > > <controller type='pci' index='1' model='pcie-root-port'>
> > > <model name='pcie-root-port'/>
> > > <target chassis='1' port='0x10'/>
> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > function='0x0' multifunction='on'/>
> > > </controller>
> > > <controller type='pci' index='2' model='pcie-root-port'>
> > > <model name='pcie-root-port'/>
> > > <target chassis='2' port='0x11'/>
> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > function='0x1'/>
> > > </controller>
> > > <controller type='pci' index='3' model='pcie-root-port'>
> > > <model name='pcie-root-port'/>
> > > <target chassis='3' port='0x12'/>
> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > function='0x2'/>
> > > </controller>
> > > <controller type='pci' index='4' model='dmi-to-pci-bridge'>
> > > <model name='i82801b11-bridge'/>
> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1e'
> > > function='0x0'/>
> > > </controller>
> > > <controller type='pci' index='5' model='pci-bridge'>
> > > <model name='pci-bridge'/>
> > > <target chassisNr='5'/>
> > > <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> > > function='0x0'/>
> > > </controller>
> > > <interface type='bridge'>
> > > <mac address='00:50:56:1b:dd:6f'/>
> > > <source bridge='br0'/>
> > > <model type='virtio'/>
> > > <boot order='1'/>
> > > <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
> > > function='0x0'/>
> > > </interface>
> > > <serial type='pty'>
> > > <target port='0'/>
> > > </serial>
> > > <console type='pty'>
> > > <target type='serial' port='0'/>
> > > </console>
> > > <input type='tablet' bus='usb'>
> > > <address type='usb' bus='0' port='1'/>
> > > </input>
> > > <input type='mouse' bus='ps2'/>
> > > <input type='keyboard' bus='ps2'/>
> > > <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'
> > > keymap='en-us'>
> > > <listen type='address' address='127.0.0.1'/>
> > > </graphics>
> > > <video>
> > > <model type='vga' vram='16384' heads='1' primary='yes'/>
> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> > > function='0x0'/>
> > > </video>
> > > <memballoon model='virtio'>
> > > <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
> > > function='0x0'/>
> > > </memballoon>
> > > </devices>
> > > <seclabel type='none' model='none'/>
> > > <seclabel type='dynamic' model='dac' relabel='yes'/>
> > > <qemu:commandline>
> > > <qemu:arg value='-device'/>
> > > <qemu:arg value='pcie-root-port,pref64-
> > > reserve=500M,slot=0,id=pcie_port.1'/>
> > > <qemu:arg value='-netdev'/>
> > > <qemu:arg value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
> > > <qemu:arg value='-device'/>
> > > <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
> > > </qemu:commandline>
> > > </domain>
> > > "
> > > ______________________
> > > General info:
> > > Host OS: RH7.0 (Kernel: 4.14.13)
> > > QEMU version: 2.11
> > > libvirt version: 3.2.0
> > > Running the following on the host shows that both nested and IOMMU are
> > > enabled:
> > > ~]#: cat /sys/module/kvm_intel/parameters/nested
> > > Y
> > > ~]# dmesg | grep -e DMAR -e IOMMU
> > > [ 0.000000] DMAR: IOMMU enabled
> > >
> > > Thanks,
> > > Elijah
> > >
>
>
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host
2018-07-26 16:38 ` Knut Omang
@ 2018-07-26 16:41 ` Michael S. Tsirkin
2018-07-26 16:42 ` Knut Omang
0 siblings, 1 reply; 17+ messages in thread
From: Michael S. Tsirkin @ 2018-07-26 16:41 UTC (permalink / raw)
To: Knut Omang
Cc: Marcel Apfelbaum, Stefan Hajnoczi, Elijah Shakkour, qemu-devel,
Tal Attaly
On Thu, Jul 26, 2018 at 06:38:32PM +0200, Knut Omang wrote:
> On Thu, 2018-07-26 at 18:51 +0300, Marcel Apfelbaum wrote:
> > Hi
> >
> > On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
> > > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
> > > > Hey,
> > > >
> > > > Our team is adding a NIC functional emulation to QEMU.
> > > > One of the features we are adding to this NIC is SRIOV.
> > > >
> > > > Here is the error message I get when checking SRIOV support of our
> > > > emulated NIC on Win2016 server (the hyper-v VM).
> > > > "
> > > > SR-IOV cannot be used on this system as the PCI Express hardware does not
> > > > support Access Control Services (ACS) at any root port.
> >
> > QEMU's emulated PCI Express Root Ports do not support ACS yet, however I
> > am not sure ACS is a prerequisite
> > for SR-IOV. We would need ARI support for allowing more than 8VFs, but
> > QEMU doesn't support that either (yet).
> >
> > Knut Omag has some working patches, he successfully implemented SR-IOV
> > with QEMU, see:
> > https://github.com/knuto/qemu/tree/sriov_patches_v7
> >
> > The code was not merged since we need at least a device with SR-IOV
> > support to justify the addition.
>
> FYI, I recently rebased these to latest master but just didn't get to push them
> out. I did just now - they are available here:
>
> https://github.com/knuto/qemu/tree/sriov_patches_v8
>
> As far as I know ARI support with my patch set works just fine - I have tested
> it with lots of VFs.
>
> One of the patches in the series (pci: Make use of the devfn property when
> registering new devices) is necessary to make ARI work as it should with SR/IOV.
>
> For the hardware model I developed the SR/IOV patches for, I also added enough
> ACS support in the root port (PCIe capability helper patch + usage in
> ioh3420) to make VFIO "happy". I haven't submitted them because they
> are "questionable" since they likely do not reflect the actual features of the
> ioh3420.
In that the actual ioh3420 doesn't support ACS?
> I can make those available if interesting.
>
> Thanks,
> Knut
>
> > > > Contact your system vendor for further information.
> > > > "
> > >
> > > I'm not sure what the status of emulated SR-IOV is so I have CCed
> > > Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.
> >
> > Thanks,
> > Marcel
> >
> > > > Could you please advise about what could be the issue here?
> > > >
> > > > BTW: I use same configuration (VM XML file attached) when running linux VM
> > > > (RH7.2) image (instead of Win Hyper-V) over the same host and SRIOV is
> > > > working for me there.
> > > >
> > > > Here the XML file I use to define the VM (our emulated NIC is added at the
> > > > end of XML):
> > > > "
> > > > <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'
> > > > >;
> > > > <name>nst105</name>
> > > > <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
> > > > <memory unit='KiB'>8388608</memory>
> > > > <currentMemory unit='KiB'>8388608</currentMemory>
> > > > <vcpu placement='static'>8</vcpu>
> > > > <resource>
> > > > <partition>/machine</partition>
> > > > </resource>
> > > > <os>
> > > > <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
> > > > </os>
> > > > <features>
> > > > <acpi/>
> > > > <apic/>
> > > > <hyperv>
> > > > <relaxed state='on'/>
> > > > <vapic state='on'/>
> > > > <spinlocks state='on' retries='8191'/>
> > > > </hyperv>
> > > > </features>
> > > > <cpu mode='custom' match='exact' check='none'>
> > > > <model fallback='forbid'>SandyBridge</model>
> > > > <feature policy='require' name='hypervisor'/>
> > > > <feature policy='require' name='vmx'/>
> > > > </cpu>
> > > > <clock offset='localtime'>
> > > > <timer name='rtc' tickpolicy='catchup'/>
> > > > <timer name='pit' tickpolicy='delay'/>
> > > > <timer name='hpet' present='no'/>
> > > > <timer name='hypervclock' present='yes'/>
> > > > </clock>
> > > > <on_poweroff>destroy</on_poweroff>
> > > > <on_reboot>restart</on_reboot>
> > > > <on_crash>destroy</on_crash>
> > > > <pm>
> > > > <suspend-to-mem enabled='no'/>
> > > > <suspend-to-disk enabled='no'/>
> > > > </pm>
> > > > <devices>
> > > > <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
> > > > <disk type='file' device='disk'>
> > > > <driver name='qemu' type='qcow2'/>
> > > > <source file='/images/nst105.qcow2'/>
> > > > <target dev='sda' bus='sata'/>
> > > > <boot order='2'/>
> > > > <address type='drive' controller='0' bus='0' target='0' unit='0'/>
> > > > </disk>
> > > > <controller type='usb' index='0' model='ich9-ehci1'>
> > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > function='0x7'/>
> > > > </controller>
> > > > <controller type='usb' index='0' model='ich9-uhci1'>
> > > > <master startport='0'/>
> > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > function='0x0' multifunction='on'/>
> > > > </controller>
> > > > <controller type='usb' index='0' model='ich9-uhci2'>
> > > > <master startport='2'/>
> > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > function='0x1'/>
> > > > </controller>
> > > > <controller type='usb' index='0' model='ich9-uhci3'>
> > > > <master startport='4'/>
> > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > function='0x2'/>
> > > > </controller>
> > > > <controller type='sata' index='0'>
> > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1f'
> > > > function='0x2'/>
> > > > </controller>
> > > > <controller type='pci' index='0' model='pcie-root'/>
> > > > <controller type='pci' index='1' model='pcie-root-port'>
> > > > <model name='pcie-root-port'/>
> > > > <target chassis='1' port='0x10'/>
> > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > function='0x0' multifunction='on'/>
> > > > </controller>
> > > > <controller type='pci' index='2' model='pcie-root-port'>
> > > > <model name='pcie-root-port'/>
> > > > <target chassis='2' port='0x11'/>
> > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > function='0x1'/>
> > > > </controller>
> > > > <controller type='pci' index='3' model='pcie-root-port'>
> > > > <model name='pcie-root-port'/>
> > > > <target chassis='3' port='0x12'/>
> > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > function='0x2'/>
> > > > </controller>
> > > > <controller type='pci' index='4' model='dmi-to-pci-bridge'>
> > > > <model name='i82801b11-bridge'/>
> > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1e'
> > > > function='0x0'/>
> > > > </controller>
> > > > <controller type='pci' index='5' model='pci-bridge'>
> > > > <model name='pci-bridge'/>
> > > > <target chassisNr='5'/>
> > > > <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> > > > function='0x0'/>
> > > > </controller>
> > > > <interface type='bridge'>
> > > > <mac address='00:50:56:1b:dd:6f'/>
> > > > <source bridge='br0'/>
> > > > <model type='virtio'/>
> > > > <boot order='1'/>
> > > > <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
> > > > function='0x0'/>
> > > > </interface>
> > > > <serial type='pty'>
> > > > <target port='0'/>
> > > > </serial>
> > > > <console type='pty'>
> > > > <target type='serial' port='0'/>
> > > > </console>
> > > > <input type='tablet' bus='usb'>
> > > > <address type='usb' bus='0' port='1'/>
> > > > </input>
> > > > <input type='mouse' bus='ps2'/>
> > > > <input type='keyboard' bus='ps2'/>
> > > > <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'
> > > > keymap='en-us'>
> > > > <listen type='address' address='127.0.0.1'/>
> > > > </graphics>
> > > > <video>
> > > > <model type='vga' vram='16384' heads='1' primary='yes'/>
> > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> > > > function='0x0'/>
> > > > </video>
> > > > <memballoon model='virtio'>
> > > > <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
> > > > function='0x0'/>
> > > > </memballoon>
> > > > </devices>
> > > > <seclabel type='none' model='none'/>
> > > > <seclabel type='dynamic' model='dac' relabel='yes'/>
> > > > <qemu:commandline>
> > > > <qemu:arg value='-device'/>
> > > > <qemu:arg value='pcie-root-port,pref64-
> > > > reserve=500M,slot=0,id=pcie_port.1'/>
> > > > <qemu:arg value='-netdev'/>
> > > > <qemu:arg value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
> > > > <qemu:arg value='-device'/>
> > > > <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
> > > > </qemu:commandline>
> > > > </domain>
> > > > "
> > > > ______________________
> > > > General info:
> > > > Host OS: RH7.0 (Kernel: 4.14.13)
> > > > QEMU version: 2.11
> > > > libvirt version: 3.2.0
> > > > Running the following on the host shows that both nested and IOMMU are
> > > > enabled:
> > > > ~]#: cat /sys/module/kvm_intel/parameters/nested
> > > > Y
> > > > ~]# dmesg | grep -e DMAR -e IOMMU
> > > > [ 0.000000] DMAR: IOMMU enabled
> > > >
> > > > Thanks,
> > > > Elijah
> > > >
> >
> >
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host
2018-07-26 16:41 ` Michael S. Tsirkin
@ 2018-07-26 16:42 ` Knut Omang
2018-07-26 17:05 ` Marcel Apfelbaum
2018-07-26 17:28 ` Michael S. Tsirkin
0 siblings, 2 replies; 17+ messages in thread
From: Knut Omang @ 2018-07-26 16:42 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Marcel Apfelbaum, Stefan Hajnoczi, Elijah Shakkour, qemu-devel,
Tal Attaly
On Thu, 2018-07-26 at 19:41 +0300, Michael S. Tsirkin wrote:
> On Thu, Jul 26, 2018 at 06:38:32PM +0200, Knut Omang wrote:
> > On Thu, 2018-07-26 at 18:51 +0300, Marcel Apfelbaum wrote:
> > > Hi
> > >
> > > On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
> > > > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
> > > > > Hey,
> > > > >
> > > > > Our team is adding a NIC functional emulation to QEMU.
> > > > > One of the features we are adding to this NIC is SRIOV.
> > > > >
> > > > > Here is the error message I get when checking SRIOV support of our
> > > > > emulated NIC on Win2016 server (the hyper-v VM).
> > > > > "
> > > > > SR-IOV cannot be used on this system as the PCI Express hardware does
> > > > > not
> > > > > support Access Control Services (ACS) at any root port.
> > >
> > > QEMU's emulated PCI Express Root Ports do not support ACS yet, however I
> > > am not sure ACS is a prerequisite
> > > for SR-IOV. We would need ARI support for allowing more than 8VFs, but
> > > QEMU doesn't support that either (yet).
> > >
> > > Knut Omag has some working patches, he successfully implemented SR-IOV
> > > with QEMU, see:
> > > https://github.com/knuto/qemu/tree/sriov_patches_v7
> > >
> > > The code was not merged since we need at least a device with SR-IOV
> > > support to justify the addition.
> >
> > FYI, I recently rebased these to latest master but just didn't get to push
> > them
> > out. I did just now - they are available here:
> >
> > https://github.com/knuto/qemu/tree/sriov_patches_v8
> >
> > As far as I know ARI support with my patch set works just fine - I have
> > tested
> > it with lots of VFs.
> >
> > One of the patches in the series (pci: Make use of the devfn property when
> > registering new devices) is necessary to make ARI work as it should with
> > SR/IOV.
> >
> > For the hardware model I developed the SR/IOV patches for, I also added
> > enough
> > ACS support in the root port (PCIe capability helper patch + usage in
> > ioh3420) to make VFIO "happy". I haven't submitted them because they
> > are "questionable" since they likely do not reflect the actual features of
> > the
> > ioh3420.
>
> In that the actual ioh3420 doesn't support ACS?
yes.. I don't have one so I don't know but that was my assumption..
Knut
> > I can make those available if interesting.
> >
> > Thanks,
> > Knut
> >
> > > > > Contact your system vendor for further information.
> > > > > "
> > > >
> > > > I'm not sure what the status of emulated SR-IOV is so I have CCed
> > > > Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.
> > >
> > > Thanks,
> > > Marcel
> > >
> > > > > Could you please advise about what could be the issue here?
> > > > >
> > > > > BTW: I use same configuration (VM XML file attached) when running
> > > > > linux VM
> > > > > (RH7.2) image (instead of Win Hyper-V) over the same host and SRIOV is
> > > > > working for me there.
> > > > >
> > > > > Here the XML file I use to define the VM (our emulated NIC is added at
> > > > > the
> > > > > end of XML):
> > > > > "
> > > > > <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/
> > > > > 1.0'
> > > > > > ;
> > > > >
> > > > > <name>nst105</name>
> > > > > <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
> > > > > <memory unit='KiB'>8388608</memory>
> > > > > <currentMemory unit='KiB'>8388608</currentMemory>
> > > > > <vcpu placement='static'>8</vcpu>
> > > > > <resource>
> > > > > <partition>/machine</partition>
> > > > > </resource>
> > > > > <os>
> > > > > <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
> > > > > </os>
> > > > > <features>
> > > > > <acpi/>
> > > > > <apic/>
> > > > > <hyperv>
> > > > > <relaxed state='on'/>
> > > > > <vapic state='on'/>
> > > > > <spinlocks state='on' retries='8191'/>
> > > > > </hyperv>
> > > > > </features>
> > > > > <cpu mode='custom' match='exact' check='none'>
> > > > > <model fallback='forbid'>SandyBridge</model>
> > > > > <feature policy='require' name='hypervisor'/>
> > > > > <feature policy='require' name='vmx'/>
> > > > > </cpu>
> > > > > <clock offset='localtime'>
> > > > > <timer name='rtc' tickpolicy='catchup'/>
> > > > > <timer name='pit' tickpolicy='delay'/>
> > > > > <timer name='hpet' present='no'/>
> > > > > <timer name='hypervclock' present='yes'/>
> > > > > </clock>
> > > > > <on_poweroff>destroy</on_poweroff>
> > > > > <on_reboot>restart</on_reboot>
> > > > > <on_crash>destroy</on_crash>
> > > > > <pm>
> > > > > <suspend-to-mem enabled='no'/>
> > > > > <suspend-to-disk enabled='no'/>
> > > > > </pm>
> > > > > <devices>
> > > > > <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
> > > > > <disk type='file' device='disk'>
> > > > > <driver name='qemu' type='qcow2'/>
> > > > > <source file='/images/nst105.qcow2'/>
> > > > > <target dev='sda' bus='sata'/>
> > > > > <boot order='2'/>
> > > > > <address type='drive' controller='0' bus='0' target='0'
> > > > > unit='0'/>
> > > > > </disk>
> > > > > <controller type='usb' index='0' model='ich9-ehci1'>
> > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > function='0x7'/>
> > > > > </controller>
> > > > > <controller type='usb' index='0' model='ich9-uhci1'>
> > > > > <master startport='0'/>
> > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > function='0x0' multifunction='on'/>
> > > > > </controller>
> > > > > <controller type='usb' index='0' model='ich9-uhci2'>
> > > > > <master startport='2'/>
> > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > function='0x1'/>
> > > > > </controller>
> > > > > <controller type='usb' index='0' model='ich9-uhci3'>
> > > > > <master startport='4'/>
> > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > function='0x2'/>
> > > > > </controller>
> > > > > <controller type='sata' index='0'>
> > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1f'
> > > > > function='0x2'/>
> > > > > </controller>
> > > > > <controller type='pci' index='0' model='pcie-root'/>
> > > > > <controller type='pci' index='1' model='pcie-root-port'>
> > > > > <model name='pcie-root-port'/>
> > > > > <target chassis='1' port='0x10'/>
> > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > > function='0x0' multifunction='on'/>
> > > > > </controller>
> > > > > <controller type='pci' index='2' model='pcie-root-port'>
> > > > > <model name='pcie-root-port'/>
> > > > > <target chassis='2' port='0x11'/>
> > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > > function='0x1'/>
> > > > > </controller>
> > > > > <controller type='pci' index='3' model='pcie-root-port'>
> > > > > <model name='pcie-root-port'/>
> > > > > <target chassis='3' port='0x12'/>
> > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > > function='0x2'/>
> > > > > </controller>
> > > > > <controller type='pci' index='4' model='dmi-to-pci-bridge'>
> > > > > <model name='i82801b11-bridge'/>
> > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1e'
> > > > > function='0x0'/>
> > > > > </controller>
> > > > > <controller type='pci' index='5' model='pci-bridge'>
> > > > > <model name='pci-bridge'/>
> > > > > <target chassisNr='5'/>
> > > > > <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> > > > > function='0x0'/>
> > > > > </controller>
> > > > > <interface type='bridge'>
> > > > > <mac address='00:50:56:1b:dd:6f'/>
> > > > > <source bridge='br0'/>
> > > > > <model type='virtio'/>
> > > > > <boot order='1'/>
> > > > > <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
> > > > > function='0x0'/>
> > > > > </interface>
> > > > > <serial type='pty'>
> > > > > <target port='0'/>
> > > > > </serial>
> > > > > <console type='pty'>
> > > > > <target type='serial' port='0'/>
> > > > > </console>
> > > > > <input type='tablet' bus='usb'>
> > > > > <address type='usb' bus='0' port='1'/>
> > > > > </input>
> > > > > <input type='mouse' bus='ps2'/>
> > > > > <input type='keyboard' bus='ps2'/>
> > > > > <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'
> > > > > keymap='en-us'>
> > > > > <listen type='address' address='127.0.0.1'/>
> > > > > </graphics>
> > > > > <video>
> > > > > <model type='vga' vram='16384' heads='1' primary='yes'/>
> > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> > > > > function='0x0'/>
> > > > > </video>
> > > > > <memballoon model='virtio'>
> > > > > <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
> > > > > function='0x0'/>
> > > > > </memballoon>
> > > > > </devices>
> > > > > <seclabel type='none' model='none'/>
> > > > > <seclabel type='dynamic' model='dac' relabel='yes'/>
> > > > > <qemu:commandline>
> > > > > <qemu:arg value='-device'/>
> > > > > <qemu:arg value='pcie-root-port,pref64-
> > > > > reserve=500M,slot=0,id=pcie_port.1'/>
> > > > > <qemu:arg value='-netdev'/>
> > > > > <qemu:arg
> > > > > value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
> > > > > <qemu:arg value='-device'/>
> > > > > <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
> > > > > </qemu:commandline>
> > > > > </domain>
> > > > > "
> > > > > ______________________
> > > > > General info:
> > > > > Host OS: RH7.0 (Kernel: 4.14.13)
> > > > > QEMU version: 2.11
> > > > > libvirt version: 3.2.0
> > > > > Running the following on the host shows that both nested and IOMMU are
> > > > > enabled:
> > > > > ~]#: cat /sys/module/kvm_intel/parameters/nested
> > > > > Y
> > > > > ~]# dmesg | grep -e DMAR -e IOMMU
> > > > > [ 0.000000] DMAR: IOMMU enabled
> > > > >
> > > > > Thanks,
> > > > > Elijah
> > > > >
> > >
> > >
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host
2018-07-26 16:14 ` Michael S. Tsirkin
2018-07-26 16:34 ` Marcel Apfelbaum
@ 2018-07-26 16:42 ` Alex Williamson
2018-07-26 17:11 ` Michael S. Tsirkin
1 sibling, 1 reply; 17+ messages in thread
From: Alex Williamson @ 2018-07-26 16:42 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Marcel Apfelbaum, Tal Attaly, Stefan Hajnoczi, Knut Omang,
qemu-devel, Elijah Shakkour
On Thu, 26 Jul 2018 19:14:45 +0300
"Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Thu, Jul 26, 2018 at 06:51:13PM +0300, Marcel Apfelbaum wrote:
> > Hi
> >
> > On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
> > > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
> > > > Hey,
> > > >
> > > > Our team is adding a NIC functional emulation to QEMU.
> > > > One of the features we are adding to this NIC is SRIOV.
> > > >
> > > > Here is the error message I get when checking SRIOV support of our emulated NIC on Win2016 server (the hyper-v VM).
> > > > "
> > > > SR-IOV cannot be used on this system as the PCI Express hardware does not support Access Control Services (ACS) at any root port.
> >
> > QEMU's emulated PCI Express Root Ports do not support ACS yet, however I am
> > not sure ACS is a prerequisite
> > for SR-IOV.
ACS is certainly not a prerequisite for SR-IOV.
> Looks like windows blocks dev assignment in nested VMs without it.
> Thinking about it, doesn't vfio do the same by default? I think vfio has
> a flag to override this though.
IOMMU grouping in Linux takes isolation via ACS and device specific
mechanisms into account, limiting the granularity with which a
userspace driver can claim ownership of devices, but it doesn't
actually prevent enabling SR-IOV on the endpoint. It just makes it
less useful if your intention is to use SR-IOV for device assignment.
The only overrides for this are out-of-tree. Thanks,
Alex
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host
2018-07-26 16:42 ` Knut Omang
@ 2018-07-26 17:05 ` Marcel Apfelbaum
2018-07-26 17:25 ` Knut Omang
2018-07-26 17:27 ` Knut Omang
2018-07-26 17:28 ` Michael S. Tsirkin
1 sibling, 2 replies; 17+ messages in thread
From: Marcel Apfelbaum @ 2018-07-26 17:05 UTC (permalink / raw)
To: Knut Omang, Michael S. Tsirkin
Cc: Stefan Hajnoczi, Elijah Shakkour, qemu-devel, Tal Attaly
On 07/26/2018 07:42 PM, Knut Omang wrote:
> On Thu, 2018-07-26 at 19:41 +0300, Michael S. Tsirkin wrote:
>> On Thu, Jul 26, 2018 at 06:38:32PM +0200, Knut Omang wrote:
>>> On Thu, 2018-07-26 at 18:51 +0300, Marcel Apfelbaum wrote:
>>>> Hi
>>>>
>>>> On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
>>>>> On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
>>>>>> Hey,
>>>>>>
>>>>>> Our team is adding a NIC functional emulation to QEMU.
>>>>>> One of the features we are adding to this NIC is SRIOV.
>>>>>>
>>>>>> Here is the error message I get when checking SRIOV support of our
>>>>>> emulated NIC on Win2016 server (the hyper-v VM).
>>>>>> "
>>>>>> SR-IOV cannot be used on this system as the PCI Express hardware does
>>>>>> not
>>>>>> support Access Control Services (ACS) at any root port.
>>>> QEMU's emulated PCI Express Root Ports do not support ACS yet, however I
>>>> am not sure ACS is a prerequisite
>>>> for SR-IOV. We would need ARI support for allowing more than 8VFs, but
>>>> QEMU doesn't support that either (yet).
>>>>
>>>> Knut Omag has some working patches, he successfully implemented SR-IOV
>>>> with QEMU, see:
>>>> https://github.com/knuto/qemu/tree/sriov_patches_v7
>>>>
>>>> The code was not merged since we need at least a device with SR-IOV
>>>> support to justify the addition.
>>> FYI, I recently rebased these to latest master but just didn't get to push
>>> them
>>> out. I did just now - they are available here:
>>>
>>> https://github.com/knuto/qemu/tree/sriov_patches_v8
>>>
>>> As far as I know ARI support with my patch set works just fine - I have
>>> tested
>>> it with lots of VFs.
>>>
>>> One of the patches in the series (pci: Make use of the devfn property when
>>> registering new devices) is necessary to make ARI work as it should with
>>> SR/IOV.
>>>
>>> For the hardware model I developed the SR/IOV patches for, I also added
>>> enough
>>> ACS support in the root port (PCIe capability helper patch + usage in
>>> ioh3420) to make VFIO "happy". I haven't submitted them because they
>>> are "questionable" since they likely do not reflect the actual features of
>>> the
>>> ioh3420.
>> In that the actual ioh3420 doesn't support ACS?
> yes.. I don't have one so I don't know but that was my assumption..
Hi Knut,
We have now a generic PCIe Root Port we can add whatever we want to it.
See please hw/pci-bridge/gen_pcie_root_port.c.
So your patches add both ARI and ACS support, nice!
Maybe it worth merging at least these features.
Thanks,
Marcel
>
> Knut
>
>>> I can make those available if interesting.
>>>
>>> Thanks,
>>> Knut
>>>
>>>>>> Contact your system vendor for further information.
>>>>>> "
>>>>> I'm not sure what the status of emulated SR-IOV is so I have CCed
>>>>> Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.
>>>> Thanks,
>>>> Marcel
>>>>
>>>>>> Could you please advise about what could be the issue here?
>>>>>>
>>>>>> BTW: I use same configuration (VM XML file attached) when running
>>>>>> linux VM
>>>>>> (RH7.2) image (instead of Win Hyper-V) over the same host and SRIOV is
>>>>>> working for me there.
>>>>>>
>>>>>> Here the XML file I use to define the VM (our emulated NIC is added at
>>>>>> the
>>>>>> end of XML):
>>>>>> "
>>>>>> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/
>>>>>> 1.0'
>>>>>>> ;
>>>>>> <name>nst105</name>
>>>>>> <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
>>>>>> <memory unit='KiB'>8388608</memory>
>>>>>> <currentMemory unit='KiB'>8388608</currentMemory>
>>>>>> <vcpu placement='static'>8</vcpu>
>>>>>> <resource>
>>>>>> <partition>/machine</partition>
>>>>>> </resource>
>>>>>> <os>
>>>>>> <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
>>>>>> </os>
>>>>>> <features>
>>>>>> <acpi/>
>>>>>> <apic/>
>>>>>> <hyperv>
>>>>>> <relaxed state='on'/>
>>>>>> <vapic state='on'/>
>>>>>> <spinlocks state='on' retries='8191'/>
>>>>>> </hyperv>
>>>>>> </features>
>>>>>> <cpu mode='custom' match='exact' check='none'>
>>>>>> <model fallback='forbid'>SandyBridge</model>
>>>>>> <feature policy='require' name='hypervisor'/>
>>>>>> <feature policy='require' name='vmx'/>
>>>>>> </cpu>
>>>>>> <clock offset='localtime'>
>>>>>> <timer name='rtc' tickpolicy='catchup'/>
>>>>>> <timer name='pit' tickpolicy='delay'/>
>>>>>> <timer name='hpet' present='no'/>
>>>>>> <timer name='hypervclock' present='yes'/>
>>>>>> </clock>
>>>>>> <on_poweroff>destroy</on_poweroff>
>>>>>> <on_reboot>restart</on_reboot>
>>>>>> <on_crash>destroy</on_crash>
>>>>>> <pm>
>>>>>> <suspend-to-mem enabled='no'/>
>>>>>> <suspend-to-disk enabled='no'/>
>>>>>> </pm>
>>>>>> <devices>
>>>>>> <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
>>>>>> <disk type='file' device='disk'>
>>>>>> <driver name='qemu' type='qcow2'/>
>>>>>> <source file='/images/nst105.qcow2'/>
>>>>>> <target dev='sda' bus='sata'/>
>>>>>> <boot order='2'/>
>>>>>> <address type='drive' controller='0' bus='0' target='0'
>>>>>> unit='0'/>
>>>>>> </disk>
>>>>>> <controller type='usb' index='0' model='ich9-ehci1'>
>>>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
>>>>>> function='0x7'/>
>>>>>> </controller>
>>>>>> <controller type='usb' index='0' model='ich9-uhci1'>
>>>>>> <master startport='0'/>
>>>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
>>>>>> function='0x0' multifunction='on'/>
>>>>>> </controller>
>>>>>> <controller type='usb' index='0' model='ich9-uhci2'>
>>>>>> <master startport='2'/>
>>>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
>>>>>> function='0x1'/>
>>>>>> </controller>
>>>>>> <controller type='usb' index='0' model='ich9-uhci3'>
>>>>>> <master startport='4'/>
>>>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
>>>>>> function='0x2'/>
>>>>>> </controller>
>>>>>> <controller type='sata' index='0'>
>>>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f'
>>>>>> function='0x2'/>
>>>>>> </controller>
>>>>>> <controller type='pci' index='0' model='pcie-root'/>
>>>>>> <controller type='pci' index='1' model='pcie-root-port'>
>>>>>> <model name='pcie-root-port'/>
>>>>>> <target chassis='1' port='0x10'/>
>>>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
>>>>>> function='0x0' multifunction='on'/>
>>>>>> </controller>
>>>>>> <controller type='pci' index='2' model='pcie-root-port'>
>>>>>> <model name='pcie-root-port'/>
>>>>>> <target chassis='2' port='0x11'/>
>>>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
>>>>>> function='0x1'/>
>>>>>> </controller>
>>>>>> <controller type='pci' index='3' model='pcie-root-port'>
>>>>>> <model name='pcie-root-port'/>
>>>>>> <target chassis='3' port='0x12'/>
>>>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
>>>>>> function='0x2'/>
>>>>>> </controller>
>>>>>> <controller type='pci' index='4' model='dmi-to-pci-bridge'>
>>>>>> <model name='i82801b11-bridge'/>
>>>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e'
>>>>>> function='0x0'/>
>>>>>> </controller>
>>>>>> <controller type='pci' index='5' model='pci-bridge'>
>>>>>> <model name='pci-bridge'/>
>>>>>> <target chassisNr='5'/>
>>>>>> <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
>>>>>> function='0x0'/>
>>>>>> </controller>
>>>>>> <interface type='bridge'>
>>>>>> <mac address='00:50:56:1b:dd:6f'/>
>>>>>> <source bridge='br0'/>
>>>>>> <model type='virtio'/>
>>>>>> <boot order='1'/>
>>>>>> <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
>>>>>> function='0x0'/>
>>>>>> </interface>
>>>>>> <serial type='pty'>
>>>>>> <target port='0'/>
>>>>>> </serial>
>>>>>> <console type='pty'>
>>>>>> <target type='serial' port='0'/>
>>>>>> </console>
>>>>>> <input type='tablet' bus='usb'>
>>>>>> <address type='usb' bus='0' port='1'/>
>>>>>> </input>
>>>>>> <input type='mouse' bus='ps2'/>
>>>>>> <input type='keyboard' bus='ps2'/>
>>>>>> <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'
>>>>>> keymap='en-us'>
>>>>>> <listen type='address' address='127.0.0.1'/>
>>>>>> </graphics>
>>>>>> <video>
>>>>>> <model type='vga' vram='16384' heads='1' primary='yes'/>
>>>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
>>>>>> function='0x0'/>
>>>>>> </video>
>>>>>> <memballoon model='virtio'>
>>>>>> <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
>>>>>> function='0x0'/>
>>>>>> </memballoon>
>>>>>> </devices>
>>>>>> <seclabel type='none' model='none'/>
>>>>>> <seclabel type='dynamic' model='dac' relabel='yes'/>
>>>>>> <qemu:commandline>
>>>>>> <qemu:arg value='-device'/>
>>>>>> <qemu:arg value='pcie-root-port,pref64-
>>>>>> reserve=500M,slot=0,id=pcie_port.1'/>
>>>>>> <qemu:arg value='-netdev'/>
>>>>>> <qemu:arg
>>>>>> value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
>>>>>> <qemu:arg value='-device'/>
>>>>>> <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
>>>>>> </qemu:commandline>
>>>>>> </domain>
>>>>>> "
>>>>>> ______________________
>>>>>> General info:
>>>>>> Host OS: RH7.0 (Kernel: 4.14.13)
>>>>>> QEMU version: 2.11
>>>>>> libvirt version: 3.2.0
>>>>>> Running the following on the host shows that both nested and IOMMU are
>>>>>> enabled:
>>>>>> ~]#: cat /sys/module/kvm_intel/parameters/nested
>>>>>> Y
>>>>>> ~]# dmesg | grep -e DMAR -e IOMMU
>>>>>> [ 0.000000] DMAR: IOMMU enabled
>>>>>>
>>>>>> Thanks,
>>>>>> Elijah
>>>>>>
>>>>
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host
2018-07-26 16:42 ` Alex Williamson
@ 2018-07-26 17:11 ` Michael S. Tsirkin
2018-07-26 17:22 ` Knut Omang
2018-07-26 17:22 ` Alex Williamson
0 siblings, 2 replies; 17+ messages in thread
From: Michael S. Tsirkin @ 2018-07-26 17:11 UTC (permalink / raw)
To: Alex Williamson
Cc: Marcel Apfelbaum, Tal Attaly, Stefan Hajnoczi, Knut Omang,
qemu-devel, Elijah Shakkour
On Thu, Jul 26, 2018 at 10:42:52AM -0600, Alex Williamson wrote:
> On Thu, 26 Jul 2018 19:14:45 +0300
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
>
> > On Thu, Jul 26, 2018 at 06:51:13PM +0300, Marcel Apfelbaum wrote:
> > > Hi
> > >
> > > On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
> > > > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
> > > > > Hey,
> > > > >
> > > > > Our team is adding a NIC functional emulation to QEMU.
> > > > > One of the features we are adding to this NIC is SRIOV.
> > > > >
> > > > > Here is the error message I get when checking SRIOV support of our emulated NIC on Win2016 server (the hyper-v VM).
> > > > > "
> > > > > SR-IOV cannot be used on this system as the PCI Express hardware does not support Access Control Services (ACS) at any root port.
> > >
> > > QEMU's emulated PCI Express Root Ports do not support ACS yet, however I am
> > > not sure ACS is a prerequisite
> > > for SR-IOV.
>
> ACS is certainly not a prerequisite for SR-IOV.
>
> > Looks like windows blocks dev assignment in nested VMs without it.
> > Thinking about it, doesn't vfio do the same by default? I think vfio has
> > a flag to override this though.
>
> IOMMU grouping in Linux takes isolation via ACS and device specific
> mechanisms into account, limiting the granularity with which a
> userspace driver can claim ownership of devices,
In that you must assign all devices behind this port as a group?
> but it doesn't
> actually prevent enabling SR-IOV on the endpoint. It just makes it
> less useful if your intention is to use SR-IOV for device assignment.
> The only overrides for this are out-of-tree. Thanks,
>
> Alex
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host
2018-07-26 17:11 ` Michael S. Tsirkin
@ 2018-07-26 17:22 ` Knut Omang
2018-07-26 17:22 ` Alex Williamson
1 sibling, 0 replies; 17+ messages in thread
From: Knut Omang @ 2018-07-26 17:22 UTC (permalink / raw)
To: Michael S. Tsirkin, Alex Williamson
Cc: Marcel Apfelbaum, Tal Attaly, Stefan Hajnoczi, qemu-devel,
Elijah Shakkour
On Thu, 2018-07-26 at 20:11 +0300, Michael S. Tsirkin wrote:
> On Thu, Jul 26, 2018 at 10:42:52AM -0600, Alex Williamson wrote:
> > On Thu, 26 Jul 2018 19:14:45 +0300
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >
> > > On Thu, Jul 26, 2018 at 06:51:13PM +0300, Marcel Apfelbaum wrote:
> > > > Hi
> > > >
> > > > On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
> > > > > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
> > > > > > Hey,
> > > > > >
> > > > > > Our team is adding a NIC functional emulation to QEMU.
> > > > > > One of the features we are adding to this NIC is SRIOV.
> > > > > >
> > > > > > Here is the error message I get when checking SRIOV support of our
> > > > > > emulated NIC on Win2016 server (the hyper-v VM).
> > > > > > "
> > > > > > SR-IOV cannot be used on this system as the PCI Express hardware
> > > > > > does not support Access Control Services (ACS) at any root port.
> > > >
> > > > QEMU's emulated PCI Express Root Ports do not support ACS yet, however I
> > > > am
> > > > not sure ACS is a prerequisite
> > > > for SR-IOV.
> >
> > ACS is certainly not a prerequisite for SR-IOV.
> >
> > > Looks like windows blocks dev assignment in nested VMs without it.
> > > Thinking about it, doesn't vfio do the same by default? I think vfio has
> > > a flag to override this though.
> >
> > IOMMU grouping in Linux takes isolation via ACS and device specific
> > mechanisms into account, limiting the granularity with which a
> > userspace driver can claim ownership of devices,
>
> In that you must assign all devices behind this port as a group?
Yes, that's why I needed to add PCIe config space ACS support in ioh3420:
to be able to use VFIO to pass through individual VFs to different L2 VMs with
"plain" kernels on the L1 host which owns the SR/IOV device.
Knut
> > but it doesn't
> > actually prevent enabling SR-IOV on the endpoint. It just makes it
> > less useful if your intention is to use SR-IOV for device assignment.
> > The only overrides for this are out-of-tree. Thanks,
> >
> >
> > Alex
>
>
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host
2018-07-26 17:11 ` Michael S. Tsirkin
2018-07-26 17:22 ` Knut Omang
@ 2018-07-26 17:22 ` Alex Williamson
1 sibling, 0 replies; 17+ messages in thread
From: Alex Williamson @ 2018-07-26 17:22 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Stefan Hajnoczi, Tal Attaly, Knut Omang, qemu-devel, Elijah Shakkour
On Thu, 26 Jul 2018 20:11:44 +0300
"Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Thu, Jul 26, 2018 at 10:42:52AM -0600, Alex Williamson wrote:
> > On Thu, 26 Jul 2018 19:14:45 +0300
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >
> > > On Thu, Jul 26, 2018 at 06:51:13PM +0300, Marcel Apfelbaum wrote:
> > > > Hi
> > > >
> > > > On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
> > > > > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
> > > > > > Hey,
> > > > > >
> > > > > > Our team is adding a NIC functional emulation to QEMU.
> > > > > > One of the features we are adding to this NIC is SRIOV.
> > > > > >
> > > > > > Here is the error message I get when checking SRIOV support of our emulated NIC on Win2016 server (the hyper-v VM).
> > > > > > "
> > > > > > SR-IOV cannot be used on this system as the PCI Express hardware does not support Access Control Services (ACS) at any root port.
> > > >
> > > > QEMU's emulated PCI Express Root Ports do not support ACS yet, however I am
> > > > not sure ACS is a prerequisite
> > > > for SR-IOV.
> >
> > ACS is certainly not a prerequisite for SR-IOV.
> >
> > > Looks like windows blocks dev assignment in nested VMs without it.
> > > Thinking about it, doesn't vfio do the same by default? I think vfio has
> > > a flag to override this though.
> >
> > IOMMU grouping in Linux takes isolation via ACS and device specific
> > mechanisms into account, limiting the granularity with which a
> > userspace driver can claim ownership of devices,
>
> In that you must assign all devices behind this port as a group?
Yup. Not very useful with SR-IOV. Thanks,
Alex
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host
2018-07-26 17:05 ` Marcel Apfelbaum
@ 2018-07-26 17:25 ` Knut Omang
2018-07-26 17:27 ` Knut Omang
1 sibling, 0 replies; 17+ messages in thread
From: Knut Omang @ 2018-07-26 17:25 UTC (permalink / raw)
To: Marcel Apfelbaum, Michael S. Tsirkin
Cc: Stefan Hajnoczi, Elijah Shakkour, qemu-devel, Tal Attaly
On Thu, 2018-07-26 at 20:05 +0300, Marcel Apfelbaum wrote:
>
> On 07/26/2018 07:42 PM, Knut Omang wrote:
> > On Thu, 2018-07-26 at 19:41 +0300, Michael S. Tsirkin wrote:
> > > On Thu, Jul 26, 2018 at 06:38:32PM +0200, Knut Omang wrote:
> > > > On Thu, 2018-07-26 at 18:51 +0300, Marcel Apfelbaum wrote:
> > > > > Hi
> > > > >
> > > > > On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
> > > > > > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
> > > > > > > Hey,
> > > > > > >
> > > > > > > Our team is adding a NIC functional emulation to QEMU.
> > > > > > > One of the features we are adding to this NIC is SRIOV.
> > > > > > >
> > > > > > > Here is the error message I get when checking SRIOV support
> > > > > > > of our
> > > > > > > emulated NIC on Win2016 server (the hyper-v VM).
> > > > > > > "
> > > > > > > SR-IOV cannot be used on this system as the PCI Express hardware
> > > > > > > does
> > > > > > > not
> > > > > > > support Access Control Services (ACS) at any root port.
> > > > >
> > > > > QEMU's emulated PCI Express Root Ports do not support ACS yet, however
> > > > > I
> > > > > am not sure ACS is a prerequisite
> > > > > for SR-IOV. We would need ARI support for allowing more than 8VFs, but
> > > > > QEMU doesn't support that either (yet).
> > > > >
> > > > > Knut Omag has some working patches, he successfully implemented SR-IOV
> > > > > with QEMU, see:
> > > > > https://github.com/knuto/qemu/tree/sriov_patches_v7
> > > > >
> > > > > The code was not merged since we need at least a device with SR-IOV
> > > > > support to justify the addition.
> > > >
> > > > FYI, I recently rebased these to latest master but just didn't get to
> > > > push
> > > > them
> > > > out. I did just now - they are available here:
> > > >
> > > > https://github.com/knuto/qemu/tree/sriov_patches_v8
> > > >
> > > > As far as I know ARI support with my patch set works just fine - I have
> > > > tested
> > > > it with lots of VFs.
> > > >
> > > > One of the patches in the series (pci: Make use of the devfn property
> > > > when
> > > > registering new devices) is necessary to make ARI work as it should with
> > > > SR/IOV.
> > > >
> > > > For the hardware model I developed the SR/IOV patches for, I also added
> > > > enough
> > > > ACS support in the root port (PCIe capability helper patch + usage in
> > > > ioh3420) to make VFIO "happy". I haven't submitted them because they
> > > > are "questionable" since they likely do not reflect the actual features
> > > > of
> > > > the
> > > > ioh3420.
> > >
> > > In that the actual ioh3420 doesn't support ACS?
> >
> > yes.. I don't have one so I don't know but that was my assumption..
>
> Hi Knut,
>
> We have now a generic PCIe Root Port we can add whatever we want to it.
> See please hw/pci-bridge/gen_pcie_root_port.c.
Ok, I see! - I wasn't aware of that. I was thinking more in terms of just
emulating a slightly newer Intel root port but never got around to that.
> So your patches add both ARI and ACS support, nice!
> Maybe it worth merging at least these features.
I'll have a look at the generic root port, it is probably a natural extension
for v9 of the sriov patch set :-)
Thanks,
Knut
>
> Thanks,
> Marcel
>
> >
> > Knut
> >
> > > > I can make those available if interesting.
> > > >
> > > > Thanks,
> > > > Knut
> > > >
> > > > > > > Contact your system vendor for further information.
> > > > > > > "
> > > > > >
> > > > > > I'm not sure what the status of emulated SR-IOV is so I have CCed
> > > > > > Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.
> > > > >
> > > > > Thanks,
> > > > > Marcel
> > > > >
> > > > > > > Could you please advise about what could be the issue here?
> > > > > > >
> > > > > > > BTW: I use same configuration (VM XML file attached) when running
> > > > > > > linux VM
> > > > > > > (RH7.2) image (instead of Win Hyper-V) over the same host and
> > > > > > > SRIOV is
> > > > > > > working for me there.
> > > > > > >
> > > > > > > Here the XML file I use to define the VM (our emulated NIC is
> > > > > > > added at
> > > > > > > the
> > > > > > > end of XML):
> > > > > > > "
> > > > > > > <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/q
> > > > > > > emu/
> > > > > > > 1.0'
> > > > > > > > ;
> > > > > > >
> > > > > > > <name>nst105</name>
> > > > > > > <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
> > > > > > > <memory unit='KiB'>8388608</memory>
> > > > > > > <currentMemory unit='KiB'>8388608</currentMemory>
> > > > > > > <vcpu placement='static'>8</vcpu>
> > > > > > > <resource>
> > > > > > > <partition>/machine</partition>
> > > > > > > </resource>
> > > > > > > <os>
> > > > > > > <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
> > > > > > > </os>
> > > > > > > <features>
> > > > > > > <acpi/>
> > > > > > > <apic/>
> > > > > > > <hyperv>
> > > > > > > <relaxed state='on'/>
> > > > > > > <vapic state='on'/>
> > > > > > > <spinlocks state='on' retries='8191'/>
> > > > > > > </hyperv>
> > > > > > > </features>
> > > > > > > <cpu mode='custom' match='exact' check='none'>
> > > > > > > <model fallback='forbid'>SandyBridge</model>
> > > > > > > <feature policy='require' name='hypervisor'/>
> > > > > > > <feature policy='require' name='vmx'/>
> > > > > > > </cpu>
> > > > > > > <clock offset='localtime'>
> > > > > > > <timer name='rtc' tickpolicy='catchup'/>
> > > > > > > <timer name='pit' tickpolicy='delay'/>
> > > > > > > <timer name='hpet' present='no'/>
> > > > > > > <timer name='hypervclock' present='yes'/>
> > > > > > > </clock>
> > > > > > > <on_poweroff>destroy</on_poweroff>
> > > > > > > <on_reboot>restart</on_reboot>
> > > > > > > <on_crash>destroy</on_crash>
> > > > > > > <pm>
> > > > > > > <suspend-to-mem enabled='no'/>
> > > > > > > <suspend-to-disk enabled='no'/>
> > > > > > > </pm>
> > > > > > > <devices>
> > > > > > > <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
> > > > > > > <disk type='file' device='disk'>
> > > > > > > <driver name='qemu' type='qcow2'/>
> > > > > > > <source file='/images/nst105.qcow2'/>
> > > > > > > <target dev='sda' bus='sata'/>
> > > > > > > <boot order='2'/>
> > > > > > > <address type='drive' controller='0' bus='0' target='0'
> > > > > > > unit='0'/>
> > > > > > > </disk>
> > > > > > > <controller type='usb' index='0' model='ich9-ehci1'>
> > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > > > function='0x7'/>
> > > > > > > </controller>
> > > > > > > <controller type='usb' index='0' model='ich9-uhci1'>
> > > > > > > <master startport='0'/>
> > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > > > function='0x0' multifunction='on'/>
> > > > > > > </controller>
> > > > > > > <controller type='usb' index='0' model='ich9-uhci2'>
> > > > > > > <master startport='2'/>
> > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > > > function='0x1'/>
> > > > > > > </controller>
> > > > > > > <controller type='usb' index='0' model='ich9-uhci3'>
> > > > > > > <master startport='4'/>
> > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > > > function='0x2'/>
> > > > > > > </controller>
> > > > > > > <controller type='sata' index='0'>
> > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1f'
> > > > > > > function='0x2'/>
> > > > > > > </controller>
> > > > > > > <controller type='pci' index='0' model='pcie-root'/>
> > > > > > > <controller type='pci' index='1' model='pcie-root-port'>
> > > > > > > <model name='pcie-root-port'/>
> > > > > > > <target chassis='1' port='0x10'/>
> > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > > > > function='0x0' multifunction='on'/>
> > > > > > > </controller>
> > > > > > > <controller type='pci' index='2' model='pcie-root-port'>
> > > > > > > <model name='pcie-root-port'/>
> > > > > > > <target chassis='2' port='0x11'/>
> > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > > > > function='0x1'/>
> > > > > > > </controller>
> > > > > > > <controller type='pci' index='3' model='pcie-root-port'>
> > > > > > > <model name='pcie-root-port'/>
> > > > > > > <target chassis='3' port='0x12'/>
> > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > > > > function='0x2'/>
> > > > > > > </controller>
> > > > > > > <controller type='pci' index='4' model='dmi-to-pci-bridge'>
> > > > > > > <model name='i82801b11-bridge'/>
> > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1e'
> > > > > > > function='0x0'/>
> > > > > > > </controller>
> > > > > > > <controller type='pci' index='5' model='pci-bridge'>
> > > > > > > <model name='pci-bridge'/>
> > > > > > > <target chassisNr='5'/>
> > > > > > > <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> > > > > > > function='0x0'/>
> > > > > > > </controller>
> > > > > > > <interface type='bridge'>
> > > > > > > <mac address='00:50:56:1b:dd:6f'/>
> > > > > > > <source bridge='br0'/>
> > > > > > > <model type='virtio'/>
> > > > > > > <boot order='1'/>
> > > > > > > <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
> > > > > > > function='0x0'/>
> > > > > > > </interface>
> > > > > > > <serial type='pty'>
> > > > > > > <target port='0'/>
> > > > > > > </serial>
> > > > > > > <console type='pty'>
> > > > > > > <target type='serial' port='0'/>
> > > > > > > </console>
> > > > > > > <input type='tablet' bus='usb'>
> > > > > > > <address type='usb' bus='0' port='1'/>
> > > > > > > </input>
> > > > > > > <input type='mouse' bus='ps2'/>
> > > > > > > <input type='keyboard' bus='ps2'/>
> > > > > > > <graphics type='vnc' port='-1' autoport='yes'
> > > > > > > listen='127.0.0.1'
> > > > > > > keymap='en-us'>
> > > > > > > <listen type='address' address='127.0.0.1'/>
> > > > > > > </graphics>
> > > > > > > <video>
> > > > > > > <model type='vga' vram='16384' heads='1' primary='yes'/>
> > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> > > > > > > function='0x0'/>
> > > > > > > </video>
> > > > > > > <memballoon model='virtio'>
> > > > > > > <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
> > > > > > > function='0x0'/>
> > > > > > > </memballoon>
> > > > > > > </devices>
> > > > > > > <seclabel type='none' model='none'/>
> > > > > > > <seclabel type='dynamic' model='dac' relabel='yes'/>
> > > > > > > <qemu:commandline>
> > > > > > > <qemu:arg value='-device'/>
> > > > > > > <qemu:arg value='pcie-root-port,pref64-
> > > > > > > reserve=500M,slot=0,id=pcie_port.1'/>
> > > > > > > <qemu:arg value='-netdev'/>
> > > > > > > <qemu:arg
> > > > > > > value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
> > > > > > > <qemu:arg value='-device'/>
> > > > > > > <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
> > > > > > > </qemu:commandline>
> > > > > > > </domain>
> > > > > > > "
> > > > > > > ______________________
> > > > > > > General info:
> > > > > > > Host OS: RH7.0 (Kernel: 4.14.13)
> > > > > > > QEMU version: 2.11
> > > > > > > libvirt version: 3.2.0
> > > > > > > Running the following on the host shows that both nested and IOMMU
> > > > > > > are
> > > > > > > enabled:
> > > > > > > ~]#: cat /sys/module/kvm_intel/parameters/nested
> > > > > > > Y
> > > > > > > ~]# dmesg | grep -e DMAR -e IOMMU
> > > > > > > [ 0.000000] DMAR: IOMMU enabled
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Elijah
> > > > > > >
>
>
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host
2018-07-26 17:05 ` Marcel Apfelbaum
2018-07-26 17:25 ` Knut Omang
@ 2018-07-26 17:27 ` Knut Omang
1 sibling, 0 replies; 17+ messages in thread
From: Knut Omang @ 2018-07-26 17:27 UTC (permalink / raw)
To: Marcel Apfelbaum, Michael S. Tsirkin
Cc: Stefan Hajnoczi, Elijah Shakkour, qemu-devel, Tal Attaly
On Thu, 2018-07-26 at 20:05 +0300, Marcel Apfelbaum wrote:
>
> On 07/26/2018 07:42 PM, Knut Omang wrote:
> > On Thu, 2018-07-26 at 19:41 +0300, Michael S. Tsirkin wrote:
> > > On Thu, Jul 26, 2018 at 06:38:32PM +0200, Knut Omang wrote:
> > > > On Thu, 2018-07-26 at 18:51 +0300, Marcel Apfelbaum wrote:
> > > > > Hi
> > > > >
> > > > > On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
> > > > > > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
> > > > > > > Hey,
> > > > > > >
> > > > > > > Our team is adding a NIC functional emulation to QEMU.
> > > > > > > One of the features we are adding to this NIC is SRIOV.
> > > > > > >
> > > > > > > Here is the error message I get when checking SRIOV support
> > > > > > > of our
> > > > > > > emulated NIC on Win2016 server (the hyper-v VM).
> > > > > > > "
> > > > > > > SR-IOV cannot be used on this system as the PCI Express hardware
> > > > > > > does
> > > > > > > not
> > > > > > > support Access Control Services (ACS) at any root port.
> > > > >
> > > > > QEMU's emulated PCI Express Root Ports do not support ACS yet, however
> > > > > I
> > > > > am not sure ACS is a prerequisite
> > > > > for SR-IOV. We would need ARI support for allowing more than 8VFs, but
> > > > > QEMU doesn't support that either (yet).
> > > > >
> > > > > Knut Omag has some working patches, he successfully implemented SR-IOV
> > > > > with QEMU, see:
> > > > > https://github.com/knuto/qemu/tree/sriov_patches_v7
> > > > >
> > > > > The code was not merged since we need at least a device with SR-IOV
> > > > > support to justify the addition.
> > > >
> > > > FYI, I recently rebased these to latest master but just didn't get to
> > > > push
> > > > them
> > > > out. I did just now - they are available here:
> > > >
> > > > https://github.com/knuto/qemu/tree/sriov_patches_v8
> > > >
> > > > As far as I know ARI support with my patch set works just fine - I have
> > > > tested
> > > > it with lots of VFs.
> > > >
> > > > One of the patches in the series (pci: Make use of the devfn property
> > > > when
> > > > registering new devices) is necessary to make ARI work as it should with
> > > > SR/IOV.
> > > >
> > > > For the hardware model I developed the SR/IOV patches for, I also added
> > > > enough
> > > > ACS support in the root port (PCIe capability helper patch + usage in
> > > > ioh3420) to make VFIO "happy". I haven't submitted them because they
> > > > are "questionable" since they likely do not reflect the actual features
> > > > of
> > > > the
> > > > ioh3420.
> > >
> > > In that the actual ioh3420 doesn't support ACS?
> >
> > yes.. I don't have one so I don't know but that was my assumption..
>
> Hi Knut,
>
> We have now a generic PCIe Root Port we can add whatever we want to it.
> See please hw/pci-bridge/gen_pcie_root_port.c.
Ok, I see! - I wasn't aware of that. I was thinking more in terms of just
emulating a slightly newer Intel root port but never got around to that.
> So your patches add both ARI and ACS support, nice!
> Maybe it worth merging at least these features.
I'll have a look at the generic root port, it is probably a natural extension
for v9 of the sriov patch set :-)
Thanks,
Knut
>
> Thanks,
> Marcel
>
> >
> > Knut
> >
> > > > I can make those available if interesting.
> > > >
> > > > Thanks,
> > > > Knut
> > > >
> > > > > > > Contact your system vendor for further information.
> > > > > > > "
> > > > > >
> > > > > > I'm not sure what the status of emulated SR-IOV is so I have CCed
> > > > > > Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.
> > > > >
> > > > > Thanks,
> > > > > Marcel
> > > > >
> > > > > > > Could you please advise about what could be the issue here?
> > > > > > >
> > > > > > > BTW: I use same configuration (VM XML file attached) when running
> > > > > > > linux VM
> > > > > > > (RH7.2) image (instead of Win Hyper-V) over the same host and
> > > > > > > SRIOV is
> > > > > > > working for me there.
> > > > > > >
> > > > > > > Here the XML file I use to define the VM (our emulated NIC is
> > > > > > > added at
> > > > > > > the
> > > > > > > end of XML):
> > > > > > > "
> > > > > > > <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/q
> > > > > > > emu/
> > > > > > > 1.0'
> > > > > > > > ;
> > > > > > >
> > > > > > > <name>nst105</name>
> > > > > > > <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
> > > > > > > <memory unit='KiB'>8388608</memory>
> > > > > > > <currentMemory unit='KiB'>8388608</currentMemory>
> > > > > > > <vcpu placement='static'>8</vcpu>
> > > > > > > <resource>
> > > > > > > <partition>/machine</partition>
> > > > > > > </resource>
> > > > > > > <os>
> > > > > > > <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
> > > > > > > </os>
> > > > > > > <features>
> > > > > > > <acpi/>
> > > > > > > <apic/>
> > > > > > > <hyperv>
> > > > > > > <relaxed state='on'/>
> > > > > > > <vapic state='on'/>
> > > > > > > <spinlocks state='on' retries='8191'/>
> > > > > > > </hyperv>
> > > > > > > </features>
> > > > > > > <cpu mode='custom' match='exact' check='none'>
> > > > > > > <model fallback='forbid'>SandyBridge</model>
> > > > > > > <feature policy='require' name='hypervisor'/>
> > > > > > > <feature policy='require' name='vmx'/>
> > > > > > > </cpu>
> > > > > > > <clock offset='localtime'>
> > > > > > > <timer name='rtc' tickpolicy='catchup'/>
> > > > > > > <timer name='pit' tickpolicy='delay'/>
> > > > > > > <timer name='hpet' present='no'/>
> > > > > > > <timer name='hypervclock' present='yes'/>
> > > > > > > </clock>
> > > > > > > <on_poweroff>destroy</on_poweroff>
> > > > > > > <on_reboot>restart</on_reboot>
> > > > > > > <on_crash>destroy</on_crash>
> > > > > > > <pm>
> > > > > > > <suspend-to-mem enabled='no'/>
> > > > > > > <suspend-to-disk enabled='no'/>
> > > > > > > </pm>
> > > > > > > <devices>
> > > > > > > <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
> > > > > > > <disk type='file' device='disk'>
> > > > > > > <driver name='qemu' type='qcow2'/>
> > > > > > > <source file='/images/nst105.qcow2'/>
> > > > > > > <target dev='sda' bus='sata'/>
> > > > > > > <boot order='2'/>
> > > > > > > <address type='drive' controller='0' bus='0' target='0'
> > > > > > > unit='0'/>
> > > > > > > </disk>
> > > > > > > <controller type='usb' index='0' model='ich9-ehci1'>
> > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > > > function='0x7'/>
> > > > > > > </controller>
> > > > > > > <controller type='usb' index='0' model='ich9-uhci1'>
> > > > > > > <master startport='0'/>
> > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > > > function='0x0' multifunction='on'/>
> > > > > > > </controller>
> > > > > > > <controller type='usb' index='0' model='ich9-uhci2'>
> > > > > > > <master startport='2'/>
> > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > > > function='0x1'/>
> > > > > > > </controller>
> > > > > > > <controller type='usb' index='0' model='ich9-uhci3'>
> > > > > > > <master startport='4'/>
> > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > > > function='0x2'/>
> > > > > > > </controller>
> > > > > > > <controller type='sata' index='0'>
> > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1f'
> > > > > > > function='0x2'/>
> > > > > > > </controller>
> > > > > > > <controller type='pci' index='0' model='pcie-root'/>
> > > > > > > <controller type='pci' index='1' model='pcie-root-port'>
> > > > > > > <model name='pcie-root-port'/>
> > > > > > > <target chassis='1' port='0x10'/>
> > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > > > > function='0x0' multifunction='on'/>
> > > > > > > </controller>
> > > > > > > <controller type='pci' index='2' model='pcie-root-port'>
> > > > > > > <model name='pcie-root-port'/>
> > > > > > > <target chassis='2' port='0x11'/>
> > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > > > > function='0x1'/>
> > > > > > > </controller>
> > > > > > > <controller type='pci' index='3' model='pcie-root-port'>
> > > > > > > <model name='pcie-root-port'/>
> > > > > > > <target chassis='3' port='0x12'/>
> > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > > > > function='0x2'/>
> > > > > > > </controller>
> > > > > > > <controller type='pci' index='4' model='dmi-to-pci-bridge'>
> > > > > > > <model name='i82801b11-bridge'/>
> > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1e'
> > > > > > > function='0x0'/>
> > > > > > > </controller>
> > > > > > > <controller type='pci' index='5' model='pci-bridge'>
> > > > > > > <model name='pci-bridge'/>
> > > > > > > <target chassisNr='5'/>
> > > > > > > <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> > > > > > > function='0x0'/>
> > > > > > > </controller>
> > > > > > > <interface type='bridge'>
> > > > > > > <mac address='00:50:56:1b:dd:6f'/>
> > > > > > > <source bridge='br0'/>
> > > > > > > <model type='virtio'/>
> > > > > > > <boot order='1'/>
> > > > > > > <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
> > > > > > > function='0x0'/>
> > > > > > > </interface>
> > > > > > > <serial type='pty'>
> > > > > > > <target port='0'/>
> > > > > > > </serial>
> > > > > > > <console type='pty'>
> > > > > > > <target type='serial' port='0'/>
> > > > > > > </console>
> > > > > > > <input type='tablet' bus='usb'>
> > > > > > > <address type='usb' bus='0' port='1'/>
> > > > > > > </input>
> > > > > > > <input type='mouse' bus='ps2'/>
> > > > > > > <input type='keyboard' bus='ps2'/>
> > > > > > > <graphics type='vnc' port='-1' autoport='yes'
> > > > > > > listen='127.0.0.1'
> > > > > > > keymap='en-us'>
> > > > > > > <listen type='address' address='127.0.0.1'/>
> > > > > > > </graphics>
> > > > > > > <video>
> > > > > > > <model type='vga' vram='16384' heads='1' primary='yes'/>
> > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> > > > > > > function='0x0'/>
> > > > > > > </video>
> > > > > > > <memballoon model='virtio'>
> > > > > > > <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
> > > > > > > function='0x0'/>
> > > > > > > </memballoon>
> > > > > > > </devices>
> > > > > > > <seclabel type='none' model='none'/>
> > > > > > > <seclabel type='dynamic' model='dac' relabel='yes'/>
> > > > > > > <qemu:commandline>
> > > > > > > <qemu:arg value='-device'/>
> > > > > > > <qemu:arg value='pcie-root-port,pref64-
> > > > > > > reserve=500M,slot=0,id=pcie_port.1'/>
> > > > > > > <qemu:arg value='-netdev'/>
> > > > > > > <qemu:arg
> > > > > > > value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
> > > > > > > <qemu:arg value='-device'/>
> > > > > > > <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
> > > > > > > </qemu:commandline>
> > > > > > > </domain>
> > > > > > > "
> > > > > > > ______________________
> > > > > > > General info:
> > > > > > > Host OS: RH7.0 (Kernel: 4.14.13)
> > > > > > > QEMU version: 2.11
> > > > > > > libvirt version: 3.2.0
> > > > > > > Running the following on the host shows that both nested and IOMMU
> > > > > > > are
> > > > > > > enabled:
> > > > > > > ~]#: cat /sys/module/kvm_intel/parameters/nested
> > > > > > > Y
> > > > > > > ~]# dmesg | grep -e DMAR -e IOMMU
> > > > > > > [ 0.000000] DMAR: IOMMU enabled
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Elijah
> > > > > > >
>
>
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host
2018-07-26 16:42 ` Knut Omang
2018-07-26 17:05 ` Marcel Apfelbaum
@ 2018-07-26 17:28 ` Michael S. Tsirkin
1 sibling, 0 replies; 17+ messages in thread
From: Michael S. Tsirkin @ 2018-07-26 17:28 UTC (permalink / raw)
To: Knut Omang
Cc: Marcel Apfelbaum, Stefan Hajnoczi, Elijah Shakkour, qemu-devel,
Tal Attaly
On Thu, Jul 26, 2018 at 06:42:51PM +0200, Knut Omang wrote:
> On Thu, 2018-07-26 at 19:41 +0300, Michael S. Tsirkin wrote:
> > On Thu, Jul 26, 2018 at 06:38:32PM +0200, Knut Omang wrote:
> > > On Thu, 2018-07-26 at 18:51 +0300, Marcel Apfelbaum wrote:
> > > > Hi
> > > >
> > > > On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
> > > > > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
> > > > > > Hey,
> > > > > >
> > > > > > Our team is adding a NIC functional emulation to QEMU.
> > > > > > One of the features we are adding to this NIC is SRIOV.
> > > > > >
> > > > > > Here is the error message I get when checking SRIOV support of our
> > > > > > emulated NIC on Win2016 server (the hyper-v VM).
> > > > > > "
> > > > > > SR-IOV cannot be used on this system as the PCI Express hardware does
> > > > > > not
> > > > > > support Access Control Services (ACS) at any root port.
> > > >
> > > > QEMU's emulated PCI Express Root Ports do not support ACS yet, however I
> > > > am not sure ACS is a prerequisite
> > > > for SR-IOV. We would need ARI support for allowing more than 8VFs, but
> > > > QEMU doesn't support that either (yet).
> > > >
> > > > Knut Omag has some working patches, he successfully implemented SR-IOV
> > > > with QEMU, see:
> > > > https://github.com/knuto/qemu/tree/sriov_patches_v7
> > > >
> > > > The code was not merged since we need at least a device with SR-IOV
> > > > support to justify the addition.
> > >
> > > FYI, I recently rebased these to latest master but just didn't get to push
> > > them
> > > out. I did just now - they are available here:
> > >
> > > https://github.com/knuto/qemu/tree/sriov_patches_v8
> > >
> > > As far as I know ARI support with my patch set works just fine - I have
> > > tested
> > > it with lots of VFs.
> > >
> > > One of the patches in the series (pci: Make use of the devfn property when
> > > registering new devices) is necessary to make ARI work as it should with
> > > SR/IOV.
> > >
> > > For the hardware model I developed the SR/IOV patches for, I also added
> > > enough
> > > ACS support in the root port (PCIe capability helper patch + usage in
> > > ioh3420) to make VFIO "happy". I haven't submitted them because they
> > > are "questionable" since they likely do not reflect the actual features of
> > > the
> > > ioh3420.
> >
> > In that the actual ioh3420 doesn't support ACS?
>
> yes.. I don't have one so I don't know but that was my assumption..
>
> Knut
As Marcel said, just add it to the generic port, we can
upstream that.
> > > I can make those available if interesting.
> > >
> > > Thanks,
> > > Knut
> > >
> > > > > > Contact your system vendor for further information.
> > > > > > "
> > > > >
> > > > > I'm not sure what the status of emulated SR-IOV is so I have CCed
> > > > > Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.
> > > >
> > > > Thanks,
> > > > Marcel
> > > >
> > > > > > Could you please advise about what could be the issue here?
> > > > > >
> > > > > > BTW: I use same configuration (VM XML file attached) when running
> > > > > > linux VM
> > > > > > (RH7.2) image (instead of Win Hyper-V) over the same host and SRIOV is
> > > > > > working for me there.
> > > > > >
> > > > > > Here the XML file I use to define the VM (our emulated NIC is added at
> > > > > > the
> > > > > > end of XML):
> > > > > > "
> > > > > > <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/
> > > > > > 1.0'
> > > > > > > ;
> > > > > >
> > > > > > <name>nst105</name>
> > > > > > <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
> > > > > > <memory unit='KiB'>8388608</memory>
> > > > > > <currentMemory unit='KiB'>8388608</currentMemory>
> > > > > > <vcpu placement='static'>8</vcpu>
> > > > > > <resource>
> > > > > > <partition>/machine</partition>
> > > > > > </resource>
> > > > > > <os>
> > > > > > <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
> > > > > > </os>
> > > > > > <features>
> > > > > > <acpi/>
> > > > > > <apic/>
> > > > > > <hyperv>
> > > > > > <relaxed state='on'/>
> > > > > > <vapic state='on'/>
> > > > > > <spinlocks state='on' retries='8191'/>
> > > > > > </hyperv>
> > > > > > </features>
> > > > > > <cpu mode='custom' match='exact' check='none'>
> > > > > > <model fallback='forbid'>SandyBridge</model>
> > > > > > <feature policy='require' name='hypervisor'/>
> > > > > > <feature policy='require' name='vmx'/>
> > > > > > </cpu>
> > > > > > <clock offset='localtime'>
> > > > > > <timer name='rtc' tickpolicy='catchup'/>
> > > > > > <timer name='pit' tickpolicy='delay'/>
> > > > > > <timer name='hpet' present='no'/>
> > > > > > <timer name='hypervclock' present='yes'/>
> > > > > > </clock>
> > > > > > <on_poweroff>destroy</on_poweroff>
> > > > > > <on_reboot>restart</on_reboot>
> > > > > > <on_crash>destroy</on_crash>
> > > > > > <pm>
> > > > > > <suspend-to-mem enabled='no'/>
> > > > > > <suspend-to-disk enabled='no'/>
> > > > > > </pm>
> > > > > > <devices>
> > > > > > <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
> > > > > > <disk type='file' device='disk'>
> > > > > > <driver name='qemu' type='qcow2'/>
> > > > > > <source file='/images/nst105.qcow2'/>
> > > > > > <target dev='sda' bus='sata'/>
> > > > > > <boot order='2'/>
> > > > > > <address type='drive' controller='0' bus='0' target='0'
> > > > > > unit='0'/>
> > > > > > </disk>
> > > > > > <controller type='usb' index='0' model='ich9-ehci1'>
> > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > > function='0x7'/>
> > > > > > </controller>
> > > > > > <controller type='usb' index='0' model='ich9-uhci1'>
> > > > > > <master startport='0'/>
> > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > > function='0x0' multifunction='on'/>
> > > > > > </controller>
> > > > > > <controller type='usb' index='0' model='ich9-uhci2'>
> > > > > > <master startport='2'/>
> > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > > function='0x1'/>
> > > > > > </controller>
> > > > > > <controller type='usb' index='0' model='ich9-uhci3'>
> > > > > > <master startport='4'/>
> > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > > function='0x2'/>
> > > > > > </controller>
> > > > > > <controller type='sata' index='0'>
> > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1f'
> > > > > > function='0x2'/>
> > > > > > </controller>
> > > > > > <controller type='pci' index='0' model='pcie-root'/>
> > > > > > <controller type='pci' index='1' model='pcie-root-port'>
> > > > > > <model name='pcie-root-port'/>
> > > > > > <target chassis='1' port='0x10'/>
> > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > > > function='0x0' multifunction='on'/>
> > > > > > </controller>
> > > > > > <controller type='pci' index='2' model='pcie-root-port'>
> > > > > > <model name='pcie-root-port'/>
> > > > > > <target chassis='2' port='0x11'/>
> > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > > > function='0x1'/>
> > > > > > </controller>
> > > > > > <controller type='pci' index='3' model='pcie-root-port'>
> > > > > > <model name='pcie-root-port'/>
> > > > > > <target chassis='3' port='0x12'/>
> > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > > > function='0x2'/>
> > > > > > </controller>
> > > > > > <controller type='pci' index='4' model='dmi-to-pci-bridge'>
> > > > > > <model name='i82801b11-bridge'/>
> > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1e'
> > > > > > function='0x0'/>
> > > > > > </controller>
> > > > > > <controller type='pci' index='5' model='pci-bridge'>
> > > > > > <model name='pci-bridge'/>
> > > > > > <target chassisNr='5'/>
> > > > > > <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> > > > > > function='0x0'/>
> > > > > > </controller>
> > > > > > <interface type='bridge'>
> > > > > > <mac address='00:50:56:1b:dd:6f'/>
> > > > > > <source bridge='br0'/>
> > > > > > <model type='virtio'/>
> > > > > > <boot order='1'/>
> > > > > > <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
> > > > > > function='0x0'/>
> > > > > > </interface>
> > > > > > <serial type='pty'>
> > > > > > <target port='0'/>
> > > > > > </serial>
> > > > > > <console type='pty'>
> > > > > > <target type='serial' port='0'/>
> > > > > > </console>
> > > > > > <input type='tablet' bus='usb'>
> > > > > > <address type='usb' bus='0' port='1'/>
> > > > > > </input>
> > > > > > <input type='mouse' bus='ps2'/>
> > > > > > <input type='keyboard' bus='ps2'/>
> > > > > > <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'
> > > > > > keymap='en-us'>
> > > > > > <listen type='address' address='127.0.0.1'/>
> > > > > > </graphics>
> > > > > > <video>
> > > > > > <model type='vga' vram='16384' heads='1' primary='yes'/>
> > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> > > > > > function='0x0'/>
> > > > > > </video>
> > > > > > <memballoon model='virtio'>
> > > > > > <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
> > > > > > function='0x0'/>
> > > > > > </memballoon>
> > > > > > </devices>
> > > > > > <seclabel type='none' model='none'/>
> > > > > > <seclabel type='dynamic' model='dac' relabel='yes'/>
> > > > > > <qemu:commandline>
> > > > > > <qemu:arg value='-device'/>
> > > > > > <qemu:arg value='pcie-root-port,pref64-
> > > > > > reserve=500M,slot=0,id=pcie_port.1'/>
> > > > > > <qemu:arg value='-netdev'/>
> > > > > > <qemu:arg
> > > > > > value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
> > > > > > <qemu:arg value='-device'/>
> > > > > > <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
> > > > > > </qemu:commandline>
> > > > > > </domain>
> > > > > > "
> > > > > > ______________________
> > > > > > General info:
> > > > > > Host OS: RH7.0 (Kernel: 4.14.13)
> > > > > > QEMU version: 2.11
> > > > > > libvirt version: 3.2.0
> > > > > > Running the following on the host shows that both nested and IOMMU are
> > > > > > enabled:
> > > > > > ~]#: cat /sys/module/kvm_intel/parameters/nested
> > > > > > Y
> > > > > > ~]# dmesg | grep -e DMAR -e IOMMU
> > > > > > [ 0.000000] DMAR: IOMMU enabled
> > > > > >
> > > > > > Thanks,
> > > > > > Elijah
> > > > > >
> > > >
> > > >
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2018-07-26 17:30 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <DB5PR05MB11605C65D7D734C9354F0421D54F0@DB5PR05MB1160.eurprd05.prod.outlook.com>
[not found] ` <1407239306.11199051.1530240887935.JavaMail.zimbra@redhat.com>
[not found] ` <20180705194311-mutt-send-email-mst@kernel.org>
[not found] ` <DB5PR05MB1160D84E747DA480435B6605D5440@DB5PR05MB1160.eurprd05.prod.outlook.com>
[not found] ` <DB5PR05MB1160BB381473C723C8336EC2D55A0@DB5PR05MB1160.eurprd05.prod.outlook.com>
[not found] ` <AM5PR0501MB2499CB4F2D01BCDA47C0ACB6C95A0@AM5PR0501MB2499.eurprd05.prod.outlook.com>
2018-07-12 7:33 ` [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host Elijah Shakkour
2018-07-26 14:52 ` Stefan Hajnoczi
2018-07-26 15:51 ` Marcel Apfelbaum
2018-07-26 16:14 ` Michael S. Tsirkin
2018-07-26 16:34 ` Marcel Apfelbaum
2018-07-26 16:38 ` Michael S. Tsirkin
2018-07-26 16:42 ` Alex Williamson
2018-07-26 17:11 ` Michael S. Tsirkin
2018-07-26 17:22 ` Knut Omang
2018-07-26 17:22 ` Alex Williamson
2018-07-26 16:38 ` Knut Omang
2018-07-26 16:41 ` Michael S. Tsirkin
2018-07-26 16:42 ` Knut Omang
2018-07-26 17:05 ` Marcel Apfelbaum
2018-07-26 17:25 ` Knut Omang
2018-07-26 17:27 ` Knut Omang
2018-07-26 17:28 ` Michael S. Tsirkin
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.