VFIO PROBLEM: pci_alloc_irq_vectors function request 32 MSI interrupts vectors, but return 1 in KVM virtual machine.

VFIO PROBLEM: pci_alloc_irq_vectors function request 32 MSI interrupts vectors, but return 1 in KVM virtual machine.

[Renjun Wang]

Hi all:
I have a question about PCI which troubled me for a few weeks.
I have a virtual machine with ubuntu 16.4.03 on KVM platform. There is a PCIe device(Xilinx PCIe IP) plugged in the host machine, and passthrough to guest via VFIO feature.
On the ubuntu operation system, I am developing the pcie driver. When I use pci_alloc_irq_vectors() function to allocate 32 msi vectors, but return 1.
The command  `lspci -vvv` output shows 
MSI: Enable+ Count=1/32 Maskable+ 64bit+

there is a similar case https://stackoverflow.com/questions/49821599/multiple-msi-vectors-linux-pci-alloc-irq-vectors-return-one-while-the-devi.
But not working for KVM virtual machine.

I do not known why the function  pci_alloc_irq_vectors() returns one ?


Best regards.

Re: VFIO PROBLEM: pci_alloc_irq_vectors function request 32 MSI interrupts vectors, but return 1 in KVM virtual machine.

[Alex Williamson]

On Sat, 28 Dec 2019 01:59:43 +0000
Renjun Wang <rwang@panyi.ai> wrote:

> Hi all:
> I have a question about PCI which troubled me for a few weeks.
> I have a virtual machine with ubuntu 16.4.03 on KVM platform. There
> is a PCIe device(Xilinx PCIe IP) plugged in the host machine, and
> passthrough to guest via VFIO feature. On the ubuntu operation
> system, I am developing the pcie driver. When I use
> pci_alloc_irq_vectors() function to allocate 32 msi vectors, but
> return 1. The command  `lspci -vvv` output shows MSI: Enable+
> Count=1/32 Maskable+ 64bit+
> 
> there is a similar case
> https://stackoverflow.com/questions/49821599/multiple-msi-vectors-linux-pci-alloc-irq-vectors-return-one-while-the-devi.
> But not working for KVM virtual machine.
> 
> I do not known why the function  pci_alloc_irq_vectors() returns one ?

When you say it's not working in the virtual machine with that
stackoverflow tip, does that mean your VM is running a Q35 machine type
with the intel-iommu device enabled in both QEMU and on the guest
command line?  You should see "IR-PCI-MSI" in /proc/interrupts on host
and guest for the interrupt type if the interrupt remapping is enabled.
Linux doesn't support multiple MSI vectors without some kind of
interrupt remapper support.  You probably have that on the host, but
you'll need it in the guest as well or else the guest kernel will limit
you to a single vector.

BTW, if you have any influence over the device, you really, really want
to use MSI-X for supporting multiple vectors.  Thanks,

Alex

回复: VFIO PROBLEM: pci_alloc_irq_vectors function request 32 MSI interrupts vectors, but return 1 in KVM virtual machine.

[Renjun Wang]

-----邮件原件-----
发件人: kvm-owner@vger.kernel.org <kvm-owner@vger.kernel.org> 代表 Alex Williamson
发送时间: 2020年1月7日 6:41
收件人: Renjun Wang <rwang@panyi.ai>
抄送: kvm@vger.kernel.org
主题: Re: VFIO PROBLEM: pci_alloc_irq_vectors function request 32 MSI interrupts vectors, but return 1 in KVM virtual machine.

On Sat, 28 Dec 2019 01:59:43 +0000
Renjun Wang <rwang@panyi.ai> wrote:

> Hi all:
> I have a question about PCI which troubled me for a few weeks.
> I have a virtual machine with ubuntu 16.4.03 on KVM platform. There is 
> a PCIe device(Xilinx PCIe IP) plugged in the host machine, and 
> passthrough to guest via VFIO feature. On the ubuntu operation system, 
> I am developing the pcie driver. When I use
> pci_alloc_irq_vectors() function to allocate 32 msi vectors, but 
> return 1. The command  `lspci -vvv` output shows MSI: Enable+
> Count=1/32 Maskable+ 64bit+
> 
> there is a similar case
> https://stackoverflow.com/questions/49821599/multiple-msi-vectors-linux-pci-alloc-irq-vectors-return-one-while-the-devi.
> But not working for KVM virtual machine.
> 
> I do not known why the function  pci_alloc_irq_vectors() returns one ?

When you say it's not working in the virtual machine with that stackoverflow tip, does that mean your VM is running a Q35 machine type with the intel-iommu device enabled in both QEMU and on the guest command line?  You should see "IR-PCI-MSI" in /proc/interrupts on host and guest for the interrupt type if the interrupt remapping is enabled.
Linux doesn't support multiple MSI vectors without some kind of interrupt remapper support.  You probably have that on the host, but you'll need it in the guest as well or else the guest kernel will limit you to a single vector.

BTW, if you have any influence over the device, you really, really want to use MSI-X for supporting multiple vectors.  Thanks,

Alex


Hi, Alex
Thanks very much for your response. It is indeed vIOMMU not enabled in guest machine. According the page as blew, I reconfigure the qemu virtual machine setting, aha it is working. Thanks again.
https://wiki.qemu.org/Features/VT-d