question about handling MSI-X by VFIO

question about handling MSI-X by VFIO

[Thanos Makatos]

I'm passing through a virtual PCI device to a QEMU guest via VFIO/mdev and I
notice that MSI-X interrupts are disabled in the device (MSIXCAP.MXC.MXE is
zero) and the BARs containing the table and PBA (4 and 5 in my case) are never
accessed.  However, whenever I fire an MSI-X interrupt from the virtual device
(although I'm not supposed to do so as they're disabled), the guest seems to
correctly receive it. I've started looking at hw/vfio/pci.c and it seems that
VFIO handles MSI-X interrupts there, including masking etc?

Re: question about handling MSI-X by VFIO

[Alex Williamson]

On Tue, 21 Jan 2020 13:36:26 +0000
Thanos Makatos <thanos.makatos@nutanix.com> wrote:

> I'm passing through a virtual PCI device to a QEMU guest via VFIO/mdev and I
> notice that MSI-X interrupts are disabled in the device (MSIXCAP.MXC.MXE is
> zero) and the BARs containing the table and PBA (4 and 5 in my case) are never
> accessed.  However, whenever I fire an MSI-X interrupt from the virtual device
> (although I'm not supposed to do so as they're disabled), the guest seems to
> correctly receive it. I've started looking at hw/vfio/pci.c and it seems that
> VFIO handles MSI-X interrupts there, including masking etc?

Yes, the vector table and PBA are emulated in QEMU, the latter lazily
only when vectors are masked, iirc.  The backing device vector table
should never be directly accessed by the user (it can be, but you can
just discard those accesses), MSI-X is configured via the
VFIO_DEVICE_SET_IRQS ioctl, which configures the eventfd through which
an mdev driver would trigger an MSI.  When you say that you "fire and
MSI-X interrupt from the virtual device" does this mean that you're
signaling via one of these eventfds?  It looks to me like emulating the
MSI-X enable bit in the MSI-X capability is probably the responsibility
of the mdev vendor driver.  With vfio-pci the VFIO_DEVICE_SET_IRQS ioctl
would enable MSI-X on the physical device and the MSI-X capability seen
by the user would reflect that.  Are you missing a bit of code that
updates the mdev config space as part of the SET_IRQS ioctl?  Thanks,

Alex

RE: question about handling MSI-X by VFIO

[Thanos Makatos]

> > I'm passing through a virtual PCI device to a QEMU guest via VFIO/mdev
> and I
> > notice that MSI-X interrupts are disabled in the device (MSIXCAP.MXC.MXE
> is
> > zero) and the BARs containing the table and PBA (4 and 5 in my case) are
> never
> > accessed.  However, whenever I fire an MSI-X interrupt from the virtual
> device
> > (although I'm not supposed to do so as they're disabled), the guest seems
> to
> > correctly receive it. I've started looking at hw/vfio/pci.c and it seems that
> > VFIO handles MSI-X interrupts there, including masking etc?
> 
> Yes, the vector table and PBA are emulated in QEMU, the latter lazily
> only when vectors are masked, iirc.  The backing device vector table
> should never be directly accessed by the user (it can be, but you can
> just discard those accesses), MSI-X is configured via the
> VFIO_DEVICE_SET_IRQS ioctl, which configures the eventfd through which
> an mdev driver would trigger an MSI.  When you say that you "fire and
> MSI-X interrupt from the virtual device" does this mean that you're
> signaling via one of these eventfds?  It looks to me like emulating the
> MSI-X enable bit in the MSI-X capability is probably the responsibility
> of the mdev vendor driver.  With vfio-pci the VFIO_DEVICE_SET_IRQS ioctl
> would enable MSI-X on the physical device and the MSI-X capability seen
> by the user would reflect that.  Are you missing a bit of code that
> updates the mdev config space as part of the SET_IRQS ioctl?  Thanks,

Indeed I fire interrupts via the eventfd and it works correctly. I just
couldn't understand how it could possibly work since the table and PBA BARs 
were never accessed and the MSI-X enable bit was not set.  It makes perfect
sense now why it works since QEMU does it all.