Re: [PATCH v5 7/9] s390x/pci: enable adapter event notification for interpreted devices

[Niklas Schnelle <schnelle@linux.ibm.com>]

On Mon, 2022-05-02 at 09:48 +0200, Pierre Morel wrote:
> 
> On 4/22/22 14:10, Matthew Rosato wrote:
> > On 4/22/22 5:39 AM, Pierre Morel wrote:
> > > 
> > > On 4/4/22 20:17, Matthew Rosato wrote:
> > > > Use the associated kvm ioctl operation to enable adapter event 
> > > > notification
> > > > and forwarding for devices when requested.  This feature will be set up
> > > > with or without firmware assist based upon the 'forwarding_assist' 
> > > > setting.
> > > > 
> > > > Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
> > > > ---
> > > >   hw/s390x/s390-pci-bus.c         | 20 ++++++++++++++---
> > > >   hw/s390x/s390-pci-inst.c        | 40 +++++++++++++++++++++++++++++++--
> > > >   hw/s390x/s390-pci-kvm.c         | 30 +++++++++++++++++++++++++
> > > >   include/hw/s390x/s390-pci-bus.h |  1 +
> > > >   include/hw/s390x/s390-pci-kvm.h | 14 ++++++++++++
> > > >   5 files changed, 100 insertions(+), 5 deletions(-)
> > > > 
> > > > diff --git a/hw/s390x/s390-pci-bus.c b/hw/s390x/s390-pci-bus.c
> > > > index 9c02d31250..47918d2ce9 100644
> > > > --- a/hw/s390x/s390-pci-bus.c
> > > > +++ b/hw/s390x/s390-pci-bus.c
> > > > @@ -190,7 +190,10 @@ void s390_pci_sclp_deconfigure(SCCB *sccb)
> > > >           rc = SCLP_RC_NO_ACTION_REQUIRED;
> > > >           break;
> > > >       default:
> > > > -        if (pbdev->summary_ind) {
> > > > +        if (pbdev->interp && (pbdev->fh & FH_MASK_ENABLE)) {
> > > > +            /* Interpreted devices were using interrupt forwarding */
> > > > +            s390_pci_kvm_aif_disable(pbdev);
> > > 
> > > Same remark as for the kernel part.
> > > The VFIO device is already initialized and the action is on this 
> > > device, Shouldn't we use the VFIO device interface instead of the KVM 
> > > interface?
> > > 
> > 
> > I don't necessarily disagree, but in v3 of the kernel series I was told 
> > not to use VFIO ioctls to accomplish tasks that are unique to KVM (e.g. 
> > AEN interpretation) and to instead use a KVM ioctl.
> > 
> > VFIO_DEVICE_SET_IRQS won't work as-is for reasons described in the 
> > kernel series (e.g. we don't see any of the config space notifiers 
> > because of instruction interpretation) -- as far as I can figure we 
> > could add our own s390 code to QEMU to issue VFIO_DEVICE_SET_IRQS 
> > directly for an interpreted device, but I think would also need 
> > s390-specific changes to VFIO_DEVICE_SET_IRQS accommodate this (e.g. 
> > maybe something like a VFIO_IRQ_SET_DATA_S390AEN where we can then 
> > specify the aen information in vfio_irq_set.data -- or something else I 
> 
> Hi,
> 
> yes this in VFIO_DEVICE_SET_IRQS is what I think should be done.
> 
> > haven't though of yet) -- I can try to look at this some more and see if 
> > I get a good idea.
> 
> 
> I understood that the demand was concerning the IOMMU but I may be wrong.
> For my opinion, the handling of AEN is not specific to KVM but specific 
> to the device, for example the code should be the same if Z ever decide 
> to use XEN or another hypervizor, except for the GISA part but this part 
> is already implemented in KVM in a way it can be used from a device like 
> in VFIO AP.
> 
> @Alex, what do you think?
> 
> Regards,
> Pierre
> 

As I understand it the question isn't if it is specific to KVM but
rather if it is specific to virtualization. As vfio-pci is also used
for non virtualization purposes such as with DPDK/SPDK or a fully
emulating QEMU, it should only be in VFIO if it is relevant for these
kinds of user-space PCI accesses too. I'm not an AEN expert but as I
understand it, this does forwarding interrupts into a SIE context which
only makes sense for virtualization not for general user-space PCI.