Re: [PATCH v5 5/9] s390x/pci: enable for load/store intepretation

[Pierre Morel <pmorel@linux.ibm.com>]

On 4/4/22 20:17, Matthew Rosato wrote:
> If the appropriate CPU facilty is available as well as the necessary
> ZPCI_OP ioctl, then the underlying KVM host will enable load/store
> intepretation for any guest device without a SHM bit in the guest
> function handle.  For a device that will be using interpretation
> support, ensure the guest function handle matches the host function
> handle; this value is re-checked every time the guest issues a SET PCI FN
> to enable the guest device as it is the only opportunity to reflect
> function handle changes.
> 
> By default, unless interpret=off is specified, interpretation support will
> always be assumed and exploited if the necessary ioctl and features are
> available on the host kernel.  When these are unavailable, we will silently
> revert to the interception model; this allows existing guest configurations
> to work unmodified on hosts with and without zPCI interpretation support,
> allowing QEMU to choose the best support model available.
> 
> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
> ---
>   hw/s390x/meson.build            |  1 +
>   hw/s390x/s390-pci-bus.c         | 66 ++++++++++++++++++++++++++++++++-
>   hw/s390x/s390-pci-inst.c        | 12 ++++++
>   hw/s390x/s390-pci-kvm.c         | 21 +++++++++++
>   include/hw/s390x/s390-pci-bus.h |  1 +
>   include/hw/s390x/s390-pci-kvm.h | 24 ++++++++++++
>   target/s390x/kvm/kvm.c          |  7 ++++
>   target/s390x/kvm/kvm_s390x.h    |  1 +
>   8 files changed, 132 insertions(+), 1 deletion(-)
>   create mode 100644 hw/s390x/s390-pci-kvm.c
>   create mode 100644 include/hw/s390x/s390-pci-kvm.h
> 

...snip...

>           if (s390_pci_msix_init(pbdev)) {
> @@ -1360,6 +1423,7 @@ static Property s390_pci_device_properties[] = {
>       DEFINE_PROP_UINT16("uid", S390PCIBusDevice, uid, UID_UNDEFINED),
>       DEFINE_PROP_S390_PCI_FID("fid", S390PCIBusDevice, fid),
>       DEFINE_PROP_STRING("target", S390PCIBusDevice, target),
> +    DEFINE_PROP_BOOL("interpret", S390PCIBusDevice, interp, true),
>       DEFINE_PROP_END_OF_LIST(),
>   };
>   
> diff --git a/hw/s390x/s390-pci-inst.c b/hw/s390x/s390-pci-inst.c
> index 6d400d4147..c898c8abe9 100644
> --- a/hw/s390x/s390-pci-inst.c
> +++ b/hw/s390x/s390-pci-inst.c
> @@ -18,6 +18,8 @@
>   #include "sysemu/hw_accel.h"
>   #include "hw/s390x/s390-pci-inst.h"
>   #include "hw/s390x/s390-pci-bus.h"
> +#include "hw/s390x/s390-pci-kvm.h"
> +#include "hw/s390x/s390-pci-vfio.h"
>   #include "hw/s390x/tod.h"
>   
>   #ifndef DEBUG_S390PCI_INST
> @@ -246,6 +248,16 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra)
>                   goto out;
>               }
>   
> +            /*
> +             * Take this opportunity to make sure we still have an accurate
> +             * host fh.  It's possible part of the handle changed while the
> +             * device was disabled to the guest (e.g. vfio hot reset for
> +             * ISM during plug)
> +             */
> +            if (pbdev->interp) {
> +                /* Take this opportunity to make sure we are sync'd with host */
> +                s390_pci_get_host_fh(pbdev, &pbdev->fh);
> +            }
>               pbdev->fh |= FH_MASK_ENABLE;

Are we sure here that the PCI device is always enabled?
Shouldn't we check?


>               pbdev->state = ZPCI_FS_ENABLED;
>               stl_p(&ressetpci->fh, pbdev->fh);
> diff --git a/hw/s390x/s390-pci-kvm.c b/hw/s390x/s390-pci-kvm.c
> new file mode 100644
> index 0000000000..8bfce9ef18
> --- /dev/null
> +++ b/hw/s390x/s390-pci-kvm.c
> @@ -0,0 +1,21 @@
> +/*
> + * s390 zPCI KVM interfaces
> + *
> + * Copyright 2022 IBM Corp.
> + * Author(s): Matthew Rosato <mjrosato@linux.ibm.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or (at
> + * your option) any later version. See the COPYING file in the top-level
> + * directory.
> + */
> +
> +#include "qemu/osdep.h"
> +
> +#include "kvm/kvm_s390x.h"
> +#include "hw/s390x/s390-pci-kvm.h"
> +#include "cpu_models.h"
> +
> +bool s390_pci_kvm_interp_allowed(void)
> +{
> +    return s390_has_feat(S390_FEAT_ZPCI_INTERP) && kvm_s390_get_zpci_op();
> +}

ZPCI is not supported by the PV currently but I do not see what could 
prevent it to be enable in the future.
As the code of QEMU zPCI is not PV compatible, I would like to add a 
check for PV.

... && !s390_is_pv())



> diff --git a/include/hw/s390x/s390-pci-bus.h b/include/hw/s390x/s390-pci-bus.h
> index da3cde2bb4..a9843dfe97 100644
> --- a/include/hw/s390x/s390-pci-bus.h
> +++ b/include/hw/s390x/s390-pci-bus.h
> @@ -350,6 +350,7 @@ struct S390PCIBusDevice {
>       IndAddr *indicator;
>       bool pci_unplug_request_processed;
>       bool unplug_requested;
> +    bool interp;
>       QTAILQ_ENTRY(S390PCIBusDevice) link;
>   };
>   

...snip...

-- 
Pierre Morel
IBM Lab Boeblingen