[PATCH v5 00/21] KVM: s390: enable zPCI for interpretive execution

[PATCH v5 00/21] KVM: s390: enable zPCI for interpretive execution

[Matthew Rosato]

Note: in this version, all IOMMU changes have been removed to be pursued
as a follow-on / in conjunction with IOMMUFD.  This series proposes to
add only exploitation of the interpretive execution facilities.

Enable interpretive execution of zPCI instructions + adapter interruption
forwarding for s390x KVM vfio-pci.  This is done by triggering a routine
when the VFIO group is associated with the KVM guest, transmitting to
firmware a special token (GISA designation) to enable that specific guest
for interpretive execution on that zPCI device.  Load/store interpreation
enablement is then controlled by userspace (based upon whether or not a
SHM bit is placed in the virtual function handle).  Adapter Event
Notification interpretation is controlled from userspace via a new KVM
ioctl.

By allowing intepretation of zPCI instructions and firmware delivery of
interrupts to guests, we can reduce the frequency of guest SIE exits for
zPCI.  

From the perspective of guest configuration, you passthrough zPCI devices
in the same manner as before, with intepretation support being used by
default if available in kernel+qemu.

Will reply with a link to the associated QEMU series.

Changelog v4->v5:
- Remove all IOMMU changes - this series will continue to use vfio type1
  for mapping; RPCIT enhancements will be pursued as a follow-on in
  coordination with the IOMMUFD project
- Remove most of the ops from the proposed KVM ioctl -- combine the notion
  of 'start' and 'enable interpretation' into a single operation that
  occurs in response to assigning a KVM association to the VFIO group
- a new kvm ioctl is still being used to register/unregister for adapter
  event notification forwarding.  But teach it to use pin_user_pages
  instead of gfn_to_page and perform accounting.
- Because we now attempt to enable interpretation for all devices (even
  if the virtual PCI device has a SHM bit on) avoid setting the GISA for
  the device and the ECB bits for the guest on hardware without SHM.
- found a bug with "s390/pci: stash associated GISA designation", namely
  that CLP SET PCI FN (disable) expects the gisa designation to always be
  0.  Now that we set the gisa earlier, vfio-pci can trip this when
  triggering restore for ISM.
- Add a single routine to determine if interpretation support is usable
  on this host, to be used in multiple spots.
- eliminated arch/s390/include/asm/kvm_pci.h, moving most elements into
  arch/s390/kvm/pci.h with one exception to asm/kvm_host.h

Matthew Rosato (21):
  s390/sclp: detect the zPCI load/store interpretation facility
  s390/sclp: detect the AISII facility
  s390/sclp: detect the AENI facility
  s390/sclp: detect the AISI facility
  s390/airq: pass more TPI info to airq handlers
  s390/airq: allow for airq structure that uses an input vector
  s390/pci: externalize the SIC operation controls and routine
  s390/pci: stash associated GISA designation
  s390/pci: stash dtsm and maxstbl
  KVM: s390: pci: add basic kvm_zdev structure
  KVM: s390: pci: do initial setup for AEN interpretation
  KVM: s390: pci: enable host forwarding of Adapter Event Notifications
  KVM: s390: mechanism to enable guest zPCI Interpretation
  KVM: s390: pci: provide routines for enabling/disabling interrupt
    forwarding
  KVM: s390: pci: add routines to start/stop interpretive execution
  KVM: vfio: add s390x hook to register KVM guest designation
  vfio-pci/zdev: add function handle to clp base capability
  vfio-pci/zdev: different maxstbl for interpreted devices
  KVM: s390: add KVM_S390_ZPCI_OP to manage guest zPCI devices
  KVM: s390: introduce CPU feature for zPCI Interpretation
  MAINTAINERS: additional files related kvm s390 pci passthrough

 Documentation/virt/kvm/api.rst   |  45 +++
 MAINTAINERS                      |   1 +
 arch/s390/include/asm/airq.h     |   7 +-
 arch/s390/include/asm/kvm_host.h |  16 +
 arch/s390/include/asm/pci.h      |  10 +
 arch/s390/include/asm/pci_clp.h  |   9 +-
 arch/s390/include/asm/pci_insn.h |  29 +-
 arch/s390/include/asm/sclp.h     |   4 +
 arch/s390/include/asm/tpi.h      |  13 +
 arch/s390/include/uapi/asm/kvm.h |   1 +
 arch/s390/kvm/Makefile           |   1 +
 arch/s390/kvm/interrupt.c        |  95 ++++-
 arch/s390/kvm/kvm-s390.c         |  81 +++-
 arch/s390/kvm/kvm-s390.h         |  10 +
 arch/s390/kvm/pci.c              | 666 +++++++++++++++++++++++++++++++
 arch/s390/kvm/pci.h              |  86 ++++
 arch/s390/pci/pci.c              |  15 +
 arch/s390/pci/pci_clp.c          |   7 +
 arch/s390/pci/pci_insn.c         |   4 +-
 arch/s390/pci/pci_irq.c          |  48 ++-
 drivers/s390/char/sclp_early.c   |   4 +
 drivers/s390/cio/airq.c          |  12 +-
 drivers/s390/cio/qdio_thinint.c  |   6 +-
 drivers/s390/crypto/ap_bus.c     |   9 +-
 drivers/s390/virtio/virtio_ccw.c |   6 +-
 drivers/vfio/pci/vfio_pci_zdev.c |  11 +-
 include/uapi/linux/kvm.h         |  31 ++
 include/uapi/linux/vfio_zdev.h   |   7 +
 virt/kvm/vfio.c                  |  35 +-
 29 files changed, 1216 insertions(+), 53 deletions(-)
 create mode 100644 arch/s390/kvm/pci.c
 create mode 100644 arch/s390/kvm/pci.h

-- 
2.27.0

[PATCH v5 01/21] s390/sclp: detect the zPCI load/store interpretation facility

[Matthew Rosato]

Detect the zPCI Load/Store Interpretation facility.

Reviewed-by: Eric Farman <farman@linux.ibm.com>
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
---
 arch/s390/include/asm/sclp.h   | 1 +
 drivers/s390/char/sclp_early.c | 1 +
 2 files changed, 2 insertions(+)

diff --git a/arch/s390/include/asm/sclp.h b/arch/s390/include/asm/sclp.h
index 04cb1e7582a6..06c0b5eb8ef5 100644
--- a/arch/s390/include/asm/sclp.h
+++ b/arch/s390/include/asm/sclp.h
@@ -87,6 +87,7 @@ struct sclp_info {
 	unsigned char has_diag318 : 1;
 	unsigned char has_sipl : 1;
 	unsigned char has_dirq : 1;
+	unsigned char has_zpci_lsi : 1;
 	unsigned int ibc;
 	unsigned int mtid;
 	unsigned int mtid_cp;
diff --git a/drivers/s390/char/sclp_early.c b/drivers/s390/char/sclp_early.c
index e9943a86c361..b88dd0da1231 100644
--- a/drivers/s390/char/sclp_early.c
+++ b/drivers/s390/char/sclp_early.c
@@ -45,6 +45,7 @@ static void __init sclp_early_facilities_detect(void)
 	sclp.has_gisaf = !!(sccb->fac118 & 0x08);
 	sclp.has_hvs = !!(sccb->fac119 & 0x80);
 	sclp.has_kss = !!(sccb->fac98 & 0x01);
+	sclp.has_zpci_lsi = !!(sccb->fac118 & 0x01);
 	if (sccb->fac85 & 0x02)
 		S390_lowcore.machine_flags |= MACHINE_FLAG_ESOP;
 	if (sccb->fac91 & 0x40)
-- 
2.27.0

[PATCH v5 02/21] s390/sclp: detect the AISII facility

[Matthew Rosato]

Detect the Adapter Interruption Source ID Interpretation facility.

Reviewed-by: Eric Farman <farman@linux.ibm.com>
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
---
 arch/s390/include/asm/sclp.h   | 1 +
 drivers/s390/char/sclp_early.c | 1 +
 2 files changed, 2 insertions(+)

diff --git a/arch/s390/include/asm/sclp.h b/arch/s390/include/asm/sclp.h
index 06c0b5eb8ef5..be14644fe113 100644
--- a/arch/s390/include/asm/sclp.h
+++ b/arch/s390/include/asm/sclp.h
@@ -88,6 +88,7 @@ struct sclp_info {
 	unsigned char has_sipl : 1;
 	unsigned char has_dirq : 1;
 	unsigned char has_zpci_lsi : 1;
+	unsigned char has_aisii : 1;
 	unsigned int ibc;
 	unsigned int mtid;
 	unsigned int mtid_cp;
diff --git a/drivers/s390/char/sclp_early.c b/drivers/s390/char/sclp_early.c
index b88dd0da1231..29fee179e197 100644
--- a/drivers/s390/char/sclp_early.c
+++ b/drivers/s390/char/sclp_early.c
@@ -45,6 +45,7 @@ static void __init sclp_early_facilities_detect(void)
 	sclp.has_gisaf = !!(sccb->fac118 & 0x08);
 	sclp.has_hvs = !!(sccb->fac119 & 0x80);
 	sclp.has_kss = !!(sccb->fac98 & 0x01);
+	sclp.has_aisii = !!(sccb->fac118 & 0x40);
 	sclp.has_zpci_lsi = !!(sccb->fac118 & 0x01);
 	if (sccb->fac85 & 0x02)
 		S390_lowcore.machine_flags |= MACHINE_FLAG_ESOP;
-- 
2.27.0

[PATCH v5 03/21] s390/sclp: detect the AENI facility

[Matthew Rosato]

Detect the Adapter Event Notification Interpretation facility.

Reviewed-by: Eric Farman <farman@linux.ibm.com>
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
---
 arch/s390/include/asm/sclp.h   | 1 +
 drivers/s390/char/sclp_early.c | 1 +
 2 files changed, 2 insertions(+)

diff --git a/arch/s390/include/asm/sclp.h b/arch/s390/include/asm/sclp.h
index be14644fe113..ff359df62154 100644
--- a/arch/s390/include/asm/sclp.h
+++ b/arch/s390/include/asm/sclp.h
@@ -89,6 +89,7 @@ struct sclp_info {
 	unsigned char has_dirq : 1;
 	unsigned char has_zpci_lsi : 1;
 	unsigned char has_aisii : 1;
+	unsigned char has_aeni : 1;
 	unsigned int ibc;
 	unsigned int mtid;
 	unsigned int mtid_cp;
diff --git a/drivers/s390/char/sclp_early.c b/drivers/s390/char/sclp_early.c
index 29fee179e197..e9af01b4c97a 100644
--- a/drivers/s390/char/sclp_early.c
+++ b/drivers/s390/char/sclp_early.c
@@ -46,6 +46,7 @@ static void __init sclp_early_facilities_detect(void)
 	sclp.has_hvs = !!(sccb->fac119 & 0x80);
 	sclp.has_kss = !!(sccb->fac98 & 0x01);
 	sclp.has_aisii = !!(sccb->fac118 & 0x40);
+	sclp.has_aeni = !!(sccb->fac118 & 0x20);
 	sclp.has_zpci_lsi = !!(sccb->fac118 & 0x01);
 	if (sccb->fac85 & 0x02)
 		S390_lowcore.machine_flags |= MACHINE_FLAG_ESOP;
-- 
2.27.0

[PATCH v5 04/21] s390/sclp: detect the AISI facility

[Matthew Rosato]

Detect the Adapter Interruption Suppression Interpretation facility.

Reviewed-by: Eric Farman <farman@linux.ibm.com>
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
---
 arch/s390/include/asm/sclp.h   | 1 +
 drivers/s390/char/sclp_early.c | 1 +
 2 files changed, 2 insertions(+)

diff --git a/arch/s390/include/asm/sclp.h b/arch/s390/include/asm/sclp.h
index ff359df62154..bdd365407176 100644
--- a/arch/s390/include/asm/sclp.h
+++ b/arch/s390/include/asm/sclp.h
@@ -90,6 +90,7 @@ struct sclp_info {
 	unsigned char has_zpci_lsi : 1;
 	unsigned char has_aisii : 1;
 	unsigned char has_aeni : 1;
+	unsigned char has_aisi : 1;
 	unsigned int ibc;
 	unsigned int mtid;
 	unsigned int mtid_cp;
diff --git a/drivers/s390/char/sclp_early.c b/drivers/s390/char/sclp_early.c
index e9af01b4c97a..c13e55cc4a5d 100644
--- a/drivers/s390/char/sclp_early.c
+++ b/drivers/s390/char/sclp_early.c
@@ -47,6 +47,7 @@ static void __init sclp_early_facilities_detect(void)
 	sclp.has_kss = !!(sccb->fac98 & 0x01);
 	sclp.has_aisii = !!(sccb->fac118 & 0x40);
 	sclp.has_aeni = !!(sccb->fac118 & 0x20);
+	sclp.has_aisi = !!(sccb->fac118 & 0x10);
 	sclp.has_zpci_lsi = !!(sccb->fac118 & 0x01);
 	if (sccb->fac85 & 0x02)
 		S390_lowcore.machine_flags |= MACHINE_FLAG_ESOP;
-- 
2.27.0

[PATCH v5 05/21] s390/airq: pass more TPI info to airq handlers

[Matthew Rosato]

A subsequent patch will introduce an airq handler that requires additional
TPI information beyond directed vs floating, so pass the entire tpi_info
structure via the handler.  Only pci actually uses this information today,
for the other airq handlers this is effectively a no-op.

Reviewed-by: Eric Farman <farman@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Acked-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Acked-by: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
---
 arch/s390/include/asm/airq.h     | 3 ++-
 arch/s390/kvm/interrupt.c        | 4 +++-
 arch/s390/pci/pci_irq.c          | 9 +++++++--
 drivers/s390/cio/airq.c          | 2 +-
 drivers/s390/cio/qdio_thinint.c  | 6 ++++--
 drivers/s390/crypto/ap_bus.c     | 9 ++++++---
 drivers/s390/virtio/virtio_ccw.c | 4 +++-
 7 files changed, 26 insertions(+), 11 deletions(-)

diff --git a/arch/s390/include/asm/airq.h b/arch/s390/include/asm/airq.h
index 01936fdfaddb..7918a7d09028 100644
--- a/arch/s390/include/asm/airq.h
+++ b/arch/s390/include/asm/airq.h
@@ -12,10 +12,11 @@
 
 #include <linux/bit_spinlock.h>
 #include <linux/dma-mapping.h>
+#include <asm/tpi.h>
 
 struct airq_struct {
 	struct hlist_node list;		/* Handler queueing. */
-	void (*handler)(struct airq_struct *airq, bool floating);
+	void (*handler)(struct airq_struct *airq, struct tpi_info *tpi_info);
 	u8 *lsi_ptr;			/* Local-Summary-Indicator pointer */
 	u8 lsi_mask;			/* Local-Summary-Indicator mask */
 	u8 isc;				/* Interrupt-subclass */
diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
index 9b30beac904d..250119a26c60 100644
--- a/arch/s390/kvm/interrupt.c
+++ b/arch/s390/kvm/interrupt.c
@@ -28,6 +28,7 @@
 #include <asm/switch_to.h>
 #include <asm/nmi.h>
 #include <asm/airq.h>
+#include <asm/tpi.h>
 #include "kvm-s390.h"
 #include "gaccess.h"
 #include "trace-s390.h"
@@ -3311,7 +3312,8 @@ int kvm_s390_gisc_unregister(struct kvm *kvm, u32 gisc)
 }
 EXPORT_SYMBOL_GPL(kvm_s390_gisc_unregister);
 
-static void gib_alert_irq_handler(struct airq_struct *airq, bool floating)
+static void gib_alert_irq_handler(struct airq_struct *airq,
+				  struct tpi_info *tpi_info)
 {
 	inc_irq_stat(IRQIO_GAL);
 	process_gib_alert_list();
diff --git a/arch/s390/pci/pci_irq.c b/arch/s390/pci/pci_irq.c
index 500cd2dbdf53..b805c75252ed 100644
--- a/arch/s390/pci/pci_irq.c
+++ b/arch/s390/pci/pci_irq.c
@@ -11,6 +11,7 @@
 
 #include <asm/isc.h>
 #include <asm/airq.h>
+#include <asm/tpi.h>
 
 static enum {FLOATING, DIRECTED} irq_delivery;
 
@@ -216,8 +217,11 @@ static void zpci_handle_fallback_irq(void)
 	}
 }
 
-static void zpci_directed_irq_handler(struct airq_struct *airq, bool floating)
+static void zpci_directed_irq_handler(struct airq_struct *airq,
+				      struct tpi_info *tpi_info)
 {
+	bool floating = !tpi_info->directed_irq;
+
 	if (floating) {
 		inc_irq_stat(IRQIO_PCF);
 		zpci_handle_fallback_irq();
@@ -227,7 +231,8 @@ static void zpci_directed_irq_handler(struct airq_struct *airq, bool floating)
 	}
 }
 
-static void zpci_floating_irq_handler(struct airq_struct *airq, bool floating)
+static void zpci_floating_irq_handler(struct airq_struct *airq,
+				      struct tpi_info *tpi_info)
 {
 	unsigned long si, ai;
 	struct airq_iv *aibv;
diff --git a/drivers/s390/cio/airq.c b/drivers/s390/cio/airq.c
index c0ed364bf446..230eb5b5b64e 100644
--- a/drivers/s390/cio/airq.c
+++ b/drivers/s390/cio/airq.c
@@ -99,7 +99,7 @@ static irqreturn_t do_airq_interrupt(int irq, void *dummy)
 	rcu_read_lock();
 	hlist_for_each_entry_rcu(airq, head, list)
 		if ((*airq->lsi_ptr & airq->lsi_mask) != 0)
-			airq->handler(airq, !tpi_info->directed_irq);
+			airq->handler(airq, tpi_info);
 	rcu_read_unlock();
 
 	return IRQ_HANDLED;
diff --git a/drivers/s390/cio/qdio_thinint.c b/drivers/s390/cio/qdio_thinint.c
index 8e09bf3a2fcd..9b9335dd06db 100644
--- a/drivers/s390/cio/qdio_thinint.c
+++ b/drivers/s390/cio/qdio_thinint.c
@@ -15,6 +15,7 @@
 #include <asm/qdio.h>
 #include <asm/airq.h>
 #include <asm/isc.h>
+#include <asm/tpi.h>
 
 #include "cio.h"
 #include "ioasm.h"
@@ -93,9 +94,10 @@ static inline u32 clear_shared_ind(void)
 /**
  * tiqdio_thinint_handler - thin interrupt handler for qdio
  * @airq: pointer to adapter interrupt descriptor
- * @floating: flag to recognize floating vs. directed interrupts (unused)
+ * @tpi_info: interrupt information (e.g. floating vs directed -- unused)
  */
-static void tiqdio_thinint_handler(struct airq_struct *airq, bool floating)
+static void tiqdio_thinint_handler(struct airq_struct *airq,
+				   struct tpi_info *tpi_info)
 {
 	u64 irq_time = S390_lowcore.int_clock;
 	u32 si_used = clear_shared_ind();
diff --git a/drivers/s390/crypto/ap_bus.c b/drivers/s390/crypto/ap_bus.c
index fdf16cb70881..6cb8a2802ef8 100644
--- a/drivers/s390/crypto/ap_bus.c
+++ b/drivers/s390/crypto/ap_bus.c
@@ -27,6 +27,7 @@
 #include <linux/kthread.h>
 #include <linux/mutex.h>
 #include <asm/airq.h>
+#include <asm/tpi.h>
 #include <linux/atomic.h>
 #include <asm/isc.h>
 #include <linux/hrtimer.h>
@@ -131,7 +132,8 @@ static int ap_max_adapter_id = 63;
 static struct bus_type ap_bus_type;
 
 /* Adapter interrupt definitions */
-static void ap_interrupt_handler(struct airq_struct *airq, bool floating);
+static void ap_interrupt_handler(struct airq_struct *airq,
+				 struct tpi_info *tpi_info);
 
 static bool ap_irq_flag;
 
@@ -452,9 +454,10 @@ static enum hrtimer_restart ap_poll_timeout(struct hrtimer *unused)
 /**
  * ap_interrupt_handler() - Schedule ap_tasklet on interrupt
  * @airq: pointer to adapter interrupt descriptor
- * @floating: ignored
+ * @tpi_info: ignored
  */
-static void ap_interrupt_handler(struct airq_struct *airq, bool floating)
+static void ap_interrupt_handler(struct airq_struct *airq,
+				 struct tpi_info *tpi_info)
 {
 	inc_irq_stat(IRQIO_APB);
 	tasklet_schedule(&ap_tasklet);
diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c
index d35e7a3f7067..52c376d15978 100644
--- a/drivers/s390/virtio/virtio_ccw.c
+++ b/drivers/s390/virtio/virtio_ccw.c
@@ -33,6 +33,7 @@
 #include <asm/virtio-ccw.h>
 #include <asm/isc.h>
 #include <asm/airq.h>
+#include <asm/tpi.h>
 
 /*
  * virtio related functions
@@ -203,7 +204,8 @@ static void drop_airq_indicator(struct virtqueue *vq, struct airq_info *info)
 	write_unlock_irqrestore(&info->lock, flags);
 }
 
-static void virtio_airq_handler(struct airq_struct *airq, bool floating)
+static void virtio_airq_handler(struct airq_struct *airq,
+				struct tpi_info *tpi_info)
 {
 	struct airq_info *info = container_of(airq, struct airq_info, airq);
 	unsigned long ai;
-- 
2.27.0

[PATCH v5 06/21] s390/airq: allow for airq structure that uses an input vector

[Matthew Rosato]

When doing device passthrough where interrupts are being forwarded from
host to guest, we wish to use a pinned section of guest memory as the
vector (the same memory used by the guest as the vector). To accomplish
this, add a new parameter for airq_iv_create which allows passing an
existing vector to be used instead of allocating a new one. The caller
is responsible for ensuring the vector is pinned in memory as well as for
unpinning the memory when the vector is no longer needed.

A subsequent patch will use this new parameter for zPCI interpretation.

Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Acked-by: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
---
 arch/s390/include/asm/airq.h     |  4 +++-
 arch/s390/pci/pci_irq.c          |  8 ++++----
 drivers/s390/cio/airq.c          | 10 +++++++---
 drivers/s390/virtio/virtio_ccw.c |  2 +-
 4 files changed, 15 insertions(+), 9 deletions(-)

diff --git a/arch/s390/include/asm/airq.h b/arch/s390/include/asm/airq.h
index 7918a7d09028..e82e5626e139 100644
--- a/arch/s390/include/asm/airq.h
+++ b/arch/s390/include/asm/airq.h
@@ -47,8 +47,10 @@ struct airq_iv {
 #define AIRQ_IV_PTR		4	/* Allocate the ptr array */
 #define AIRQ_IV_DATA		8	/* Allocate the data array */
 #define AIRQ_IV_CACHELINE	16	/* Cacheline alignment for the vector */
+#define AIRQ_IV_GUESTVEC	32	/* Vector is a pinned guest page */
 
-struct airq_iv *airq_iv_create(unsigned long bits, unsigned long flags);
+struct airq_iv *airq_iv_create(unsigned long bits, unsigned long flags,
+			       unsigned long *vec);
 void airq_iv_release(struct airq_iv *iv);
 unsigned long airq_iv_alloc(struct airq_iv *iv, unsigned long num);
 void airq_iv_free(struct airq_iv *iv, unsigned long bit, unsigned long num);
diff --git a/arch/s390/pci/pci_irq.c b/arch/s390/pci/pci_irq.c
index b805c75252ed..87c7d121c255 100644
--- a/arch/s390/pci/pci_irq.c
+++ b/arch/s390/pci/pci_irq.c
@@ -296,7 +296,7 @@ int arch_setup_msi_irqs(struct pci_dev *pdev, int nvec, int type)
 		zdev->aisb = bit;
 
 		/* Create adapter interrupt vector */
-		zdev->aibv = airq_iv_create(msi_vecs, AIRQ_IV_DATA | AIRQ_IV_BITLOCK);
+		zdev->aibv = airq_iv_create(msi_vecs, AIRQ_IV_DATA | AIRQ_IV_BITLOCK, NULL);
 		if (!zdev->aibv)
 			return -ENOMEM;
 
@@ -419,7 +419,7 @@ static int __init zpci_directed_irq_init(void)
 	union zpci_sic_iib iib = {{0}};
 	unsigned int cpu;
 
-	zpci_sbv = airq_iv_create(num_possible_cpus(), 0);
+	zpci_sbv = airq_iv_create(num_possible_cpus(), 0, NULL);
 	if (!zpci_sbv)
 		return -ENOMEM;
 
@@ -441,7 +441,7 @@ static int __init zpci_directed_irq_init(void)
 		zpci_ibv[cpu] = airq_iv_create(cache_line_size() * BITS_PER_BYTE,
 					       AIRQ_IV_DATA |
 					       AIRQ_IV_CACHELINE |
-					       (!cpu ? AIRQ_IV_ALLOC : 0));
+					       (!cpu ? AIRQ_IV_ALLOC : 0), NULL);
 		if (!zpci_ibv[cpu])
 			return -ENOMEM;
 	}
@@ -458,7 +458,7 @@ static int __init zpci_floating_irq_init(void)
 	if (!zpci_ibv)
 		return -ENOMEM;
 
-	zpci_sbv = airq_iv_create(ZPCI_NR_DEVICES, AIRQ_IV_ALLOC);
+	zpci_sbv = airq_iv_create(ZPCI_NR_DEVICES, AIRQ_IV_ALLOC, NULL);
 	if (!zpci_sbv)
 		goto out_free;
 
diff --git a/drivers/s390/cio/airq.c b/drivers/s390/cio/airq.c
index 230eb5b5b64e..34967e67249e 100644
--- a/drivers/s390/cio/airq.c
+++ b/drivers/s390/cio/airq.c
@@ -122,10 +122,12 @@ static inline unsigned long iv_size(unsigned long bits)
  * airq_iv_create - create an interrupt vector
  * @bits: number of bits in the interrupt vector
  * @flags: allocation flags
+ * @vec: pointer to pinned guest memory if AIRQ_IV_GUESTVEC
  *
  * Returns a pointer to an interrupt vector structure
  */
-struct airq_iv *airq_iv_create(unsigned long bits, unsigned long flags)
+struct airq_iv *airq_iv_create(unsigned long bits, unsigned long flags,
+			       unsigned long *vec)
 {
 	struct airq_iv *iv;
 	unsigned long size;
@@ -146,6 +148,8 @@ struct airq_iv *airq_iv_create(unsigned long bits, unsigned long flags)
 					     &iv->vector_dma);
 		if (!iv->vector)
 			goto out_free;
+	} else if (flags & AIRQ_IV_GUESTVEC) {
+		iv->vector = vec;
 	} else {
 		iv->vector = cio_dma_zalloc(size);
 		if (!iv->vector)
@@ -185,7 +189,7 @@ struct airq_iv *airq_iv_create(unsigned long bits, unsigned long flags)
 	kfree(iv->avail);
 	if (iv->flags & AIRQ_IV_CACHELINE && iv->vector)
 		dma_pool_free(airq_iv_cache, iv->vector, iv->vector_dma);
-	else
+	else if (!(iv->flags & AIRQ_IV_GUESTVEC))
 		cio_dma_free(iv->vector, size);
 	kfree(iv);
 out:
@@ -204,7 +208,7 @@ void airq_iv_release(struct airq_iv *iv)
 	kfree(iv->bitlock);
 	if (iv->flags & AIRQ_IV_CACHELINE)
 		dma_pool_free(airq_iv_cache, iv->vector, iv->vector_dma);
-	else
+	else if (!(iv->flags & AIRQ_IV_GUESTVEC))
 		cio_dma_free(iv->vector, iv_size(iv->bits));
 	kfree(iv->avail);
 	kfree(iv);
diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c
index 52c376d15978..410498d693f8 100644
--- a/drivers/s390/virtio/virtio_ccw.c
+++ b/drivers/s390/virtio/virtio_ccw.c
@@ -241,7 +241,7 @@ static struct airq_info *new_airq_info(int index)
 		return NULL;
 	rwlock_init(&info->lock);
 	info->aiv = airq_iv_create(VIRTIO_IV_BITS, AIRQ_IV_ALLOC | AIRQ_IV_PTR
-				   | AIRQ_IV_CACHELINE);
+				   | AIRQ_IV_CACHELINE, NULL);
 	if (!info->aiv) {
 		kfree(info);
 		return NULL;
-- 
2.27.0

[PATCH v5 07/21] s390/pci: externalize the SIC operation controls and routine

[Matthew Rosato]

A subsequent patch will be issuing SIC from KVM -- export the necessary
routine and make the operation control definitions available from a header.
Because the routine will now be exported, let's rename __zpci_set_irq_ctrl
to zpci_set_irq_ctrl and get rid of the zero'd iib wrapper function of
the same name.

Reviewed-by: Niklas Schnelle <schnelle@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
---
 arch/s390/include/asm/pci_insn.h | 17 +++++++++--------
 arch/s390/pci/pci_insn.c         |  3 ++-
 arch/s390/pci/pci_irq.c          | 26 ++++++++++++--------------
 3 files changed, 23 insertions(+), 23 deletions(-)

diff --git a/arch/s390/include/asm/pci_insn.h b/arch/s390/include/asm/pci_insn.h
index 61cf9531f68f..5331082fa516 100644
--- a/arch/s390/include/asm/pci_insn.h
+++ b/arch/s390/include/asm/pci_insn.h
@@ -98,6 +98,14 @@ struct zpci_fib {
 	u32 gd;
 } __packed __aligned(8);
 
+/* Set Interruption Controls Operation Controls  */
+#define	SIC_IRQ_MODE_ALL		0
+#define	SIC_IRQ_MODE_SINGLE		1
+#define	SIC_IRQ_MODE_DIRECT		4
+#define	SIC_IRQ_MODE_D_ALL		16
+#define	SIC_IRQ_MODE_D_SINGLE		17
+#define	SIC_IRQ_MODE_SET_CPU		18
+
 /* directed interruption information block */
 struct zpci_diib {
 	u32 : 1;
@@ -134,13 +142,6 @@ int __zpci_store(u64 data, u64 req, u64 offset);
 int zpci_store(const volatile void __iomem *addr, u64 data, unsigned long len);
 int __zpci_store_block(const u64 *data, u64 req, u64 offset);
 void zpci_barrier(void);
-int __zpci_set_irq_ctrl(u16 ctl, u8 isc, union zpci_sic_iib *iib);
-
-static inline int zpci_set_irq_ctrl(u16 ctl, u8 isc)
-{
-	union zpci_sic_iib iib = {{0}};
-
-	return __zpci_set_irq_ctrl(ctl, isc, &iib);
-}
+int zpci_set_irq_ctrl(u16 ctl, u8 isc, union zpci_sic_iib *iib);
 
 #endif
diff --git a/arch/s390/pci/pci_insn.c b/arch/s390/pci/pci_insn.c
index 1710d006ee93..4c6967b73932 100644
--- a/arch/s390/pci/pci_insn.c
+++ b/arch/s390/pci/pci_insn.c
@@ -98,7 +98,7 @@ int zpci_refresh_trans(u64 fn, u64 addr, u64 range)
 }
 
 /* Set Interruption Controls */
-int __zpci_set_irq_ctrl(u16 ctl, u8 isc, union zpci_sic_iib *iib)
+int zpci_set_irq_ctrl(u16 ctl, u8 isc, union zpci_sic_iib *iib)
 {
 	if (!test_facility(72))
 		return -EIO;
@@ -109,6 +109,7 @@ int __zpci_set_irq_ctrl(u16 ctl, u8 isc, union zpci_sic_iib *iib)
 
 	return 0;
 }
+EXPORT_SYMBOL_GPL(zpci_set_irq_ctrl);
 
 /* PCI Load */
 static inline int ____pcilg(u64 *data, u64 req, u64 offset, u8 *status)
diff --git a/arch/s390/pci/pci_irq.c b/arch/s390/pci/pci_irq.c
index 87c7d121c255..f2b3145b6697 100644
--- a/arch/s390/pci/pci_irq.c
+++ b/arch/s390/pci/pci_irq.c
@@ -15,13 +15,6 @@
 
 static enum {FLOATING, DIRECTED} irq_delivery;
 
-#define	SIC_IRQ_MODE_ALL		0
-#define	SIC_IRQ_MODE_SINGLE		1
-#define	SIC_IRQ_MODE_DIRECT		4
-#define	SIC_IRQ_MODE_D_ALL		16
-#define	SIC_IRQ_MODE_D_SINGLE		17
-#define	SIC_IRQ_MODE_SET_CPU		18
-
 /*
  * summary bit vector
  * FLOATING - summary bit per function
@@ -154,6 +147,7 @@ static struct irq_chip zpci_irq_chip = {
 static void zpci_handle_cpu_local_irq(bool rescan)
 {
 	struct airq_iv *dibv = zpci_ibv[smp_processor_id()];
+	union zpci_sic_iib iib = {{0}};
 	unsigned long bit;
 	int irqs_on = 0;
 
@@ -165,7 +159,7 @@ static void zpci_handle_cpu_local_irq(bool rescan)
 				/* End of second scan with interrupts on. */
 				break;
 			/* First scan complete, reenable interrupts. */
-			if (zpci_set_irq_ctrl(SIC_IRQ_MODE_D_SINGLE, PCI_ISC))
+			if (zpci_set_irq_ctrl(SIC_IRQ_MODE_D_SINGLE, PCI_ISC, &iib))
 				break;
 			bit = 0;
 			continue;
@@ -193,6 +187,7 @@ static void zpci_handle_remote_irq(void *data)
 static void zpci_handle_fallback_irq(void)
 {
 	struct cpu_irq_data *cpu_data;
+	union zpci_sic_iib iib = {{0}};
 	unsigned long cpu;
 	int irqs_on = 0;
 
@@ -203,7 +198,7 @@ static void zpci_handle_fallback_irq(void)
 				/* End of second scan with interrupts on. */
 				break;
 			/* First scan complete, reenable interrupts. */
-			if (zpci_set_irq_ctrl(SIC_IRQ_MODE_SINGLE, PCI_ISC))
+			if (zpci_set_irq_ctrl(SIC_IRQ_MODE_SINGLE, PCI_ISC, &iib))
 				break;
 			cpu = 0;
 			continue;
@@ -234,6 +229,7 @@ static void zpci_directed_irq_handler(struct airq_struct *airq,
 static void zpci_floating_irq_handler(struct airq_struct *airq,
 				      struct tpi_info *tpi_info)
 {
+	union zpci_sic_iib iib = {{0}};
 	unsigned long si, ai;
 	struct airq_iv *aibv;
 	int irqs_on = 0;
@@ -247,7 +243,7 @@ static void zpci_floating_irq_handler(struct airq_struct *airq,
 				/* End of second scan with interrupts on. */
 				break;
 			/* First scan complete, reenable interrupts. */
-			if (zpci_set_irq_ctrl(SIC_IRQ_MODE_SINGLE, PCI_ISC))
+			if (zpci_set_irq_ctrl(SIC_IRQ_MODE_SINGLE, PCI_ISC, &iib))
 				break;
 			si = 0;
 			continue;
@@ -407,11 +403,12 @@ static struct airq_struct zpci_airq = {
 static void __init cpu_enable_directed_irq(void *unused)
 {
 	union zpci_sic_iib iib = {{0}};
+	union zpci_sic_iib ziib = {{0}};
 
 	iib.cdiib.dibv_addr = (u64) zpci_ibv[smp_processor_id()]->vector;
 
-	__zpci_set_irq_ctrl(SIC_IRQ_MODE_SET_CPU, 0, &iib);
-	zpci_set_irq_ctrl(SIC_IRQ_MODE_D_SINGLE, PCI_ISC);
+	zpci_set_irq_ctrl(SIC_IRQ_MODE_SET_CPU, 0, &iib);
+	zpci_set_irq_ctrl(SIC_IRQ_MODE_D_SINGLE, PCI_ISC, &ziib);
 }
 
 static int __init zpci_directed_irq_init(void)
@@ -426,7 +423,7 @@ static int __init zpci_directed_irq_init(void)
 	iib.diib.isc = PCI_ISC;
 	iib.diib.nr_cpus = num_possible_cpus();
 	iib.diib.disb_addr = virt_to_phys(zpci_sbv->vector);
-	__zpci_set_irq_ctrl(SIC_IRQ_MODE_DIRECT, 0, &iib);
+	zpci_set_irq_ctrl(SIC_IRQ_MODE_DIRECT, 0, &iib);
 
 	zpci_ibv = kcalloc(num_possible_cpus(), sizeof(*zpci_ibv),
 			   GFP_KERNEL);
@@ -471,6 +468,7 @@ static int __init zpci_floating_irq_init(void)
 
 int __init zpci_irq_init(void)
 {
+	union zpci_sic_iib iib = {{0}};
 	int rc;
 
 	irq_delivery = sclp.has_dirq ? DIRECTED : FLOATING;
@@ -502,7 +500,7 @@ int __init zpci_irq_init(void)
 	 * Enable floating IRQs (with suppression after one IRQ). When using
 	 * directed IRQs this enables the fallback path.
 	 */
-	zpci_set_irq_ctrl(SIC_IRQ_MODE_SINGLE, PCI_ISC);
+	zpci_set_irq_ctrl(SIC_IRQ_MODE_SINGLE, PCI_ISC, &iib);
 
 	return 0;
 out_airq:
-- 
2.27.0

[PATCH v5 08/21] s390/pci: stash associated GISA designation

[Matthew Rosato]

For passthrough devices, we will need to know the GISA designation of the
guest if interpretation facilities are to be used.  Setup to stash this in
the zdev and set a default of 0 (no GISA designation) for now; a subsequent
patch will set a valid GISA designation for passthrough devices.
Also, extend mpcific routines to specify this stashed designation as part
of the mpcific command.

Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
---
 arch/s390/include/asm/pci.h     | 1 +
 arch/s390/include/asm/pci_clp.h | 3 ++-
 arch/s390/pci/pci.c             | 6 ++++++
 arch/s390/pci/pci_clp.c         | 5 +++++
 arch/s390/pci/pci_irq.c         | 5 +++++
 5 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h
index fdb9745ee998..42a4a312a6dd 100644
--- a/arch/s390/include/asm/pci.h
+++ b/arch/s390/include/asm/pci.h
@@ -123,6 +123,7 @@ struct zpci_dev {
 	enum zpci_state state;
 	u32		fid;		/* function ID, used by sclp */
 	u32		fh;		/* function handle, used by insn's */
+	u32		gisa;		/* GISA designation for passthrough */
 	u16		vfn;		/* virtual function number */
 	u16		pchid;		/* physical channel ID */
 	u8		pfgid;		/* function group ID */
diff --git a/arch/s390/include/asm/pci_clp.h b/arch/s390/include/asm/pci_clp.h
index 1f4b666e85ee..f3286bc5ba6e 100644
--- a/arch/s390/include/asm/pci_clp.h
+++ b/arch/s390/include/asm/pci_clp.h
@@ -173,7 +173,8 @@ struct clp_req_set_pci {
 	u16 reserved2;
 	u8 oc;				/* operation controls */
 	u8 ndas;			/* number of dma spaces */
-	u64 reserved3;
+	u32 reserved3;
+	u32 gisa;			/* GISA designation */
 } __packed;
 
 /* Set PCI function response */
diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c
index e563cb65c0c4..a86cd1cbb80e 100644
--- a/arch/s390/pci/pci.c
+++ b/arch/s390/pci/pci.c
@@ -120,6 +120,7 @@ int zpci_register_ioat(struct zpci_dev *zdev, u8 dmaas,
 	fib.pba = base;
 	fib.pal = limit;
 	fib.iota = iota | ZPCI_IOTA_RTTO_FLAG;
+	fib.gd = zdev->gisa;
 	cc = zpci_mod_fc(req, &fib, &status);
 	if (cc)
 		zpci_dbg(3, "reg ioat fid:%x, cc:%d, status:%d\n", zdev->fid, cc, status);
@@ -133,6 +134,8 @@ int zpci_unregister_ioat(struct zpci_dev *zdev, u8 dmaas)
 	struct zpci_fib fib = {0};
 	u8 cc, status;
 
+	fib.gd = zdev->gisa;
+
 	cc = zpci_mod_fc(req, &fib, &status);
 	if (cc)
 		zpci_dbg(3, "unreg ioat fid:%x, cc:%d, status:%d\n", zdev->fid, cc, status);
@@ -160,6 +163,7 @@ int zpci_fmb_enable_device(struct zpci_dev *zdev)
 	atomic64_set(&zdev->unmapped_pages, 0);
 
 	fib.fmb_addr = virt_to_phys(zdev->fmb);
+	fib.gd = zdev->gisa;
 	cc = zpci_mod_fc(req, &fib, &status);
 	if (cc) {
 		kmem_cache_free(zdev_fmb_cache, zdev->fmb);
@@ -178,6 +182,8 @@ int zpci_fmb_disable_device(struct zpci_dev *zdev)
 	if (!zdev->fmb)
 		return -EINVAL;
 
+	fib.gd = zdev->gisa;
+
 	/* Function measurement is disabled if fmb address is zero */
 	cc = zpci_mod_fc(req, &fib, &status);
 	if (cc == 3) /* Function already gone. */
diff --git a/arch/s390/pci/pci_clp.c b/arch/s390/pci/pci_clp.c
index 1057d7af4a55..deed35edea14 100644
--- a/arch/s390/pci/pci_clp.c
+++ b/arch/s390/pci/pci_clp.c
@@ -229,12 +229,16 @@ static int clp_set_pci_fn(struct zpci_dev *zdev, u32 *fh, u8 nr_dma_as, u8 comma
 {
 	struct clp_req_rsp_set_pci *rrb;
 	int rc, retries = 100;
+	u32 gisa = 0;
 
 	*fh = 0;
 	rrb = clp_alloc_block(GFP_KERNEL);
 	if (!rrb)
 		return -ENOMEM;
 
+	if (command != CLP_SET_DISABLE_PCI_FN)
+		gisa = zdev->gisa;
+
 	do {
 		memset(rrb, 0, sizeof(*rrb));
 		rrb->request.hdr.len = sizeof(rrb->request);
@@ -243,6 +247,7 @@ static int clp_set_pci_fn(struct zpci_dev *zdev, u32 *fh, u8 nr_dma_as, u8 comma
 		rrb->request.fh = zdev->fh;
 		rrb->request.oc = command;
 		rrb->request.ndas = nr_dma_as;
+		rrb->request.gisa = gisa;
 
 		rc = clp_req(rrb, CLP_LPS_PCI);
 		if (rrb->response.hdr.rsp == CLP_RC_SETPCIFN_BUSY) {
diff --git a/arch/s390/pci/pci_irq.c b/arch/s390/pci/pci_irq.c
index f2b3145b6697..a2b42a63a53b 100644
--- a/arch/s390/pci/pci_irq.c
+++ b/arch/s390/pci/pci_irq.c
@@ -43,6 +43,7 @@ static int zpci_set_airq(struct zpci_dev *zdev)
 	fib.fmt0.aibvo = 0;	/* each zdev has its own interrupt vector */
 	fib.fmt0.aisb = virt_to_phys(zpci_sbv->vector) + (zdev->aisb / 64) * 8;
 	fib.fmt0.aisbo = zdev->aisb & 63;
+	fib.gd = zdev->gisa;
 
 	return zpci_mod_fc(req, &fib, &status) ? -EIO : 0;
 }
@@ -54,6 +55,8 @@ static int zpci_clear_airq(struct zpci_dev *zdev)
 	struct zpci_fib fib = {0};
 	u8 cc, status;
 
+	fib.gd = zdev->gisa;
+
 	cc = zpci_mod_fc(req, &fib, &status);
 	if (cc == 3 || (cc == 1 && status == 24))
 		/* Function already gone or IRQs already deregistered. */
@@ -72,6 +75,7 @@ static int zpci_set_directed_irq(struct zpci_dev *zdev)
 	fib.fmt = 1;
 	fib.fmt1.noi = zdev->msi_nr_irqs;
 	fib.fmt1.dibvo = zdev->msi_first_bit;
+	fib.gd = zdev->gisa;
 
 	return zpci_mod_fc(req, &fib, &status) ? -EIO : 0;
 }
@@ -84,6 +88,7 @@ static int zpci_clear_directed_irq(struct zpci_dev *zdev)
 	u8 cc, status;
 
 	fib.fmt = 1;
+	fib.gd = zdev->gisa;
 	cc = zpci_mod_fc(req, &fib, &status);
 	if (cc == 3 || (cc == 1 && status == 24))
 		/* Function already gone or IRQs already deregistered. */
-- 
2.27.0

[PATCH v5 09/21] s390/pci: stash dtsm and maxstbl

[Matthew Rosato]

Store information about what IOAT designation types are supported by
underlying hardware as well as the largest store block size allowed.
These values will be needed by passthrough.

Reviewed-by: Niklas Schnelle <schnelle@linux.ibm.com>
Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>
Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
---
 arch/s390/include/asm/pci.h     | 2 ++
 arch/s390/include/asm/pci_clp.h | 6 ++++--
 arch/s390/pci/pci_clp.c         | 2 ++
 3 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h
index 42a4a312a6dd..4c5b8fbc2079 100644
--- a/arch/s390/include/asm/pci.h
+++ b/arch/s390/include/asm/pci.h
@@ -126,9 +126,11 @@ struct zpci_dev {
 	u32		gisa;		/* GISA designation for passthrough */
 	u16		vfn;		/* virtual function number */
 	u16		pchid;		/* physical channel ID */
+	u16		maxstbl;	/* Maximum store block size */
 	u8		pfgid;		/* function group ID */
 	u8		pft;		/* pci function type */
 	u8		port;
+	u8		dtsm;		/* Supported DT mask */
 	u8		rid_available	: 1;
 	u8		has_hp_slot	: 1;
 	u8		has_resources	: 1;
diff --git a/arch/s390/include/asm/pci_clp.h b/arch/s390/include/asm/pci_clp.h
index f3286bc5ba6e..d6189ed14f84 100644
--- a/arch/s390/include/asm/pci_clp.h
+++ b/arch/s390/include/asm/pci_clp.h
@@ -153,9 +153,11 @@ struct clp_rsp_query_pci_grp {
 	u8			:  6;
 	u8 frame		:  1;
 	u8 refresh		:  1;	/* TLB refresh mode */
-	u16 reserved2;
+	u16			:  3;
+	u16 maxstbl		: 13;	/* Maximum store block size */
 	u16 mui;
-	u16			: 16;
+	u8 dtsm;			/* Supported DT mask */
+	u8 reserved3;
 	u16 maxfaal;
 	u16			:  4;
 	u16 dnoi		: 12;
diff --git a/arch/s390/pci/pci_clp.c b/arch/s390/pci/pci_clp.c
index deed35edea14..1edb07ceb766 100644
--- a/arch/s390/pci/pci_clp.c
+++ b/arch/s390/pci/pci_clp.c
@@ -106,6 +106,8 @@ static void clp_store_query_pci_fngrp(struct zpci_dev *zdev,
 	zdev->max_msi = response->noi;
 	zdev->fmb_update = response->mui;
 	zdev->version = response->version;
+	zdev->maxstbl = response->maxstbl;
+	zdev->dtsm = response->dtsm;
 
 	switch (response->version) {
 	case 1:
-- 
2.27.0

[PATCH v5 10/21] KVM: s390: pci: add basic kvm_zdev structure

[Matthew Rosato]

This structure will be used to carry kvm passthrough information related to
zPCI devices.

Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
---
 arch/s390/include/asm/pci.h |  3 +++
 arch/s390/kvm/Makefile      |  1 +
 arch/s390/kvm/pci.c         | 38 +++++++++++++++++++++++++++++++++++++
 arch/s390/kvm/pci.h         | 21 ++++++++++++++++++++
 4 files changed, 63 insertions(+)
 create mode 100644 arch/s390/kvm/pci.c
 create mode 100644 arch/s390/kvm/pci.h

diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h
index 4c5b8fbc2079..9eb20cebaa18 100644
--- a/arch/s390/include/asm/pci.h
+++ b/arch/s390/include/asm/pci.h
@@ -97,6 +97,7 @@ struct zpci_bar_struct {
 };
 
 struct s390_domain;
+struct kvm_zdev;
 
 #define ZPCI_FUNCTIONS_PER_BUS 256
 struct zpci_bus {
@@ -190,6 +191,8 @@ struct zpci_dev {
 	struct dentry	*debugfs_dev;
 
 	struct s390_domain *s390_domain; /* s390 IOMMU domain data */
+
+	struct kvm_zdev *kzdev; /* passthrough data */
 };
 
 static inline bool zdev_enabled(struct zpci_dev *zdev)
diff --git a/arch/s390/kvm/Makefile b/arch/s390/kvm/Makefile
index 26f4a74e5ce4..00cf6853d93f 100644
--- a/arch/s390/kvm/Makefile
+++ b/arch/s390/kvm/Makefile
@@ -10,4 +10,5 @@ ccflags-y := -Ivirt/kvm -Iarch/s390/kvm
 kvm-y += kvm-s390.o intercept.o interrupt.o priv.o sigp.o
 kvm-y += diag.o gaccess.o guestdbg.o vsie.o pv.o
 
+kvm-$(CONFIG_PCI) += pci.o
 obj-$(CONFIG_KVM) += kvm.o
diff --git a/arch/s390/kvm/pci.c b/arch/s390/kvm/pci.c
new file mode 100644
index 000000000000..213be236c05a
--- /dev/null
+++ b/arch/s390/kvm/pci.c
@@ -0,0 +1,38 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * s390 kvm PCI passthrough support
+ *
+ * Copyright IBM Corp. 2022
+ *
+ *    Author(s): Matthew Rosato <mjrosato@linux.ibm.com>
+ */
+
+#include <linux/kvm_host.h>
+#include <linux/pci.h>
+#include "pci.h"
+
+int kvm_s390_pci_dev_open(struct zpci_dev *zdev)
+{
+	struct kvm_zdev *kzdev;
+
+	kzdev = kzalloc(sizeof(struct kvm_zdev), GFP_KERNEL);
+	if (!kzdev)
+		return -ENOMEM;
+
+	kzdev->zdev = zdev;
+	zdev->kzdev = kzdev;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(kvm_s390_pci_dev_open);
+
+void kvm_s390_pci_dev_release(struct zpci_dev *zdev)
+{
+	struct kvm_zdev *kzdev;
+
+	kzdev = zdev->kzdev;
+	WARN_ON(kzdev->zdev != zdev);
+	zdev->kzdev = 0;
+	kfree(kzdev);
+}
+EXPORT_SYMBOL_GPL(kvm_s390_pci_dev_release);
diff --git a/arch/s390/kvm/pci.h b/arch/s390/kvm/pci.h
new file mode 100644
index 000000000000..ce93978e8913
--- /dev/null
+++ b/arch/s390/kvm/pci.h
@@ -0,0 +1,21 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * s390 kvm PCI passthrough support
+ *
+ * Copyright IBM Corp. 2022
+ *
+ *    Author(s): Matthew Rosato <mjrosato@linux.ibm.com>
+ */
+
+#ifndef __KVM_S390_PCI_H
+#define __KVM_S390_PCI_H
+
+#include <linux/kvm_host.h>
+#include <linux/pci.h>
+
+struct kvm_zdev {
+	struct zpci_dev *zdev;
+	struct kvm *kvm;
+};
+
+#endif /* __KVM_S390_PCI_H */
-- 
2.27.0

[PATCH v5 11/21] KVM: s390: pci: do initial setup for AEN interpretation

[Matthew Rosato]

Initial setup for Adapter Event Notification Interpretation for zPCI
passthrough devices.  Specifically, allocate a structure for forwarding of
adapter events and pass the address of this structure to firmware.

Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
---
 arch/s390/include/asm/pci.h      |   4 +
 arch/s390/include/asm/pci_insn.h |  12 +++
 arch/s390/kvm/interrupt.c        |  14 +++
 arch/s390/kvm/kvm-s390.c         |   9 ++
 arch/s390/kvm/pci.c              | 153 +++++++++++++++++++++++++++++++
 arch/s390/kvm/pci.h              |  47 ++++++++++
 arch/s390/pci/pci.c              |   6 ++
 7 files changed, 245 insertions(+)

diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h
index 9eb20cebaa18..557b0ffb32d2 100644
--- a/arch/s390/include/asm/pci.h
+++ b/arch/s390/include/asm/pci.h
@@ -9,6 +9,7 @@
 #include <asm-generic/pci.h>
 #include <asm/pci_clp.h>
 #include <asm/pci_debug.h>
+#include <asm/pci_insn.h>
 #include <asm/sclp.h>
 
 #define PCIBIOS_MIN_IO		0x1000
@@ -204,6 +205,9 @@ extern const struct attribute_group *zpci_attr_groups[];
 extern unsigned int s390_pci_force_floating __initdata;
 extern unsigned int s390_pci_no_rid;
 
+extern union zpci_sic_iib *zpci_aipb;
+extern struct airq_iv *zpci_aif_sbv;
+
 /* -----------------------------------------------------------------------------
   Prototypes
 ----------------------------------------------------------------------------- */
diff --git a/arch/s390/include/asm/pci_insn.h b/arch/s390/include/asm/pci_insn.h
index 5331082fa516..e5f57cfe1d45 100644
--- a/arch/s390/include/asm/pci_insn.h
+++ b/arch/s390/include/asm/pci_insn.h
@@ -101,6 +101,7 @@ struct zpci_fib {
 /* Set Interruption Controls Operation Controls  */
 #define	SIC_IRQ_MODE_ALL		0
 #define	SIC_IRQ_MODE_SINGLE		1
+#define	SIC_SET_AENI_CONTROLS		2
 #define	SIC_IRQ_MODE_DIRECT		4
 #define	SIC_IRQ_MODE_D_ALL		16
 #define	SIC_IRQ_MODE_D_SINGLE		17
@@ -127,9 +128,20 @@ struct zpci_cdiib {
 	u64 : 64;
 } __packed __aligned(8);
 
+/* adapter interruption parameters block */
+struct zpci_aipb {
+	u64 faisb;
+	u64 gait;
+	u16 : 13;
+	u16 afi : 3;
+	u32 : 32;
+	u16 faal;
+} __packed __aligned(8);
+
 union zpci_sic_iib {
 	struct zpci_diib diib;
 	struct zpci_cdiib cdiib;
+	struct zpci_aipb aipb;
 };
 
 DECLARE_STATIC_KEY_FALSE(have_mio);
diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
index 250119a26c60..57a27dfc85ea 100644
--- a/arch/s390/kvm/interrupt.c
+++ b/arch/s390/kvm/interrupt.c
@@ -32,6 +32,7 @@
 #include "kvm-s390.h"
 #include "gaccess.h"
 #include "trace-s390.h"
+#include "pci.h"
 
 #define PFAULT_INIT 0x0600
 #define PFAULT_DONE 0x0680
@@ -3328,6 +3329,11 @@ void kvm_s390_gib_destroy(void)
 {
 	if (!gib)
 		return;
+	if (kvm_s390_pci_interp_allowed() && aift) {
+		mutex_lock(&aift->aift_lock);
+		kvm_s390_pci_aen_exit();
+		mutex_unlock(&aift->aift_lock);
+	}
 	chsc_sgib(0);
 	unregister_adapter_interrupt(&gib_alert_irq);
 	free_page((unsigned long)gib);
@@ -3365,6 +3371,14 @@ int kvm_s390_gib_init(u8 nisc)
 		goto out_unreg_gal;
 	}
 
+	if (kvm_s390_pci_interp_allowed()) {
+		if (kvm_s390_pci_aen_init(nisc)) {
+			pr_err("Initializing AEN for PCI failed\n");
+			rc = -EIO;
+			goto out_unreg_gal;
+		}
+	}
+
 	KVM_EVENT(3, "gib 0x%pK (nisc=%d) initialized", gib, gib->nisc);
 	goto out;
 
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 156d1c25a3c1..9db6f8080f71 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -47,6 +47,7 @@
 #include <asm/fpu/api.h>
 #include "kvm-s390.h"
 #include "gaccess.h"
+#include "pci.h"
 
 #define CREATE_TRACE_POINTS
 #include "trace.h"
@@ -502,6 +503,14 @@ int kvm_arch_init(void *opaque)
 		goto out;
 	}
 
+	if (kvm_s390_pci_interp_allowed()) {
+		rc = kvm_s390_pci_init();
+		if (rc) {
+			pr_err("Unable to allocate AIFT for PCI\n");
+			goto out;
+		}
+	}
+
 	rc = kvm_s390_gib_init(GAL_ISC);
 	if (rc)
 		goto out;
diff --git a/arch/s390/kvm/pci.c b/arch/s390/kvm/pci.c
index 213be236c05a..01bd8a2f503b 100644
--- a/arch/s390/kvm/pci.c
+++ b/arch/s390/kvm/pci.c
@@ -9,8 +9,149 @@
 
 #include <linux/kvm_host.h>
 #include <linux/pci.h>
+#include <asm/pci.h>
+#include <asm/pci_insn.h>
 #include "pci.h"
 
+struct zpci_aift *aift;
+
+static inline int __set_irq_noiib(u16 ctl, u8 isc)
+{
+	union zpci_sic_iib iib = {{0}};
+
+	return zpci_set_irq_ctrl(ctl, isc, &iib);
+}
+
+/* Caller must hold the aift lock before calling this function */
+void kvm_s390_pci_aen_exit(void)
+{
+	unsigned long flags;
+	struct kvm_zdev **gait_kzdev;
+
+	/*
+	 * Contents of the aipb remain registered for the life of the host
+	 * kernel, the information preserved in zpci_aipb and zpci_aif_sbv
+	 * in case we insert the KVM module again later.  Clear the AIFT
+	 * information and free anything not registered with underlying
+	 * firmware.
+	 */
+	spin_lock_irqsave(&aift->gait_lock, flags);
+	gait_kzdev = aift->kzdev;
+	aift->gait = 0;
+	aift->sbv = 0;
+	aift->kzdev = 0;
+	spin_unlock_irqrestore(&aift->gait_lock, flags);
+
+	kfree(gait_kzdev);
+}
+
+static int zpci_setup_aipb(u8 nisc)
+{
+	struct page *page;
+	int size, rc;
+
+	zpci_aipb = kzalloc(sizeof(union zpci_sic_iib), GFP_KERNEL);
+	if (!zpci_aipb)
+		return -ENOMEM;
+
+	aift->sbv = airq_iv_create(ZPCI_NR_DEVICES, AIRQ_IV_ALLOC, 0);
+	if (!aift->sbv) {
+		rc = -ENOMEM;
+		goto free_aipb;
+	}
+	zpci_aif_sbv = aift->sbv;
+	size = get_order(PAGE_ALIGN(ZPCI_NR_DEVICES *
+						sizeof(struct zpci_gaite)));
+	page = alloc_pages(GFP_KERNEL | __GFP_ZERO, size);
+	if (!page) {
+		rc = -ENOMEM;
+		goto free_sbv;
+	}
+	aift->gait = (struct zpci_gaite *)page_to_phys(page);
+
+	zpci_aipb->aipb.faisb = virt_to_phys(aift->sbv->vector);
+	zpci_aipb->aipb.gait = virt_to_phys(aift->gait);
+	zpci_aipb->aipb.afi = nisc;
+	zpci_aipb->aipb.faal = ZPCI_NR_DEVICES;
+
+	/* Setup Adapter Event Notification Interpretation */
+	if (zpci_set_irq_ctrl(SIC_SET_AENI_CONTROLS, 0, zpci_aipb)) {
+		rc = -EIO;
+		goto free_gait;
+	}
+
+	return 0;
+
+free_gait:
+	size = get_order(PAGE_ALIGN(ZPCI_NR_DEVICES *
+				    sizeof(struct zpci_gaite)));
+	free_pages((unsigned long)aift->gait, size);
+free_sbv:
+	airq_iv_release(aift->sbv);
+	zpci_aif_sbv = 0;
+free_aipb:
+	kfree(zpci_aipb);
+	zpci_aipb = 0;
+
+	return rc;
+}
+
+static int zpci_reset_aipb(u8 nisc)
+{
+	/*
+	 * AEN registration can only happen once per system boot.  If
+	 * an aipb already exists then AEN was already registered and
+	 * we can re-use the aipb contents.  This can only happen if
+	 * the KVM module was removed and re-inserted.
+	 */
+	if (zpci_aipb->aipb.faal != ZPCI_NR_DEVICES ||
+	    zpci_aipb->aipb.afi != nisc) {
+		return -EINVAL;
+	}
+	aift->sbv = zpci_aif_sbv;
+	aift->gait = (struct zpci_gaite *)zpci_aipb->aipb.gait;
+
+	return 0;
+}
+
+int kvm_s390_pci_aen_init(u8 nisc)
+{
+	int rc = 0;
+
+	/* If already enabled for AEN, bail out now */
+	if (aift->gait || aift->sbv)
+		return -EPERM;
+
+	mutex_lock(&aift->aift_lock);
+	aift->kzdev = kcalloc(ZPCI_NR_DEVICES, sizeof(struct kvm_zdev),
+			      GFP_KERNEL);
+	if (!aift->kzdev) {
+		rc = -ENOMEM;
+		goto unlock;
+	}
+
+	if (!zpci_aipb)
+		rc = zpci_setup_aipb(nisc);
+	else
+		rc = zpci_reset_aipb(nisc);
+	if (rc)
+		goto free_zdev;
+
+	/* Enable floating IRQs */
+	if (__set_irq_noiib(SIC_IRQ_MODE_SINGLE, nisc)) {
+		rc = -EIO;
+		kvm_s390_pci_aen_exit();
+	}
+
+	goto unlock;
+
+free_zdev:
+	kfree(aift->kzdev);
+unlock:
+	mutex_unlock(&aift->aift_lock);
+	return rc;
+}
+
 int kvm_s390_pci_dev_open(struct zpci_dev *zdev)
 {
 	struct kvm_zdev *kzdev;
@@ -36,3 +177,15 @@ void kvm_s390_pci_dev_release(struct zpci_dev *zdev)
 	kfree(kzdev);
 }
 EXPORT_SYMBOL_GPL(kvm_s390_pci_dev_release);
+
+int kvm_s390_pci_init(void)
+{
+	aift = kzalloc(sizeof(struct zpci_aift), GFP_KERNEL);
+	if (!aift)
+		return -ENOMEM;
+
+	spin_lock_init(&aift->gait_lock);
+	mutex_init(&aift->aift_lock);
+
+	return 0;
+}
diff --git a/arch/s390/kvm/pci.h b/arch/s390/kvm/pci.h
index ce93978e8913..a6a62db792b6 100644
--- a/arch/s390/kvm/pci.h
+++ b/arch/s390/kvm/pci.h
@@ -12,10 +12,57 @@
 
 #include <linux/kvm_host.h>
 #include <linux/pci.h>
+#include <linux/mutex.h>
+#include <asm/airq.h>
+#include <asm/cpu.h>
 
 struct kvm_zdev {
 	struct zpci_dev *zdev;
 	struct kvm *kvm;
 };
 
+struct zpci_gaite {
+	u32 gisa;
+	u8 gisc;
+	u8 count;
+	u8 reserved;
+	u8 aisbo;
+	u64 aisb;
+};
+
+struct zpci_aift {
+	struct zpci_gaite *gait;
+	struct airq_iv *sbv;
+	struct kvm_zdev **kzdev;
+	spinlock_t gait_lock; /* Protects the gait, used during AEN forward */
+	struct mutex aift_lock; /* Protects the other structures in aift */
+};
+
+extern struct zpci_aift *aift;
+
+int kvm_s390_pci_aen_init(u8 nisc);
+void kvm_s390_pci_aen_exit(void);
+
+int kvm_s390_pci_init(void);
+
+static inline bool kvm_s390_pci_interp_allowed(void)
+{
+	struct cpuid cpu_id;
+
+	get_cpu_id(&cpu_id);
+	switch (cpu_id.machine) {
+	case 0x2817:
+	case 0x2818:
+	case 0x2827:
+	case 0x2828:
+	case 0x2964:
+	case 0x2965:
+		/* No SHM on certain machines */
+		return false;
+	default:
+		return (IS_ENABLED(CONFIG_VFIO_PCI) && sclp.has_zpci_lsi &&
+			sclp.has_aeni && sclp.has_aisi && sclp.has_aisii);
+	}
+}
+
 #endif /* __KVM_S390_PCI_H */
diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c
index a86cd1cbb80e..f0a439c43395 100644
--- a/arch/s390/pci/pci.c
+++ b/arch/s390/pci/pci.c
@@ -61,6 +61,12 @@ DEFINE_STATIC_KEY_FALSE(have_mio);
 
 static struct kmem_cache *zdev_fmb_cache;
 
+/* AEN structures that must be preserved over KVM module re-insertion */
+union zpci_sic_iib *zpci_aipb;
+EXPORT_SYMBOL_GPL(zpci_aipb);
+struct airq_iv *zpci_aif_sbv;
+EXPORT_SYMBOL_GPL(zpci_aif_sbv);
+
 struct zpci_dev *get_zdev_by_fid(u32 fid)
 {
 	struct zpci_dev *tmp, *zdev = NULL;
-- 
2.27.0

[PATCH v5 12/21] KVM: s390: pci: enable host forwarding of Adapter Event Notifications

[Matthew Rosato]

In cases where interrupts are not forwarded to the guest via firmware,
KVM is responsible for ensuring delivery.  When an interrupt presents
with the forwarding bit, we must process the forwarding tables until
all interrupts are delivered.

Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
---
 arch/s390/include/asm/kvm_host.h |  1 +
 arch/s390/include/asm/tpi.h      | 13 ++++++
 arch/s390/kvm/interrupt.c        | 77 +++++++++++++++++++++++++++++++-
 arch/s390/kvm/kvm-s390.c         |  3 +-
 arch/s390/kvm/pci.h              | 10 +++++
 5 files changed, 102 insertions(+), 2 deletions(-)

diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index 766028d54a3e..c1518a505060 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -759,6 +759,7 @@ struct kvm_vm_stat {
 	u64 inject_pfault_done;
 	u64 inject_service_signal;
 	u64 inject_virtio;
+	u64 aen_forward;
 };
 
 struct kvm_arch_memory_slot {
diff --git a/arch/s390/include/asm/tpi.h b/arch/s390/include/asm/tpi.h
index 1ac538b8cbf5..f76e5fdff23a 100644
--- a/arch/s390/include/asm/tpi.h
+++ b/arch/s390/include/asm/tpi.h
@@ -19,6 +19,19 @@ struct tpi_info {
 	u32 :12;
 } __packed __aligned(4);
 
+/* I/O-Interruption Code as stored by TPI for an Adapter I/O */
+struct tpi_adapter_info {
+	u32 aism:8;
+	u32 :22;
+	u32 error:1;
+	u32 forward:1;
+	u32 reserved;
+	u32 adapter_IO:1;
+	u32 directed_irq:1;
+	u32 isc:3;
+	u32 :27;
+} __packed __aligned(4);
+
 #endif /* __ASSEMBLY__ */
 
 #endif /* _ASM_S390_TPI_H */
diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
index 57a27dfc85ea..bea0cc59eeba 100644
--- a/arch/s390/kvm/interrupt.c
+++ b/arch/s390/kvm/interrupt.c
@@ -3313,11 +3313,86 @@ int kvm_s390_gisc_unregister(struct kvm *kvm, u32 gisc)
 }
 EXPORT_SYMBOL_GPL(kvm_s390_gisc_unregister);
 
+static void aen_host_forward(unsigned long si)
+{
+	struct kvm_s390_gisa_interrupt *gi;
+	struct zpci_gaite *gaite;
+	struct kvm *kvm;
+
+	gaite = (struct zpci_gaite *)aift->gait +
+		(si * sizeof(struct zpci_gaite));
+	if (gaite->count == 0)
+		return;
+	if (gaite->aisb != 0)
+		set_bit_inv(gaite->aisbo, (unsigned long *)gaite->aisb);
+
+	kvm = kvm_s390_pci_si_to_kvm(aift, si);
+	if (!kvm)
+		return;
+	gi = &kvm->arch.gisa_int;
+
+	if (!(gi->origin->g1.simm & AIS_MODE_MASK(gaite->gisc)) ||
+	    !(gi->origin->g1.nimm & AIS_MODE_MASK(gaite->gisc))) {
+		gisa_set_ipm_gisc(gi->origin, gaite->gisc);
+		if (hrtimer_active(&gi->timer))
+			hrtimer_cancel(&gi->timer);
+		hrtimer_start(&gi->timer, 0, HRTIMER_MODE_REL);
+		kvm->stat.aen_forward++;
+	}
+}
+
+static void aen_process_gait(u8 isc)
+{
+	bool found = false, first = true;
+	union zpci_sic_iib iib = {{0}};
+	unsigned long si, flags;
+
+	spin_lock_irqsave(&aift->gait_lock, flags);
+
+	if (!aift->gait) {
+		spin_unlock_irqrestore(&aift->gait_lock, flags);
+		return;
+	}
+
+	for (si = 0;;) {
+		/* Scan adapter summary indicator bit vector */
+		si = airq_iv_scan(aift->sbv, si, airq_iv_end(aift->sbv));
+		if (si == -1UL) {
+			if (first || found) {
+				/* Re-enable interrupts. */
+				zpci_set_irq_ctrl(SIC_IRQ_MODE_SINGLE, isc,
+						  &iib);
+				first = found = false;
+			} else {
+				/* Interrupts on and all bits processed */
+				break;
+			}
+			found = false;
+			si = 0;
+			/* Scan again after re-enabling interrupts */
+			continue;
+		}
+		found = true;
+		aen_host_forward(si);
+	}
+
+	spin_unlock_irqrestore(&aift->gait_lock, flags);
+}
+
 static void gib_alert_irq_handler(struct airq_struct *airq,
 				  struct tpi_info *tpi_info)
 {
+	struct tpi_adapter_info *info = (struct tpi_adapter_info *)tpi_info;
+
 	inc_irq_stat(IRQIO_GAL);
-	process_gib_alert_list();
+
+	if (IS_ENABLED(CONFIG_VFIO_PCI) && (info->forward || info->error)) {
+		aen_process_gait(info->isc);
+		if (info->aism != 0)
+			process_gib_alert_list();
+	} else {
+		process_gib_alert_list();
+	}
 }
 
 static struct airq_struct gib_alert_irq = {
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 9db6f8080f71..a9eb8b75af93 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -64,7 +64,8 @@ const struct _kvm_stats_desc kvm_vm_stats_desc[] = {
 	STATS_DESC_COUNTER(VM, inject_float_mchk),
 	STATS_DESC_COUNTER(VM, inject_pfault_done),
 	STATS_DESC_COUNTER(VM, inject_service_signal),
-	STATS_DESC_COUNTER(VM, inject_virtio)
+	STATS_DESC_COUNTER(VM, inject_virtio),
+	STATS_DESC_COUNTER(VM, aen_forward)
 };
 
 const struct kvm_stats_header kvm_vm_stats_header = {
diff --git a/arch/s390/kvm/pci.h b/arch/s390/kvm/pci.h
index a6a62db792b6..d4997e2236ef 100644
--- a/arch/s390/kvm/pci.h
+++ b/arch/s390/kvm/pci.h
@@ -13,6 +13,7 @@
 #include <linux/kvm_host.h>
 #include <linux/pci.h>
 #include <linux/mutex.h>
+#include <linux/kvm_host.h>
 #include <asm/airq.h>
 #include <asm/cpu.h>
 
@@ -40,6 +41,15 @@ struct zpci_aift {
 
 extern struct zpci_aift *aift;
 
+static inline struct kvm *kvm_s390_pci_si_to_kvm(struct zpci_aift *aift,
+						 unsigned long si)
+{
+	if (!IS_ENABLED(CONFIG_VFIO_PCI) || aift->kzdev == 0 ||
+	    aift->kzdev[si] == 0)
+		return 0;
+	return aift->kzdev[si]->kvm;
+};
+
 int kvm_s390_pci_aen_init(u8 nisc);
 void kvm_s390_pci_aen_exit(void);
 
-- 
2.27.0

[PATCH v5 13/21] KVM: s390: mechanism to enable guest zPCI Interpretation

[Matthew Rosato]

The guest must have access to certain facilities in order to allow
interpretive execution of zPCI instructions and adapter event
notifications.  However, there are some cases where a guest might
disable interpretation -- provide a mechanism via which we can defer
enabling the associated zPCI interpretation facilities until the guest
indicates it wishes to use them.

Acked-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
---
 arch/s390/include/asm/kvm_host.h |  4 ++++
 arch/s390/kvm/kvm-s390.c         | 37 ++++++++++++++++++++++++++++++++
 arch/s390/kvm/kvm-s390.h         | 10 +++++++++
 3 files changed, 51 insertions(+)

diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index c1518a505060..8e381603b6a7 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -254,7 +254,10 @@ struct kvm_s390_sie_block {
 #define ECB2_IEP	0x20
 #define ECB2_PFMFI	0x08
 #define ECB2_ESCA	0x04
+#define ECB2_ZPCI_LSI	0x02
 	__u8    ecb2;                   /* 0x0062 */
+#define ECB3_AISI	0x20
+#define ECB3_AISII	0x10
 #define ECB3_DEA 0x08
 #define ECB3_AES 0x04
 #define ECB3_RI  0x01
@@ -940,6 +943,7 @@ struct kvm_arch{
 	int use_cmma;
 	int use_pfmfi;
 	int use_skf;
+	int use_zpci_interp;
 	int user_cpu_state_ctrl;
 	int user_sigp;
 	int user_stsi;
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index a9eb8b75af93..327649ddddce 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -1029,6 +1029,41 @@ static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
 	return 0;
 }
 
+static void kvm_s390_vcpu_pci_setup(struct kvm_vcpu *vcpu)
+{
+	/* Only set the ECB bits after guest requests zPCI interpretation */
+	if (!vcpu->kvm->arch.use_zpci_interp)
+		return;
+
+	vcpu->arch.sie_block->ecb2 |= ECB2_ZPCI_LSI;
+	vcpu->arch.sie_block->ecb3 |= ECB3_AISII + ECB3_AISI;
+}
+
+/* Must be called with kvm->lock held */
+void kvm_s390_vcpu_pci_enable_interp(struct kvm *kvm)
+{
+	struct kvm_vcpu *vcpu;
+	unsigned long i;
+
+	/*
+	 * If host is configured for PCI and the necessary facilities are
+	 * available, turn on interpretation for the life of this guest
+	 */
+	if (!kvm_s390_pci_interp_allowed())
+		return;
+
+	kvm->arch.use_zpci_interp = 1;
+
+	kvm_s390_vcpu_block_all(kvm);
+
+	kvm_for_each_vcpu(i, vcpu, kvm) {
+		kvm_s390_vcpu_pci_setup(vcpu);
+		kvm_s390_sync_request(KVM_REQ_VSIE_RESTART, vcpu);
+	}
+
+	kvm_s390_vcpu_unblock_all(kvm);
+}
+
 static void kvm_s390_sync_request_broadcast(struct kvm *kvm, int req)
 {
 	unsigned long cx;
@@ -3329,6 +3364,8 @@ static int kvm_s390_vcpu_setup(struct kvm_vcpu *vcpu)
 
 	kvm_s390_vcpu_crypto_setup(vcpu);
 
+	kvm_s390_vcpu_pci_setup(vcpu);
+
 	mutex_lock(&vcpu->kvm->lock);
 	if (kvm_s390_pv_is_protected(vcpu->kvm)) {
 		rc = kvm_s390_pv_create_cpu(vcpu, &uvrc, &uvrrc);
diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
index 497d52a83c78..975cffaf13de 100644
--- a/arch/s390/kvm/kvm-s390.h
+++ b/arch/s390/kvm/kvm-s390.h
@@ -507,6 +507,16 @@ void kvm_s390_reinject_machine_check(struct kvm_vcpu *vcpu,
  */
 void kvm_s390_vcpu_crypto_reset_all(struct kvm *kvm);
 
+/**
+ * kvm_s390_vcpu_pci_enable_interp
+ *
+ * Set the associated PCI attributes for each vcpu to allow for zPCI Load/Store
+ * interpretation as well as adapter interruption forwarding.
+ *
+ * @kvm: the KVM guest
+ */
+void kvm_s390_vcpu_pci_enable_interp(struct kvm *kvm);
+
 /**
  * diag9c_forwarding_hz
  *
-- 
2.27.0

[PATCH v5 14/21] KVM: s390: pci: provide routines for enabling/disabling interrupt forwarding

[Matthew Rosato]

These routines will be wired into a kvm ioctl in order to respond to
requests to enable / disable a device for Adapter Event Notifications /
Adapter Interuption Forwarding.

Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
---
 arch/s390/kvm/pci.c      | 247 +++++++++++++++++++++++++++++++++++++++
 arch/s390/kvm/pci.h      |   1 +
 arch/s390/pci/pci_insn.c |   1 +
 3 files changed, 249 insertions(+)

diff --git a/arch/s390/kvm/pci.c b/arch/s390/kvm/pci.c
index 01bd8a2f503b..f0fd68569a9d 100644
--- a/arch/s390/kvm/pci.c
+++ b/arch/s390/kvm/pci.c
@@ -11,6 +11,7 @@
 #include <linux/pci.h>
 #include <asm/pci.h>
 #include <asm/pci_insn.h>
+#include <asm/pci_io.h>
 #include "pci.h"
 
 struct zpci_aift *aift;
@@ -152,6 +153,252 @@ int kvm_s390_pci_aen_init(u8 nisc)
 	return rc;
 }
 
+/* Modify PCI: Register floating adapter interruption forwarding */
+static int kvm_zpci_set_airq(struct zpci_dev *zdev)
+{
+	u64 req = ZPCI_CREATE_REQ(zdev->fh, 0, ZPCI_MOD_FC_REG_INT);
+	struct zpci_fib fib = {};
+	u8 status;
+
+	fib.fmt0.isc = zdev->kzdev->fib.fmt0.isc;
+	fib.fmt0.sum = 1;       /* enable summary notifications */
+	fib.fmt0.noi = airq_iv_end(zdev->aibv);
+	fib.fmt0.aibv = virt_to_phys(zdev->aibv->vector);
+	fib.fmt0.aibvo = 0;
+	fib.fmt0.aisb = virt_to_phys(aift->sbv->vector + (zdev->aisb / 64) * 8);
+	fib.fmt0.aisbo = zdev->aisb & 63;
+	fib.gd = zdev->gisa;
+
+	return zpci_mod_fc(req, &fib, &status) ? -EIO : 0;
+}
+
+/* Modify PCI: Unregister floating adapter interruption forwarding */
+static int kvm_zpci_clear_airq(struct zpci_dev *zdev)
+{
+	u64 req = ZPCI_CREATE_REQ(zdev->fh, 0, ZPCI_MOD_FC_DEREG_INT);
+	struct zpci_fib fib = {};
+	u8 cc, status;
+
+	fib.gd = zdev->gisa;
+
+	cc = zpci_mod_fc(req, &fib, &status);
+	if (cc == 3 || (cc == 1 && status == 24))
+		/* Function already gone or IRQs already deregistered. */
+		cc = 0;
+
+	return cc ? -EIO : 0;
+}
+
+static inline void unaccount_mem(unsigned long nr_pages)
+{
+	struct user_struct *user = get_uid(current_user());
+
+	if (user)
+		atomic_long_sub(nr_pages, &user->locked_vm);
+	if (current->mm)
+		atomic64_sub(nr_pages, &current->mm->pinned_vm);
+}
+
+static inline int account_mem(unsigned long nr_pages)
+{
+	struct user_struct *user = get_uid(current_user());
+	unsigned long page_limit, cur_pages, new_pages;
+
+	page_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
+
+	do {
+		cur_pages = atomic_long_read(&user->locked_vm);
+		new_pages = cur_pages + nr_pages;
+		if (new_pages > page_limit)
+			return -ENOMEM;
+	} while (atomic_long_cmpxchg(&user->locked_vm, cur_pages,
+					new_pages) != cur_pages);
+
+	atomic64_add(nr_pages, &current->mm->pinned_vm);
+
+	return 0;
+}
+
+static int kvm_s390_pci_aif_enable(struct zpci_dev *zdev, struct zpci_fib *fib,
+				   bool assist)
+{
+	struct page *pages[1], *aibv_page, *aisb_page = NULL;
+	unsigned int msi_vecs, idx;
+	struct zpci_gaite *gaite;
+	unsigned long hva, bit;
+	struct kvm *kvm;
+	phys_addr_t gaddr;
+	int rc = 0, gisc, npages, pcount = 0;
+
+	/*
+	 * Interrupt forwarding is only applicable if the device is already
+	 * enabled for interpretation
+	 */
+	if (zdev->gisa == 0)
+		return -EINVAL;
+
+	kvm = zdev->kzdev->kvm;
+	msi_vecs = min_t(unsigned int, fib->fmt0.noi, zdev->max_msi);
+
+	/* Get the associated forwarding ISC - if invalid, return the error */
+	gisc = kvm_s390_gisc_register(kvm, fib->fmt0.isc);
+	if (gisc < 0)
+		return gisc;
+
+	/* Replace AIBV address */
+	idx = srcu_read_lock(&kvm->srcu);
+	hva = gfn_to_hva(kvm, gpa_to_gfn((gpa_t)fib->fmt0.aibv));
+	npages = pin_user_pages_fast(hva, 1, FOLL_WRITE | FOLL_LONGTERM, pages);
+	srcu_read_unlock(&kvm->srcu, idx);
+	if (npages < 1) {
+		rc = -EIO;
+		goto out;
+	}
+	aibv_page = pages[0];
+	pcount++;
+	gaddr = page_to_phys(aibv_page) + (fib->fmt0.aibv & ~PAGE_MASK);
+	fib->fmt0.aibv = gaddr;
+
+	/* Pin the guest AISB if one was specified */
+	if (fib->fmt0.sum == 1) {
+		idx = srcu_read_lock(&kvm->srcu);
+		hva = gfn_to_hva(kvm, gpa_to_gfn((gpa_t)fib->fmt0.aisb));
+		npages = pin_user_pages_fast(hva, 1, FOLL_WRITE | FOLL_LONGTERM,
+					     pages);
+		srcu_read_unlock(&kvm->srcu, idx);
+		if (npages < 1) {
+			rc = -EIO;
+			goto unpin1;
+		}
+		aisb_page = pages[0];
+		pcount++;
+	}
+
+	/* Account for pinned pages, roll back on failure */
+	if (account_mem(pcount))
+		goto unpin2;
+
+	/* AISB must be allocated before we can fill in GAITE */
+	mutex_lock(&aift->aift_lock);
+	bit = airq_iv_alloc_bit(aift->sbv);
+	if (bit == -1UL)
+		goto unlock;
+	zdev->aisb = bit; /* store the summary bit number */
+	zdev->aibv = airq_iv_create(msi_vecs, AIRQ_IV_DATA |
+				    AIRQ_IV_BITLOCK |
+				    AIRQ_IV_GUESTVEC,
+				    phys_to_virt(fib->fmt0.aibv));
+
+	spin_lock_irq(&aift->gait_lock);
+	gaite = (struct zpci_gaite *)aift->gait + (zdev->aisb *
+						   sizeof(struct zpci_gaite));
+
+	/* If assist not requested, host will get all alerts */
+	if (assist)
+		gaite->gisa = (u32)virt_to_phys(&kvm->arch.sie_page2->gisa);
+	else
+		gaite->gisa = 0;
+
+	gaite->gisc = fib->fmt0.isc;
+	gaite->count++;
+	gaite->aisbo = fib->fmt0.aisbo;
+	gaite->aisb = virt_to_phys(page_address(aisb_page) + (fib->fmt0.aisb &
+							      ~PAGE_MASK));
+	aift->kzdev[zdev->aisb] = zdev->kzdev;
+	spin_unlock_irq(&aift->gait_lock);
+
+	/* Update guest FIB for re-issue */
+	fib->fmt0.aisbo = zdev->aisb & 63;
+	fib->fmt0.aisb = virt_to_phys(aift->sbv->vector + (zdev->aisb / 64) * 8);
+	fib->fmt0.isc = gisc;
+
+	/* Save some guest fib values in the host for later use */
+	zdev->kzdev->fib.fmt0.isc = fib->fmt0.isc;
+	zdev->kzdev->fib.fmt0.aibv = fib->fmt0.aibv;
+	mutex_unlock(&aift->aift_lock);
+
+	/* Issue the clp to setup the irq now */
+	rc = kvm_zpci_set_airq(zdev);
+	return rc;
+
+unlock:
+	mutex_unlock(&aift->aift_lock);
+unpin2:
+	if (fib->fmt0.sum == 1)
+		unpin_user_page(aisb_page);
+unpin1:
+	unpin_user_page(aibv_page);
+out:
+	return rc;
+}
+
+static int kvm_s390_pci_aif_disable(struct zpci_dev *zdev, bool force)
+{
+	struct kvm_zdev *kzdev = zdev->kzdev;
+	struct zpci_gaite *gaite;
+	struct page *vpage = NULL, *spage = NULL;
+	int rc, pcount = 0;
+	u8 isc;
+
+	if (zdev->gisa == 0)
+		return -EINVAL;
+
+	mutex_lock(&aift->aift_lock);
+
+	/*
+	 * If the clear fails due to an error, leave now unless we know this
+	 * device is about to go away (force) -- In that case clear the GAITE
+	 * regardless.
+	 */
+	rc = kvm_zpci_clear_airq(zdev);
+	if (rc && !force)
+		goto out;
+
+	if (zdev->kzdev->fib.fmt0.aibv == 0)
+		goto out;
+	spin_lock_irq(&aift->gait_lock);
+	gaite = (struct zpci_gaite *)aift->gait + (zdev->aisb *
+						   sizeof(struct zpci_gaite));
+	isc = gaite->gisc;
+	gaite->count--;
+	if (gaite->count == 0) {
+		/* Release guest AIBV and AISB */
+		vpage = phys_to_page(kzdev->fib.fmt0.aibv);
+		if (gaite->aisb != 0)
+			spage = phys_to_page(gaite->aisb);
+		/* Clear the GAIT entry */
+		gaite->aisb = 0;
+		gaite->gisc = 0;
+		gaite->aisbo = 0;
+		gaite->gisa = 0;
+		aift->kzdev[zdev->aisb] = 0;
+		/* Clear zdev info */
+		airq_iv_free_bit(aift->sbv, zdev->aisb);
+		airq_iv_release(zdev->aibv);
+		zdev->aisb = 0;
+		zdev->aibv = NULL;
+	}
+	spin_unlock_irq(&aift->gait_lock);
+	kvm_s390_gisc_unregister(kzdev->kvm, isc);
+	kzdev->fib.fmt0.isc = 0;
+	kzdev->fib.fmt0.aibv = 0;
+
+	if (vpage) {
+		unpin_user_page(vpage);
+		pcount++;
+	}
+	if (spage) {
+		unpin_user_page(spage);
+		pcount++;
+	}
+	if (pcount > 0)
+		unaccount_mem(pcount);
+out:
+	mutex_unlock(&aift->aift_lock);
+
+	return rc;
+}
+
 int kvm_s390_pci_dev_open(struct zpci_dev *zdev)
 {
 	struct kvm_zdev *kzdev;
diff --git a/arch/s390/kvm/pci.h b/arch/s390/kvm/pci.h
index d4997e2236ef..b4bf3d1d4b66 100644
--- a/arch/s390/kvm/pci.h
+++ b/arch/s390/kvm/pci.h
@@ -20,6 +20,7 @@
 struct kvm_zdev {
 	struct zpci_dev *zdev;
 	struct kvm *kvm;
+	struct zpci_fib fib;
 };
 
 struct zpci_gaite {
diff --git a/arch/s390/pci/pci_insn.c b/arch/s390/pci/pci_insn.c
index 4c6967b73932..cd9fb186a6be 100644
--- a/arch/s390/pci/pci_insn.c
+++ b/arch/s390/pci/pci_insn.c
@@ -60,6 +60,7 @@ u8 zpci_mod_fc(u64 req, struct zpci_fib *fib, u8 *status)
 
 	return cc;
 }
+EXPORT_SYMBOL_GPL(zpci_mod_fc);
 
 /* Refresh PCI Translations */
 static inline u8 __rpcit(u64 fn, u64 addr, u64 range, u8 *status)
-- 
2.27.0

[PATCH v5 15/21] KVM: s390: pci: add routines to start/stop interpretive execution

[Matthew Rosato]

These routines will be invoked at the time an s390x vfio-pci device is
associated with a KVM (or when the association is removed), allowing
the zPCI device to enable or disable load/store intepretation mode;
this requires the host zPCI device to inform firmware of the unique
token (GISA designation) that is associated with the owning KVM.

Furthemore, add/remove these devices from a list associated with the
kvm and ensure proper cleanup always occurs during vm exit.

Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
---
 arch/s390/include/asm/kvm_host.h |  11 +++
 arch/s390/kvm/kvm-s390.c         |   5 +
 arch/s390/kvm/pci.c              | 163 +++++++++++++++++++++++++++++++
 arch/s390/kvm/pci.h              |   5 +
 arch/s390/pci/pci.c              |   3 +
 5 files changed, 187 insertions(+)

diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index 8e381603b6a7..ef3364af6b34 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -967,6 +967,8 @@ struct kvm_arch{
 	DECLARE_BITMAP(idle_mask, KVM_MAX_VCPUS);
 	struct kvm_s390_gisa_interrupt gisa_int;
 	struct kvm_s390_pv pv;
+	struct list_head kzdev_list;
+	spinlock_t kzdev_list_lock;
 };
 
 #define KVM_HVA_ERR_BAD		(-1UL)
@@ -1017,4 +1019,13 @@ static inline void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
 static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) {}
 static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
 
+#ifdef CONFIG_PCI
+int kvm_s390_pci_register_kvm(struct device *dev, void *data);
+#else
+static inline int kvm_s390_pci_register_kvm(struct device *dev, void *data)
+{
+	return -EPERM;
+}
+#endif
+
 #endif
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 327649ddddce..704d85214f4f 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -2864,6 +2864,9 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 
 	kvm_s390_crypto_init(kvm);
 
+	if (IS_ENABLED(CONFIG_VFIO_PCI))
+		kvm_s390_pci_init_list(kvm);
+
 	mutex_init(&kvm->arch.float_int.ais_lock);
 	spin_lock_init(&kvm->arch.float_int.lock);
 	for (i = 0; i < FIRQ_LIST_COUNT; i++)
@@ -2949,6 +2952,8 @@ void kvm_arch_destroy_vm(struct kvm *kvm)
 	if (!kvm_is_ucontrol(kvm))
 		gmap_remove(kvm->arch.gmap);
 	kvm_s390_destroy_adapters(kvm);
+	if (IS_ENABLED(CONFIG_VFIO_PCI))
+		kvm_s390_pci_clear_list(kvm);
 	kvm_s390_clear_float_irqs(kvm);
 	kvm_s390_vsie_destroy(kvm);
 	KVM_EVENT(3, "vm 0x%pK destroyed", kvm);
diff --git a/arch/s390/kvm/pci.c b/arch/s390/kvm/pci.c
index f0fd68569a9d..66565f5f3f43 100644
--- a/arch/s390/kvm/pci.c
+++ b/arch/s390/kvm/pci.c
@@ -12,7 +12,9 @@
 #include <asm/pci.h>
 #include <asm/pci_insn.h>
 #include <asm/pci_io.h>
+#include <asm/sclp.h>
 #include "pci.h"
+#include "kvm-s390.h"
 
 struct zpci_aift *aift;
 
@@ -425,6 +427,167 @@ void kvm_s390_pci_dev_release(struct zpci_dev *zdev)
 }
 EXPORT_SYMBOL_GPL(kvm_s390_pci_dev_release);
 
+static inline int register_kvm(struct zpci_dev *zdev, struct kvm *kvm)
+{
+	int rc;
+
+	if (zdev->kzdev || zdev->gisa != 0)
+		return -EINVAL;
+
+	mutex_lock(&kvm->lock);
+
+	rc = kvm_s390_pci_dev_open(zdev);
+	if (rc)
+		goto err;
+
+	/*
+	 * If interpretation facilities aren't available, add the device to
+	 * the kzdev list but don't enable for interpretation.
+	 */
+	if (!kvm_s390_pci_interp_allowed())
+		goto out;
+
+	/*
+	 * If this is the first request to use an interpreted device, make the
+	 * necessary vcpu changes
+	 */
+	if (!kvm->arch.use_zpci_interp)
+		kvm_s390_vcpu_pci_enable_interp(kvm);
+
+	if (zdev_enabled(zdev)) {
+		rc = zpci_disable_device(zdev);
+		if (rc)
+			goto err;
+	}
+
+	/*
+	 * Store information about the identity of the kvm guest allowed to
+	 * access this device via interpretation to be used by host CLP
+	 */
+	zdev->gisa = (u32)virt_to_phys(&kvm->arch.sie_page2->gisa);
+
+	rc = zpci_enable_device(zdev);
+	if (rc)
+		goto clear_gisa;
+
+	/* Re-register the IOMMU that was already created */
+	rc = zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
+				virt_to_phys(zdev->dma_table));
+	if (rc)
+		goto clear_gisa;
+
+out:
+	zdev->kzdev->kvm = kvm;
+
+	spin_lock(&kvm->arch.kzdev_list_lock);
+	list_add_tail(&zdev->kzdev->entry, &kvm->arch.kzdev_list);
+	spin_unlock(&kvm->arch.kzdev_list_lock);
+
+	mutex_unlock(&kvm->lock);
+	return 0;
+
+clear_gisa:
+	zdev->gisa = 0;
+err:
+	if (zdev->kzdev)
+		kvm_s390_pci_dev_release(zdev);
+	mutex_unlock(&kvm->lock);
+	return rc;
+}
+
+static inline int unregister_kvm(struct zpci_dev *zdev)
+{
+	struct kvm *kvm;
+	int rc;
+
+	if (!zdev->kzdev)
+		return -EINVAL;
+
+	kvm = zdev->kzdev->kvm;
+	mutex_lock(&kvm->lock);
+
+	/*
+	 * A 0 gisa means interpretation was never enabled, just remove the
+	 * device from the list.
+	 */
+	if (zdev->gisa == 0)
+		goto out;
+
+	/* Forwarding must be turned off before interpretation */
+	if (zdev->kzdev->fib.fmt0.aibv != 0)
+		kvm_s390_pci_aif_disable(zdev, true);
+
+	/* Remove the host CLP guest designation */
+	zdev->gisa = 0;
+
+	if (zdev_enabled(zdev)) {
+		rc = zpci_disable_device(zdev);
+		if (rc)
+			goto out;
+	}
+
+	rc = zpci_enable_device(zdev);
+	if (rc)
+		goto out;
+
+	/* Re-register the IOMMU that was already created */
+	rc = zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
+				virt_to_phys(zdev->dma_table));
+
+out:
+	spin_lock(&kvm->arch.kzdev_list_lock);
+	list_del(&zdev->kzdev->entry);
+	spin_unlock(&kvm->arch.kzdev_list_lock);
+	kvm_s390_pci_dev_release(zdev);
+
+	mutex_unlock(&kvm->lock);
+
+	return rc;
+}
+
+int kvm_s390_pci_register_kvm(struct device *dev, void *data)
+{
+	struct zpci_dev *zdev = NULL;
+	struct kvm *kvm = data;
+
+	/* Only proceed for zPCI devices, quietly ignore others */
+	if (dev_is_pci(dev))
+		zdev = to_zpci_dev(dev);
+	if (!zdev)
+		return 0;
+
+	/*
+	 * Register all devices with this KVM.  If interpetation facilities
+	 * are available, enable them and let userspace indicate whether or
+	 * not they will be used (specify SHM bit to disable).
+	 */
+	if (kvm)
+		return register_kvm(zdev, kvm);
+	else
+		return unregister_kvm(zdev);
+}
+EXPORT_SYMBOL_GPL(kvm_s390_pci_register_kvm);
+
+void kvm_s390_pci_init_list(struct kvm *kvm)
+{
+	spin_lock_init(&kvm->arch.kzdev_list_lock);
+	INIT_LIST_HEAD(&kvm->arch.kzdev_list);
+}
+
+void kvm_s390_pci_clear_list(struct kvm *kvm)
+{
+	struct kvm_zdev *tmp, *kzdev;
+	LIST_HEAD(remove);
+
+	spin_lock(&kvm->arch.kzdev_list_lock);
+	list_for_each_entry_safe(kzdev, tmp, &kvm->arch.kzdev_list, entry)
+		list_move_tail(&kzdev->entry, &remove);
+	spin_unlock(&kvm->arch.kzdev_list_lock);
+
+	list_for_each_entry_safe(kzdev, tmp, &remove, entry)
+		unregister_kvm(kzdev->zdev);
+}
+
 int kvm_s390_pci_init(void)
 {
 	aift = kzalloc(sizeof(struct zpci_aift), GFP_KERNEL);
diff --git a/arch/s390/kvm/pci.h b/arch/s390/kvm/pci.h
index b4bf3d1d4b66..cb5ec3208923 100644
--- a/arch/s390/kvm/pci.h
+++ b/arch/s390/kvm/pci.h
@@ -13,6 +13,7 @@
 #include <linux/kvm_host.h>
 #include <linux/pci.h>
 #include <linux/mutex.h>
+#include <linux/kvm.h>
 #include <linux/kvm_host.h>
 #include <asm/airq.h>
 #include <asm/cpu.h>
@@ -21,6 +22,7 @@ struct kvm_zdev {
 	struct zpci_dev *zdev;
 	struct kvm *kvm;
 	struct zpci_fib fib;
+	struct list_head entry;
 };
 
 struct zpci_gaite {
@@ -54,6 +56,9 @@ static inline struct kvm *kvm_s390_pci_si_to_kvm(struct zpci_aift *aift,
 int kvm_s390_pci_aen_init(u8 nisc);
 void kvm_s390_pci_aen_exit(void);
 
+void kvm_s390_pci_init_list(struct kvm *kvm);
+void kvm_s390_pci_clear_list(struct kvm *kvm);
+
 int kvm_s390_pci_init(void);
 
 static inline bool kvm_s390_pci_interp_allowed(void)
diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c
index f0a439c43395..d9b021fb84d5 100644
--- a/arch/s390/pci/pci.c
+++ b/arch/s390/pci/pci.c
@@ -132,6 +132,7 @@ int zpci_register_ioat(struct zpci_dev *zdev, u8 dmaas,
 		zpci_dbg(3, "reg ioat fid:%x, cc:%d, status:%d\n", zdev->fid, cc, status);
 	return cc;
 }
+EXPORT_SYMBOL_GPL(zpci_register_ioat);
 
 /* Modify PCI: Unregister I/O address translation parameters */
 int zpci_unregister_ioat(struct zpci_dev *zdev, u8 dmaas)
@@ -712,6 +713,7 @@ int zpci_enable_device(struct zpci_dev *zdev)
 		zpci_update_fh(zdev, fh);
 	return rc;
 }
+EXPORT_SYMBOL_GPL(zpci_enable_device);
 
 int zpci_disable_device(struct zpci_dev *zdev)
 {
@@ -735,6 +737,7 @@ int zpci_disable_device(struct zpci_dev *zdev)
 	}
 	return rc;
 }
+EXPORT_SYMBOL_GPL(zpci_disable_device);
 
 /**
  * zpci_hot_reset_device - perform a reset of the given zPCI function
-- 
2.27.0

[PATCH v5 16/21] KVM: vfio: add s390x hook to register KVM guest designation

[Matthew Rosato]

At the time a KVM is associated with a vfio group, s390x zPCI devices
must register a special guest indication (GISA designation) to allow
for the use of interpretive execution facilities.  This indication is
used to ensure that only the specified KVM can interact with the device.
Similarly, the indication must be removed once the KVM is no longer
associated with the device.

This patch adds an s390-specific hook to invoke a KVM registration routine
for each device associated with the iommu group; in reality, it will be a
NOP for all but zPCI devices on s390x.

Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
---
 virt/kvm/vfio.c | 35 ++++++++++++++++++++++++++++++++++-
 1 file changed, 34 insertions(+), 1 deletion(-)

diff --git a/virt/kvm/vfio.c b/virt/kvm/vfio.c
index 8fcbc50221c2..2efe5be5a6ee 100644
--- a/virt/kvm/vfio.c
+++ b/virt/kvm/vfio.c
@@ -106,7 +106,7 @@ static bool kvm_vfio_group_is_coherent(struct vfio_group *vfio_group)
 	return ret > 0;
 }
 
-#ifdef CONFIG_SPAPR_TCE_IOMMU
+#if defined(CONFIG_SPAPR_TCE_IOMMU) || defined(CONFIG_S390)
 static int kvm_vfio_external_user_iommu_id(struct vfio_group *vfio_group)
 {
 	int (*fn)(struct vfio_group *);
@@ -133,7 +133,9 @@ static struct iommu_group *kvm_vfio_group_get_iommu_group(
 
 	return iommu_group_get_by_id(group_id);
 }
+#endif
 
+#ifdef CONFIG_SPAPR_TCE_IOMMU
 static void kvm_spapr_tce_release_vfio_group(struct kvm *kvm,
 		struct vfio_group *vfio_group)
 {
@@ -147,6 +149,24 @@ static void kvm_spapr_tce_release_vfio_group(struct kvm *kvm,
 }
 #endif
 
+#ifdef CONFIG_S390
+static int kvm_s390_register_kvm(struct vfio_group *vfio_group,
+				  struct kvm *kvm)
+{
+	int rc;
+
+	struct iommu_group *grp = kvm_vfio_group_get_iommu_group(vfio_group);
+
+	if (WARN_ON_ONCE(!grp))
+		return -EPERM;
+
+	rc = iommu_group_for_each_dev(grp, kvm, kvm_s390_pci_register_kvm);
+	iommu_group_put(grp);
+
+	return rc;
+}
+#endif
+
 /*
  * Groups can use the same or different IOMMU domains.  If the same then
  * adding a new group may change the coherency of groups we've previously
@@ -223,6 +243,16 @@ static int kvm_vfio_set_group(struct kvm_device *dev, long attr, u64 arg)
 			return -ENOMEM;
 		}
 
+#ifdef CONFIG_S390
+		ret = kvm_s390_register_kvm(vfio_group, dev->kvm);
+		if (ret) {
+			kfree(kvg);
+			mutex_unlock(&kv->lock);
+			kvm_vfio_group_put_external_user(vfio_group);
+			return ret;
+		}
+#endif
+
 		list_add_tail(&kvg->node, &kv->group_list);
 		kvg->vfio_group = vfio_group;
 
@@ -258,6 +288,9 @@ static int kvm_vfio_set_group(struct kvm_device *dev, long attr, u64 arg)
 #ifdef CONFIG_SPAPR_TCE_IOMMU
 			kvm_spapr_tce_release_vfio_group(dev->kvm,
 							 kvg->vfio_group);
+#endif
+#ifdef CONFIG_S390
+			kvm_s390_register_kvm(vfio_group, NULL);
 #endif
 			kvm_vfio_group_set_kvm(kvg->vfio_group, NULL);
 			kvm_vfio_group_put_external_user(kvg->vfio_group);
-- 
2.27.0

[PATCH v5 17/21] vfio-pci/zdev: add function handle to clp base capability

[Matthew Rosato]

The function handle is a system-wide unique identifier for a zPCI
device.  It is used as input for various zPCI operations.

Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
---
 drivers/vfio/pci/vfio_pci_zdev.c | 5 +++--
 include/uapi/linux/vfio_zdev.h   | 3 +++
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/pci/vfio_pci_zdev.c b/drivers/vfio/pci/vfio_pci_zdev.c
index ea4c0d2b0663..4a653ce480c7 100644
--- a/drivers/vfio/pci/vfio_pci_zdev.c
+++ b/drivers/vfio/pci/vfio_pci_zdev.c
@@ -23,14 +23,15 @@ static int zpci_base_cap(struct zpci_dev *zdev, struct vfio_info_cap *caps)
 {
 	struct vfio_device_info_cap_zpci_base cap = {
 		.header.id = VFIO_DEVICE_INFO_CAP_ZPCI_BASE,
-		.header.version = 1,
+		.header.version = 2,
 		.start_dma = zdev->start_dma,
 		.end_dma = zdev->end_dma,
 		.pchid = zdev->pchid,
 		.vfn = zdev->vfn,
 		.fmb_length = zdev->fmb_length,
 		.pft = zdev->pft,
-		.gid = zdev->pfgid
+		.gid = zdev->pfgid,
+		.fh = zdev->fh
 	};
 
 	return vfio_info_add_capability(caps, &cap.header, sizeof(cap));
diff --git a/include/uapi/linux/vfio_zdev.h b/include/uapi/linux/vfio_zdev.h
index b4309397b6b2..78c022af3d29 100644
--- a/include/uapi/linux/vfio_zdev.h
+++ b/include/uapi/linux/vfio_zdev.h
@@ -29,6 +29,9 @@ struct vfio_device_info_cap_zpci_base {
 	__u16 fmb_length;	/* Measurement Block Length (in bytes) */
 	__u8 pft;		/* PCI Function Type */
 	__u8 gid;		/* PCI function group ID */
+	/* End of version 1 */
+	__u32 fh;		/* PCI function handle */
+	/* End of version 2 */
 };
 
 /**
-- 
2.27.0

[PATCH v5 18/21] vfio-pci/zdev: different maxstbl for interpreted devices

[Matthew Rosato]

When doing load/store interpretation, the maximum store block length is
determined by the underlying firmware, not the host kernel API.  Reflect
that in the associated Query PCI Function Group clp capability and let
userspace decide which is appropriate to present to the guest.

Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
---
 drivers/vfio/pci/vfio_pci_zdev.c | 6 ++++--
 include/uapi/linux/vfio_zdev.h   | 4 ++++
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/pci/vfio_pci_zdev.c b/drivers/vfio/pci/vfio_pci_zdev.c
index 4a653ce480c7..d3ca58d9d8ec 100644
--- a/drivers/vfio/pci/vfio_pci_zdev.c
+++ b/drivers/vfio/pci/vfio_pci_zdev.c
@@ -44,14 +44,16 @@ static int zpci_group_cap(struct zpci_dev *zdev, struct vfio_info_cap *caps)
 {
 	struct vfio_device_info_cap_zpci_group cap = {
 		.header.id = VFIO_DEVICE_INFO_CAP_ZPCI_GROUP,
-		.header.version = 1,
+		.header.version = 2,
 		.dasm = zdev->dma_mask,
 		.msi_addr = zdev->msi_addr,
 		.flags = VFIO_DEVICE_INFO_ZPCI_FLAG_REFRESH,
 		.mui = zdev->fmb_update,
 		.noi = zdev->max_msi,
 		.maxstbl = ZPCI_MAX_WRITE_SIZE,
-		.version = zdev->version
+		.version = zdev->version,
+		.reserved = 0,
+		.imaxstbl = zdev->maxstbl
 	};
 
 	return vfio_info_add_capability(caps, &cap.header, sizeof(cap));
diff --git a/include/uapi/linux/vfio_zdev.h b/include/uapi/linux/vfio_zdev.h
index 78c022af3d29..77f2aff1f27e 100644
--- a/include/uapi/linux/vfio_zdev.h
+++ b/include/uapi/linux/vfio_zdev.h
@@ -50,6 +50,10 @@ struct vfio_device_info_cap_zpci_group {
 	__u16 noi;		/* Maximum number of MSIs */
 	__u16 maxstbl;		/* Maximum Store Block Length */
 	__u8 version;		/* Supported PCI Version */
+	/* End of version 1 */
+	__u8 reserved;
+	__u16 imaxstbl;		/* Maximum Interpreted Store Block Length */
+	/* End of version 2 */
 };
 
 /**
-- 
2.27.0

[PATCH v5 19/21] KVM: s390: add KVM_S390_ZPCI_OP to manage guest zPCI devices

[Matthew Rosato]

The KVM_S390_ZPCI_OP ioctl provides a mechanism for managing
hardware-assisted virtualization features for s390X zPCI passthrough.
Add the first 2 operations, which can be used to enable/disable
the specified device for Adapter Event Notification interpretation.

Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
---
 Documentation/virt/kvm/api.rst | 45 +++++++++++++++++++++++
 arch/s390/kvm/kvm-s390.c       | 23 ++++++++++++
 arch/s390/kvm/pci.c            | 65 ++++++++++++++++++++++++++++++++++
 arch/s390/kvm/pci.h            |  2 ++
 include/uapi/linux/kvm.h       | 31 ++++++++++++++++
 5 files changed, 166 insertions(+)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index d13fa6600467..474f502c2ea0 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -5645,6 +5645,51 @@ enabled with ``arch_prctl()``, but this may change in the future.
 The offsets of the state save areas in struct kvm_xsave follow the contents
 of CPUID leaf 0xD on the host.
 
+4.135 KVM_S390_ZPCI_OP
+--------------------
+
+:Capability: KVM_CAP_S390_ZPCI_OP
+:Architectures: s390
+:Type: vcpu ioctl
+:Parameters: struct kvm_s390_zpci_op (in)
+:Returns: 0 on success, <0 on error
+
+Used to manage hardware-assisted virtualization features for zPCI devices.
+
+Parameters are specified via the following structure::
+
+  struct kvm_s390_zpci_op {
+	/* in */
+	__u32 fh;		/* target device */
+	__u8  op;		/* operation to perform */
+	__u8  pad[3];
+	union {
+		/* for KVM_S390_ZPCIOP_REG_AEN */
+		struct {
+			__u64 ibv;	/* Guest addr of interrupt bit vector */
+			__u64 sb;	/* Guest addr of summary bit */
+			__u32 flags;
+			__u32 noi;	/* Number of interrupts */
+			__u8 isc;	/* Guest interrupt subclass */
+			__u8 sbo;	/* Offset of guest summary bit vector */
+			__u16 pad;
+		} reg_aen;
+		__u64 reserved[8];
+	} u;
+  };
+
+The type of operation is specified in the "op" field.
+KVM_S390_ZPCIOP_REG_AEN is used to register the VM for adapter event
+notification interpretation, which will allow firmware delivery of adapter
+events directly to the vm, with KVM providing a backup delivery mechanism;
+KVM_S390_ZPCIOP_DEREG_AEN is used to subsequently disable interpretation of
+adapter event notifications.
+
+The target zPCI function must also be specified via the "fh" field.  For the
+KVM_S390_ZPCIOP_REG_AEN operation, additional information to establish firmware
+delivery must be provided via the "reg_aen" struct.
+
+The "reserved" field is meant for future extensions.
 
 5. The kvm_run structure
 ========================
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 704d85214f4f..65a53e22f686 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -616,6 +616,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_S390_PROTECTED:
 		r = is_prot_virt_host();
 		break;
+	case KVM_CAP_S390_ZPCI_OP:
+		if (kvm_s390_pci_interp_allowed())
+			r = 1;
+		else
+			r = 0;
+		break;
 	default:
 		r = 0;
 	}
@@ -2621,6 +2627,23 @@ long kvm_arch_vm_ioctl(struct file *filp,
 			r = -EFAULT;
 		break;
 	}
+	case KVM_S390_ZPCI_OP: {
+		struct kvm_s390_zpci_op args;
+
+		r = -EINVAL;
+		if (!IS_ENABLED(CONFIG_VFIO_PCI))
+			break;
+		if (copy_from_user(&args, argp, sizeof(args))) {
+			r = -EFAULT;
+			break;
+		}
+		r = kvm_s390_pci_zpci_op(kvm, &args);
+		if (r)
+			break;
+		if (copy_to_user(argp, &args, sizeof(args)))
+			r = -EFAULT;
+		break;
+	}
 	default:
 		r = -ENOTTY;
 	}
diff --git a/arch/s390/kvm/pci.c b/arch/s390/kvm/pci.c
index 66565f5f3f43..d941657bcb39 100644
--- a/arch/s390/kvm/pci.c
+++ b/arch/s390/kvm/pci.c
@@ -588,6 +588,71 @@ void kvm_s390_pci_clear_list(struct kvm *kvm)
 		unregister_kvm(kzdev->zdev);
 }
 
+static struct kvm_zdev *get_kzdev_by_fh(struct kvm *kvm, u32 fh)
+{
+	struct kvm_zdev *kzdev, *retval = NULL;
+
+	spin_lock(&kvm->arch.kzdev_list_lock);
+	list_for_each_entry(kzdev, &kvm->arch.kzdev_list, entry) {
+		if (kzdev->zdev->fh == fh) {
+			retval = kzdev;
+			break;
+		}
+	}
+	spin_unlock(&kvm->arch.kzdev_list_lock);
+
+	return retval;
+}
+
+static int kvm_s390_pci_zpci_reg_aen(struct zpci_dev *zdev,
+				     struct kvm_s390_zpci_op *args)
+{
+	struct zpci_fib fib = {};
+
+	fib.fmt0.aibv = args->u.reg_aen.ibv;
+	fib.fmt0.isc = args->u.reg_aen.isc;
+	fib.fmt0.noi = args->u.reg_aen.noi;
+	if (args->u.reg_aen.sb != 0) {
+		fib.fmt0.aisb = args->u.reg_aen.sb;
+		fib.fmt0.aisbo = args->u.reg_aen.sbo;
+		fib.fmt0.sum = 1;
+	} else {
+		fib.fmt0.aisb = 0;
+		fib.fmt0.aisbo = 0;
+		fib.fmt0.sum = 0;
+	}
+
+	if (args->u.reg_aen.flags & KVM_S390_ZPCIOP_REGAEN_HOST)
+		return kvm_s390_pci_aif_enable(zdev, &fib, true);
+	else
+		return kvm_s390_pci_aif_enable(zdev, &fib, false);
+}
+
+int kvm_s390_pci_zpci_op(struct kvm *kvm, struct kvm_s390_zpci_op *args)
+{
+	struct kvm_zdev *kzdev;
+	struct zpci_dev *zdev;
+	int r;
+
+	kzdev = get_kzdev_by_fh(kvm, args->fh);
+	if (!kzdev)
+		return -ENODEV;
+	zdev = kzdev->zdev;
+
+	switch (args->op) {
+	case KVM_S390_ZPCIOP_REG_AEN:
+		r = kvm_s390_pci_zpci_reg_aen(zdev, args);
+		break;
+	case KVM_S390_ZPCIOP_DEREG_AEN:
+		r = kvm_s390_pci_aif_disable(zdev, false);
+		break;
+	default:
+		r = -EINVAL;
+	}
+
+	return r;
+}
+
 int kvm_s390_pci_init(void)
 {
 	aift = kzalloc(sizeof(struct zpci_aift), GFP_KERNEL);
diff --git a/arch/s390/kvm/pci.h b/arch/s390/kvm/pci.h
index cb5ec3208923..b053b50c0904 100644
--- a/arch/s390/kvm/pci.h
+++ b/arch/s390/kvm/pci.h
@@ -59,6 +59,8 @@ void kvm_s390_pci_aen_exit(void);
 void kvm_s390_pci_init_list(struct kvm *kvm);
 void kvm_s390_pci_clear_list(struct kvm *kvm);
 
+int kvm_s390_pci_zpci_op(struct kvm *kvm, struct kvm_s390_zpci_op *args);
+
 int kvm_s390_pci_init(void);
 
 static inline bool kvm_s390_pci_interp_allowed(void)
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 91a6fe4e02c0..014adb30d42c 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1144,6 +1144,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_S390_MEM_OP_EXTENSION 211
 #define KVM_CAP_PMU_CAPABILITY 212
 #define KVM_CAP_DISABLE_QUIRKS2 213
+#define KVM_CAP_S390_ZPCI_OP 214
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
@@ -2060,4 +2061,34 @@ struct kvm_stats_desc {
 /* Available with KVM_CAP_XSAVE2 */
 #define KVM_GET_XSAVE2		  _IOR(KVMIO,  0xcf, struct kvm_xsave)
 
+/* Available with KVM_CAP_S390_ZPCI_OP */
+#define KVM_S390_ZPCI_OP	  _IOW(KVMIO,  0xd0, struct kvm_s390_zpci_op)
+
+struct kvm_s390_zpci_op {
+	/* in */
+	__u32 fh;		/* target device */
+	__u8  op;		/* operation to perform */
+	__u8  pad[3];
+	union {
+		/* for KVM_S390_ZPCIOP_REG_AEN */
+		struct {
+			__u64 ibv;	/* Guest addr of interrupt bit vector */
+			__u64 sb;	/* Guest addr of summary bit */
+			__u32 flags;
+			__u32 noi;	/* Number of interrupts */
+			__u8 isc;	/* Guest interrupt subclass */
+			__u8 sbo;	/* Offset of guest summary bit vector */
+			__u16 pad;
+		} reg_aen;
+		__u64 reserved[8];
+	} u;
+};
+
+/* types for kvm_s390_zpci_op->op */
+#define KVM_S390_ZPCIOP_REG_AEN		0
+#define KVM_S390_ZPCIOP_DEREG_AEN	1
+
+/* flags for kvm_s390_zpci_op->u.reg_aen.flags */
+#define KVM_S390_ZPCIOP_REGAEN_HOST	(1 << 0)
+
 #endif /* __LINUX_KVM_H */
-- 
2.27.0

[PATCH v5 20/21] KVM: s390: introduce CPU feature for zPCI Interpretation

[Matthew Rosato]

KVM_S390_VM_CPU_FEAT_ZPCI_INTERP relays whether zPCI interpretive
execution is possible based on the available hardware facilities.

Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
---
 arch/s390/include/uapi/asm/kvm.h | 1 +
 arch/s390/kvm/kvm-s390.c         | 4 ++++
 2 files changed, 5 insertions(+)

diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h
index 7a6b14874d65..ed06458a871f 100644
--- a/arch/s390/include/uapi/asm/kvm.h
+++ b/arch/s390/include/uapi/asm/kvm.h
@@ -130,6 +130,7 @@ struct kvm_s390_vm_cpu_machine {
 #define KVM_S390_VM_CPU_FEAT_PFMFI	11
 #define KVM_S390_VM_CPU_FEAT_SIGPIF	12
 #define KVM_S390_VM_CPU_FEAT_KSS	13
+#define KVM_S390_VM_CPU_FEAT_ZPCI_INTERP 14
 struct kvm_s390_vm_cpu_feat {
 	__u64 feat[16];
 };
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 65a53e22f686..d6988bd22df8 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -433,6 +433,10 @@ static void kvm_s390_cpu_feat_init(void)
 	if (test_facility(151)) /* DFLTCC */
 		__insn32_query(INSN_DFLTCC, kvm_s390_available_subfunc.dfltcc);
 
+	/* zPCI Interpretation */
+	if (kvm_s390_pci_interp_allowed())
+		allow_cpu_feat(KVM_S390_VM_CPU_FEAT_ZPCI_INTERP);
+
 	if (MACHINE_HAS_ESOP)
 		allow_cpu_feat(KVM_S390_VM_CPU_FEAT_ESOP);
 	/*
-- 
2.27.0

[PATCH v5 21/21] MAINTAINERS: additional files related kvm s390 pci passthrough

[Matthew Rosato]

Add entries from the s390 kvm subdirectory related to pci passthrough.

Acked-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index fd768d43e048..0d9e991820a8 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -17259,6 +17259,7 @@ M:	Eric Farman <farman@linux.ibm.com>
 L:	linux-s390@vger.kernel.org
 L:	kvm@vger.kernel.org
 S:	Supported
+F:	arch/s390/kvm/pci*
 F:	drivers/vfio/pci/vfio_pci_zdev.c
 F:	include/uapi/linux/vfio_zdev.h
 
-- 
2.27.0

Re: [PATCH v5 08/21] s390/pci: stash associated GISA designation

[Niklas Schnelle]

On Mon, 2022-04-04 at 13:43 -0400, Matthew Rosato wrote:
> For passthrough devices, we will need to know the GISA designation of the
> guest if interpretation facilities are to be used.  Setup to stash this in
> the zdev and set a default of 0 (no GISA designation) for now; a subsequent
> patch will set a valid GISA designation for passthrough devices.
> Also, extend mpcific routines to specify this stashed designation as part
> of the mpcific command.
> 
> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
> ---
>  arch/s390/include/asm/pci.h     | 1 +
>  arch/s390/include/asm/pci_clp.h | 3 ++-
>  arch/s390/pci/pci.c             | 6 ++++++
>  arch/s390/pci/pci_clp.c         | 5 +++++
>  arch/s390/pci/pci_irq.c         | 5 +++++
>  5 files changed, 19 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h
> index fdb9745ee998..42a4a312a6dd 100644
> --- a/arch/s390/include/asm/pci.h
> +++ b/arch/s390/include/asm/pci.h
> @@ -123,6 +123,7 @@ struct zpci_dev {
>  	enum zpci_state state;
>  	u32		fid;		/* function ID, used by sclp */
>  	u32		fh;		/* function handle, used by insn's */
> +	u32		gisa;		/* GISA designation for passthrough */
>  	u16		vfn;		/* virtual function number */
>  	u16		pchid;		/* physical channel ID */
>  	u8		pfgid;		/* function group ID */
> diff --git a/arch/s390/include/asm/pci_clp.h b/arch/s390/include/asm/pci_clp.h
> index 1f4b666e85ee..f3286bc5ba6e 100644
> --- a/arch/s390/include/asm/pci_clp.h
> +++ b/arch/s390/include/asm/pci_clp.h
> @@ -173,7 +173,8 @@ struct clp_req_set_pci {
>  	u16 reserved2;
>  	u8 oc;				/* operation controls */
>  	u8 ndas;			/* number of dma spaces */
> -	u64 reserved3;
> +	u32 reserved3;
> +	u32 gisa;			/* GISA designation */
>  } __packed;
>  
>  /* Set PCI function response */
> diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c
> index e563cb65c0c4..a86cd1cbb80e 100644
> --- a/arch/s390/pci/pci.c
> +++ b/arch/s390/pci/pci.c
> @@ -120,6 +120,7 @@ int zpci_register_ioat(struct zpci_dev *zdev, u8 dmaas,
>  	fib.pba = base;
>  	fib.pal = limit;
>  	fib.iota = iota | ZPCI_IOTA_RTTO_FLAG;
> +	fib.gd = zdev->gisa;
>  	cc = zpci_mod_fc(req, &fib, &status);
>  	if (cc)
>  		zpci_dbg(3, "reg ioat fid:%x, cc:%d, status:%d\n", zdev->fid, cc, status);
> @@ -133,6 +134,8 @@ int zpci_unregister_ioat(struct zpci_dev *zdev, u8 dmaas)
>  	struct zpci_fib fib = {0};
>  	u8 cc, status;
>  
> +	fib.gd = zdev->gisa;

This could go into the initializer which becomes:

   struct zpci_fib fib = {.gd = zdev->gisa};

> +
>  	cc = zpci_mod_fc(req, &fib, &status);
>  	if (cc)
>  		zpci_dbg(3, "unreg ioat fid:%x, cc:%d, status:%d\n", zdev->fid, cc, status);
> @@ -160,6 +163,7 @@ int zpci_fmb_enable_device(struct zpci_dev *zdev)
>  	atomic64_set(&zdev->unmapped_pages, 0);
>  
>  	fib.fmb_addr = virt_to_phys(zdev->fmb);
> +	fib.gd = zdev->gisa;
>  	cc = zpci_mod_fc(req, &fib, &status);
>  	if (cc) {
>  		kmem_cache_free(zdev_fmb_cache, zdev->fmb);
> @@ -178,6 +182,8 @@ int zpci_fmb_disable_device(struct zpci_dev *zdev)
>  	if (!zdev->fmb)
>  		return -EINVAL;
>  
> +	fib.gd = zdev->gisa;
> +
>  	/* Function measurement is disabled if fmb address is zero */
>  	cc = zpci_mod_fc(req, &fib, &status);
>  	if (cc == 3) /* Function already gone. */
> diff --git a/arch/s390/pci/pci_clp.c b/arch/s390/pci/pci_clp.c
> index 1057d7af4a55..deed35edea14 100644
> --- a/arch/s390/pci/pci_clp.c
> +++ b/arch/s390/pci/pci_clp.c
> @@ -229,12 +229,16 @@ static int clp_set_pci_fn(struct zpci_dev *zdev, u32 *fh, u8 nr_dma_as, u8 comma
>  {
>  	struct clp_req_rsp_set_pci *rrb;
>  	int rc, retries = 100;
> +	u32 gisa = 0;
>  
>  	*fh = 0;
>  	rrb = clp_alloc_block(GFP_KERNEL);
>  	if (!rrb)
>  		return -ENOMEM;
>  
> +	if (command != CLP_SET_DISABLE_PCI_FN)
> +		gisa = zdev->gisa;
> +
>  	do {
>  		memset(rrb, 0, sizeof(*rrb));
>  		rrb->request.hdr.len = sizeof(rrb->request);
> @@ -243,6 +247,7 @@ static int clp_set_pci_fn(struct zpci_dev *zdev, u32 *fh, u8 nr_dma_as, u8 comma
>  		rrb->request.fh = zdev->fh;
>  		rrb->request.oc = command;
>  		rrb->request.ndas = nr_dma_as;
> +		rrb->request.gisa = gisa;
>  
>  		rc = clp_req(rrb, CLP_LPS_PCI);
>  		if (rrb->response.hdr.rsp == CLP_RC_SETPCIFN_BUSY) {
> diff --git a/arch/s390/pci/pci_irq.c b/arch/s390/pci/pci_irq.c
> index f2b3145b6697..a2b42a63a53b 100644
> --- a/arch/s390/pci/pci_irq.c
> +++ b/arch/s390/pci/pci_irq.c
> @@ -43,6 +43,7 @@ static int zpci_set_airq(struct zpci_dev *zdev)
>  	fib.fmt0.aibvo = 0;	/* each zdev has its own interrupt vector */
>  	fib.fmt0.aisb = virt_to_phys(zpci_sbv->vector) + (zdev->aisb / 64) * 8;
>  	fib.fmt0.aisbo = zdev->aisb & 63;
> +	fib.gd = zdev->gisa;
>  
>  	return zpci_mod_fc(req, &fib, &status) ? -EIO : 0;
>  }
> @@ -54,6 +55,8 @@ static int zpci_clear_airq(struct zpci_dev *zdev)
>  	struct zpci_fib fib = {0};
>  	u8 cc, status;
>  
> +	fib.gd = zdev->gisa;
> +

Same as in zpci_register_ioat()

>  	cc = zpci_mod_fc(req, &fib, &status);
>  	if (cc == 3 || (cc == 1 && status == 24))
>  		/* Function already gone or IRQs already deregistered. */
> @@ -72,6 +75,7 @@ static int zpci_set_directed_irq(struct zpci_dev *zdev)
>  	fib.fmt = 1;
>  	fib.fmt1.noi = zdev->msi_nr_irqs;
>  	fib.fmt1.dibvo = zdev->msi_first_bit;
> +	fib.gd = zdev->gisa;
>  
>  	return zpci_mod_fc(req, &fib, &status) ? -EIO : 0;
>  }
> @@ -84,6 +88,7 @@ static int zpci_clear_directed_irq(struct zpci_dev *zdev)
>  	u8 cc, status;
>  
>  	fib.fmt = 1;
> +	fib.gd = zdev->gisa;
>  	cc = zpci_mod_fc(req, &fib, &status);
>  	if (cc == 3 || (cc == 1 && status == 24))
>  		/* Function already gone or IRQs already deregistered. */

Looks good with or without using an initializer:
Reviewed-by: Niklas Schnelle <schnelle@linux.ibm.com>

Re: [PATCH v5 10/21] KVM: s390: pci: add basic kvm_zdev structure

[Niklas Schnelle]

On Mon, 2022-04-04 at 13:43 -0400, Matthew Rosato wrote:
> This structure will be used to carry kvm passthrough information related to
> zPCI devices.
> 
> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
> ---
>  arch/s390/include/asm/pci.h |  3 +++
>  arch/s390/kvm/Makefile      |  1 +
>  arch/s390/kvm/pci.c         | 38 +++++++++++++++++++++++++++++++++++++
>  arch/s390/kvm/pci.h         | 21 ++++++++++++++++++++
>  4 files changed, 63 insertions(+)
>  create mode 100644 arch/s390/kvm/pci.c
>  create mode 100644 arch/s390/kvm/pci.h
> 
> diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h
> index 4c5b8fbc2079..9eb20cebaa18 100644
> --- a/arch/s390/include/asm/pci.h
> +++ b/arch/s390/include/asm/pci.h
> @@ -97,6 +97,7 @@ struct zpci_bar_struct {
>  };
>  
>  struct s390_domain;
> +struct kvm_zdev;
>  
>  #define ZPCI_FUNCTIONS_PER_BUS 256
>  struct zpci_bus {
> @@ -190,6 +191,8 @@ struct zpci_dev {
>  	struct dentry	*debugfs_dev;
>  
>  	struct s390_domain *s390_domain; /* s390 IOMMU domain data */
> +
> +	struct kvm_zdev *kzdev; /* passthrough data */
>  };

The struct zpci_dev tries to use semantic groups in its formatting.
It's not perfect and we probably need to clean this up to remove some
holes in the future. For now let's put the new kzdev without a blank
line together with s390_domain and add a "section comment" like
"IOMMU and passthrough".
Also I'd drop the "... data" part of the line end comment or even drop
it entirely, the name is pretty clear already when combined with the
section comment.

With that Reviewed-by: Niklas Schnelle <schnelle@linux.ibm.com>

Re: [PATCH v5 14/21] KVM: s390: pci: provide routines for enabling/disabling interrupt forwarding

[Niklas Schnelle]

On Mon, 2022-04-04 at 13:43 -0400, Matthew Rosato wrote:
> These routines will be wired into a kvm ioctl in order to respond to
> requests to enable / disable a device for Adapter Event Notifications /
> Adapter Interuption Forwarding.
> 
> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
> ---
>  arch/s390/kvm/pci.c      | 247 +++++++++++++++++++++++++++++++++++++++
>  arch/s390/kvm/pci.h      |   1 +
>  arch/s390/pci/pci_insn.c |   1 +
>  3 files changed, 249 insertions(+)
> 
> diff --git a/arch/s390/kvm/pci.c b/arch/s390/kvm/pci.c
> index 01bd8a2f503b..f0fd68569a9d 100644
> --- a/arch/s390/kvm/pci.c
> +++ b/arch/s390/kvm/pci.c
> @@ -11,6 +11,7 @@
>  #include <linux/pci.h>
>  #include <asm/pci.h>
>  #include <asm/pci_insn.h>
> +#include <asm/pci_io.h>
>  #include "pci.h"
>  
>  struct zpci_aift *aift;
> @@ -152,6 +153,252 @@ int kvm_s390_pci_aen_init(u8 nisc)
>  	return rc;
>  }
>  
> +/* Modify PCI: Register floating adapter interruption forwarding */
> +static int kvm_zpci_set_airq(struct zpci_dev *zdev)
> +{
> +	u64 req = ZPCI_CREATE_REQ(zdev->fh, 0, ZPCI_MOD_FC_REG_INT);
> +	struct zpci_fib fib = {};

Hmm this one uses '{}' as initializer while all current callers of
zpci_mod_fc() use '{0}'. As far as I know the empty braces are a GNU
extension so should work for the kernel but for consistency I'd go with
'{0}' or possibly '{.foo = bar, ...}' where that is more readable.
There too uninitialized fields will be set to 0. Unless of course there
is a conflicting KVM convention that I don't know about.

> +	u8 status;
> +
> +	fib.fmt0.isc = zdev->kzdev->fib.fmt0.isc;
> +	fib.fmt0.sum = 1;       /* enable summary notifications */
> +	fib.fmt0.noi = airq_iv_end(zdev->aibv);
> +	fib.fmt0.aibv = virt_to_phys(zdev->aibv->vector);
> +	fib.fmt0.aibvo = 0;
> +	fib.fmt0.aisb = virt_to_phys(aift->sbv->vector + (zdev->aisb / 64) * 8);
> +	fib.fmt0.aisbo = zdev->aisb & 63;
> +	fib.gd = zdev->gisa;
> +
> +	return zpci_mod_fc(req, &fib, &status) ? -EIO : 0;
> +}
> +
> +/* Modify PCI: Unregister floating adapter interruption forwarding */
> +static int kvm_zpci_clear_airq(struct zpci_dev *zdev)
> +{
> +	u64 req = ZPCI_CREATE_REQ(zdev->fh, 0, ZPCI_MOD_FC_DEREG_INT);
> +	struct zpci_fib fib = {};

Same here

> +	u8 cc, status;
> +
> +	fib.gd = zdev->gisa;
> +
> +	cc = zpci_mod_fc(req, &fib, &status);
> +	if (cc == 3 || (cc == 1 && status == 24))
> +		/* Function already gone or IRQs already deregistered. */
> +		cc = 0;
> +
> +	return cc ? -EIO : 0;
> +}
> +
> 
---8<---
>  int kvm_s390_pci_dev_open(struct zpci_dev *zdev)
>  {
>  	struct kvm_zdev *kzdev;
> diff --git a/arch/s390/kvm/pci.h b/arch/s390/kvm/pci.h
> index d4997e2236ef..b4bf3d1d4b66 100644
> --- a/arch/s390/kvm/pci.h
> +++ b/arch/s390/kvm/pci.h
> @@ -20,6 +20,7 @@
>  struct kvm_zdev {
>  	struct zpci_dev *zdev;
>  	struct kvm *kvm;
> +	struct zpci_fib fib;
>  };
>  
>  struct zpci_gaite {
> diff --git a/arch/s390/pci/pci_insn.c b/arch/s390/pci/pci_insn.c
> index 4c6967b73932..cd9fb186a6be 100644
> --- a/arch/s390/pci/pci_insn.c
> +++ b/arch/s390/pci/pci_insn.c
> @@ -60,6 +60,7 @@ u8 zpci_mod_fc(u64 req, struct zpci_fib *fib, u8 *status)
>  
>  	return cc;
>  }
> +EXPORT_SYMBOL_GPL(zpci_mod_fc);
>  
>  /* Refresh PCI Translations */
>  static inline u8 __rpcit(u64 fn, u64 addr, u64 range, u8 *status)

Acked-by: Niklas Schnelle <schnelle@linux.ibm.com>

Re: [PATCH v5 14/21] KVM: s390: pci: provide routines for enabling/disabling interrupt forwarding

[Matthew Rosato]

On 4/5/22 9:39 AM, Niklas Schnelle wrote:
> On Mon, 2022-04-04 at 13:43 -0400, Matthew Rosato wrote:
>> These routines will be wired into a kvm ioctl in order to respond to
>> requests to enable / disable a device for Adapter Event Notifications /
>> Adapter Interuption Forwarding.
>>
>> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
>> ---
>>   arch/s390/kvm/pci.c      | 247 +++++++++++++++++++++++++++++++++++++++
>>   arch/s390/kvm/pci.h      |   1 +
>>   arch/s390/pci/pci_insn.c |   1 +
>>   3 files changed, 249 insertions(+)
>>
>> diff --git a/arch/s390/kvm/pci.c b/arch/s390/kvm/pci.c
>> index 01bd8a2f503b..f0fd68569a9d 100644
>> --- a/arch/s390/kvm/pci.c
>> +++ b/arch/s390/kvm/pci.c
>> @@ -11,6 +11,7 @@
>>   #include <linux/pci.h>
>>   #include <asm/pci.h>
>>   #include <asm/pci_insn.h>
>> +#include <asm/pci_io.h>
>>   #include "pci.h"
>>   
>>   struct zpci_aift *aift;
>> @@ -152,6 +153,252 @@ int kvm_s390_pci_aen_init(u8 nisc)
>>   	return rc;
>>   }
>>   
>> +/* Modify PCI: Register floating adapter interruption forwarding */
>> +static int kvm_zpci_set_airq(struct zpci_dev *zdev)
>> +{
>> +	u64 req = ZPCI_CREATE_REQ(zdev->fh, 0, ZPCI_MOD_FC_REG_INT);
>> +	struct zpci_fib fib = {};
> 
> Hmm this one uses '{}' as initializer while all current callers of
> zpci_mod_fc() use '{0}'. As far as I know the empty braces are a GNU
> extension so should work for the kernel but for consistency I'd go with
> '{0}' or possibly '{.foo = bar, ...}' where that is more readable.
> There too uninitialized fields will be set to 0. Unless of course there
> is a conflicting KVM convention that I don't know about.

No convention that I'm aware of, I previously had fib = {0} based on the 
same rationale you describe and changed to fib = {} per review request 
from Pierre a few versions back.  I don't have a strong preference, but 
I did not note any functional difference between the two and see a bunch 
of examples of both methods throughout the kernel.

Re: [PATCH v5 10/21] KVM: s390: pci: add basic kvm_zdev structure

[Matthew Rosato]

On 4/5/22 4:20 AM, Niklas Schnelle wrote:
> On Mon, 2022-04-04 at 13:43 -0400, Matthew Rosato wrote:
>> This structure will be used to carry kvm passthrough information related to
>> zPCI devices.
>>
>> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
>> ---
>>   arch/s390/include/asm/pci.h |  3 +++
>>   arch/s390/kvm/Makefile      |  1 +
>>   arch/s390/kvm/pci.c         | 38 +++++++++++++++++++++++++++++++++++++
>>   arch/s390/kvm/pci.h         | 21 ++++++++++++++++++++
>>   4 files changed, 63 insertions(+)
>>   create mode 100644 arch/s390/kvm/pci.c
>>   create mode 100644 arch/s390/kvm/pci.h
>>
>> diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h
>> index 4c5b8fbc2079..9eb20cebaa18 100644
>> --- a/arch/s390/include/asm/pci.h
>> +++ b/arch/s390/include/asm/pci.h
>> @@ -97,6 +97,7 @@ struct zpci_bar_struct {
>>   };
>>   
>>   struct s390_domain;
>> +struct kvm_zdev;
>>   
>>   #define ZPCI_FUNCTIONS_PER_BUS 256
>>   struct zpci_bus {
>> @@ -190,6 +191,8 @@ struct zpci_dev {
>>   	struct dentry	*debugfs_dev;
>>   
>>   	struct s390_domain *s390_domain; /* s390 IOMMU domain data */
>> +
>> +	struct kvm_zdev *kzdev; /* passthrough data */
>>   };
> 
> The struct zpci_dev tries to use semantic groups in its formatting.
> It's not perfect and we probably need to clean this up to remove some
> holes in the future. For now let's put the new kzdev without a blank
> line together with s390_domain and add a "section comment" like
> "IOMMU and passthrough".
> Also I'd drop the "... data" part of the line end comment or even drop
> it entirely, the name is pretty clear already when combined with the
> section comment.

Sure, will do

> 
> With that Reviewed-by: Niklas Schnelle <schnelle@linux.ibm.com>
> 

Thanks!

Re: [PATCH v5 14/21] KVM: s390: pci: provide routines for enabling/disabling interrupt forwarding

[Pierre Morel]

On 4/5/22 15:48, Matthew Rosato wrote:
> On 4/5/22 9:39 AM, Niklas Schnelle wrote:
>> On Mon, 2022-04-04 at 13:43 -0400, Matthew Rosato wrote:
>>> These routines will be wired into a kvm ioctl in order to respond to
>>> requests to enable / disable a device for Adapter Event Notifications /
>>> Adapter Interuption Forwarding.
>>>
>>> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
>>> ---
>>>   arch/s390/kvm/pci.c      | 247 +++++++++++++++++++++++++++++++++++++++
>>>   arch/s390/kvm/pci.h      |   1 +
>>>   arch/s390/pci/pci_insn.c |   1 +
>>>   3 files changed, 249 insertions(+)
>>>
>>> diff --git a/arch/s390/kvm/pci.c b/arch/s390/kvm/pci.c
>>> index 01bd8a2f503b..f0fd68569a9d 100644
>>> --- a/arch/s390/kvm/pci.c
>>> +++ b/arch/s390/kvm/pci.c
>>> @@ -11,6 +11,7 @@
>>>   #include <linux/pci.h>
>>>   #include <asm/pci.h>
>>>   #include <asm/pci_insn.h>
>>> +#include <asm/pci_io.h>
>>>   #include "pci.h"
>>>   struct zpci_aift *aift;
>>> @@ -152,6 +153,252 @@ int kvm_s390_pci_aen_init(u8 nisc)
>>>       return rc;
>>>   }
>>> +/* Modify PCI: Register floating adapter interruption forwarding */
>>> +static int kvm_zpci_set_airq(struct zpci_dev *zdev)
>>> +{
>>> +    u64 req = ZPCI_CREATE_REQ(zdev->fh, 0, ZPCI_MOD_FC_REG_INT);
>>> +    struct zpci_fib fib = {};
>>
>> Hmm this one uses '{}' as initializer while all current callers of
>> zpci_mod_fc() use '{0}'. As far as I know the empty braces are a GNU
>> extension so should work for the kernel but for consistency I'd go with
>> '{0}' or possibly '{.foo = bar, ...}' where that is more readable.
>> There too uninitialized fields will be set to 0. Unless of course there
>> is a conflicting KVM convention that I don't know about.
> 
> No convention that I'm aware of, I previously had fib = {0} based on the 
> same rationale you describe and changed to fib = {} per review request 
> from Pierre a few versions back.  I don't have a strong preference, but 
> I did not note any functional difference between the two and see a bunch 
> of examples of both methods throughout the kernel.
> 

Was stupid of me to comment that, as you said there are no difference, 
so do as you want.


-- 
Pierre Morel
IBM Lab Boeblingen

Re: [PATCH v5 16/21] KVM: vfio: add s390x hook to register KVM guest designation

[Jason Gunthorpe]

On Mon, Apr 04, 2022 at 01:43:44PM -0400, Matthew Rosato wrote:
> At the time a KVM is associated with a vfio group, s390x zPCI devices
> must register a special guest indication (GISA designation) to allow
> for the use of interpretive execution facilities.  This indication is
> used to ensure that only the specified KVM can interact with the device.
> Similarly, the indication must be removed once the KVM is no longer
> associated with the device.
> 
> This patch adds an s390-specific hook to invoke a KVM registration routine
> for each device associated with the iommu group; in reality, it will be a
> NOP for all but zPCI devices on s390x.
> 
> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
> ---
>  virt/kvm/vfio.c | 35 ++++++++++++++++++++++++++++++++++-
>  1 file changed, 34 insertions(+), 1 deletion(-)

I wonder if this should be done in the vfio_pci side from the existing
kvm notifier

Jason

Re: [PATCH v5 15/21] KVM: s390: pci: add routines to start/stop interpretive execution

[Jason Gunthorpe]

On Mon, Apr 04, 2022 at 01:43:43PM -0400, Matthew Rosato wrote:
> +int kvm_s390_pci_register_kvm(struct device *dev, void *data)
> +{
> +	struct zpci_dev *zdev = NULL;
> +	struct kvm *kvm = data;
> +
> +	/* Only proceed for zPCI devices, quietly ignore others */
> +	if (dev_is_pci(dev))
> +		zdev = to_zpci_dev(dev);
> +	if (!zdev)
> +		return 0;

Especially since this only works if we have zpci device

So having the zpci code hook the kvm notifier and then call the arch
code from the zpci area seems pretty OK

Also why is a struct kvm * being passed as a void *?

Jason

Re: [PATCH v5 14/21] KVM: s390: pci: provide routines for enabling/disabling interrupt forwarding

[Jason Gunthorpe]

On Tue, Apr 05, 2022 at 03:39:19PM +0200, Niklas Schnelle wrote:
> On Mon, 2022-04-04 at 13:43 -0400, Matthew Rosato wrote:
> > These routines will be wired into a kvm ioctl in order to respond to
> > requests to enable / disable a device for Adapter Event Notifications /
> > Adapter Interuption Forwarding.
> > 
> > Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
> >  arch/s390/kvm/pci.c      | 247 +++++++++++++++++++++++++++++++++++++++
> >  arch/s390/kvm/pci.h      |   1 +
> >  arch/s390/pci/pci_insn.c |   1 +
> >  3 files changed, 249 insertions(+)
> > 
> > diff --git a/arch/s390/kvm/pci.c b/arch/s390/kvm/pci.c
> > index 01bd8a2f503b..f0fd68569a9d 100644
> > +++ b/arch/s390/kvm/pci.c
> > @@ -11,6 +11,7 @@
> >  #include <linux/pci.h>
> >  #include <asm/pci.h>
> >  #include <asm/pci_insn.h>
> > +#include <asm/pci_io.h>
> >  #include "pci.h"
> >  
> >  struct zpci_aift *aift;
> > @@ -152,6 +153,252 @@ int kvm_s390_pci_aen_init(u8 nisc)
> >  	return rc;
> >  }
> >  
> > +/* Modify PCI: Register floating adapter interruption forwarding */
> > +static int kvm_zpci_set_airq(struct zpci_dev *zdev)
> > +{
> > +	u64 req = ZPCI_CREATE_REQ(zdev->fh, 0, ZPCI_MOD_FC_REG_INT);
> > +	struct zpci_fib fib = {};
> 
> Hmm this one uses '{}' as initializer while all current callers of
> zpci_mod_fc() use '{0}'. As far as I know the empty braces are a GNU
> extension so should work for the kernel but for consistency I'd go with
> '{0}' or possibly '{.foo = bar, ...}' where that is more readable.
> There too uninitialized fields will be set to 0. Unless of course there
> is a conflicting KVM convention that I don't know about.

{} is not a GNU extension, it is the preferred way to write it.

The standard has a weird distinction between {} and {0} that results
in different behavior.

Jason

Re: [PATCH v5 14/21] KVM: s390: pci: provide routines for enabling/disabling interrupt forwarding

[Heiko Carstens]

On Fri, Apr 08, 2022 at 09:48:24AM -0300, Jason Gunthorpe wrote:
> On Tue, Apr 05, 2022 at 03:39:19PM +0200, Niklas Schnelle wrote:
> > On Mon, 2022-04-04 at 13:43 -0400, Matthew Rosato wrote:
> > > +	struct zpci_fib fib = {};
> > 
> > Hmm this one uses '{}' as initializer while all current callers of
> > zpci_mod_fc() use '{0}'. As far as I know the empty braces are a GNU
> > extension so should work for the kernel but for consistency I'd go with
> > '{0}' or possibly '{.foo = bar, ...}' where that is more readable.
> > There too uninitialized fields will be set to 0. Unless of course there
> > is a conflicting KVM convention that I don't know about.
> 
> {} is not a GNU extension, it is the preferred way to write it.
> 
> The standard has a weird distinction between {} and {0} that results
> in different behavior.

Whoever cares: details are described in "6.7.8 Initialization" within
the C standard.

Re: [PATCH v5 15/21] KVM: s390: pci: add routines to start/stop interpretive execution

[Matthew Rosato]

On 4/8/22 8:47 AM, Jason Gunthorpe wrote:
> On Mon, Apr 04, 2022 at 01:43:43PM -0400, Matthew Rosato wrote:
>> +int kvm_s390_pci_register_kvm(struct device *dev, void *data)
>> +{
>> +	struct zpci_dev *zdev = NULL;
>> +	struct kvm *kvm = data;
>> +
>> +	/* Only proceed for zPCI devices, quietly ignore others */
>> +	if (dev_is_pci(dev))
>> +		zdev = to_zpci_dev(dev);
>> +	if (!zdev)
>> +		return 0;
> 
> Especially since this only works if we have zpci device
> 
> So having the zpci code hook the kvm notifier and then call the arch
> code from the zpci area seems pretty OK
> 
> Also why is a struct kvm * being passed as a void *?

Only because the function is intended to be called via 
iommu_group_for_each_dev (next patch) which requires int (*fn)(struct 
device *, void *)

Re: [PATCH v5 15/21] KVM: s390: pci: add routines to start/stop interpretive execution

[Jason Gunthorpe]

On Tue, Apr 12, 2022 at 09:14:36AM -0400, Matthew Rosato wrote:
> On 4/8/22 8:47 AM, Jason Gunthorpe wrote:
> > On Mon, Apr 04, 2022 at 01:43:43PM -0400, Matthew Rosato wrote:
> > > +int kvm_s390_pci_register_kvm(struct device *dev, void *data)
> > > +{
> > > +	struct zpci_dev *zdev = NULL;
> > > +	struct kvm *kvm = data;
> > > +
> > > +	/* Only proceed for zPCI devices, quietly ignore others */
> > > +	if (dev_is_pci(dev))
> > > +		zdev = to_zpci_dev(dev);
> > > +	if (!zdev)
> > > +		return 0;
> > 
> > Especially since this only works if we have zpci device
> > 
> > So having the zpci code hook the kvm notifier and then call the arch
> > code from the zpci area seems pretty OK
> > 
> > Also why is a struct kvm * being passed as a void *?
> 
> Only because the function is intended to be called via
> iommu_group_for_each_dev (next patch) which requires int (*fn)(struct device
> *, void *)

I think this further says this should be called from vfio on the
actual struct device that is assigned to the KVM, not try to deduce it
from the gorup..

Jason

Re: [PATCH v5 16/21] KVM: vfio: add s390x hook to register KVM guest designation

[Matthew Rosato]

On 4/8/22 8:45 AM, Jason Gunthorpe wrote:
> On Mon, Apr 04, 2022 at 01:43:44PM -0400, Matthew Rosato wrote:
>> At the time a KVM is associated with a vfio group, s390x zPCI devices
>> must register a special guest indication (GISA designation) to allow
>> for the use of interpretive execution facilities.  This indication is
>> used to ensure that only the specified KVM can interact with the device.
>> Similarly, the indication must be removed once the KVM is no longer
>> associated with the device.
>>
>> This patch adds an s390-specific hook to invoke a KVM registration routine
>> for each device associated with the iommu group; in reality, it will be a
>> NOP for all but zPCI devices on s390x.
>>
>> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
>> ---
>>   virt/kvm/vfio.c | 35 ++++++++++++++++++++++++++++++++++-
>>   1 file changed, 34 insertions(+), 1 deletion(-)
> 
> I wonder if this should be done in the vfio_pci side from the existing
> kvm notifier
> 

So you mean rather than hooking into virt as I do here, drive something 
out of drivers/vfio/vfio.c:vfio_group_set_kvm?  Note, the kvm notifier 
is handled in vfio, not vfio_pci, so if you want to handle it in 
vfio_pci I think we'd need to add a new routine to vfio_device_ops and 
only define it vfio_pci for s390

e.g.

static const struct vfio_device_ops vfio_pci_ops = {
	.name		= "vfio-pci",
[...]
#ifdef CONFIG_S390
	.set_kvm = vfio_pci_zdev_set_kvm,
#endif
};

and something like...

void vfio_group_set_kvm(struct vfio_group *group, struct kvm *kvm)
{
	struct vfio_device *vdev;
	group->kvm = kvm;

	mutex_lock(&group->device_lock);
	list_for_each_entry(vdev, &group->device_list, group_next) {
		if (vdev->ops->set_kvm)
			it->ops->set_kvm(vdev, kvm);
	}
	mutex_unlock(&group->device_lock);

	blocking_notifier_call_chain(&group->notifier,
				VFIO_GROUP_NOTIFY_SET_KVM, kvm);
}

Re: [PATCH v5 16/21] KVM: vfio: add s390x hook to register KVM guest designation

[Jason Gunthorpe]

On Tue, Apr 12, 2022 at 09:39:44AM -0400, Matthew Rosato wrote:
> On 4/8/22 8:45 AM, Jason Gunthorpe wrote:
> > On Mon, Apr 04, 2022 at 01:43:44PM -0400, Matthew Rosato wrote:
> > > At the time a KVM is associated with a vfio group, s390x zPCI devices
> > > must register a special guest indication (GISA designation) to allow
> > > for the use of interpretive execution facilities.  This indication is
> > > used to ensure that only the specified KVM can interact with the device.
> > > Similarly, the indication must be removed once the KVM is no longer
> > > associated with the device.
> > > 
> > > This patch adds an s390-specific hook to invoke a KVM registration routine
> > > for each device associated with the iommu group; in reality, it will be a
> > > NOP for all but zPCI devices on s390x.
> > > 
> > > Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
> > >   virt/kvm/vfio.c | 35 ++++++++++++++++++++++++++++++++++-
> > >   1 file changed, 34 insertions(+), 1 deletion(-)
> > 
> > I wonder if this should be done in the vfio_pci side from the existing
> > kvm notifier
> > 
> 
> So you mean rather than hooking into virt as I do here, drive something out
> of drivers/vfio/vfio.c:vfio_group_set_kvm?  Note, the kvm notifier is
> handled in vfio, not vfio_pci, so if you want to handle it in vfio_pci I
> think we'd need to add a new routine to vfio_device_ops and only define it
> vfio_pci for s390

I've been thinking about doing that anyhow, exactly for reasons like
this..

> static const struct vfio_device_ops vfio_pci_ops = {
> 	.name		= "vfio-pci",
> [...]
> #ifdef CONFIG_S390
> 	.set_kvm = vfio_pci_zdev_set_kvm,
> #endif
> };
> 
> and something like...
> 
> void vfio_group_set_kvm(struct vfio_group *group, struct kvm *kvm)
> {
> 	struct vfio_device *vdev;
> 	group->kvm = kvm;
> 
> 	mutex_lock(&group->device_lock);
> 	list_for_each_entry(vdev, &group->device_list, group_next) {
> 		if (vdev->ops->set_kvm)
> 			it->ops->set_kvm(vdev, kvm);
> 	}
> 	mutex_unlock(&group->device_lock);

Almost, the device should be open before calling the callback

And you have to inject a callback during open if the device is opened
after the kvm was set.

But I don't think you need to do this, you can just register a
notifier in zpci when it hooks open_device like everything else,
right?

Jason

Re: [PATCH v5 16/21] KVM: vfio: add s390x hook to register KVM guest designation

[Matthew Rosato]

On 4/12/22 9:55 AM, Jason Gunthorpe wrote:
> On Tue, Apr 12, 2022 at 09:39:44AM -0400, Matthew Rosato wrote:
>> On 4/8/22 8:45 AM, Jason Gunthorpe wrote:
>>> On Mon, Apr 04, 2022 at 01:43:44PM -0400, Matthew Rosato wrote:
>>>> At the time a KVM is associated with a vfio group, s390x zPCI devices
>>>> must register a special guest indication (GISA designation) to allow
>>>> for the use of interpretive execution facilities.  This indication is
>>>> used to ensure that only the specified KVM can interact with the device.
>>>> Similarly, the indication must be removed once the KVM is no longer
>>>> associated with the device.
>>>>
>>>> This patch adds an s390-specific hook to invoke a KVM registration routine
>>>> for each device associated with the iommu group; in reality, it will be a
>>>> NOP for all but zPCI devices on s390x.
>>>>
>>>> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
>>>>    virt/kvm/vfio.c | 35 ++++++++++++++++++++++++++++++++++-
>>>>    1 file changed, 34 insertions(+), 1 deletion(-)
>>>
>>> I wonder if this should be done in the vfio_pci side from the existing
>>> kvm notifier
>>>
>>
>> So you mean rather than hooking into virt as I do here, drive something out
>> of drivers/vfio/vfio.c:vfio_group_set_kvm?  Note, the kvm notifier is
>> handled in vfio, not vfio_pci, so if you want to handle it in vfio_pci I
>> think we'd need to add a new routine to vfio_device_ops and only define it
>> vfio_pci for s390
> 
> I've been thinking about doing that anyhow, exactly for reasons like
> this..
> 
>> static const struct vfio_device_ops vfio_pci_ops = {
>> 	.name		= "vfio-pci",
>> [...]
>> #ifdef CONFIG_S390
>> 	.set_kvm = vfio_pci_zdev_set_kvm,
>> #endif
>> };
>>
>> and something like...
>>
>> void vfio_group_set_kvm(struct vfio_group *group, struct kvm *kvm)
>> {
>> 	struct vfio_device *vdev;
>> 	group->kvm = kvm;
>>
>> 	mutex_lock(&group->device_lock);
>> 	list_for_each_entry(vdev, &group->device_list, group_next) {
>> 		if (vdev->ops->set_kvm)
>> 			it->ops->set_kvm(vdev, kvm);
>> 	}
>> 	mutex_unlock(&group->device_lock);
> 
> Almost, the device should be open before calling the callback
> 
> And you have to inject a callback during open if the device is opened
> after the kvm was set.
> 
> But I don't think you need to do this, you can just register a
> notifier in zpci when it hooks open_device like everything else,
> right?

Yes, that would also work -- I was registering a notifier for a few 
prior versions of this series (granted, not from open_device) but got 
the impression I should avoid registering a notifier from within 
vfio_pci_zdev.

I will go ahead and add register/unregister notifiers hooked from 
vfio_pci_core_finish_enable/vfio_pci_core_close_device for zpci (e.g. 
vfio_pci_zdev_{open,close}) and use the notifier events to drive the 
routines from patch 15.

Re: [PATCH v5 08/21] s390/pci: stash associated GISA designation

[Christian Borntraeger]

Am 05.04.22 um 10:03 schrieb Niklas Schnelle:
> On Mon, 2022-04-04 at 13:43 -0400, Matthew Rosato wrote:
>> For passthrough devices, we will need to know the GISA designation of the
>> guest if interpretation facilities are to be used.  Setup to stash this in
>> the zdev and set a default of 0 (no GISA designation) for now; a subsequent
>> patch will set a valid GISA designation for passthrough devices.
>> Also, extend mpcific routines to specify this stashed designation as part
>> of the mpcific command.
>>
>> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>

Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com>

>> ---
>>   arch/s390/include/asm/pci.h     | 1 +
>>   arch/s390/include/asm/pci_clp.h | 3 ++-
>>   arch/s390/pci/pci.c             | 6 ++++++
>>   arch/s390/pci/pci_clp.c         | 5 +++++
>>   arch/s390/pci/pci_irq.c         | 5 +++++
>>   5 files changed, 19 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h
>> index fdb9745ee998..42a4a312a6dd 100644
>> --- a/arch/s390/include/asm/pci.h
>> +++ b/arch/s390/include/asm/pci.h
>> @@ -123,6 +123,7 @@ struct zpci_dev {
>>   	enum zpci_state state;
>>   	u32		fid;		/* function ID, used by sclp */
>>   	u32		fh;		/* function handle, used by insn's */
>> +	u32		gisa;		/* GISA designation for passthrough */
>>   	u16		vfn;		/* virtual function number */
>>   	u16		pchid;		/* physical channel ID */
>>   	u8		pfgid;		/* function group ID */
>> diff --git a/arch/s390/include/asm/pci_clp.h b/arch/s390/include/asm/pci_clp.h
>> index 1f4b666e85ee..f3286bc5ba6e 100644
>> --- a/arch/s390/include/asm/pci_clp.h
>> +++ b/arch/s390/include/asm/pci_clp.h
>> @@ -173,7 +173,8 @@ struct clp_req_set_pci {
>>   	u16 reserved2;
>>   	u8 oc;				/* operation controls */
>>   	u8 ndas;			/* number of dma spaces */
>> -	u64 reserved3;
>> +	u32 reserved3;
>> +	u32 gisa;			/* GISA designation */
>>   } __packed;
>>   
>>   /* Set PCI function response */
>> diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c
>> index e563cb65c0c4..a86cd1cbb80e 100644
>> --- a/arch/s390/pci/pci.c
>> +++ b/arch/s390/pci/pci.c
>> @@ -120,6 +120,7 @@ int zpci_register_ioat(struct zpci_dev *zdev, u8 dmaas,
>>   	fib.pba = base;
>>   	fib.pal = limit;
>>   	fib.iota = iota | ZPCI_IOTA_RTTO_FLAG;
>> +	fib.gd = zdev->gisa;
>>   	cc = zpci_mod_fc(req, &fib, &status);
>>   	if (cc)
>>   		zpci_dbg(3, "reg ioat fid:%x, cc:%d, status:%d\n", zdev->fid, cc, status);
>> @@ -133,6 +134,8 @@ int zpci_unregister_ioat(struct zpci_dev *zdev, u8 dmaas)
>>   	struct zpci_fib fib = {0};
>>   	u8 cc, status;
>>   
>> +	fib.gd = zdev->gisa;
> 
> This could go into the initializer which becomes:
> 
>     struct zpci_fib fib = {.gd = zdev->gisa};
> 
>> +
>>   	cc = zpci_mod_fc(req, &fib, &status);
>>   	if (cc)
>>   		zpci_dbg(3, "unreg ioat fid:%x, cc:%d, status:%d\n", zdev->fid, cc, status);
>> @@ -160,6 +163,7 @@ int zpci_fmb_enable_device(struct zpci_dev *zdev)
>>   	atomic64_set(&zdev->unmapped_pages, 0);
>>   
>>   	fib.fmb_addr = virt_to_phys(zdev->fmb);
>> +	fib.gd = zdev->gisa;
>>   	cc = zpci_mod_fc(req, &fib, &status);
>>   	if (cc) {
>>   		kmem_cache_free(zdev_fmb_cache, zdev->fmb);
>> @@ -178,6 +182,8 @@ int zpci_fmb_disable_device(struct zpci_dev *zdev)
>>   	if (!zdev->fmb)
>>   		return -EINVAL;
>>   
>> +	fib.gd = zdev->gisa;
>> +
>>   	/* Function measurement is disabled if fmb address is zero */
>>   	cc = zpci_mod_fc(req, &fib, &status);
>>   	if (cc == 3) /* Function already gone. */
>> diff --git a/arch/s390/pci/pci_clp.c b/arch/s390/pci/pci_clp.c
>> index 1057d7af4a55..deed35edea14 100644
>> --- a/arch/s390/pci/pci_clp.c
>> +++ b/arch/s390/pci/pci_clp.c
>> @@ -229,12 +229,16 @@ static int clp_set_pci_fn(struct zpci_dev *zdev, u32 *fh, u8 nr_dma_as, u8 comma
>>   {
>>   	struct clp_req_rsp_set_pci *rrb;
>>   	int rc, retries = 100;
>> +	u32 gisa = 0;
>>   
>>   	*fh = 0;
>>   	rrb = clp_alloc_block(GFP_KERNEL);
>>   	if (!rrb)
>>   		return -ENOMEM;
>>   
>> +	if (command != CLP_SET_DISABLE_PCI_FN)
>> +		gisa = zdev->gisa;
>> +
>>   	do {
>>   		memset(rrb, 0, sizeof(*rrb));
>>   		rrb->request.hdr.len = sizeof(rrb->request);
>> @@ -243,6 +247,7 @@ static int clp_set_pci_fn(struct zpci_dev *zdev, u32 *fh, u8 nr_dma_as, u8 comma
>>   		rrb->request.fh = zdev->fh;
>>   		rrb->request.oc = command;
>>   		rrb->request.ndas = nr_dma_as;
>> +		rrb->request.gisa = gisa;
>>   
>>   		rc = clp_req(rrb, CLP_LPS_PCI);
>>   		if (rrb->response.hdr.rsp == CLP_RC_SETPCIFN_BUSY) {
>> diff --git a/arch/s390/pci/pci_irq.c b/arch/s390/pci/pci_irq.c
>> index f2b3145b6697..a2b42a63a53b 100644
>> --- a/arch/s390/pci/pci_irq.c
>> +++ b/arch/s390/pci/pci_irq.c
>> @@ -43,6 +43,7 @@ static int zpci_set_airq(struct zpci_dev *zdev)
>>   	fib.fmt0.aibvo = 0;	/* each zdev has its own interrupt vector */
>>   	fib.fmt0.aisb = virt_to_phys(zpci_sbv->vector) + (zdev->aisb / 64) * 8;
>>   	fib.fmt0.aisbo = zdev->aisb & 63;
>> +	fib.gd = zdev->gisa;
>>   
>>   	return zpci_mod_fc(req, &fib, &status) ? -EIO : 0;
>>   }
>> @@ -54,6 +55,8 @@ static int zpci_clear_airq(struct zpci_dev *zdev)
>>   	struct zpci_fib fib = {0};
>>   	u8 cc, status;
>>   
>> +	fib.gd = zdev->gisa;
>> +
> 
> Same as in zpci_register_ioat()
> 
>>   	cc = zpci_mod_fc(req, &fib, &status);
>>   	if (cc == 3 || (cc == 1 && status == 24))
>>   		/* Function already gone or IRQs already deregistered. */
>> @@ -72,6 +75,7 @@ static int zpci_set_directed_irq(struct zpci_dev *zdev)
>>   	fib.fmt = 1;
>>   	fib.fmt1.noi = zdev->msi_nr_irqs;
>>   	fib.fmt1.dibvo = zdev->msi_first_bit;
>> +	fib.gd = zdev->gisa;
>>   
>>   	return zpci_mod_fc(req, &fib, &status) ? -EIO : 0;
>>   }
>> @@ -84,6 +88,7 @@ static int zpci_clear_directed_irq(struct zpci_dev *zdev)
>>   	u8 cc, status;
>>   
>>   	fib.fmt = 1;
>> +	fib.gd = zdev->gisa;
>>   	cc = zpci_mod_fc(req, &fib, &status);
>>   	if (cc == 3 || (cc == 1 && status == 24))
>>   		/* Function already gone or IRQs already deregistered. */
> 
> Looks good with or without using an initializer:
> Reviewed-by: Niklas Schnelle <schnelle@linux.ibm.com>
> 
> 

Re: [PATCH v5 10/21] KVM: s390: pci: add basic kvm_zdev structure

[Christian Borntraeger]

Am 04.04.22 um 19:43 schrieb Matthew Rosato:
> This structure will be used to carry kvm passthrough information related to
> zPCI devices.
> 
> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>

Reviewed-by: Christian Borntraeger <borntraeger@linux.ibm.com>

Re: [PATCH v5 11/21] KVM: s390: pci: do initial setup for AEN interpretation

[Christian Borntraeger]

Am 04.04.22 um 19:43 schrieb Matthew Rosato:
> Initial setup for Adapter Event Notification Interpretation for zPCI
> passthrough devices.  Specifically, allocate a structure for forwarding of
> adapter events and pass the address of this structure to firmware.
> 
> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
[...]
>   
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 156d1c25a3c1..9db6f8080f71 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -47,6 +47,7 @@
>   #include <asm/fpu/api.h>
>   #include "kvm-s390.h"
>   #include "gaccess.h"
> +#include "pci.h"
>   
>   #define CREATE_TRACE_POINTS
>   #include "trace.h"
> @@ -502,6 +503,14 @@ int kvm_arch_init(void *opaque)
>   		goto out;
>   	}
>   
> +	if (kvm_s390_pci_interp_allowed()) {
> +		rc = kvm_s390_pci_init();
> +		if (rc) {
> +			pr_err("Unable to allocate AIFT for PCI\n");
> +			goto out;
> +		}
> +	}
> +
>   	rc = kvm_s390_gib_init(GAL_ISC);
>   	if (rc)
>   		goto out;

We would not free the aift that was allocated by kvm_s390_pci_init
in kvm_arch_exit.
Wouldnt we re-allocate a new aift when we unload/reload kvm forgetting about the old one?


> diff --git a/arch/s390/kvm/pci.c b/arch/s390/kvm/pci.c
[...]
> +static int zpci_setup_aipb(u8 nisc)
[...]
> +	size = get_order(PAGE_ALIGN(ZPCI_NR_DEVICES *
> +						sizeof(struct zpci_gaite)));
[...]
> +	if (zpci_set_irq_ctrl(SIC_SET_AENI_CONTROLS, 0, zpci_aipb)) {
> +		rc = -EIO;
> +		goto free_gait;
> +	}
> +
> +	return 0;
> +
> +free_gait:
> +	size = get_order(PAGE_ALIGN(ZPCI_NR_DEVICES *
> +				    sizeof(struct zpci_gaite)));

size should still be valid here?

> +	free_pages((unsigned long)aift->gait, size);
> +free_sbv:
> +	airq_iv_release(aift->sbv);
> +	zpci_aif_sbv = 0;
> +free_aipb:
> +	kfree(zpci_aipb);
> +	zpci_aipb = 0;
> +
> +	return rc;
> +}
> +

The remaining parts look sane.

Re: [PATCH v5 11/21] KVM: s390: pci: do initial setup for AEN interpretation

[Matthew Rosato]

On 4/14/22 3:20 AM, Christian Borntraeger wrote:
> 
> 
> Am 04.04.22 um 19:43 schrieb Matthew Rosato:
>> Initial setup for Adapter Event Notification Interpretation for zPCI
>> passthrough devices.  Specifically, allocate a structure for 
>> forwarding of
>> adapter events and pass the address of this structure to firmware.
>>
>> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
> [...]
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index 156d1c25a3c1..9db6f8080f71 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -47,6 +47,7 @@
>>   #include <asm/fpu/api.h>
>>   #include "kvm-s390.h"
>>   #include "gaccess.h"
>> +#include "pci.h"
>>   #define CREATE_TRACE_POINTS
>>   #include "trace.h"
>> @@ -502,6 +503,14 @@ int kvm_arch_init(void *opaque)
>>           goto out;
>>       }
>> +    if (kvm_s390_pci_interp_allowed()) {
>> +        rc = kvm_s390_pci_init();
>> +        if (rc) {
>> +            pr_err("Unable to allocate AIFT for PCI\n");
>> +            goto out;
>> +        }
>> +    }
>> +
>>       rc = kvm_s390_gib_init(GAL_ISC);
>>       if (rc)
>>           goto out;
> 
> We would not free the aift that was allocated by kvm_s390_pci_init
> in kvm_arch_exit.
> Wouldnt we re-allocate a new aift when we unload/reload kvm forgetting 
> about the old one?

Oops, yes it looks like that's the case.  We must back-pocket a certain 
subset of firmware-shared structures (e.g. zpci_aipb and zpci_aif_sbv) 
as these cannot change for the life of the system once registered with 
firmware; but the aift is a kernel-only structure that should be safe to 
free until next module load.  I think this can be done at the end of 
kvm_s390_pci_aen_exit (with some caller adjustments re: the aift mutex)
> 
> 
>> diff --git a/arch/s390/kvm/pci.c b/arch/s390/kvm/pci.c
> [...]
>> +static int zpci_setup_aipb(u8 nisc)
> [...]
>> +    size = get_order(PAGE_ALIGN(ZPCI_NR_DEVICES *
>> +                        sizeof(struct zpci_gaite)));
> [...]
>> +    if (zpci_set_irq_ctrl(SIC_SET_AENI_CONTROLS, 0, zpci_aipb)) {
>> +        rc = -EIO;
>> +        goto free_gait;
>> +    }
>> +
>> +    return 0;
>> +
>> +free_gait:
>> +    size = get_order(PAGE_ALIGN(ZPCI_NR_DEVICES *
>> +                    sizeof(struct zpci_gaite)));
> 
> size should still be valid here?

Good point

> 
>> +    free_pages((unsigned long)aift->gait, size);
>> +free_sbv:
>> +    airq_iv_release(aift->sbv);
>> +    zpci_aif_sbv = 0;
>> +free_aipb:
>> +    kfree(zpci_aipb);
>> +    zpci_aipb = 0;
>> +
>> +    return rc;
>> +}
>> +
> 
> The remaining parts look sane.

Re: [PATCH v5 08/21] s390/pci: stash associated GISA designation

[Pierre Morel]

On 4/4/22 19:43, Matthew Rosato wrote:
> For passthrough devices, we will need to know the GISA designation of the
> guest if interpretation facilities are to be used.  Setup to stash this in
> the zdev and set a default of 0 (no GISA designation) for now; a subsequent
> patch will set a valid GISA designation for passthrough devices.
> Also, extend mpcific routines to specify this stashed designation as part
> of the mpcific command.
> 
> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>


Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>

> ---
>   arch/s390/include/asm/pci.h     | 1 +
>   arch/s390/include/asm/pci_clp.h | 3 ++-
>   arch/s390/pci/pci.c             | 6 ++++++
>   arch/s390/pci/pci_clp.c         | 5 +++++
>   arch/s390/pci/pci_irq.c         | 5 +++++
>   5 files changed, 19 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h
> index fdb9745ee998..42a4a312a6dd 100644
> --- a/arch/s390/include/asm/pci.h
> +++ b/arch/s390/include/asm/pci.h
> @@ -123,6 +123,7 @@ struct zpci_dev {
>   	enum zpci_state state;
>   	u32		fid;		/* function ID, used by sclp */
>   	u32		fh;		/* function handle, used by insn's */
> +	u32		gisa;		/* GISA designation for passthrough */
>   	u16		vfn;		/* virtual function number */
>   	u16		pchid;		/* physical channel ID */
>   	u8		pfgid;		/* function group ID */
> diff --git a/arch/s390/include/asm/pci_clp.h b/arch/s390/include/asm/pci_clp.h
> index 1f4b666e85ee..f3286bc5ba6e 100644
> --- a/arch/s390/include/asm/pci_clp.h
> +++ b/arch/s390/include/asm/pci_clp.h
> @@ -173,7 +173,8 @@ struct clp_req_set_pci {
>   	u16 reserved2;
>   	u8 oc;				/* operation controls */
>   	u8 ndas;			/* number of dma spaces */
> -	u64 reserved3;
> +	u32 reserved3;
> +	u32 gisa;			/* GISA designation */
>   } __packed;
>   
>   /* Set PCI function response */
> diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c
> index e563cb65c0c4..a86cd1cbb80e 100644
> --- a/arch/s390/pci/pci.c
> +++ b/arch/s390/pci/pci.c
> @@ -120,6 +120,7 @@ int zpci_register_ioat(struct zpci_dev *zdev, u8 dmaas,
>   	fib.pba = base;
>   	fib.pal = limit;
>   	fib.iota = iota | ZPCI_IOTA_RTTO_FLAG;
> +	fib.gd = zdev->gisa;
>   	cc = zpci_mod_fc(req, &fib, &status);
>   	if (cc)
>   		zpci_dbg(3, "reg ioat fid:%x, cc:%d, status:%d\n", zdev->fid, cc, status);
> @@ -133,6 +134,8 @@ int zpci_unregister_ioat(struct zpci_dev *zdev, u8 dmaas)
>   	struct zpci_fib fib = {0};
>   	u8 cc, status;
>   
> +	fib.gd = zdev->gisa;
> +
>   	cc = zpci_mod_fc(req, &fib, &status);
>   	if (cc)
>   		zpci_dbg(3, "unreg ioat fid:%x, cc:%d, status:%d\n", zdev->fid, cc, status);
> @@ -160,6 +163,7 @@ int zpci_fmb_enable_device(struct zpci_dev *zdev)
>   	atomic64_set(&zdev->unmapped_pages, 0);
>   
>   	fib.fmb_addr = virt_to_phys(zdev->fmb);
> +	fib.gd = zdev->gisa;
>   	cc = zpci_mod_fc(req, &fib, &status);
>   	if (cc) {
>   		kmem_cache_free(zdev_fmb_cache, zdev->fmb);
> @@ -178,6 +182,8 @@ int zpci_fmb_disable_device(struct zpci_dev *zdev)
>   	if (!zdev->fmb)
>   		return -EINVAL;
>   
> +	fib.gd = zdev->gisa;
> +
>   	/* Function measurement is disabled if fmb address is zero */
>   	cc = zpci_mod_fc(req, &fib, &status);
>   	if (cc == 3) /* Function already gone. */
> diff --git a/arch/s390/pci/pci_clp.c b/arch/s390/pci/pci_clp.c
> index 1057d7af4a55..deed35edea14 100644
> --- a/arch/s390/pci/pci_clp.c
> +++ b/arch/s390/pci/pci_clp.c
> @@ -229,12 +229,16 @@ static int clp_set_pci_fn(struct zpci_dev *zdev, u32 *fh, u8 nr_dma_as, u8 comma
>   {
>   	struct clp_req_rsp_set_pci *rrb;
>   	int rc, retries = 100;
> +	u32 gisa = 0;
>   
>   	*fh = 0;
>   	rrb = clp_alloc_block(GFP_KERNEL);
>   	if (!rrb)
>   		return -ENOMEM;
>   
> +	if (command != CLP_SET_DISABLE_PCI_FN)
> +		gisa = zdev->gisa;
> +
>   	do {
>   		memset(rrb, 0, sizeof(*rrb));
>   		rrb->request.hdr.len = sizeof(rrb->request);
> @@ -243,6 +247,7 @@ static int clp_set_pci_fn(struct zpci_dev *zdev, u32 *fh, u8 nr_dma_as, u8 comma
>   		rrb->request.fh = zdev->fh;
>   		rrb->request.oc = command;
>   		rrb->request.ndas = nr_dma_as;
> +		rrb->request.gisa = gisa;
>   
>   		rc = clp_req(rrb, CLP_LPS_PCI);
>   		if (rrb->response.hdr.rsp == CLP_RC_SETPCIFN_BUSY) {
> diff --git a/arch/s390/pci/pci_irq.c b/arch/s390/pci/pci_irq.c
> index f2b3145b6697..a2b42a63a53b 100644
> --- a/arch/s390/pci/pci_irq.c
> +++ b/arch/s390/pci/pci_irq.c
> @@ -43,6 +43,7 @@ static int zpci_set_airq(struct zpci_dev *zdev)
>   	fib.fmt0.aibvo = 0;	/* each zdev has its own interrupt vector */
>   	fib.fmt0.aisb = virt_to_phys(zpci_sbv->vector) + (zdev->aisb / 64) * 8;
>   	fib.fmt0.aisbo = zdev->aisb & 63;
> +	fib.gd = zdev->gisa;
>   
>   	return zpci_mod_fc(req, &fib, &status) ? -EIO : 0;
>   }
> @@ -54,6 +55,8 @@ static int zpci_clear_airq(struct zpci_dev *zdev)
>   	struct zpci_fib fib = {0};
>   	u8 cc, status;
>   
> +	fib.gd = zdev->gisa;
> +
>   	cc = zpci_mod_fc(req, &fib, &status);
>   	if (cc == 3 || (cc == 1 && status == 24))
>   		/* Function already gone or IRQs already deregistered. */
> @@ -72,6 +75,7 @@ static int zpci_set_directed_irq(struct zpci_dev *zdev)
>   	fib.fmt = 1;
>   	fib.fmt1.noi = zdev->msi_nr_irqs;
>   	fib.fmt1.dibvo = zdev->msi_first_bit;
> +	fib.gd = zdev->gisa;
>   
>   	return zpci_mod_fc(req, &fib, &status) ? -EIO : 0;
>   }
> @@ -84,6 +88,7 @@ static int zpci_clear_directed_irq(struct zpci_dev *zdev)
>   	u8 cc, status;
>   
>   	fib.fmt = 1;
> +	fib.gd = zdev->gisa;
>   	cc = zpci_mod_fc(req, &fib, &status);
>   	if (cc == 3 || (cc == 1 && status == 24))
>   		/* Function already gone or IRQs already deregistered. */
> 

-- 
Pierre Morel
IBM Lab Boeblingen

Re: [PATCH v5 10/21] KVM: s390: pci: add basic kvm_zdev structure

[Pierre Morel]

On 4/4/22 19:43, Matthew Rosato wrote:
> This structure will be used to carry kvm passthrough information related to
> zPCI devices.
> 
> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>


Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>

> ---
>   arch/s390/include/asm/pci.h |  3 +++
>   arch/s390/kvm/Makefile      |  1 +
>   arch/s390/kvm/pci.c         | 38 +++++++++++++++++++++++++++++++++++++
>   arch/s390/kvm/pci.h         | 21 ++++++++++++++++++++
>   4 files changed, 63 insertions(+)
>   create mode 100644 arch/s390/kvm/pci.c
>   create mode 100644 arch/s390/kvm/pci.h
> 
> diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h
> index 4c5b8fbc2079..9eb20cebaa18 100644
> --- a/arch/s390/include/asm/pci.h
> +++ b/arch/s390/include/asm/pci.h
> @@ -97,6 +97,7 @@ struct zpci_bar_struct {
>   };
>   
>   struct s390_domain;
> +struct kvm_zdev;
>   
>   #define ZPCI_FUNCTIONS_PER_BUS 256
>   struct zpci_bus {
> @@ -190,6 +191,8 @@ struct zpci_dev {
>   	struct dentry	*debugfs_dev;
>   
>   	struct s390_domain *s390_domain; /* s390 IOMMU domain data */
> +
> +	struct kvm_zdev *kzdev; /* passthrough data */
>   };
>   
>   static inline bool zdev_enabled(struct zpci_dev *zdev)
> diff --git a/arch/s390/kvm/Makefile b/arch/s390/kvm/Makefile
> index 26f4a74e5ce4..00cf6853d93f 100644
> --- a/arch/s390/kvm/Makefile
> +++ b/arch/s390/kvm/Makefile
> @@ -10,4 +10,5 @@ ccflags-y := -Ivirt/kvm -Iarch/s390/kvm
>   kvm-y += kvm-s390.o intercept.o interrupt.o priv.o sigp.o
>   kvm-y += diag.o gaccess.o guestdbg.o vsie.o pv.o
>   
> +kvm-$(CONFIG_PCI) += pci.o
>   obj-$(CONFIG_KVM) += kvm.o
> diff --git a/arch/s390/kvm/pci.c b/arch/s390/kvm/pci.c
> new file mode 100644
> index 000000000000..213be236c05a
> --- /dev/null
> +++ b/arch/s390/kvm/pci.c
> @@ -0,0 +1,38 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * s390 kvm PCI passthrough support
> + *
> + * Copyright IBM Corp. 2022
> + *
> + *    Author(s): Matthew Rosato <mjrosato@linux.ibm.com>
> + */
> +
> +#include <linux/kvm_host.h>
> +#include <linux/pci.h>
> +#include "pci.h"
> +
> +int kvm_s390_pci_dev_open(struct zpci_dev *zdev)
> +{
> +	struct kvm_zdev *kzdev;
> +
> +	kzdev = kzalloc(sizeof(struct kvm_zdev), GFP_KERNEL);
> +	if (!kzdev)
> +		return -ENOMEM;
> +
> +	kzdev->zdev = zdev;
> +	zdev->kzdev = kzdev;
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(kvm_s390_pci_dev_open);
> +
> +void kvm_s390_pci_dev_release(struct zpci_dev *zdev)
> +{
> +	struct kvm_zdev *kzdev;
> +
> +	kzdev = zdev->kzdev;
> +	WARN_ON(kzdev->zdev != zdev);
> +	zdev->kzdev = 0;
> +	kfree(kzdev);
> +}
> +EXPORT_SYMBOL_GPL(kvm_s390_pci_dev_release);
> diff --git a/arch/s390/kvm/pci.h b/arch/s390/kvm/pci.h
> new file mode 100644
> index 000000000000..ce93978e8913
> --- /dev/null
> +++ b/arch/s390/kvm/pci.h
> @@ -0,0 +1,21 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * s390 kvm PCI passthrough support
> + *
> + * Copyright IBM Corp. 2022
> + *
> + *    Author(s): Matthew Rosato <mjrosato@linux.ibm.com>
> + */
> +
> +#ifndef __KVM_S390_PCI_H
> +#define __KVM_S390_PCI_H
> +
> +#include <linux/kvm_host.h>
> +#include <linux/pci.h>
> +
> +struct kvm_zdev {
> +	struct zpci_dev *zdev;
> +	struct kvm *kvm;
> +};
> +
> +#endif /* __KVM_S390_PCI_H */
> 

-- 
Pierre Morel
IBM Lab Boeblingen

Re: [PATCH v5 11/21] KVM: s390: pci: do initial setup for AEN interpretation

[Pierre Morel]

On 4/4/22 19:43, Matthew Rosato wrote:
> Initial setup for Adapter Event Notification Interpretation for zPCI
> passthrough devices.  Specifically, allocate a structure for forwarding of
> adapter events and pass the address of this structure to firmware.
> 
> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
> ---
>   arch/s390/include/asm/pci.h      |   4 +
>   arch/s390/include/asm/pci_insn.h |  12 +++
>   arch/s390/kvm/interrupt.c        |  14 +++
>   arch/s390/kvm/kvm-s390.c         |   9 ++
>   arch/s390/kvm/pci.c              | 153 +++++++++++++++++++++++++++++++
>   arch/s390/kvm/pci.h              |  47 ++++++++++
>   arch/s390/pci/pci.c              |   6 ++
>   7 files changed, 245 insertions(+)
> 
> diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h
> index 9eb20cebaa18..557b0ffb32d2 100644
> --- a/arch/s390/include/asm/pci.h
> +++ b/arch/s390/include/asm/pci.h
> @@ -9,6 +9,7 @@
>   #include <asm-generic/pci.h>
>   #include <asm/pci_clp.h>
>   #include <asm/pci_debug.h>
> +#include <asm/pci_insn.h>
>   #include <asm/sclp.h>
>   
>   #define PCIBIOS_MIN_IO		0x1000
> @@ -204,6 +205,9 @@ extern const struct attribute_group *zpci_attr_groups[];
>   extern unsigned int s390_pci_force_floating __initdata;
>   extern unsigned int s390_pci_no_rid;
>   
> +extern union zpci_sic_iib *zpci_aipb;
> +extern struct airq_iv *zpci_aif_sbv;
> +
>   /* -----------------------------------------------------------------------------
>     Prototypes
>   ----------------------------------------------------------------------------- */
> diff --git a/arch/s390/include/asm/pci_insn.h b/arch/s390/include/asm/pci_insn.h
> index 5331082fa516..e5f57cfe1d45 100644
> --- a/arch/s390/include/asm/pci_insn.h
> +++ b/arch/s390/include/asm/pci_insn.h
> @@ -101,6 +101,7 @@ struct zpci_fib {
>   /* Set Interruption Controls Operation Controls  */
>   #define	SIC_IRQ_MODE_ALL		0
>   #define	SIC_IRQ_MODE_SINGLE		1
> +#define	SIC_SET_AENI_CONTROLS		2
>   #define	SIC_IRQ_MODE_DIRECT		4
>   #define	SIC_IRQ_MODE_D_ALL		16
>   #define	SIC_IRQ_MODE_D_SINGLE		17
> @@ -127,9 +128,20 @@ struct zpci_cdiib {
>   	u64 : 64;
>   } __packed __aligned(8);
>   
> +/* adapter interruption parameters block */
> +struct zpci_aipb {
> +	u64 faisb;
> +	u64 gait;
> +	u16 : 13;
> +	u16 afi : 3;
> +	u32 : 32;
> +	u16 faal;
> +} __packed __aligned(8);
> +
>   union zpci_sic_iib {
>   	struct zpci_diib diib;
>   	struct zpci_cdiib cdiib;
> +	struct zpci_aipb aipb;
>   };
>   
>   DECLARE_STATIC_KEY_FALSE(have_mio);
> diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
> index 250119a26c60..57a27dfc85ea 100644
> --- a/arch/s390/kvm/interrupt.c
> +++ b/arch/s390/kvm/interrupt.c
> @@ -32,6 +32,7 @@
>   #include "kvm-s390.h"
>   #include "gaccess.h"
>   #include "trace-s390.h"
> +#include "pci.h"
>   
>   #define PFAULT_INIT 0x0600
>   #define PFAULT_DONE 0x0680
> @@ -3328,6 +3329,11 @@ void kvm_s390_gib_destroy(void)
>   {
>   	if (!gib)
>   		return;
> +	if (kvm_s390_pci_interp_allowed() && aift) {
> +		mutex_lock(&aift->aift_lock);
> +		kvm_s390_pci_aen_exit();
> +		mutex_unlock(&aift->aift_lock);
> +	}
>   	chsc_sgib(0);
>   	unregister_adapter_interrupt(&gib_alert_irq);
>   	free_page((unsigned long)gib);
> @@ -3365,6 +3371,14 @@ int kvm_s390_gib_init(u8 nisc)
>   		goto out_unreg_gal;
>   	}
>   
> +	if (kvm_s390_pci_interp_allowed()) {
> +		if (kvm_s390_pci_aen_init(nisc)) {
> +			pr_err("Initializing AEN for PCI failed\n");
> +			rc = -EIO;
> +			goto out_unreg_gal;
> +		}
> +	}
> +
>   	KVM_EVENT(3, "gib 0x%pK (nisc=%d) initialized", gib, gib->nisc);
>   	goto out;
>   
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 156d1c25a3c1..9db6f8080f71 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -47,6 +47,7 @@
>   #include <asm/fpu/api.h>
>   #include "kvm-s390.h"
>   #include "gaccess.h"
> +#include "pci.h"
>   
>   #define CREATE_TRACE_POINTS
>   #include "trace.h"
> @@ -502,6 +503,14 @@ int kvm_arch_init(void *opaque)
>   		goto out;
>   	}
>   
> +	if (kvm_s390_pci_interp_allowed()) {
> +		rc = kvm_s390_pci_init();
> +		if (rc) {
> +			pr_err("Unable to allocate AIFT for PCI\n");
> +			goto out;
> +		}
> +	}
> +
>   	rc = kvm_s390_gib_init(GAL_ISC);
>   	if (rc)
>   		goto out;
> diff --git a/arch/s390/kvm/pci.c b/arch/s390/kvm/pci.c
> index 213be236c05a..01bd8a2f503b 100644
> --- a/arch/s390/kvm/pci.c
> +++ b/arch/s390/kvm/pci.c
> @@ -9,8 +9,149 @@
>   
>   #include <linux/kvm_host.h>
>   #include <linux/pci.h>
> +#include <asm/pci.h>
> +#include <asm/pci_insn.h>
>   #include "pci.h"
>   
> +struct zpci_aift *aift;
> +
> +static inline int __set_irq_noiib(u16 ctl, u8 isc)
> +{
> +	union zpci_sic_iib iib = {{0}};
> +
> +	return zpci_set_irq_ctrl(ctl, isc, &iib);
> +}
> +
> +/* Caller must hold the aift lock before calling this function */
> +void kvm_s390_pci_aen_exit(void)
> +{
> +	unsigned long flags;
> +	struct kvm_zdev **gait_kzdev;
> +
> +	/*
> +	 * Contents of the aipb remain registered for the life of the host
> +	 * kernel, the information preserved in zpci_aipb and zpci_aif_sbv
> +	 * in case we insert the KVM module again later.  Clear the AIFT
> +	 * information and free anything not registered with underlying
> +	 * firmware.
> +	 */
> +	spin_lock_irqsave(&aift->gait_lock, flags);
> +	gait_kzdev = aift->kzdev;
> +	aift->gait = 0;
> +	aift->sbv = 0;
> +	aift->kzdev = 0;
> +	spin_unlock_irqrestore(&aift->gait_lock, flags);
> +
> +	kfree(gait_kzdev);
> +}
> +
> +static int zpci_setup_aipb(u8 nisc)
> +{
> +	struct page *page;
> +	int size, rc;
> +
> +	zpci_aipb = kzalloc(sizeof(union zpci_sic_iib), GFP_KERNEL);
> +	if (!zpci_aipb)
> +		return -ENOMEM;
> +
> +	aift->sbv = airq_iv_create(ZPCI_NR_DEVICES, AIRQ_IV_ALLOC, 0);
> +	if (!aift->sbv) {
> +		rc = -ENOMEM;
> +		goto free_aipb;
> +	}
> +	zpci_aif_sbv = aift->sbv;
> +	size = get_order(PAGE_ALIGN(ZPCI_NR_DEVICES *
> +						sizeof(struct zpci_gaite)));
> +	page = alloc_pages(GFP_KERNEL | __GFP_ZERO, size);
> +	if (!page) {
> +		rc = -ENOMEM;
> +		goto free_sbv;
> +	}
> +	aift->gait = (struct zpci_gaite *)page_to_phys(page);
> +
> +	zpci_aipb->aipb.faisb = virt_to_phys(aift->sbv->vector);
> +	zpci_aipb->aipb.gait = virt_to_phys(aift->gait);
> +	zpci_aipb->aipb.afi = nisc;
> +	zpci_aipb->aipb.faal = ZPCI_NR_DEVICES;
> +
> +	/* Setup Adapter Event Notification Interpretation */
> +	if (zpci_set_irq_ctrl(SIC_SET_AENI_CONTROLS, 0, zpci_aipb)) {
> +		rc = -EIO;
> +		goto free_gait;
> +	}
> +
> +	return 0;
> +
> +free_gait:
> +	size = get_order(PAGE_ALIGN(ZPCI_NR_DEVICES *
> +				    sizeof(struct zpci_gaite)));
> +	free_pages((unsigned long)aift->gait, size);
> +free_sbv:
> +	airq_iv_release(aift->sbv);
> +	zpci_aif_sbv = 0;
> +free_aipb:
> +	kfree(zpci_aipb);
> +	zpci_aipb = 0;
> +
> +	return rc;
> +}
> +
> +static int zpci_reset_aipb(u8 nisc)
> +{
> +	/*
> +	 * AEN registration can only happen once per system boot.  If
> +	 * an aipb already exists then AEN was already registered and
> +	 * we can re-use the aipb contents.  This can only happen if
> +	 * the KVM module was removed and re-inserted.
> +	 */
> +	if (zpci_aipb->aipb.faal != ZPCI_NR_DEVICES ||
> +	    zpci_aipb->aipb.afi != nisc) {
> +		return -EINVAL;
> +	}

I do not understand how faal cound be different of ZPCI_NR_DEVICES if 
aipb has been already initialised.
Same for afi.
Can you please explain?


> +	aift->sbv = zpci_aif_sbv;
> +	aift->gait = (struct zpci_gaite *)zpci_aipb->aipb.gait;
> +
> +	return 0;
> +}
> +
> +int kvm_s390_pci_aen_init(u8 nisc)
> +{
> +	int rc = 0;
> +
> +	/* If already enabled for AEN, bail out now */
> +	if (aift->gait || aift->sbv)
> +		return -EPERM;
> +
> +	mutex_lock(&aift->aift_lock);
> +	aift->kzdev = kcalloc(ZPCI_NR_DEVICES, sizeof(struct kvm_zdev),
> +			      GFP_KERNEL);
> +	if (!aift->kzdev) {
> +		rc = -ENOMEM;
> +		goto unlock;
> +	}
> +
> +	if (!zpci_aipb)
> +		rc = zpci_setup_aipb(nisc);
> +	else
> +		rc = zpci_reset_aipb(nisc);
> +	if (rc)
> +		goto free_zdev;
> +
> +	/* Enable floating IRQs */
> +	if (__set_irq_noiib(SIC_IRQ_MODE_SINGLE, nisc)) {
> +		rc = -EIO;
> +		kvm_s390_pci_aen_exit();
> +	}
> +
> +	goto unlock;
> +
> +free_zdev:
> +	kfree(aift->kzdev);
> +unlock:
> +	mutex_unlock(&aift->aift_lock);
> +	return rc;
> +}
> +
>   int kvm_s390_pci_dev_open(struct zpci_dev *zdev)
>   {
>   	struct kvm_zdev *kzdev;
> @@ -36,3 +177,15 @@ void kvm_s390_pci_dev_release(struct zpci_dev *zdev)
>   	kfree(kzdev);
>   }
>   EXPORT_SYMBOL_GPL(kvm_s390_pci_dev_release);
> +
> +int kvm_s390_pci_init(void)
> +{
> +	aift = kzalloc(sizeof(struct zpci_aift), GFP_KERNEL);
> +	if (!aift)
> +		return -ENOMEM;
> +
> +	spin_lock_init(&aift->gait_lock);
> +	mutex_init(&aift->aift_lock);
> +
> +	return 0;
> +}
> diff --git a/arch/s390/kvm/pci.h b/arch/s390/kvm/pci.h
> index ce93978e8913..a6a62db792b6 100644
> --- a/arch/s390/kvm/pci.h
> +++ b/arch/s390/kvm/pci.h
> @@ -12,10 +12,57 @@
>   
>   #include <linux/kvm_host.h>
>   #include <linux/pci.h>
> +#include <linux/mutex.h>
> +#include <asm/airq.h>
> +#include <asm/cpu.h>
>   
>   struct kvm_zdev {
>   	struct zpci_dev *zdev;
>   	struct kvm *kvm;
>   };
>   
> +struct zpci_gaite {
> +	u32 gisa;
> +	u8 gisc;
> +	u8 count;
> +	u8 reserved;
> +	u8 aisbo;
> +	u64 aisb;
> +};
> +
> +struct zpci_aift {
> +	struct zpci_gaite *gait;
> +	struct airq_iv *sbv;
> +	struct kvm_zdev **kzdev;
> +	spinlock_t gait_lock; /* Protects the gait, used during AEN forward */
> +	struct mutex aift_lock; /* Protects the other structures in aift */
> +};
> +
> +extern struct zpci_aift *aift;
> +
> +int kvm_s390_pci_aen_init(u8 nisc);
> +void kvm_s390_pci_aen_exit(void);
> +
> +int kvm_s390_pci_init(void);
> +
> +static inline bool kvm_s390_pci_interp_allowed(void)
> +{
> +	struct cpuid cpu_id;
> +
> +	get_cpu_id(&cpu_id);
> +	switch (cpu_id.machine) {
> +	case 0x2817:
> +	case 0x2818:
> +	case 0x2827:
> +	case 0x2828:
> +	case 0x2964:
> +	case 0x2965:
> +		/* No SHM on certain machines */
> +		return false;
> +	default:
> +		return (IS_ENABLED(CONFIG_VFIO_PCI) && sclp.has_zpci_lsi &&
> +			sclp.has_aeni && sclp.has_aisi && sclp.has_aisii);
> +	}
> +}
> +
>   #endif /* __KVM_S390_PCI_H */
> diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c
> index a86cd1cbb80e..f0a439c43395 100644
> --- a/arch/s390/pci/pci.c
> +++ b/arch/s390/pci/pci.c
> @@ -61,6 +61,12 @@ DEFINE_STATIC_KEY_FALSE(have_mio);
>   
>   static struct kmem_cache *zdev_fmb_cache;
>   
> +/* AEN structures that must be preserved over KVM module re-insertion */
> +union zpci_sic_iib *zpci_aipb;
> +EXPORT_SYMBOL_GPL(zpci_aipb);
> +struct airq_iv *zpci_aif_sbv;
> +EXPORT_SYMBOL_GPL(zpci_aif_sbv);
> +
>   struct zpci_dev *get_zdev_by_fid(u32 fid)
>   {
>   	struct zpci_dev *tmp, *zdev = NULL;
> 

-- 
Pierre Morel
IBM Lab Boeblingen

Re: [PATCH v5 12/21] KVM: s390: pci: enable host forwarding of Adapter Event Notifications

[Pierre Morel]

On 4/4/22 19:43, Matthew Rosato wrote:
> In cases where interrupts are not forwarded to the guest via firmware,
> KVM is responsible for ensuring delivery.  When an interrupt presents
> with the forwarding bit, we must process the forwarding tables until
> all interrupts are delivered.
> 
> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>


Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>

> ---
>   arch/s390/include/asm/kvm_host.h |  1 +
>   arch/s390/include/asm/tpi.h      | 13 ++++++
>   arch/s390/kvm/interrupt.c        | 77 +++++++++++++++++++++++++++++++-
>   arch/s390/kvm/kvm-s390.c         |  3 +-
>   arch/s390/kvm/pci.h              | 10 +++++
>   5 files changed, 102 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index 766028d54a3e..c1518a505060 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -759,6 +759,7 @@ struct kvm_vm_stat {
>   	u64 inject_pfault_done;
>   	u64 inject_service_signal;
>   	u64 inject_virtio;
> +	u64 aen_forward;
>   };
>   
>   struct kvm_arch_memory_slot {
> diff --git a/arch/s390/include/asm/tpi.h b/arch/s390/include/asm/tpi.h
> index 1ac538b8cbf5..f76e5fdff23a 100644
> --- a/arch/s390/include/asm/tpi.h
> +++ b/arch/s390/include/asm/tpi.h
> @@ -19,6 +19,19 @@ struct tpi_info {
>   	u32 :12;
>   } __packed __aligned(4);
>   
> +/* I/O-Interruption Code as stored by TPI for an Adapter I/O */
> +struct tpi_adapter_info {
> +	u32 aism:8;
> +	u32 :22;
> +	u32 error:1;
> +	u32 forward:1;
> +	u32 reserved;
> +	u32 adapter_IO:1;
> +	u32 directed_irq:1;
> +	u32 isc:3;
> +	u32 :27;
> +} __packed __aligned(4);
> +
>   #endif /* __ASSEMBLY__ */
>   
>   #endif /* _ASM_S390_TPI_H */
> diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
> index 57a27dfc85ea..bea0cc59eeba 100644
> --- a/arch/s390/kvm/interrupt.c
> +++ b/arch/s390/kvm/interrupt.c
> @@ -3313,11 +3313,86 @@ int kvm_s390_gisc_unregister(struct kvm *kvm, u32 gisc)
>   }
>   EXPORT_SYMBOL_GPL(kvm_s390_gisc_unregister);
>   
> +static void aen_host_forward(unsigned long si)
> +{
> +	struct kvm_s390_gisa_interrupt *gi;
> +	struct zpci_gaite *gaite;
> +	struct kvm *kvm;
> +
> +	gaite = (struct zpci_gaite *)aift->gait +
> +		(si * sizeof(struct zpci_gaite));
> +	if (gaite->count == 0)
> +		return;
> +	if (gaite->aisb != 0)
> +		set_bit_inv(gaite->aisbo, (unsigned long *)gaite->aisb);
> +
> +	kvm = kvm_s390_pci_si_to_kvm(aift, si);
> +	if (!kvm)
> +		return;
> +	gi = &kvm->arch.gisa_int;
> +
> +	if (!(gi->origin->g1.simm & AIS_MODE_MASK(gaite->gisc)) ||
> +	    !(gi->origin->g1.nimm & AIS_MODE_MASK(gaite->gisc))) {
> +		gisa_set_ipm_gisc(gi->origin, gaite->gisc);
> +		if (hrtimer_active(&gi->timer))
> +			hrtimer_cancel(&gi->timer);
> +		hrtimer_start(&gi->timer, 0, HRTIMER_MODE_REL);
> +		kvm->stat.aen_forward++;
> +	}
> +}
> +
> +static void aen_process_gait(u8 isc)
> +{
> +	bool found = false, first = true;
> +	union zpci_sic_iib iib = {{0}};
> +	unsigned long si, flags;
> +
> +	spin_lock_irqsave(&aift->gait_lock, flags);
> +
> +	if (!aift->gait) {
> +		spin_unlock_irqrestore(&aift->gait_lock, flags);
> +		return;
> +	}
> +
> +	for (si = 0;;) {
> +		/* Scan adapter summary indicator bit vector */
> +		si = airq_iv_scan(aift->sbv, si, airq_iv_end(aift->sbv));
> +		if (si == -1UL) {
> +			if (first || found) {
> +				/* Re-enable interrupts. */
> +				zpci_set_irq_ctrl(SIC_IRQ_MODE_SINGLE, isc,
> +						  &iib);
> +				first = found = false;
> +			} else {
> +				/* Interrupts on and all bits processed */
> +				break;
> +			}
> +			found = false;
> +			si = 0;
> +			/* Scan again after re-enabling interrupts */
> +			continue;
> +		}
> +		found = true;
> +		aen_host_forward(si);
> +	}
> +
> +	spin_unlock_irqrestore(&aift->gait_lock, flags);
> +}
> +
>   static void gib_alert_irq_handler(struct airq_struct *airq,
>   				  struct tpi_info *tpi_info)
>   {
> +	struct tpi_adapter_info *info = (struct tpi_adapter_info *)tpi_info;
> +
>   	inc_irq_stat(IRQIO_GAL);
> -	process_gib_alert_list();
> +
> +	if (IS_ENABLED(CONFIG_VFIO_PCI) && (info->forward || info->error)) {
> +		aen_process_gait(info->isc);
> +		if (info->aism != 0)
> +			process_gib_alert_list();
> +	} else {
> +		process_gib_alert_list();
> +	}
>   }
>   
>   static struct airq_struct gib_alert_irq = {
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 9db6f8080f71..a9eb8b75af93 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -64,7 +64,8 @@ const struct _kvm_stats_desc kvm_vm_stats_desc[] = {
>   	STATS_DESC_COUNTER(VM, inject_float_mchk),
>   	STATS_DESC_COUNTER(VM, inject_pfault_done),
>   	STATS_DESC_COUNTER(VM, inject_service_signal),
> -	STATS_DESC_COUNTER(VM, inject_virtio)
> +	STATS_DESC_COUNTER(VM, inject_virtio),
> +	STATS_DESC_COUNTER(VM, aen_forward)
>   };
>   
>   const struct kvm_stats_header kvm_vm_stats_header = {
> diff --git a/arch/s390/kvm/pci.h b/arch/s390/kvm/pci.h
> index a6a62db792b6..d4997e2236ef 100644
> --- a/arch/s390/kvm/pci.h
> +++ b/arch/s390/kvm/pci.h
> @@ -13,6 +13,7 @@
>   #include <linux/kvm_host.h>
>   #include <linux/pci.h>
>   #include <linux/mutex.h>
> +#include <linux/kvm_host.h>
>   #include <asm/airq.h>
>   #include <asm/cpu.h>
>   
> @@ -40,6 +41,15 @@ struct zpci_aift {
>   
>   extern struct zpci_aift *aift;
>   
> +static inline struct kvm *kvm_s390_pci_si_to_kvm(struct zpci_aift *aift,
> +						 unsigned long si)
> +{
> +	if (!IS_ENABLED(CONFIG_VFIO_PCI) || aift->kzdev == 0 ||
> +	    aift->kzdev[si] == 0)
> +		return 0;
> +	return aift->kzdev[si]->kvm;
> +};
> +
>   int kvm_s390_pci_aen_init(u8 nisc);
>   void kvm_s390_pci_aen_exit(void);
>   
> 

-- 
Pierre Morel
IBM Lab Boeblingen

Re: [PATCH v5 13/21] KVM: s390: mechanism to enable guest zPCI Interpretation

[Pierre Morel]

On 4/4/22 19:43, Matthew Rosato wrote:
> The guest must have access to certain facilities in order to allow
> interpretive execution of zPCI instructions and adapter event
> notifications.  However, there are some cases where a guest might
> disable interpretation -- provide a mechanism via which we can defer
> enabling the associated zPCI interpretation facilities until the guest
> indicates it wishes to use them.
> 
> Acked-by: Pierre Morel <pmorel@linux.ibm.com>
> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>

You can change the ACK with a RB (I think it already was a RB)

Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>


> ---
>   arch/s390/include/asm/kvm_host.h |  4 ++++
>   arch/s390/kvm/kvm-s390.c         | 37 ++++++++++++++++++++++++++++++++
>   arch/s390/kvm/kvm-s390.h         | 10 +++++++++
>   3 files changed, 51 insertions(+)
> 
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index c1518a505060..8e381603b6a7 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -254,7 +254,10 @@ struct kvm_s390_sie_block {
>   #define ECB2_IEP	0x20
>   #define ECB2_PFMFI	0x08
>   #define ECB2_ESCA	0x04
> +#define ECB2_ZPCI_LSI	0x02
>   	__u8    ecb2;                   /* 0x0062 */
> +#define ECB3_AISI	0x20
> +#define ECB3_AISII	0x10
>   #define ECB3_DEA 0x08
>   #define ECB3_AES 0x04
>   #define ECB3_RI  0x01
> @@ -940,6 +943,7 @@ struct kvm_arch{
>   	int use_cmma;
>   	int use_pfmfi;
>   	int use_skf;
> +	int use_zpci_interp;
>   	int user_cpu_state_ctrl;
>   	int user_sigp;
>   	int user_stsi;
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index a9eb8b75af93..327649ddddce 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -1029,6 +1029,41 @@ static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
>   	return 0;
>   }
>   
> +static void kvm_s390_vcpu_pci_setup(struct kvm_vcpu *vcpu)
> +{
> +	/* Only set the ECB bits after guest requests zPCI interpretation */
> +	if (!vcpu->kvm->arch.use_zpci_interp)
> +		return;
> +
> +	vcpu->arch.sie_block->ecb2 |= ECB2_ZPCI_LSI;
> +	vcpu->arch.sie_block->ecb3 |= ECB3_AISII + ECB3_AISI;
> +}
> +
> +/* Must be called with kvm->lock held */
> +void kvm_s390_vcpu_pci_enable_interp(struct kvm *kvm)
> +{
> +	struct kvm_vcpu *vcpu;
> +	unsigned long i;
> +
> +	/*
> +	 * If host is configured for PCI and the necessary facilities are
> +	 * available, turn on interpretation for the life of this guest
> +	 */
> +	if (!kvm_s390_pci_interp_allowed())
> +		return;
> +
> +	kvm->arch.use_zpci_interp = 1;
> +
> +	kvm_s390_vcpu_block_all(kvm);
> +
> +	kvm_for_each_vcpu(i, vcpu, kvm) {
> +		kvm_s390_vcpu_pci_setup(vcpu);
> +		kvm_s390_sync_request(KVM_REQ_VSIE_RESTART, vcpu);
> +	}
> +
> +	kvm_s390_vcpu_unblock_all(kvm);
> +}
> +
>   static void kvm_s390_sync_request_broadcast(struct kvm *kvm, int req)
>   {
>   	unsigned long cx;
> @@ -3329,6 +3364,8 @@ static int kvm_s390_vcpu_setup(struct kvm_vcpu *vcpu)
>   
>   	kvm_s390_vcpu_crypto_setup(vcpu);
>   
> +	kvm_s390_vcpu_pci_setup(vcpu);
> +
>   	mutex_lock(&vcpu->kvm->lock);
>   	if (kvm_s390_pv_is_protected(vcpu->kvm)) {
>   		rc = kvm_s390_pv_create_cpu(vcpu, &uvrc, &uvrrc);
> diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
> index 497d52a83c78..975cffaf13de 100644
> --- a/arch/s390/kvm/kvm-s390.h
> +++ b/arch/s390/kvm/kvm-s390.h
> @@ -507,6 +507,16 @@ void kvm_s390_reinject_machine_check(struct kvm_vcpu *vcpu,
>    */
>   void kvm_s390_vcpu_crypto_reset_all(struct kvm *kvm);
>   
> +/**
> + * kvm_s390_vcpu_pci_enable_interp
> + *
> + * Set the associated PCI attributes for each vcpu to allow for zPCI Load/Store
> + * interpretation as well as adapter interruption forwarding.
> + *
> + * @kvm: the KVM guest
> + */
> +void kvm_s390_vcpu_pci_enable_interp(struct kvm *kvm);
> +
>   /**
>    * diag9c_forwarding_hz
>    *
> 

-- 
Pierre Morel
IBM Lab Boeblingen

Re: [PATCH v5 17/21] vfio-pci/zdev: add function handle to clp base capability

[Pierre Morel]

On 4/4/22 19:43, Matthew Rosato wrote:
> The function handle is a system-wide unique identifier for a zPCI
> device.  It is used as input for various zPCI operations.

The comment is a little too much obvious.
May be add something like.
"
With the legacy instruction interception the PCI instructions used to be 
executed by the host.
With PCI instruction interpretation, the guest needs to use the real 
function handle.
Let's give it to the guest.
"
Or something like that.

With a better comment:

Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>

> 
> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
> ---
>   drivers/vfio/pci/vfio_pci_zdev.c | 5 +++--
>   include/uapi/linux/vfio_zdev.h   | 3 +++
>   2 files changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/vfio/pci/vfio_pci_zdev.c b/drivers/vfio/pci/vfio_pci_zdev.c
> index ea4c0d2b0663..4a653ce480c7 100644
> --- a/drivers/vfio/pci/vfio_pci_zdev.c
> +++ b/drivers/vfio/pci/vfio_pci_zdev.c
> @@ -23,14 +23,15 @@ static int zpci_base_cap(struct zpci_dev *zdev, struct vfio_info_cap *caps)
>   {
>   	struct vfio_device_info_cap_zpci_base cap = {
>   		.header.id = VFIO_DEVICE_INFO_CAP_ZPCI_BASE,
> -		.header.version = 1,
> +		.header.version = 2,
>   		.start_dma = zdev->start_dma,
>   		.end_dma = zdev->end_dma,
>   		.pchid = zdev->pchid,
>   		.vfn = zdev->vfn,
>   		.fmb_length = zdev->fmb_length,
>   		.pft = zdev->pft,
> -		.gid = zdev->pfgid
> +		.gid = zdev->pfgid,
> +		.fh = zdev->fh
>   	};
>   
>   	return vfio_info_add_capability(caps, &cap.header, sizeof(cap));
> diff --git a/include/uapi/linux/vfio_zdev.h b/include/uapi/linux/vfio_zdev.h
> index b4309397b6b2..78c022af3d29 100644
> --- a/include/uapi/linux/vfio_zdev.h
> +++ b/include/uapi/linux/vfio_zdev.h
> @@ -29,6 +29,9 @@ struct vfio_device_info_cap_zpci_base {
>   	__u16 fmb_length;	/* Measurement Block Length (in bytes) */
>   	__u8 pft;		/* PCI Function Type */
>   	__u8 gid;		/* PCI function group ID */
> +	/* End of version 1 */
> +	__u32 fh;		/* PCI function handle */
> +	/* End of version 2 */
>   };
>   
>   /**
> 

-- 
Pierre Morel
IBM Lab Boeblingen

Re: [PATCH v5 18/21] vfio-pci/zdev: different maxstbl for interpreted devices

[Pierre Morel]

On 4/4/22 19:43, Matthew Rosato wrote:
> When doing load/store interpretation, the maximum store block length is
> determined by the underlying firmware, not the host kernel API.  Reflect
> that in the associated Query PCI Function Group clp capability and let
> userspace decide which is appropriate to present to the guest.
> 
> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>


Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>


> ---
>   drivers/vfio/pci/vfio_pci_zdev.c | 6 ++++--
>   include/uapi/linux/vfio_zdev.h   | 4 ++++
>   2 files changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/vfio/pci/vfio_pci_zdev.c b/drivers/vfio/pci/vfio_pci_zdev.c
> index 4a653ce480c7..d3ca58d9d8ec 100644
> --- a/drivers/vfio/pci/vfio_pci_zdev.c
> +++ b/drivers/vfio/pci/vfio_pci_zdev.c
> @@ -44,14 +44,16 @@ static int zpci_group_cap(struct zpci_dev *zdev, struct vfio_info_cap *caps)
>   {
>   	struct vfio_device_info_cap_zpci_group cap = {
>   		.header.id = VFIO_DEVICE_INFO_CAP_ZPCI_GROUP,
> -		.header.version = 1,
> +		.header.version = 2,
>   		.dasm = zdev->dma_mask,
>   		.msi_addr = zdev->msi_addr,
>   		.flags = VFIO_DEVICE_INFO_ZPCI_FLAG_REFRESH,
>   		.mui = zdev->fmb_update,
>   		.noi = zdev->max_msi,
>   		.maxstbl = ZPCI_MAX_WRITE_SIZE,
> -		.version = zdev->version
> +		.version = zdev->version,
> +		.reserved = 0,
> +		.imaxstbl = zdev->maxstbl
>   	};
>   
>   	return vfio_info_add_capability(caps, &cap.header, sizeof(cap));
> diff --git a/include/uapi/linux/vfio_zdev.h b/include/uapi/linux/vfio_zdev.h
> index 78c022af3d29..77f2aff1f27e 100644
> --- a/include/uapi/linux/vfio_zdev.h
> +++ b/include/uapi/linux/vfio_zdev.h
> @@ -50,6 +50,10 @@ struct vfio_device_info_cap_zpci_group {
>   	__u16 noi;		/* Maximum number of MSIs */
>   	__u16 maxstbl;		/* Maximum Store Block Length */
>   	__u8 version;		/* Supported PCI Version */
> +	/* End of version 1 */
> +	__u8 reserved;
> +	__u16 imaxstbl;		/* Maximum Interpreted Store Block Length */
> +	/* End of version 2 */
>   };
>   
>   /**
> 

-- 
Pierre Morel
IBM Lab Boeblingen

Re: [PATCH v5 19/21] KVM: s390: add KVM_S390_ZPCI_OP to manage guest zPCI devices

[Pierre Morel]

On 4/4/22 19:43, Matthew Rosato wrote:
> The KVM_S390_ZPCI_OP ioctl provides a mechanism for managing
> hardware-assisted virtualization features for s390X zPCI passthrough.
> Add the first 2 operations, which can be used to enable/disable
> the specified device for Adapter Event Notification interpretation.
> 
> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
> ---
>   Documentation/virt/kvm/api.rst | 45 +++++++++++++++++++++++
>   arch/s390/kvm/kvm-s390.c       | 23 ++++++++++++
>   arch/s390/kvm/pci.c            | 65 ++++++++++++++++++++++++++++++++++
>   arch/s390/kvm/pci.h            |  2 ++
>   include/uapi/linux/kvm.h       | 31 ++++++++++++++++
>   5 files changed, 166 insertions(+)
> 
> diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
> index d13fa6600467..474f502c2ea0 100644
> --- a/Documentation/virt/kvm/api.rst
> +++ b/Documentation/virt/kvm/api.rst
> @@ -5645,6 +5645,51 @@ enabled with ``arch_prctl()``, but this may change in the future.
>   The offsets of the state save areas in struct kvm_xsave follow the contents
>   of CPUID leaf 0xD on the host.
>   
> +4.135 KVM_S390_ZPCI_OP
> +--------------------
> +
> +:Capability: KVM_CAP_S390_ZPCI_OP
> +:Architectures: s390
> +:Type: vcpu ioctl
> +:Parameters: struct kvm_s390_zpci_op (in)
> +:Returns: 0 on success, <0 on error
> +
> +Used to manage hardware-assisted virtualization features for zPCI devices.
> +
> +Parameters are specified via the following structure::
> +
> +  struct kvm_s390_zpci_op {
> +	/* in */
> +	__u32 fh;		/* target device */
> +	__u8  op;		/* operation to perform */
> +	__u8  pad[3];
> +	union {
> +		/* for KVM_S390_ZPCIOP_REG_AEN */
> +		struct {
> +			__u64 ibv;	/* Guest addr of interrupt bit vector */
> +			__u64 sb;	/* Guest addr of summary bit */
> +			__u32 flags;
> +			__u32 noi;	/* Number of interrupts */
> +			__u8 isc;	/* Guest interrupt subclass */
> +			__u8 sbo;	/* Offset of guest summary bit vector */
> +			__u16 pad;
> +		} reg_aen;
> +		__u64 reserved[8];
> +	} u;
> +  };
> +
> +The type of operation is specified in the "op" field.
> +KVM_S390_ZPCIOP_REG_AEN is used to register the VM for adapter event
> +notification interpretation, which will allow firmware delivery of adapter
> +events directly to the vm, with KVM providing a backup delivery mechanism;
> +KVM_S390_ZPCIOP_DEREG_AEN is used to subsequently disable interpretation of
> +adapter event notifications.
> +
> +The target zPCI function must also be specified via the "fh" field.  For the
> +KVM_S390_ZPCIOP_REG_AEN operation, additional information to establish firmware
> +delivery must be provided via the "reg_aen" struct.
> +
> +The "reserved" field is meant for future extensions.
>   
>   5. The kvm_run structure
>   ========================
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 704d85214f4f..65a53e22f686 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -616,6 +616,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
>   	case KVM_CAP_S390_PROTECTED:
>   		r = is_prot_virt_host();
>   		break;
> +	case KVM_CAP_S390_ZPCI_OP:
> +		if (kvm_s390_pci_interp_allowed())
> +			r = 1;
> +		else
> +			r = 0;
> +		break;
>   	default:
>   		r = 0;
>   	}
> @@ -2621,6 +2627,23 @@ long kvm_arch_vm_ioctl(struct file *filp,
>   			r = -EFAULT;
>   		break;
>   	}
> +	case KVM_S390_ZPCI_OP: {
> +		struct kvm_s390_zpci_op args;
> +
> +		r = -EINVAL;
> +		if (!IS_ENABLED(CONFIG_VFIO_PCI))
> +			break;
> +		if (copy_from_user(&args, argp, sizeof(args))) {
> +			r = -EFAULT;
> +			break;
> +		}
> +		r = kvm_s390_pci_zpci_op(kvm, &args);
> +		if (r)
> +			break;
> +		if (copy_to_user(argp, &args, sizeof(args)))
> +			r = -EFAULT;
> +		break;
> +	}
>   	default:
>   		r = -ENOTTY;
>   	}
> diff --git a/arch/s390/kvm/pci.c b/arch/s390/kvm/pci.c
> index 66565f5f3f43..d941657bcb39 100644
> --- a/arch/s390/kvm/pci.c
> +++ b/arch/s390/kvm/pci.c
> @@ -588,6 +588,71 @@ void kvm_s390_pci_clear_list(struct kvm *kvm)
>   		unregister_kvm(kzdev->zdev);
>   }
>   
> +static struct kvm_zdev *get_kzdev_by_fh(struct kvm *kvm, u32 fh)
> +{
> +	struct kvm_zdev *kzdev, *retval = NULL;
> +
> +	spin_lock(&kvm->arch.kzdev_list_lock);
> +	list_for_each_entry(kzdev, &kvm->arch.kzdev_list, entry) {
> +		if (kzdev->zdev->fh == fh) {
> +			retval = kzdev;
> +			break;
> +		}
> +	}
> +	spin_unlock(&kvm->arch.kzdev_list_lock);
> +
> +	return retval;
> +}
> +
> +static int kvm_s390_pci_zpci_reg_aen(struct zpci_dev *zdev,
> +				     struct kvm_s390_zpci_op *args)
> +{
> +	struct zpci_fib fib = {};
> +
> +	fib.fmt0.aibv = args->u.reg_aen.ibv;
> +	fib.fmt0.isc = args->u.reg_aen.isc;
> +	fib.fmt0.noi = args->u.reg_aen.noi;
> +	if (args->u.reg_aen.sb != 0) {
> +		fib.fmt0.aisb = args->u.reg_aen.sb;
> +		fib.fmt0.aisbo = args->u.reg_aen.sbo;
> +		fib.fmt0.sum = 1;
> +	} else {
> +		fib.fmt0.aisb = 0;
> +		fib.fmt0.aisbo = 0;
> +		fib.fmt0.sum = 0;
> +	}
> +
> +	if (args->u.reg_aen.flags & KVM_S390_ZPCIOP_REGAEN_HOST)
> +		return kvm_s390_pci_aif_enable(zdev, &fib, true);
> +	else
> +		return kvm_s390_pci_aif_enable(zdev, &fib, false);
> +}
> +
> +int kvm_s390_pci_zpci_op(struct kvm *kvm, struct kvm_s390_zpci_op *args)
> +{
> +	struct kvm_zdev *kzdev;
> +	struct zpci_dev *zdev;
> +	int r;
> +
> +	kzdev = get_kzdev_by_fh(kvm, args->fh);
> +	if (!kzdev)
> +		return -ENODEV;
> +	zdev = kzdev->zdev;
> +
> +	switch (args->op) {
> +	case KVM_S390_ZPCIOP_REG_AEN:
> +		r = kvm_s390_pci_zpci_reg_aen(zdev, args);
> +		break;
> +	case KVM_S390_ZPCIOP_DEREG_AEN:
> +		r = kvm_s390_pci_aif_disable(zdev, false);
> +		break;
> +	default:
> +		r = -EINVAL;
> +	}
> +
> +	return r;
> +}
> +
>   int kvm_s390_pci_init(void)
>   {
>   	aift = kzalloc(sizeof(struct zpci_aift), GFP_KERNEL);
> diff --git a/arch/s390/kvm/pci.h b/arch/s390/kvm/pci.h
> index cb5ec3208923..b053b50c0904 100644
> --- a/arch/s390/kvm/pci.h
> +++ b/arch/s390/kvm/pci.h
> @@ -59,6 +59,8 @@ void kvm_s390_pci_aen_exit(void);
>   void kvm_s390_pci_init_list(struct kvm *kvm);
>   void kvm_s390_pci_clear_list(struct kvm *kvm);
>   
> +int kvm_s390_pci_zpci_op(struct kvm *kvm, struct kvm_s390_zpci_op *args);
> +
>   int kvm_s390_pci_init(void);
>   
>   static inline bool kvm_s390_pci_interp_allowed(void)
> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> index 91a6fe4e02c0..014adb30d42c 100644
> --- a/include/uapi/linux/kvm.h
> +++ b/include/uapi/linux/kvm.h
> @@ -1144,6 +1144,7 @@ struct kvm_ppc_resize_hpt {
>   #define KVM_CAP_S390_MEM_OP_EXTENSION 211
>   #define KVM_CAP_PMU_CAPABILITY 212
>   #define KVM_CAP_DISABLE_QUIRKS2 213
> +#define KVM_CAP_S390_ZPCI_OP 214
>   
>   #ifdef KVM_CAP_IRQ_ROUTING
>   
> @@ -2060,4 +2061,34 @@ struct kvm_stats_desc {
>   /* Available with KVM_CAP_XSAVE2 */
>   #define KVM_GET_XSAVE2		  _IOR(KVMIO,  0xcf, struct kvm_xsave)
>   
> +/* Available with KVM_CAP_S390_ZPCI_OP */
> +#define KVM_S390_ZPCI_OP	  _IOW(KVMIO,  0xd0, struct kvm_s390_zpci_op)
> +
> +struct kvm_s390_zpci_op {
> +	/* in */
> +	__u32 fh;		/* target device */
> +	__u8  op;		/* operation to perform */
> +	__u8  pad[3];
> +	union {
> +		/* for KVM_S390_ZPCIOP_REG_AEN */
> +		struct {
> +			__u64 ibv;	/* Guest addr of interrupt bit vector */
> +			__u64 sb;	/* Guest addr of summary bit */
> +			__u32 flags;
> +			__u32 noi;	/* Number of interrupts */
> +			__u8 isc;	/* Guest interrupt subclass */
> +			__u8 sbo;	/* Offset of guest summary bit vector */
> +			__u16 pad;
> +		} reg_aen;
> +		__u64 reserved[8];
> +	} u;
> +};
> +
> +/* types for kvm_s390_zpci_op->op */
> +#define KVM_S390_ZPCIOP_REG_AEN		0
> +#define KVM_S390_ZPCIOP_DEREG_AEN	1
> +
> +/* flags for kvm_s390_zpci_op->u.reg_aen.flags */
> +#define KVM_S390_ZPCIOP_REGAEN_HOST	(1 << 0)
> +
>   #endif /* __LINUX_KVM_H */
> 


The purpose is to setup the IRQ for the zPCI device wouldn't it be more 
logical to go through VFIO_DEVICE_SET_IRQS and there jump to the zPCI 
handling, like it is done for the capabilities, instead to go directly 
through KVM?
We would also spare the research for the real FH in KVM.






-- 
Pierre Morel
IBM Lab Boeblingen

Re: [PATCH v5 20/21] KVM: s390: introduce CPU feature for zPCI Interpretation

[Pierre Morel]

On 4/4/22 19:43, Matthew Rosato wrote:
> KVM_S390_VM_CPU_FEAT_ZPCI_INTERP relays whether zPCI interpretive
> execution is possible based on the available hardware facilities.
> 
> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
> ---
>   arch/s390/include/uapi/asm/kvm.h | 1 +
>   arch/s390/kvm/kvm-s390.c         | 4 ++++
>   2 files changed, 5 insertions(+)
> 
> diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h
> index 7a6b14874d65..ed06458a871f 100644
> --- a/arch/s390/include/uapi/asm/kvm.h
> +++ b/arch/s390/include/uapi/asm/kvm.h
> @@ -130,6 +130,7 @@ struct kvm_s390_vm_cpu_machine {
>   #define KVM_S390_VM_CPU_FEAT_PFMFI	11
>   #define KVM_S390_VM_CPU_FEAT_SIGPIF	12
>   #define KVM_S390_VM_CPU_FEAT_KSS	13
> +#define KVM_S390_VM_CPU_FEAT_ZPCI_INTERP 14
>   struct kvm_s390_vm_cpu_feat {
>   	__u64 feat[16];
>   };
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 65a53e22f686..d6988bd22df8 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -433,6 +433,10 @@ static void kvm_s390_cpu_feat_init(void)
>   	if (test_facility(151)) /* DFLTCC */
>   		__insn32_query(INSN_DFLTCC, kvm_s390_available_subfunc.dfltcc);
>   
> +	/* zPCI Interpretation */
> +	if (kvm_s390_pci_interp_allowed())
> +		allow_cpu_feat(KVM_S390_VM_CPU_FEAT_ZPCI_INTERP);
> +
>   	if (MACHINE_HAS_ESOP)
>   		allow_cpu_feat(KVM_S390_VM_CPU_FEAT_ESOP);
>   	/*
> 

Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>

-- 
Pierre Morel
IBM Lab Boeblingen

Re: [PATCH v5 11/21] KVM: s390: pci: do initial setup for AEN interpretation

[Matthew Rosato]

On 4/19/22 4:16 AM, Pierre Morel wrote:
> 
> 
> On 4/4/22 19:43, Matthew Rosato wrote:
>> Initial setup for Adapter Event Notification Interpretation for zPCI
>> passthrough devices.  Specifically, allocate a structure for 
>> forwarding of
>> adapter events and pass the address of this structure to firmware.
>>
>> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
>> ---

...

>> +
>> +static int zpci_reset_aipb(u8 nisc)
>> +{
>> +    /*
>> +     * AEN registration can only happen once per system boot.  If
>> +     * an aipb already exists then AEN was already registered and
>> +     * we can re-use the aipb contents.  This can only happen if
>> +     * the KVM module was removed and re-inserted.
>> +     */
>> +    if (zpci_aipb->aipb.faal != ZPCI_NR_DEVICES ||
>> +        zpci_aipb->aipb.afi != nisc) {
>> +        return -EINVAL;
>> +    }
> 
> I do not understand how faal cound be different of ZPCI_NR_DEVICES if 
> aipb has been already initialised.
> Same for afi.
> Can you please explain?

Well, my concern was along the lines of 'what if we rmmod kvm and then 
insmod a different version of the kvm module' -- These are really sanity 
checks.

Now, ZPCI_NR_DEVICES/faal is built in with PCI, so yeah this check is 
probably unnecessary as we shouldn't be able to change this value 
without a new kernel.

afi is however derived from nisc, which was passed in all the way from 
kvm_s390_gib_init during kvm_arch_init.  Today, this is hard-coded as 
GAL_ISC; but the point is that this is hard-coded within the kvm module, 
so we can't be quite sure that it's the same value every time we insmod 
kvm.  In an (admittedly, far-fetched) scenario where we insmod kvm, 
initialize AEN with GAL_ISC, rmmod kvm, then insmod a kvm where, for 
example, GAL_ISC was changed to a different number, we would need to 
trigger a failure here because we have no way to update the forwarding 
isc with firmware until the kernel is rebooted.

Re: [PATCH v5 19/21] KVM: s390: add KVM_S390_ZPCI_OP to manage guest zPCI devices

[Matthew Rosato]

On 4/19/22 6:07 AM, Pierre Morel wrote:
> 
> 
> On 4/4/22 19:43, Matthew Rosato wrote:
>> The KVM_S390_ZPCI_OP ioctl provides a mechanism for managing
>> hardware-assisted virtualization features for s390X zPCI passthrough.
>> Add the first 2 operations, which can be used to enable/disable
>> the specified device for Adapter Event Notification interpretation.
>>
>> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
>> ---
>>   Documentation/virt/kvm/api.rst | 45 +++++++++++++++++++++++
>>   arch/s390/kvm/kvm-s390.c       | 23 ++++++++++++
>>   arch/s390/kvm/pci.c            | 65 ++++++++++++++++++++++++++++++++++
>>   arch/s390/kvm/pci.h            |  2 ++
>>   include/uapi/linux/kvm.h       | 31 ++++++++++++++++
>>   5 files changed, 166 insertions(+)
>>
>> diff --git a/Documentation/virt/kvm/api.rst 
>> b/Documentation/virt/kvm/api.rst
>> index d13fa6600467..474f502c2ea0 100644
>> --- a/Documentation/virt/kvm/api.rst
>> +++ b/Documentation/virt/kvm/api.rst
>> @@ -5645,6 +5645,51 @@ enabled with ``arch_prctl()``, but this may 
>> change in the future.
>>   The offsets of the state save areas in struct kvm_xsave follow the 
>> contents
>>   of CPUID leaf 0xD on the host.
>> +4.135 KVM_S390_ZPCI_OP
>> +--------------------
>> +
>> +:Capability: KVM_CAP_S390_ZPCI_OP
>> +:Architectures: s390
>> +:Type: vcpu ioctl
>> +:Parameters: struct kvm_s390_zpci_op (in)
>> +:Returns: 0 on success, <0 on error
>> +
>> +Used to manage hardware-assisted virtualization features for zPCI 
>> devices.
>> +
>> +Parameters are specified via the following structure::
>> +
>> +  struct kvm_s390_zpci_op {
>> +    /* in */
>> +    __u32 fh;        /* target device */
>> +    __u8  op;        /* operation to perform */
>> +    __u8  pad[3];
>> +    union {
>> +        /* for KVM_S390_ZPCIOP_REG_AEN */
>> +        struct {
>> +            __u64 ibv;    /* Guest addr of interrupt bit vector */
>> +            __u64 sb;    /* Guest addr of summary bit */
>> +            __u32 flags;
>> +            __u32 noi;    /* Number of interrupts */
>> +            __u8 isc;    /* Guest interrupt subclass */
>> +            __u8 sbo;    /* Offset of guest summary bit vector */
>> +            __u16 pad;
>> +        } reg_aen;
>> +        __u64 reserved[8];
>> +    } u;
>> +  };
>> +
>> +The type of operation is specified in the "op" field.
>> +KVM_S390_ZPCIOP_REG_AEN is used to register the VM for adapter event
>> +notification interpretation, which will allow firmware delivery of 
>> adapter
>> +events directly to the vm, with KVM providing a backup delivery 
>> mechanism;
>> +KVM_S390_ZPCIOP_DEREG_AEN is used to subsequently disable 
>> interpretation of
>> +adapter event notifications.
>> +
>> +The target zPCI function must also be specified via the "fh" field.  
>> For the
>> +KVM_S390_ZPCIOP_REG_AEN operation, additional information to 
>> establish firmware
>> +delivery must be provided via the "reg_aen" struct.
>> +
>> +The "reserved" field is meant for future extensions.
>>   5. The kvm_run structure
>>   ========================
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index 704d85214f4f..65a53e22f686 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -616,6 +616,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, 
>> long ext)
>>       case KVM_CAP_S390_PROTECTED:
>>           r = is_prot_virt_host();
>>           break;
>> +    case KVM_CAP_S390_ZPCI_OP:
>> +        if (kvm_s390_pci_interp_allowed())
>> +            r = 1;
>> +        else
>> +            r = 0;
>> +        break;
>>       default:
>>           r = 0;
>>       }
>> @@ -2621,6 +2627,23 @@ long kvm_arch_vm_ioctl(struct file *filp,
>>               r = -EFAULT;
>>           break;
>>       }
>> +    case KVM_S390_ZPCI_OP: {
>> +        struct kvm_s390_zpci_op args;
>> +
>> +        r = -EINVAL;
>> +        if (!IS_ENABLED(CONFIG_VFIO_PCI))
>> +            break;
>> +        if (copy_from_user(&args, argp, sizeof(args))) {
>> +            r = -EFAULT;
>> +            break;
>> +        }
>> +        r = kvm_s390_pci_zpci_op(kvm, &args);
>> +        if (r)
>> +            break;
>> +        if (copy_to_user(argp, &args, sizeof(args)))
>> +            r = -EFAULT;
>> +        break;
>> +    }
>>       default:
>>           r = -ENOTTY;
>>       }
>> diff --git a/arch/s390/kvm/pci.c b/arch/s390/kvm/pci.c
>> index 66565f5f3f43..d941657bcb39 100644
>> --- a/arch/s390/kvm/pci.c
>> +++ b/arch/s390/kvm/pci.c
>> @@ -588,6 +588,71 @@ void kvm_s390_pci_clear_list(struct kvm *kvm)
>>           unregister_kvm(kzdev->zdev);
>>   }
>> +static struct kvm_zdev *get_kzdev_by_fh(struct kvm *kvm, u32 fh)
>> +{
>> +    struct kvm_zdev *kzdev, *retval = NULL;
>> +
>> +    spin_lock(&kvm->arch.kzdev_list_lock);
>> +    list_for_each_entry(kzdev, &kvm->arch.kzdev_list, entry) {
>> +        if (kzdev->zdev->fh == fh) {
>> +            retval = kzdev;
>> +            break;
>> +        }
>> +    }
>> +    spin_unlock(&kvm->arch.kzdev_list_lock);
>> +
>> +    return retval;
>> +}
>> +
>> +static int kvm_s390_pci_zpci_reg_aen(struct zpci_dev *zdev,
>> +                     struct kvm_s390_zpci_op *args)
>> +{
>> +    struct zpci_fib fib = {};
>> +
>> +    fib.fmt0.aibv = args->u.reg_aen.ibv;
>> +    fib.fmt0.isc = args->u.reg_aen.isc;
>> +    fib.fmt0.noi = args->u.reg_aen.noi;
>> +    if (args->u.reg_aen.sb != 0) {
>> +        fib.fmt0.aisb = args->u.reg_aen.sb;
>> +        fib.fmt0.aisbo = args->u.reg_aen.sbo;
>> +        fib.fmt0.sum = 1;
>> +    } else {
>> +        fib.fmt0.aisb = 0;
>> +        fib.fmt0.aisbo = 0;
>> +        fib.fmt0.sum = 0;
>> +    }
>> +
>> +    if (args->u.reg_aen.flags & KVM_S390_ZPCIOP_REGAEN_HOST)
>> +        return kvm_s390_pci_aif_enable(zdev, &fib, true);
>> +    else
>> +        return kvm_s390_pci_aif_enable(zdev, &fib, false);
>> +}
>> +
>> +int kvm_s390_pci_zpci_op(struct kvm *kvm, struct kvm_s390_zpci_op *args)
>> +{
>> +    struct kvm_zdev *kzdev;
>> +    struct zpci_dev *zdev;
>> +    int r;
>> +
>> +    kzdev = get_kzdev_by_fh(kvm, args->fh);
>> +    if (!kzdev)
>> +        return -ENODEV;
>> +    zdev = kzdev->zdev;
>> +
>> +    switch (args->op) {
>> +    case KVM_S390_ZPCIOP_REG_AEN:
>> +        r = kvm_s390_pci_zpci_reg_aen(zdev, args);
>> +        break;
>> +    case KVM_S390_ZPCIOP_DEREG_AEN:
>> +        r = kvm_s390_pci_aif_disable(zdev, false);
>> +        break;
>> +    default:
>> +        r = -EINVAL;
>> +    }
>> +
>> +    return r;
>> +}
>> +
>>   int kvm_s390_pci_init(void)
>>   {
>>       aift = kzalloc(sizeof(struct zpci_aift), GFP_KERNEL);
>> diff --git a/arch/s390/kvm/pci.h b/arch/s390/kvm/pci.h
>> index cb5ec3208923..b053b50c0904 100644
>> --- a/arch/s390/kvm/pci.h
>> +++ b/arch/s390/kvm/pci.h
>> @@ -59,6 +59,8 @@ void kvm_s390_pci_aen_exit(void);
>>   void kvm_s390_pci_init_list(struct kvm *kvm);
>>   void kvm_s390_pci_clear_list(struct kvm *kvm);
>> +int kvm_s390_pci_zpci_op(struct kvm *kvm, struct kvm_s390_zpci_op 
>> *args);
>> +
>>   int kvm_s390_pci_init(void);
>>   static inline bool kvm_s390_pci_interp_allowed(void)
>> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
>> index 91a6fe4e02c0..014adb30d42c 100644
>> --- a/include/uapi/linux/kvm.h
>> +++ b/include/uapi/linux/kvm.h
>> @@ -1144,6 +1144,7 @@ struct kvm_ppc_resize_hpt {
>>   #define KVM_CAP_S390_MEM_OP_EXTENSION 211
>>   #define KVM_CAP_PMU_CAPABILITY 212
>>   #define KVM_CAP_DISABLE_QUIRKS2 213
>> +#define KVM_CAP_S390_ZPCI_OP 214
>>   #ifdef KVM_CAP_IRQ_ROUTING
>> @@ -2060,4 +2061,34 @@ struct kvm_stats_desc {
>>   /* Available with KVM_CAP_XSAVE2 */
>>   #define KVM_GET_XSAVE2          _IOR(KVMIO,  0xcf, struct kvm_xsave)
>> +/* Available with KVM_CAP_S390_ZPCI_OP */
>> +#define KVM_S390_ZPCI_OP      _IOW(KVMIO,  0xd0, struct 
>> kvm_s390_zpci_op)
>> +
>> +struct kvm_s390_zpci_op {
>> +    /* in */
>> +    __u32 fh;        /* target device */
>> +    __u8  op;        /* operation to perform */
>> +    __u8  pad[3];
>> +    union {
>> +        /* for KVM_S390_ZPCIOP_REG_AEN */
>> +        struct {
>> +            __u64 ibv;    /* Guest addr of interrupt bit vector */
>> +            __u64 sb;    /* Guest addr of summary bit */
>> +            __u32 flags;
>> +            __u32 noi;    /* Number of interrupts */
>> +            __u8 isc;    /* Guest interrupt subclass */
>> +            __u8 sbo;    /* Offset of guest summary bit vector */
>> +            __u16 pad;
>> +        } reg_aen;
>> +        __u64 reserved[8];
>> +    } u;
>> +};
>> +
>> +/* types for kvm_s390_zpci_op->op */
>> +#define KVM_S390_ZPCIOP_REG_AEN        0
>> +#define KVM_S390_ZPCIOP_DEREG_AEN    1
>> +
>> +/* flags for kvm_s390_zpci_op->u.reg_aen.flags */
>> +#define KVM_S390_ZPCIOP_REGAEN_HOST    (1 << 0)
>> +
>>   #endif /* __LINUX_KVM_H */
>>
> 
> 
> The purpose is to setup the IRQ for the zPCI device wouldn't it be more 
> logical to go through VFIO_DEVICE_SET_IRQS and there jump to the zPCI 
> handling, like it is done for the capabilities, instead to go directly 
> through KVM?
> We would also spare the research for the real FH in KVM.
> 

We need the host FH anyway -- for load/store intepretation, the guest 
must be using the FH that the firmware expects.  Plus we need that FH 
when issuing the MPCIFC op 2.  I guess maybe you mean the lookup (get 
the exact device associated with this fd vs scanning the list of PCI 
devices associated with this KVM).

One of the issues here is that, with load/store interpretation enabled, 
vfio in QEMU will never get the config space notifiers triggered; these 
are the things that end up calling vfio_enable_vectors() which will in 
turn issue the VFIO_DEVICE_SET_IRQS (instead these guest config space 
writes are executed interpretively)

I did play around with this for a while, but if we were to call 
VFIO_DEVICE_SET_IRQS ourselves from QEMU s390-pci instead of relying on 
the config space notifier, we still have the issue of associating the 
previously-established routes with the host function - because at the 
end of it we need to issue a MPCIFC(2) specifying the host FH and 
including the forwarding information (e.g. the work done in 
kvm_s390_pci_aif_enable).  AFAICT today we can't derive the FH from 
routes, that's where I got stuck.  But if you have an idea please share.

Re: [PATCH v5 19/21] KVM: s390: add KVM_S390_ZPCI_OP to manage guest zPCI devices

[Pierre Morel]

On 4/20/22 16:10, Matthew Rosato wrote:
> On 4/19/22 6:07 AM, Pierre Morel wrote:
>>
>>
>> On 4/4/22 19:43, Matthew Rosato wrote:
>>> The KVM_S390_ZPCI_OP ioctl provides a mechanism for managing
>>> hardware-assisted virtualization features for s390X zPCI passthrough.
>>> Add the first 2 operations, which can be used to enable/disable
>>> the specified device for Adapter Event Notification interpretation.
>>>
>>> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
>>> ---
>>>   Documentation/virt/kvm/api.rst | 45 +++++++++++++++++++++++
>>>   arch/s390/kvm/kvm-s390.c       | 23 ++++++++++++
>>>   arch/s390/kvm/pci.c            | 65 ++++++++++++++++++++++++++++++++++
>>>   arch/s390/kvm/pci.h            |  2 ++
>>>   include/uapi/linux/kvm.h       | 31 ++++++++++++++++
>>>   5 files changed, 166 insertions(+)
>>>
>>> diff --git a/Documentation/virt/kvm/api.rst 
>>> b/Documentation/virt/kvm/api.rst
>>> index d13fa6600467..474f502c2ea0 100644
>>> --- a/Documentation/virt/kvm/api.rst
>>> +++ b/Documentation/virt/kvm/api.rst
>>> @@ -5645,6 +5645,51 @@ enabled with ``arch_prctl()``, but this may 
>>> change in the future.
>>>   The offsets of the state save areas in struct kvm_xsave follow the 
>>> contents
>>>   of CPUID leaf 0xD on the host.
>>> +4.135 KVM_S390_ZPCI_OP
>>> +--------------------
>>> +
>>> +:Capability: KVM_CAP_S390_ZPCI_OP
>>> +:Architectures: s390
>>> +:Type: vcpu ioctl
>>> +:Parameters: struct kvm_s390_zpci_op (in)
>>> +:Returns: 0 on success, <0 on error
>>> +
>>> +Used to manage hardware-assisted virtualization features for zPCI 
>>> devices.
>>> +
>>> +Parameters are specified via the following structure::
>>> +
>>> +  struct kvm_s390_zpci_op {
>>> +    /* in */
>>> +    __u32 fh;        /* target device */
>>> +    __u8  op;        /* operation to perform */
>>> +    __u8  pad[3];
>>> +    union {
>>> +        /* for KVM_S390_ZPCIOP_REG_AEN */
>>> +        struct {
>>> +            __u64 ibv;    /* Guest addr of interrupt bit vector */
>>> +            __u64 sb;    /* Guest addr of summary bit */
>>> +            __u32 flags;
>>> +            __u32 noi;    /* Number of interrupts */
>>> +            __u8 isc;    /* Guest interrupt subclass */
>>> +            __u8 sbo;    /* Offset of guest summary bit vector */
>>> +            __u16 pad;
>>> +        } reg_aen;
>>> +        __u64 reserved[8];
>>> +    } u;
>>> +  };
>>> +
>>> +The type of operation is specified in the "op" field.
>>> +KVM_S390_ZPCIOP_REG_AEN is used to register the VM for adapter event
>>> +notification interpretation, which will allow firmware delivery of 
>>> adapter
>>> +events directly to the vm, with KVM providing a backup delivery 
>>> mechanism;
>>> +KVM_S390_ZPCIOP_DEREG_AEN is used to subsequently disable 
>>> interpretation of
>>> +adapter event notifications.
>>> +
>>> +The target zPCI function must also be specified via the "fh" field. 
>>> For the
>>> +KVM_S390_ZPCIOP_REG_AEN operation, additional information to 
>>> establish firmware
>>> +delivery must be provided via the "reg_aen" struct.
>>> +
>>> +The "reserved" field is meant for future extensions.
>>>   5. The kvm_run structure
>>>   ========================
>>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>>> index 704d85214f4f..65a53e22f686 100644
>>> --- a/arch/s390/kvm/kvm-s390.c
>>> +++ b/arch/s390/kvm/kvm-s390.c
>>> @@ -616,6 +616,12 @@ int kvm_vm_ioctl_check_extension(struct kvm 
>>> *kvm, long ext)
>>>       case KVM_CAP_S390_PROTECTED:
>>>           r = is_prot_virt_host();
>>>           break;
>>> +    case KVM_CAP_S390_ZPCI_OP:
>>> +        if (kvm_s390_pci_interp_allowed())
>>> +            r = 1;
>>> +        else
>>> +            r = 0;
>>> +        break;
>>>       default:
>>>           r = 0;
>>>       }
>>> @@ -2621,6 +2627,23 @@ long kvm_arch_vm_ioctl(struct file *filp,
>>>               r = -EFAULT;
>>>           break;
>>>       }
>>> +    case KVM_S390_ZPCI_OP: {
>>> +        struct kvm_s390_zpci_op args;
>>> +
>>> +        r = -EINVAL;
>>> +        if (!IS_ENABLED(CONFIG_VFIO_PCI))
>>> +            break;
>>> +        if (copy_from_user(&args, argp, sizeof(args))) {
>>> +            r = -EFAULT;
>>> +            break;
>>> +        }
>>> +        r = kvm_s390_pci_zpci_op(kvm, &args);
>>> +        if (r)
>>> +            break;
>>> +        if (copy_to_user(argp, &args, sizeof(args)))
>>> +            r = -EFAULT;
>>> +        break;
>>> +    }
>>>       default:
>>>           r = -ENOTTY;
>>>       }
>>> diff --git a/arch/s390/kvm/pci.c b/arch/s390/kvm/pci.c
>>> index 66565f5f3f43..d941657bcb39 100644
>>> --- a/arch/s390/kvm/pci.c
>>> +++ b/arch/s390/kvm/pci.c
>>> @@ -588,6 +588,71 @@ void kvm_s390_pci_clear_list(struct kvm *kvm)
>>>           unregister_kvm(kzdev->zdev);
>>>   }
>>> +static struct kvm_zdev *get_kzdev_by_fh(struct kvm *kvm, u32 fh)
>>> +{
>>> +    struct kvm_zdev *kzdev, *retval = NULL;
>>> +
>>> +    spin_lock(&kvm->arch.kzdev_list_lock);
>>> +    list_for_each_entry(kzdev, &kvm->arch.kzdev_list, entry) {
>>> +        if (kzdev->zdev->fh == fh) {
>>> +            retval = kzdev;
>>> +            break;
>>> +        }
>>> +    }
>>> +    spin_unlock(&kvm->arch.kzdev_list_lock);
>>> +
>>> +    return retval;
>>> +}
>>> +
>>> +static int kvm_s390_pci_zpci_reg_aen(struct zpci_dev *zdev,
>>> +                     struct kvm_s390_zpci_op *args)
>>> +{
>>> +    struct zpci_fib fib = {};
>>> +
>>> +    fib.fmt0.aibv = args->u.reg_aen.ibv;
>>> +    fib.fmt0.isc = args->u.reg_aen.isc;
>>> +    fib.fmt0.noi = args->u.reg_aen.noi;
>>> +    if (args->u.reg_aen.sb != 0) {
>>> +        fib.fmt0.aisb = args->u.reg_aen.sb;
>>> +        fib.fmt0.aisbo = args->u.reg_aen.sbo;
>>> +        fib.fmt0.sum = 1;
>>> +    } else {
>>> +        fib.fmt0.aisb = 0;
>>> +        fib.fmt0.aisbo = 0;
>>> +        fib.fmt0.sum = 0;
>>> +    }
>>> +
>>> +    if (args->u.reg_aen.flags & KVM_S390_ZPCIOP_REGAEN_HOST)
>>> +        return kvm_s390_pci_aif_enable(zdev, &fib, true);
>>> +    else
>>> +        return kvm_s390_pci_aif_enable(zdev, &fib, false);
>>> +}
>>> +
>>> +int kvm_s390_pci_zpci_op(struct kvm *kvm, struct kvm_s390_zpci_op 
>>> *args)
>>> +{
>>> +    struct kvm_zdev *kzdev;
>>> +    struct zpci_dev *zdev;
>>> +    int r;
>>> +
>>> +    kzdev = get_kzdev_by_fh(kvm, args->fh);
>>> +    if (!kzdev)
>>> +        return -ENODEV;
>>> +    zdev = kzdev->zdev;
>>> +
>>> +    switch (args->op) {
>>> +    case KVM_S390_ZPCIOP_REG_AEN:
>>> +        r = kvm_s390_pci_zpci_reg_aen(zdev, args);
>>> +        break;
>>> +    case KVM_S390_ZPCIOP_DEREG_AEN:
>>> +        r = kvm_s390_pci_aif_disable(zdev, false);
>>> +        break;
>>> +    default:
>>> +        r = -EINVAL;
>>> +    }
>>> +
>>> +    return r;
>>> +}
>>> +
>>>   int kvm_s390_pci_init(void)
>>>   {
>>>       aift = kzalloc(sizeof(struct zpci_aift), GFP_KERNEL);
>>> diff --git a/arch/s390/kvm/pci.h b/arch/s390/kvm/pci.h
>>> index cb5ec3208923..b053b50c0904 100644
>>> --- a/arch/s390/kvm/pci.h
>>> +++ b/arch/s390/kvm/pci.h
>>> @@ -59,6 +59,8 @@ void kvm_s390_pci_aen_exit(void);
>>>   void kvm_s390_pci_init_list(struct kvm *kvm);
>>>   void kvm_s390_pci_clear_list(struct kvm *kvm);
>>> +int kvm_s390_pci_zpci_op(struct kvm *kvm, struct kvm_s390_zpci_op 
>>> *args);
>>> +
>>>   int kvm_s390_pci_init(void);
>>>   static inline bool kvm_s390_pci_interp_allowed(void)
>>> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
>>> index 91a6fe4e02c0..014adb30d42c 100644
>>> --- a/include/uapi/linux/kvm.h
>>> +++ b/include/uapi/linux/kvm.h
>>> @@ -1144,6 +1144,7 @@ struct kvm_ppc_resize_hpt {
>>>   #define KVM_CAP_S390_MEM_OP_EXTENSION 211
>>>   #define KVM_CAP_PMU_CAPABILITY 212
>>>   #define KVM_CAP_DISABLE_QUIRKS2 213
>>> +#define KVM_CAP_S390_ZPCI_OP 214
>>>   #ifdef KVM_CAP_IRQ_ROUTING
>>> @@ -2060,4 +2061,34 @@ struct kvm_stats_desc {
>>>   /* Available with KVM_CAP_XSAVE2 */
>>>   #define KVM_GET_XSAVE2          _IOR(KVMIO,  0xcf, struct kvm_xsave)
>>> +/* Available with KVM_CAP_S390_ZPCI_OP */
>>> +#define KVM_S390_ZPCI_OP      _IOW(KVMIO,  0xd0, struct 
>>> kvm_s390_zpci_op)
>>> +
>>> +struct kvm_s390_zpci_op {
>>> +    /* in */
>>> +    __u32 fh;        /* target device */
>>> +    __u8  op;        /* operation to perform */
>>> +    __u8  pad[3];
>>> +    union {
>>> +        /* for KVM_S390_ZPCIOP_REG_AEN */
>>> +        struct {
>>> +            __u64 ibv;    /* Guest addr of interrupt bit vector */
>>> +            __u64 sb;    /* Guest addr of summary bit */
>>> +            __u32 flags;
>>> +            __u32 noi;    /* Number of interrupts */
>>> +            __u8 isc;    /* Guest interrupt subclass */
>>> +            __u8 sbo;    /* Offset of guest summary bit vector */
>>> +            __u16 pad;
>>> +        } reg_aen;
>>> +        __u64 reserved[8];
>>> +    } u;
>>> +};
>>> +
>>> +/* types for kvm_s390_zpci_op->op */
>>> +#define KVM_S390_ZPCIOP_REG_AEN        0
>>> +#define KVM_S390_ZPCIOP_DEREG_AEN    1
>>> +
>>> +/* flags for kvm_s390_zpci_op->u.reg_aen.flags */
>>> +#define KVM_S390_ZPCIOP_REGAEN_HOST    (1 << 0)
>>> +
>>>   #endif /* __LINUX_KVM_H */
>>>
>>
>>
>> The purpose is to setup the IRQ for the zPCI device wouldn't it be 
>> more logical to go through VFIO_DEVICE_SET_IRQS and there jump to the 
>> zPCI handling, like it is done for the capabilities, instead to go 
>> directly through KVM?
>> We would also spare the research for the real FH in KVM.
>>
> 
> We need the host FH anyway -- for load/store intepretation, the guest 
> must be using the FH that the firmware expects.  Plus we need that FH 
> when issuing the MPCIFC op 2.  I guess maybe you mean the lookup (get 
> the exact device associated with this fd vs scanning the list of PCI 
> devices associated with this KVM).

Obvious ly we need to get the real FH, yes, and yes I was speaking of 
scanning the list-pci but t is not the main idea.
The main idea is that going through the VFIO device ioctl seems to me 
the right way to go to handle IRQ for a device.

One can argue that it is not really PCI specific.
Right but we have no possibility to intercept it inside the vfio code.

unless we add some arch specific hook in vfio_device_fops_unl_ioctl().

> 
> One of the issues here is that, with load/store interpretation enabled, 
> vfio in QEMU will never get the config space notifiers triggered; these 
> are the things that end up calling vfio_enable_vectors() which will in 
> turn issue the VFIO_DEVICE_SET_IRQS (instead these guest config space 
> writes are executed interpretively)

What I mean is to add in vfio_pci_core_ioctl() a treatment for 
VFIO_DEVICE_SET_IRQS like you did for VFIO_DEVICE_GET_INFO with a 
specific operation or flag to call kvm_s390_pci_zpci_reg_aen().

Then, in QEMU we add a new call to VFIO_DEVICE_SET_IRQS with this 
special operation/flag in s390_pci_kvm_aif_enable() / disable


> 
> I did play around with this for a while, but if we were to call 
> VFIO_DEVICE_SET_IRQS ourselves from QEMU s390-pci instead of relying on 
> the config space notifier, we still have the issue of associating the 
> previously-established routes with the host function - because at the 
> end of it we need to issue a MPCIFC(2) specifying the host FH and 
> including the forwarding information (e.g. the work done in 
> kvm_s390_pci_aif_enable).  AFAICT today we can't derive the FH from 
> routes, that's where I got stuck.  But if you have an idea please share.

As I said no change about getting the FH, so no change for the MPCIFC(2)

Anyway, it touches VFIO and/or VFIO_PCI so @alex do you think it is 
interesting?
Knowing that the current version works it could also be done later.

Regards,
Pierre


-- 
Pierre Morel
IBM Lab Boeblingen