[Qemu-devel] [PATCH v3 0/7] s390x: vfio-ap: guest dedicated crypto adapters

[Qemu-devel] [PATCH v3 0/7] s390x: vfio-ap: guest dedicated crypto adapters

[Tony Krowiak]

This patch series is the QEMU counterpart to the KVM/kernel support for 
guest dedicated crypto adapters. The KVM/kernel model is built on the 
VFIO mediated device framework and provides the infrastructure for 
granting exclusive guest access to crypto devices installed on the linux 
host. This patch series introduces a new QEMU command line option, QEMU 
object model and CPU model features to exploit the KVM/kernel model.

See the detailed specifications for AP virtualization provided by this 
patch set in docs/vfio-ap.txt for a more complete discussion of the 
design introduced by this patch series.

v2 -> v3 Change log:
===================
* The corresponding KVM/kernel patch series defined a new device attribute
  to the KVM_S390_VM_CRYPTO attribute group to set interpretive execution
  of AP instructions for the guest. Now, when the vfio-ap device is 
  realized, the KVM_SET_DEVICE_ATTR ioctl is invoked to enable 
  interpretation of AP instructions executed on the guest.

* When the CPU model feature indicating AP instructions are supported for
  the guest (i.e., -cpu xxxx,ap=on), the expectation is that the AP bus 
  will be initialized on the guest. In v2, however; AP instructions were
  intercepted unless a vfio-ap device (i.e., -device vfio-ap,sysfsdev=$path)
  was defined for the guest. Since there were no handlers defined to 
  process intercepted AP instructions, an operation exception was injected
  into the guest causing the AP bus to fail initialization. The v3 version
  adds handlers for intercepted AP instructions that incorporate the 
  following logic:

  If the CPU model indicates AP instructions are installed 
     Set the status response code for the instruction to indicate that
     the APQN contained in the instruction is not valid. This is actually
     true because the AP devices are configured for the guest via the  
     sysfs attribute files of the vfio-ap device, so any APQN would not
     be valid.
   
  Else
     Return an error from the handler. This will cause an operation 
     exception to be injected into the guest in which case the AP bus
     would not initialize. This is the same behavior as exists prior to 
     the introduction of this patch series.

* Miscellaneous code formatting and other trivial changes.

Tony Krowiak (7):
  linux-headers: linux header updates for AP support
  s390x/ap: base Adjunct Processor (AP) object
  s390x/cpumodel: Set up CPU model for AP device support
  s390x/kvm: interface to interpret AP instructions
  s390x/vfio: ap: Introduce VFIO AP device
  s390x/kvm: handle AP instruction interception
  s390: doc: detailed specifications for AP virtualization

 default-configs/s390x-softmmu.mak |    1 +
 docs/vfio-ap.txt                  |  624 +++++++++++++++++++++++++++++++++++++
 hw/s390x/Makefile.objs            |    1 +
 hw/s390x/ap-device.c              |   38 +++
 hw/vfio/Makefile.objs             |    1 +
 hw/vfio/ap.c                      |  229 ++++++++++++++
 include/hw/s390x/ap-device.h      |   44 +++
 include/hw/vfio/vfio-common.h     |    1 +
 linux-headers/asm-s390/kvm.h      |    2 +
 linux-headers/linux/vfio.h        |    2 +
 target/s390x/cpu_features.c       |    3 +
 target/s390x/cpu_features_def.h   |    3 +
 target/s390x/cpu_models.c         |    2 +
 target/s390x/gen-features.c       |    3 +
 target/s390x/kvm.c                |   31 ++
 target/s390x/kvm_s390x.h          |    2 +
 16 files changed, 987 insertions(+), 0 deletions(-)
 create mode 100644 docs/vfio-ap.txt
 create mode 100644 hw/s390x/ap-device.c
 create mode 100644 hw/vfio/ap.c
 create mode 100644 include/hw/s390x/ap-device.h

[Qemu-devel] [PATCH v3 1/7] linux-headers: linux header updates for AP support

[Tony Krowiak]

Updates the linux header files in preparation for introduction
of the VFIO AP device:

* Added a feature ID to indicate AP facilities are installed

* Added a device attribute to the KVM_S390_VM_CRYPTO group to
  indicate whether AP instructions are to be interpreted

* Added VFIO device information for AP devices

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 linux-headers/asm-s390/kvm.h |    2 ++
 linux-headers/linux/vfio.h   |    2 ++
 2 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/linux-headers/asm-s390/kvm.h b/linux-headers/asm-s390/kvm.h
index 11def14..391b250 100644
--- a/linux-headers/asm-s390/kvm.h
+++ b/linux-headers/asm-s390/kvm.h
@@ -130,6 +130,7 @@ struct kvm_s390_vm_cpu_machine {
 #define KVM_S390_VM_CPU_FEAT_PFMFI	11
 #define KVM_S390_VM_CPU_FEAT_SIGPIF	12
 #define KVM_S390_VM_CPU_FEAT_KSS	13
+#define KVM_S390_VM_CPU_FEAT_AP		14
 struct kvm_s390_vm_cpu_feat {
 	__u64 feat[16];
 };
@@ -160,6 +161,7 @@ struct kvm_s390_vm_cpu_subfunc {
 #define KVM_S390_VM_CRYPTO_ENABLE_DEA_KW	1
 #define KVM_S390_VM_CRYPTO_DISABLE_AES_KW	2
 #define KVM_S390_VM_CRYPTO_DISABLE_DEA_KW	3
+#define KVM_S390_VM_CRYPTO_INTERPRET_AP		4
 
 /* kvm attributes for migration mode */
 #define KVM_S390_VM_MIGRATION_STOP	0
diff --git a/linux-headers/linux/vfio.h b/linux-headers/linux/vfio.h
index 4312e96..91298dc 100644
--- a/linux-headers/linux/vfio.h
+++ b/linux-headers/linux/vfio.h
@@ -200,6 +200,7 @@ struct vfio_device_info {
 #define VFIO_DEVICE_FLAGS_PLATFORM (1 << 2)	/* vfio-platform device */
 #define VFIO_DEVICE_FLAGS_AMBA  (1 << 3)	/* vfio-amba device */
 #define VFIO_DEVICE_FLAGS_CCW	(1 << 4)	/* vfio-ccw device */
+#define VFIO_DEVICE_FLAGS_AP (1 << 5)		/* vfio-ap device */
 	__u32	num_regions;	/* Max region index + 1 */
 	__u32	num_irqs;	/* Max IRQ index + 1 */
 };
@@ -215,6 +216,7 @@ struct vfio_device_info {
 #define VFIO_DEVICE_API_PLATFORM_STRING		"vfio-platform"
 #define VFIO_DEVICE_API_AMBA_STRING		"vfio-amba"
 #define VFIO_DEVICE_API_CCW_STRING		"vfio-ccw"
+#define VFIO_DEVICE_API_AP_STRING		"vfio-ap"
 
 /**
  * VFIO_DEVICE_GET_REGION_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 8,
-- 
1.7.1

[Qemu-devel] [PATCH v3 2/7] s390x/ap: base Adjunct Processor (AP) object

[Tony Krowiak]

This patch introduces the base object for an AP device.

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 hw/s390x/Makefile.objs       |    1 +
 hw/s390x/ap-device.c         |   38 ++++++++++++++++++++++++++++++++++++++
 include/hw/s390x/ap-device.h |   38 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 77 insertions(+), 0 deletions(-)
 create mode 100644 hw/s390x/ap-device.c
 create mode 100644 include/hw/s390x/ap-device.h

diff --git a/hw/s390x/Makefile.objs b/hw/s390x/Makefile.objs
index dc704b5..3247a07 100644
--- a/hw/s390x/Makefile.objs
+++ b/hw/s390x/Makefile.objs
@@ -17,3 +17,4 @@ obj-y += s390-stattrib.o
 obj-$(CONFIG_KVM) += s390-skeys-kvm.o
 obj-$(CONFIG_KVM) += s390-stattrib-kvm.o
 obj-y += s390-ccw.o
+obj-y += ap-device.o
diff --git a/hw/s390x/ap-device.c b/hw/s390x/ap-device.c
new file mode 100644
index 0000000..448f6db
--- /dev/null
+++ b/hw/s390x/ap-device.c
@@ -0,0 +1,38 @@
+/*
+ * Adjunct Processor (AP) matrix device
+ *
+ * Copyright 2018 IBM Corp.
+ * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or (at
+ * your option) any later version. See the COPYING file in the top-level
+ * directory.
+ */
+#include "qemu/osdep.h"
+#include "qemu/module.h"
+#include "qapi/error.h"
+#include "hw/qdev.h"
+#include "hw/s390x/ap-device.h"
+
+static void ap_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+
+    dc->desc = "AP device class";
+}
+
+static const TypeInfo ap_device_info = {
+    .name = AP_DEVICE_TYPE,
+    .parent = TYPE_DEVICE,
+    .instance_size = sizeof(APDevice),
+    .class_size = sizeof(APDeviceClass),
+    .class_init = ap_class_init,
+    .abstract = true,
+};
+
+static void ap_device_register(void)
+{
+    type_register_static(&ap_device_info);
+}
+
+type_init(ap_device_register)
diff --git a/include/hw/s390x/ap-device.h b/include/hw/s390x/ap-device.h
new file mode 100644
index 0000000..693df90
--- /dev/null
+++ b/include/hw/s390x/ap-device.h
@@ -0,0 +1,38 @@
+/*
+ * Adjunct Processor (AP) matrix device interfaces
+ *
+ * Copyright 2018 IBM Corp.
+ * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or (at
+ * your option) any later version. See the COPYING file in the top-level
+ * directory.
+ */
+#ifndef HW_S390X_AP_DEVICE_H
+#define HW_S390X_AP_DEVICE_H
+
+#define AP_DEVICE_TYPE       "ap-device"
+
+typedef struct APDevice {
+    DeviceState parent_obj;
+} APDevice;
+
+typedef struct APDeviceClass {
+    DeviceClass parent_class;
+} APDeviceClass;
+
+static inline APDevice *to_ap_dev(DeviceState *dev)
+{
+    return container_of(dev, APDevice, parent_obj);
+}
+
+#define AP_DEVICE(obj) \
+    OBJECT_CHECK(APDevice, (obj), AP_DEVICE_TYPE)
+
+#define AP_DEVICE_GET_CLASS(obj) \
+    OBJECT_GET_CLASS(APDeviceClass, (obj), AP_DEVICE_TYPE)
+
+#define AP_DEVICE_CLASS(klass) \
+    OBJECT_CLASS_CHECK(APDeviceClass, (klass), AP_DEVICE_TYPE)
+
+#endif /* HW_S390X_AP_DEVICE_H */
-- 
1.7.1

[Qemu-devel] [PATCH v3 3/7] s390x/cpumodel: Set up CPU model for AP device support

[Tony Krowiak]

A new CPU model feature and two new CPU model facilities are
introduced to support AP devices for a KVM guest.

CPU model features:

1. The KVM_S390_VM_CPU_FEAT_AP CPU model feature indicates that
   AP facilities are installed. This feature will be enabled by
   the kernel only if the AP facilities are installed on the linux
   host. This feature must be turned on from userspace to access
   AP devices from the KVM guest. The QEMU command line to turn
   this feature looks something like this:

	qemu-system-s390x ... -cpu xxx,ap=on

CPU model facilities:

1. The S390_FEAT_AP_QUERY_CONFIG_INFO feature indicates the AP Query
   Configuration Information (QCI) facility is installed. This feature
   will be enabled by the kernel only if the QCI is installed on
   the host.

2. The S390_FEAT_AP_FACILITY_TEST feature indicates that the AP
   Facility Test (APFT) facility is installed. This feature will
   be enabled by the kernel only if the APFT facility is installed
   on the host.

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 target/s390x/cpu_features.c     |    3 +++
 target/s390x/cpu_features_def.h |    3 +++
 target/s390x/cpu_models.c       |    2 ++
 target/s390x/gen-features.c     |    3 +++
 target/s390x/kvm.c              |    1 +
 5 files changed, 12 insertions(+), 0 deletions(-)

diff --git a/target/s390x/cpu_features.c b/target/s390x/cpu_features.c
index a5619f2..1abe987 100644
--- a/target/s390x/cpu_features.c
+++ b/target/s390x/cpu_features.c
@@ -36,8 +36,10 @@ static const S390FeatDef s390_features[] = {
     FEAT_INIT("srs", S390_FEAT_TYPE_STFL, 9, "Sense-running-status facility"),
     FEAT_INIT("csske", S390_FEAT_TYPE_STFL, 10, "Conditional-SSKE facility"),
     FEAT_INIT("ctop", S390_FEAT_TYPE_STFL, 11, "Configuration-topology facility"),
+    FEAT_INIT("qci", S390_FEAT_TYPE_STFL, 12, "Query AP Configuration facility"),
     FEAT_INIT("ipter", S390_FEAT_TYPE_STFL, 13, "IPTE-range facility"),
     FEAT_INIT("nonqks", S390_FEAT_TYPE_STFL, 14, "Nonquiescing key-setting facility"),
+    FEAT_INIT("apft", S390_FEAT_TYPE_STFL, 15, "Adjunct Processor Facilities Test facility"),
     FEAT_INIT("etf2", S390_FEAT_TYPE_STFL, 16, "Extended-translation facility 2"),
     FEAT_INIT("msa-base", S390_FEAT_TYPE_STFL, 17, "Message-security-assist facility (excluding subfunctions)"),
     FEAT_INIT("ldisp", S390_FEAT_TYPE_STFL, 18, "Long-displacement facility"),
@@ -125,6 +127,7 @@ static const S390FeatDef s390_features[] = {
 
     FEAT_INIT("dateh2", S390_FEAT_TYPE_MISC, 0, "DAT-enhancement facility 2"),
     FEAT_INIT("cmm", S390_FEAT_TYPE_MISC, 0, "Collaborative-memory-management facility"),
+    FEAT_INIT("ap", S390_FEAT_TYPE_MISC, 0, "AP facilities installed"),
 
     FEAT_INIT("plo-cl", S390_FEAT_TYPE_PLO, 0, "PLO Compare and load (32 bit in general registers)"),
     FEAT_INIT("plo-clg", S390_FEAT_TYPE_PLO, 1, "PLO Compare and load (64 bit in parameter list)"),
diff --git a/target/s390x/cpu_features_def.h b/target/s390x/cpu_features_def.h
index 7c5915c..8998b65 100644
--- a/target/s390x/cpu_features_def.h
+++ b/target/s390x/cpu_features_def.h
@@ -27,8 +27,10 @@ typedef enum {
     S390_FEAT_SENSE_RUNNING_STATUS,
     S390_FEAT_CONDITIONAL_SSKE,
     S390_FEAT_CONFIGURATION_TOPOLOGY,
+    S390_FEAT_AP_QUERY_CONFIG_INFO,
     S390_FEAT_IPTE_RANGE,
     S390_FEAT_NONQ_KEY_SETTING,
+    S390_FEAT_AP_FACILITIES_TEST,
     S390_FEAT_EXTENDED_TRANSLATION_2,
     S390_FEAT_MSA,
     S390_FEAT_LONG_DISPLACEMENT,
@@ -118,6 +120,7 @@ typedef enum {
     /* Misc */
     S390_FEAT_DAT_ENH_2,
     S390_FEAT_CMM,
+    S390_FEAT_AP,
 
     /* PLO */
     S390_FEAT_PLO_CL,
diff --git a/target/s390x/cpu_models.c b/target/s390x/cpu_models.c
index cf82589..7e2af09 100644
--- a/target/s390x/cpu_models.c
+++ b/target/s390x/cpu_models.c
@@ -769,6 +769,8 @@ static void check_consistency(const S390CPUModel *model)
         { S390_FEAT_PRNO_TRNG_QRTCR, S390_FEAT_MSA_EXT_5 },
         { S390_FEAT_PRNO_TRNG, S390_FEAT_MSA_EXT_5 },
         { S390_FEAT_SIE_KSS, S390_FEAT_SIE_F2 },
+        { S390_FEAT_AP_QUERY_CONFIG_INFO, S390_FEAT_AP },
+        { S390_FEAT_AP_FACILITIES_TEST, S390_FEAT_AP },
     };
     int i;
 
diff --git a/target/s390x/gen-features.c b/target/s390x/gen-features.c
index 0cdbc15..0d5b0f7 100644
--- a/target/s390x/gen-features.c
+++ b/target/s390x/gen-features.c
@@ -447,6 +447,9 @@ static uint16_t full_GEN12_GA1[] = {
     S390_FEAT_ADAPTER_INT_SUPPRESSION,
     S390_FEAT_EDAT_2,
     S390_FEAT_SIDE_EFFECT_ACCESS_ESOP2,
+    S390_FEAT_AP_QUERY_CONFIG_INFO,
+    S390_FEAT_AP_FACILITIES_TEST,
+    S390_FEAT_AP,
 };
 
 static uint16_t full_GEN12_GA2[] = {
diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c
index fbc887e..33e5ec3 100644
--- a/target/s390x/kvm.c
+++ b/target/s390x/kvm.c
@@ -2178,6 +2178,7 @@ static int kvm_to_feat[][2] = {
     { KVM_S390_VM_CPU_FEAT_PFMFI, S390_FEAT_SIE_PFMFI},
     { KVM_S390_VM_CPU_FEAT_SIGPIF, S390_FEAT_SIE_SIGPIF},
     { KVM_S390_VM_CPU_FEAT_KSS, S390_FEAT_SIE_KSS},
+    { KVM_S390_VM_CPU_FEAT_AP, S390_FEAT_AP},
 };
 
 static int query_cpu_feat(S390FeatBitmap features)
-- 
1.7.1

[Qemu-devel] [PATCH v3 4/7] s390x/kvm: interface to interpret AP instructions

[Tony Krowiak]

The VFIO AP device exploits interpretive execution of AP
instructions (APIE). APIE is enabled by setting a device attribute
via the KVM_SET_DEVICE_ATTR ioctl.

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 target/s390x/kvm.c       |   16 ++++++++++++++++
 target/s390x/kvm_s390x.h |    2 ++
 2 files changed, 18 insertions(+), 0 deletions(-)

diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c
index 33e5ec3..2812e28 100644
--- a/target/s390x/kvm.c
+++ b/target/s390x/kvm.c
@@ -277,6 +277,22 @@ static void kvm_s390_init_dea_kw(void)
     }
 }
 
+int kvm_s390_set_interpret_ap(uint8_t enable)
+{
+    struct kvm_device_attr attribute = {
+        .group = KVM_S390_VM_CRYPTO,
+        .attr  = KVM_S390_VM_CRYPTO_INTERPRET_AP,
+        .addr = 1,
+    };
+
+    if (!kvm_vm_check_attr(kvm_state, KVM_S390_VM_CRYPTO,
+                           KVM_S390_VM_CRYPTO_INTERPRET_AP)) {
+        return -EOPNOTSUPP;
+    }
+
+    return kvm_vm_ioctl(kvm_state, KVM_SET_DEVICE_ATTR, &attribute);
+}
+
 void kvm_s390_crypto_reset(void)
 {
     if (s390_has_feat(S390_FEAT_MSA_EXT_3)) {
diff --git a/target/s390x/kvm_s390x.h b/target/s390x/kvm_s390x.h
index 34ee7e7..0d6c6e7 100644
--- a/target/s390x/kvm_s390x.h
+++ b/target/s390x/kvm_s390x.h
@@ -40,4 +40,6 @@ void kvm_s390_crypto_reset(void);
 void kvm_s390_restart_interrupt(S390CPU *cpu);
 void kvm_s390_stop_interrupt(S390CPU *cpu);
 
+int kvm_s390_set_interpret_ap(uint8_t enable);
+
 #endif /* KVM_S390X_H */
-- 
1.7.1

[Qemu-devel] [PATCH v3 5/7] s390x/vfio: ap: Introduce VFIO AP device

[Tony Krowiak]

Introduces a VFIO based AP device. The device is defined via
the QEMU command line by specifying:

    -device vfio-ap,sysfsdev=<path-to-mediated-matrix-device>

The mediated matrix device is created by the VFIO AP device
driver by writing a UUID to a sysfs attribute file (see
docs/vfio-ap.txt). The mediated matrix device will be named
after the UUID. Symbolic links to the $uuid are created in
many places, so the path to the mediated matrix device $uuid
can be specified in any of the following ways:

/sys/devices/vfio_ap/matrix/$uuid
/sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/devices/$uuid
/sys/bus/mdev/devices/$uuid
/sys/bus/mdev/drivers/vfio_mdev/$uuid

When the vfio-ap device is realized, it acquires and opens the
VFIO iommu group to which the mediated matrix device is
bound. This causes a VFIO group notification event to be
signaled. The vfio_ap device driver's group notification
handler will get called at which time the device driver
will configure the the AP devices to which the guest will
be granted access.

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 default-configs/s390x-softmmu.mak |    1 +
 hw/vfio/Makefile.objs             |    1 +
 hw/vfio/ap.c                      |  184 +++++++++++++++++++++++++++++++++++++
 include/hw/vfio/vfio-common.h     |    1 +
 4 files changed, 187 insertions(+), 0 deletions(-)
 create mode 100644 hw/vfio/ap.c

diff --git a/default-configs/s390x-softmmu.mak b/default-configs/s390x-softmmu.mak
index 2f4bfe7..0b784b6 100644
--- a/default-configs/s390x-softmmu.mak
+++ b/default-configs/s390x-softmmu.mak
@@ -9,3 +9,4 @@ CONFIG_S390_FLIC=y
 CONFIG_S390_FLIC_KVM=$(CONFIG_KVM)
 CONFIG_VFIO_CCW=$(CONFIG_LINUX)
 CONFIG_WDT_DIAG288=y
+CONFIG_VFIO_AP=$(CONFIG_LINUX)
diff --git a/hw/vfio/Makefile.objs b/hw/vfio/Makefile.objs
index c3ab909..7300860 100644
--- a/hw/vfio/Makefile.objs
+++ b/hw/vfio/Makefile.objs
@@ -6,4 +6,5 @@ obj-$(CONFIG_SOFTMMU) += platform.o
 obj-$(CONFIG_VFIO_XGMAC) += calxeda-xgmac.o
 obj-$(CONFIG_VFIO_AMD_XGBE) += amd-xgbe.o
 obj-$(CONFIG_SOFTMMU) += spapr.o
+obj-$(CONFIG_VFIO_AP) += ap.o
 endif
diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c
new file mode 100644
index 0000000..b397bb1
--- /dev/null
+++ b/hw/vfio/ap.c
@@ -0,0 +1,184 @@
+/*
+ * VFIO based AP matrix device assignment
+ *
+ * Copyright 2018 IBM Corp.
+ * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or (at
+ * your option) any later version. See the COPYING file in the top-level
+ * directory.
+ */
+
+#include <linux/vfio.h>
+#include <sys/ioctl.h>
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "hw/sysbus.h"
+#include "hw/vfio/vfio.h"
+#include "hw/vfio/vfio-common.h"
+#include "hw/s390x/ap-device.h"
+#include "qemu/error-report.h"
+#include "qemu/queue.h"
+#include "cpu.h"
+#include "kvm_s390x.h"
+
+#define VFIO_AP_DEVICE_TYPE      "vfio-ap"
+
+typedef struct VFIOAPDevice {
+    APDevice apdev;
+    VFIODevice vdev;
+    QTAILQ_ENTRY(VFIOAPDevice) sibling;
+} VFIOAPDevice;
+
+static void vfio_ap_compute_needs_reset(VFIODevice *vdev)
+{
+    vdev->needs_reset = false;
+}
+
+/*
+ * We don't need vfio_hot_reset_multi and vfio_eoi operations for
+ * vfio-ap-matrix device now.
+ */
+struct VFIODeviceOps vfio_ap_ops = {
+    .vfio_compute_needs_reset = vfio_ap_compute_needs_reset,
+};
+
+static QTAILQ_HEAD(, VFIOAPDevice) vfio_ap_devices =
+        QTAILQ_HEAD_INITIALIZER(vfio_ap_devices);
+
+static void vfio_put_device(VFIOAPDevice *apdev)
+{
+    g_free(apdev->vdev.name);
+    vfio_put_base_device(&apdev->vdev);
+}
+
+static VFIOGroup *vfio_ap_get_group(VFIOAPDevice *vapdev, Error **errp)
+{
+    char *tmp, group_path[PATH_MAX];
+    ssize_t len;
+    int groupid;
+
+    tmp = g_strdup_printf("%s/iommu_group", vapdev->vdev.sysfsdev);
+    len = readlink(tmp, group_path, sizeof(group_path));
+    g_free(tmp);
+
+    if (len <= 0 || len >= sizeof(group_path)) {
+        error_setg(errp, "%s: no iommu_group found for %s",
+                   VFIO_AP_DEVICE_TYPE, vapdev->vdev.sysfsdev);
+        return NULL;
+    }
+
+    group_path[len] = 0;
+
+    if (sscanf(basename(group_path), "%d", &groupid) != 1) {
+        error_setg(errp, "vfio: failed to read %s", group_path);
+        return NULL;
+    }
+
+    return vfio_get_group(groupid, &address_space_memory, errp);
+}
+
+static void vfio_ap_realize(DeviceState *dev, Error **errp)
+{
+    VFIODevice *vbasedev;
+    VFIOGroup *vfio_group;
+    APDevice *apdev = DO_UPCAST(APDevice, parent_obj, dev);
+    VFIOAPDevice *vapdev = DO_UPCAST(VFIOAPDevice, apdev, apdev);
+    char *mdevid;
+    Error *local_err = NULL;
+    int ret;
+
+    if (!s390_has_feat(S390_FEAT_AP)) {
+        error_setg(&local_err, "AP support not enabled");
+        goto out_err;
+    }
+
+    ret = kvm_s390_set_interpret_ap(1);
+    if (ret) {
+        error_setg_errno(&local_err, errno,
+                         "error setting interpretive execution of AP instructions");
+        goto out_err;
+    }
+
+    vfio_group = vfio_ap_get_group(vapdev, &local_err);
+    if (!vfio_group) {
+        goto out_group_err;
+    }
+
+    vapdev->vdev.ops = &vfio_ap_ops;
+    vapdev->vdev.type = VFIO_DEVICE_TYPE_AP;
+    mdevid = basename(vapdev->vdev.sysfsdev);
+    vapdev->vdev.name = g_strdup_printf("%s", mdevid);
+    vapdev->vdev.dev = dev;
+    QLIST_FOREACH(vbasedev, &vfio_group->device_list, next) {
+        if (strcmp(vbasedev->name, vapdev->vdev.name) == 0) {
+            error_setg(&local_err,
+                       "%s: AP device %s has already been realized",
+                       VFIO_AP_DEVICE_TYPE, vapdev->vdev.name);
+            goto out_device_err;
+        }
+    }
+
+    ret = vfio_get_device(vfio_group, mdevid, &vapdev->vdev, &local_err);
+    if (ret) {
+        goto out_device_err;
+    }
+
+    QTAILQ_INSERT_TAIL(&vfio_ap_devices, vapdev, sibling);
+
+    return;
+
+
+out_device_err:
+    vfio_put_group(vfio_group);
+out_group_err:
+    kvm_s390_set_interpret_ap(0);
+out_err:
+    error_propagate(errp, local_err);
+}
+
+static void vfio_ap_unrealize(DeviceState *dev, Error **errp)
+{
+    APDevice *apdev = DO_UPCAST(APDevice, parent_obj, dev);
+    VFIOAPDevice *vapdev = DO_UPCAST(VFIOAPDevice, apdev, apdev);
+    VFIOGroup *group = vapdev->vdev.group;
+
+    vfio_put_device(vapdev);
+    vfio_put_group(group);
+    kvm_s390_set_interpret_ap(0);
+}
+
+static Property vfio_ap_properties[] = {
+    DEFINE_PROP_STRING("sysfsdev", VFIOAPDevice, vdev.sysfsdev),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static const VMStateDescription vfio_ap_vmstate = {
+    .name = VFIO_AP_DEVICE_TYPE,
+    .unmigratable = 1,
+};
+
+static void vfio_ap_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+
+    dc->props = vfio_ap_properties;
+    dc->vmsd = &vfio_ap_vmstate;
+    dc->desc = "VFIO-based AP device assignment";
+    dc->realize = vfio_ap_realize;
+    dc->unrealize = vfio_ap_unrealize;
+}
+
+static const TypeInfo vfio_ap_info = {
+    .name = VFIO_AP_DEVICE_TYPE,
+    .parent = AP_DEVICE_TYPE,
+    .instance_size = sizeof(VFIOAPDevice),
+    .class_init = vfio_ap_class_init,
+};
+
+static void register_vfio_ap_type(void)
+{
+    type_register_static(&vfio_ap_info);
+}
+
+type_init(register_vfio_ap_type)
diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h
index f3a2ac9..f1f22d9 100644
--- a/include/hw/vfio/vfio-common.h
+++ b/include/hw/vfio/vfio-common.h
@@ -46,6 +46,7 @@ enum {
     VFIO_DEVICE_TYPE_PCI = 0,
     VFIO_DEVICE_TYPE_PLATFORM = 1,
     VFIO_DEVICE_TYPE_CCW = 2,
+    VFIO_DEVICE_TYPE_AP = 3,
 };
 
 typedef struct VFIOMmap {
-- 
1.7.1

[Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Tony Krowiak]

If the CPU model indicates that AP facility is installed on
the guest (i.e., -cpu xxxx,ap=on), then the expectation is that
the AP bus running in the guest will initialize; however, if the
AP instructions are not being interpreted by the firmware, then
they will be intercepted and routed back to QEMU for handling.
If a handler is not defined to process the intercepted instruciton,
then an operation exception will be injected into the
guest, in which case the AP bus will not initialize.

There are two situations where AP instructions will not be
interpreted:

1. The guest is not configured with a vfio-ap device (i.e.,
   -device vfio-ap,sysfsdev=$path-to-mdev). The realize function for
   the vfio-ap device enables interpretive execution of AP
   instructions.

2. The guest is a second level guest but the first level guest has
   not enabled interpretive execution.

This patch introduces AP instruction handlers to ensure the AP bus
module initializes on the guest when the AP facility is installed
on the guest but AP instructions are not being interpreted. The logic
incorporated is:

* If the CPU model indicates AP instructions are
  installed

  * Set the status response code for the instruction to indicate that
    the APQN contained in the instruction is not valid. This is
    a valid response because there will be no devices configured for
    the guest in any of the above scenarios.

* Else return an error from the handler. This will result in an
  operation being injected into the guest and the AP bus will not
  initialize on the guest. That is commensurate with how things work
  today.

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 hw/vfio/ap.c                 |   45 ++++++++++++++++++++++++++++++++++++++++++
 include/hw/s390x/ap-device.h |    6 +++++
 target/s390x/kvm.c           |   14 +++++++++++++
 3 files changed, 65 insertions(+), 0 deletions(-)

diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c
index b397bb1..88e744d 100644
--- a/hw/vfio/ap.c
+++ b/hw/vfio/ap.c
@@ -148,6 +148,51 @@ static void vfio_ap_unrealize(DeviceState *dev, Error **errp)
     kvm_s390_set_interpret_ap(0);
 }
 
+int ap_device_handle_nqap(S390CPU *cpu)
+{
+    CPUS390XState *env = &cpu->env;
+
+    if (s390_has_feat(S390_FEAT_AP)) {
+        env->regs[1] = 0x10000;
+
+        return 0;
+    }
+
+    return -EOPNOTSUPP;
+}
+
+int ap_device_handle_dqap(S390CPU *cpu)
+{
+    CPUS390XState *env = &cpu->env;
+
+    if (s390_has_feat(S390_FEAT_AP)) {
+        env->regs[1] = 0x10000;
+
+        return 0;
+    }
+
+    return -EOPNOTSUPP;
+}
+
+int ap_device_handle_pqap(S390CPU *cpu)
+{
+    CPUS390XState *env = &cpu->env;
+    int fc = 4 & (env->regs[0] >> 24);
+
+    /*
+     * The Query Configuration Information (QCI) function (fc == 4) does not
+     * set a response code in reg 1, so check for that along with the
+     * AP feature.
+     */
+    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
+        env->regs[1] = 0x10000;
+
+        return 0;
+    }
+
+    return -EOPNOTSUPP;
+}
+
 static Property vfio_ap_properties[] = {
     DEFINE_PROP_STRING("sysfsdev", VFIOAPDevice, vdev.sysfsdev),
     DEFINE_PROP_END_OF_LIST(),
diff --git a/include/hw/s390x/ap-device.h b/include/hw/s390x/ap-device.h
index 693df90..d45ae38 100644
--- a/include/hw/s390x/ap-device.h
+++ b/include/hw/s390x/ap-device.h
@@ -11,6 +11,8 @@
 #ifndef HW_S390X_AP_DEVICE_H
 #define HW_S390X_AP_DEVICE_H
 
+#include "cpu.h"
+
 #define AP_DEVICE_TYPE       "ap-device"
 
 typedef struct APDevice {
@@ -35,4 +37,8 @@ static inline APDevice *to_ap_dev(DeviceState *dev)
 #define AP_DEVICE_CLASS(klass) \
     OBJECT_CLASS_CHECK(APDeviceClass, (klass), AP_DEVICE_TYPE)
 
+int ap_device_handle_nqap(S390CPU *cpu);
+int ap_device_handle_dqap(S390CPU *cpu);
+int ap_device_handle_pqap(S390CPU *cpu);
+
 #endif /* HW_S390X_AP_DEVICE_H */
diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c
index 2812e28..a636394 100644
--- a/target/s390x/kvm.c
+++ b/target/s390x/kvm.c
@@ -50,6 +50,7 @@
 #include "exec/memattrs.h"
 #include "hw/s390x/s390-virtio-ccw.h"
 #include "hw/s390x/s390-virtio-hcall.h"
+#include "hw/s390x/ap-device.h"
 
 #ifndef DEBUG_KVM
 #define DEBUG_KVM  0
@@ -88,6 +89,9 @@
 #define PRIV_B2_CHSC                    0x5f
 #define PRIV_B2_SIGA                    0x74
 #define PRIV_B2_XSCH                    0x76
+#define PRIV_B2_NQAP                    0xad
+#define PRIV_B2_DQAP                    0xae
+#define PRIV_B2_PQAP                    0xaf
 
 #define PRIV_EB_SQBS                    0x8a
 #define PRIV_EB_PCISTB                  0xd0
@@ -1245,6 +1249,16 @@ static int handle_b2(S390CPU *cpu, struct kvm_run *run, uint8_t ipa1)
     case PRIV_B2_SCLP_CALL:
         rc = kvm_sclp_service_call(cpu, run, ipbh0);
         break;
+    case PRIV_B2_NQAP:
+        rc = ap_device_handle_nqap(cpu);
+        break;
+    case PRIV_B2_DQAP:
+        rc = ap_device_handle_dqap(cpu);
+        break;
+    case PRIV_B2_PQAP:
+        rc = ap_device_handle_pqap(cpu);
+        break;
+        break;
     default:
         rc = -1;
         DPRINTF("KVM: unhandled PRIV: 0xb2%x\n", ipa1);
-- 
1.7.1

[Qemu-devel] [PATCH v3 7/7] s390: doc: detailed specifications for AP virtualization

[Tony Krowiak]

This patch provides documentation describing the AP architecture and
design concepts behind the virtualization of AP devices. It also
includes an example of how to configure AP devices for exclusive
use of KVM guests.

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 docs/vfio-ap.txt |  624 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 624 insertions(+), 0 deletions(-)
 create mode 100644 docs/vfio-ap.txt

diff --git a/docs/vfio-ap.txt b/docs/vfio-ap.txt
new file mode 100644
index 0000000..54e7523
--- /dev/null
+++ b/docs/vfio-ap.txt
@@ -0,0 +1,624 @@
+Adjunct Processor (AP) Device
+=============================
+
+Contents:
+=========
+* Introduction
+* AP Architectural Overview
+* Start Interpretive Execution (SIE) Instruction
+* AP Matrix Configuration on Linux Host
+* AP Matrix Configuration for a Linux Guest
+* Starting a Linux Guest Configured with an AP Matrix
+* Example: Configure AP Matrices for Two Linux Guests
+
+Introduction:
+============
+The IBM Adjunct Processor (AP) Cryptographic Facility is comprised
+of three AP instructions and from 1 to 256 PCIe cryptographic adapter cards.
+These AP devices provide cryptographic functions to all CPUs assigned to a
+linux system running in an IBM Z system LPAR.
+
+On s390x, AP adapter cards are exposed via the AP bus. This document
+describes how those cards may be made available to KVM guests using the
+VFIO mediated device framework.
+
+AP Architectural Overview:
+=========================
+In order understand the terminology used in the rest of this document, let's
+start with some definitions:
+
+* AP adapter
+
+  An AP adapter is an IBM Z adapter card that can perform cryptographic
+  functions. There can be from 0 to 256 adapters assigned to an LPAR. Adapters
+  assigned to the LPAR in which a linux host is running will be available to
+  the linux host. Each adapter is identified by a number from 0 to 255. When
+  installed, an AP adapter is accessed by AP instructions executed by any CPU.
+
+* AP domain
+
+  An adapter is partitioned into domains. Each domain can be thought of as
+  a set of hardware registers for processing AP instructions. An adapter can
+  hold up to 256 domains. Each domain is identified by a number from 0 to 255.
+  Domains can be further classified into two types:
+
+    * Usage domains are domains that can be accessed directly to process AP
+      commands
+
+    * Control domains are domains that are accessed indirectly by AP
+      commands sent to a usage domain to control or change the domain, for
+      example; to set a secure private key for the domain.
+
+* AP Queue
+
+  An AP queue is the means by which an AP command-request message is sent to an
+  AP usage domain inside a specific AP. An AP queue is identified by a tuple
+  comprised of an AP adapter ID (APID) and an AP queue index (APQI). The
+  APQI corresponds to a given usage domain number within the adapter. This tuple
+  forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP
+  instructions include a field containing the APQN to identify the AP queue to
+  which the AP command-request message is to be sent for processing.
+
+* AP Instructions:
+
+  There are three AP instructions:
+
+  * NQAP: to enqueue an AP command-request message to a queue
+  * DQAP: to dequeue an AP command-reply message from a queue
+  * PQAP: to administer the queues
+
+Start Interpretive Execution (SIE) Instruction
+==============================================
+A KVM guest is started by executing the Start Interpretive Execution (SIE)
+instruction. The SIE state description is a control block that contains the
+state information for a KVM guest and is supplied as input to the SIE
+instruction. The SIE state description contains a field that references
+a Crypto Control Block (CRYCB). The CRYCB contains three fields to identify the
+adapters, usage domains and control domains assigned to the KVM guest:
+
+* The AP Mask (APM) field is a bit mask that identifies the AP adapters assigned
+  to the KVM guest. Each bit in the mask, from most significant to least
+  significant bit, corresponds to an APID from 0-255. If a bit is set, the
+  corresponding adapter is valid for use by the KVM guest.
+
+* The AP Queue Mask (AQM) field is a bit mask identifying the AP queues assigned
+  to the KVM guest. Each bit in the mask, from most significant to least
+  significant bit, corresponds to an AP queue index (APQI) from 0-255. If a bit
+  is set, the corresponding queue is valid for use by the KVM guest.
+
+* The AP Domain Mask field is a bit mask that identifies the AP control domains
+  assigned to the KVM guest. The ADM bit mask controls which domains can be
+  changed by an AP command-request message sent to a usage domain from the
+  guest. Each bit in the mask, from least significant to most significant bit,
+  corresponds to a domain from 0-255. If a bit is set, the corresponding domain
+  can be modified by an AP command-request message sent to a usage domain
+  configured for the KVM guest.
+
+If you recall from the description of an AP Queue, AP instructions include
+an APQN to identify the AP adapter and AP queue to which an AP command-request
+message is to be sent (NQAP and PQAP instructions), or from which a
+command-reply message is to be received (DQAP instruction). The validity of an
+APQN is defined by the matrix calculated from the APM and AQM; it is the
+intersection of all assigned adapter numbers (APM) with all assigned queue
+indexes (AQM). For example, if adapters 1 and 2 and usage domains 5 and 6 are
+assigned to a guest, the APQNs (1,5), (1,6), (2,5) and (2,6) will be valid for
+the guest.
+
+The APQNs provide secure key functionality - i.e., a private key is stored on
+the adapter card for each of its domains - so each APQN must be assigned to at
+most one guest or the linux host.
+
+   Example 1: Valid configuration:
+   ------------------------------
+   Guest1: adapters 1,2  domains 5,6
+   Guest2: adapter  1,2  domain 7
+
+   This is valid because both guests have a unique set of APQNs: Guest1 has
+   APQNs (1,5), (1,6), (2,5) and (2,6); Guest2 has APQNs (1,7) and (2,7).
+
+   Example 2: Invalid configuration:
+   --------------------------------
+   Guest1: adapters 1,2  domains 5,6
+   Guest2: adapter  1    domains 6,7
+
+   This is an invalid configuration because both guests have access to
+   APQN (1,6).
+
+AP device Configuration on Linux Host:
+=====================================
+A linux system is a guest of the LPAR in which it is running and has access to
+the AP resources configured for the LPAR. The LPAR's AP matrix is
+configured using the 'Customize/Delete Activation Profiles' dialog from the HMC.
+This dialog displays the activation profiles configured for the linux system.
+Selecting the specific activation profile to be edited and clicking the
+'Customize Profile' button will open the 'Customize Image Profiles' dialog.
+Selecting the 'Crypto' link in the tree view on the left hand side of the dialog
+will display the AP matrix configuration in the right hand panel. There, one can
+assign AP adapters - called Cryptos - and domains to the LPAR. When the linux
+system is started using this activation profile, it will have access to the
+matrix of AP adapters and domains configured via the activation profile.
+
+When the linux system is started, the AP adapter devices will be connected to
+the AP bus and the following AP matrix interfaces will be created in sysfs:
+
+/sys/bus/ap
+... [devices]
+...... xx.yyyy
+...... ...
+...... cardxx
+...... ...
+
+Where:
+    cardxx     is adapter number xx (in hex)
+    yyyy       is a usage domain number yyyy (in hex)
+....xx.yyyy    is APQN (xx,yyyy)
+
+For example, if AP adapters 5 and 6 and domains 4 and 71 (0x47) are configured
+for the LPAR, the sysfs representation on the linux system would look like this:
+
+/sys/bus/ap
+... [devices]
+...... 05.0004
+...... 05.0047
+...... 06.0004
+...... 06.0047
+...... card05
+...... card06
+
+There will also be AP device drivers created to control each type of AP matrix
+interface available to the IBM Z system:
+
+/sys/bus/ap
+... [drivers]
+...... [cex2acard]        for Crypto Express 2/3 accelerator cards
+...... [cex2aqueue]       for AP queues served by Crypto Express 2/3
+                          accelerator cards
+...... [cex4card]         for Crypto Express 4/5/6 accelerator and coprocessor
+                          cards
+...... [cex4queue]        for AP queues served by Crypto Express 4/5/6
+                          accelerator and coprocessor cards
+...... [pcixcccard]       for Crypto Express 2/3 coprocessor cards
+...... [pcixccqueue]      for AP queues served by Crypto Express 2/3
+                          coprocessor cards
+
+Links to the AP interfaces controlled by each AP device driver will be created
+in the device driver's sysfs directory. For example, if AP adapter 5 and domains
+4 and 71 (0x47) are assigned to the LPAR and adapter 5 is a CEX5 card, the
+following links will be created in the CEX5 drivers' sysfs directories:
+
+/sys/bus/ap
+... [drivers]
+...... [cex4card]
+......... [card05]
+...... [cex4queue]
+......... [05.0004]
+......... [05.0047]
+
+AP Matrix Configuration for a Linux Guest:
+=========================================
+In order to configure the AP matrix for a guest, the adapters, usage domains
+and control domains to be used by the guest must be assigned to the guest. This
+section describes how to configure a guest's AP matrix.
+
+The kernel interfaces for configuring an AP matrix for a linux guest are built
+on the VFIO mediated device framework and are provided by the vfio_ap
+kernel module. By default, the vfio_ap module is a loadable module, The
+dependency chain for the vfio_ap module is:
+* vfio
+* mdev
+* vfio_mdev
+* vfio_ap
+
+When installed, the vfio_ap module is initialized. During module initialization,
+a vfio_ap driver is created and registered with the AP bus creating the
+following sysfs interfaces:
+
+ /sys/bus/ap/drivers/
+...[vfio_ap]
+...... bind
+...... unbind
+
+The vfio_ap device driver will create a 'matrix' device to hold the APQNs
+reserved for exclusive use by KVM guests:
+
+/sys/devices/
+... [vfio_ap]
+......[matrix] symlink to the matrix device directory
+
+The vfio_ap device driver serves several purposes:
+1. Provides an interface for securing APQNs preventing their use by the host
+   linux system and reserving their use by one or more guests.
+2. Creates the sysfs interfaces for configuring an AP matrix for a linux guest.
+
+Securing APQNs
+--------------
+   An APQN is reserved by unbinding an AP queue device AP bus device driver and
+   binding it to the vfio_ap device driver. For example, suppose we want to
+   secure APQN (05,0004). Assuming that the AP adapter card 5 is a CEX5
+   coprocessor card:
+
+   echo 05.0004 > /sys/bus/ap/drivers/cex4queue/unbind
+   echo 05.0004 > /sys/bus/ap/drivers/vfio_ap/bind
+
+   This action will store the APQN in the /sys/devices/vfio_ap/matrix device
+   which makes it available for use by a linux guest.
+
+Configuring an AP matrix for a linux guest.
+------------------------------------------
+These sysfs interfaces are built on the VFIO mediated device framework. To
+configure an AP matrix for a guest, a mediated matrix device must first be
+created for the /sys/devices/vfio_ap/matrix device. The sysfs interfaceAPQI corresponding to
+for creating a mediated matrix device is in:
+
+/sys/devices
+... [vfio_ap]
+......[matrix]
+......... [mdev_supported_types]
+............ [vfio_ap-passthrough]
+............... create
+............... [devices]
+
+A mediated AP matrix device is created by writing a UUID to the attribute
+file named 'create', for example:
+
+   uuidgen > create
+
+When a mediated AP matrix device is created, a sysfs directory named after
+the UUID:
+
+/sys/devices
+... [vfio_ap]
+......[matrix]
+......... [mdev_supported_types]
+............ [vfio_ap-passthrough]
+............... create
+............... [devices]
+.................. [$uuid]
+
+There will also be three sets of attribute files created in the mediated
+matrix device's sysfs directory to configure an AP matrix for the
+KVM guest:
+
+/sys/devices
+... [vfio_ap]
+......[matrix]
+......... [mdev_supported_types]
+............ [vfio_ap-passthrough]
+............... create
+............... [devices]
+.................. [$uuid]
+..................... assign_adapter
+..................... assign_control_domain
+..................... assign_domain
+..................... matrix
+..................... unassign_adapter
+..................... unassign_control_domain
+..................... unassign_domain
+
+assign_adapter
+   To assign an AP adapter to the mediated matrix device, its APID is written
+   'assign_adapter' file. This may be done multiple times to assign more than
+   one adapter. The APID may be specified using conventional semantics
+   as a decimal, hexidecimal, or octal number. For example, to assign adapters
+   4, 5 and 16 to mediated matrix device $uuid in decimal, hexidecimal and octal
+   respectively:
+
+       echo 4 > assign_adapter
+       echo 0x5 > assign_adapter
+       echo 020
+
+unassign_adapter
+   To unassign an AP adapter, its APID is written to the 'unassign_adapter'
+   file. This may also be done multiple times to unassign more than one adapter.
+
+assign_domain
+   To assign a usage domain, the APQI corresponding to the domain number is
+   written into the 'assign_domain' file. This may be done multiple times to
+   assign more than one usage domain. The APQI may be specified using
+   conventional semantics as a decimal, hexidecimal, or octal number. For
+   example, to assign usage domains 4, 8, and 71 to mediated matrix device
+   $uuid in decimal, hexidecimal and octal respectively:
+
+      echo 4 > assign_domain
+      echo 0x8 > assign_domain
+      echo 0107 > assign_domain
+
+unassign_domain
+   To unassign a usage domain, the APQI corresponding to the domain number is
+   written into the 'unassign_domain' file. This may be done multiple times to
+   unassign more than one usage domain.
+
+assign_control_domain
+   To assign a control domain, the domain number is written into the
+   'assign_control_domain' file. This may be done multiple times to
+   assign more than one control domain. The domain number may be specified using
+   conventional semantics as a decimal, hexidecimal, or octal number. For
+   example, to assign  control domains 4, 8, and 71 to mediated matrix device
+   $uuid in decimal, hexidecimal and octal respectively:
+
+      echo 4 > assign_domain
+      echo 0x8 > assign_domain
+      echo 0107 > assign_domain
+
+unassign_control_domain
+   To unassign a control domain, the domain number is written into the
+   'unassign_domain' file. This may be done multiple times to unassign more than
+   one control domain.
+
+Notes:
+* Hot plug/unplug is not currently supported for mediated AP matrix devices,
+  so the AP matrix resulting from assignment and/or unassignment of AP
+  adapters, usage domains and control domains to a mediated AP matrix device
+  while the guest is running will not take affect until the linux guest is
+  rebooted.
+* By architectural convention, all usage domains configured for a KVM guest
+  will also be implicitly assigned as control domains also, to there is no
+  need to assign control domains that are assigned as usage domains.
+
+Starting a Linux Guest Configured with an AP Matrix:
+===================================================
+In addition to providing the sysfs interfaces for configuring the AP matrix for
+a linux guest, a mediated matrix device also acts as a communication pathway
+between QEMU and the vfio_ap device driver. To gain access to the
+device driver, the following option must be specified on the QEMU command line:
+
+   -device vfio_ap,sysfsdev=$path-to-mdev
+
+The sysfsdev parameter specifies the path to the mediated matrix device.
+There are a number of ways to specify this path:
+
+/sys/devices/vfio_ap/matrix/$uuid
+/sys/bus/mdev/devices/$uuid
+/sys/bus/mdev/drivers/vfio_mdev/$uuid
+/sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/devices/$uuid
+
+When the linux guest is subsequently started, the guest will open the mediated
+matrix device's file descriptor to get information about the mediated matrix
+device. The vfio_ap device driver will update the APM, AQM, and ADM fields in the
+guest's CRYCB with the adapter, usage domain and control domains assigned to
+via the mediated matrix device's sysfs attribute files. Programs running on the
+linux guest will then:
+
+1. Have direct access to the APQNs derived from the intersection of the AP
+   adapter and usage domain numbers specified in the APM and AQM respectively
+
+2. Have authorization to process AP commands to change - e.g., store a new
+   secure key - a control domain identified in an AP instruction sent to a valid
+   APQN.
+
+CPU model features:
+
+Three CPU model features are available for controlling guest access to AP
+facilities:
+
+1. AP facilities feature
+
+   The AP facilities feature indicates that AP facilities are installed on the
+   guest. This feature will be enabled by the kernel only if the AP facilities
+   are installed on the host system. It will turned on automatically for guests
+   started with CPU model zEC12 or newer. The feature is s390-specific and is
+   represented as a parameter of the -cpu option on the QEMU command line:
+
+      qemu-system-s390x -cpu $model,ap=on|off
+
+      Where:
+
+         $model is the CPU model defined for the guest (defaults to the model of
+                the host system if not specified).
+
+         ap=on|off indicates whether AP facilities are installed (on) or not
+                   installed (off). The default for CPU models zEC12 or newer
+                   is ap=on. AP facilities must be installed when this parameter
+                   is used in conjunction with -device vfio-ap,sysfsdev=$path or
+                   the guest will not start.
+
+2. Query Configuration Information (QCI) facility
+
+   The QCI facility is used by the AP bus running on the guest to query the
+   configuration of the AP facilities. This facility will be enabled by
+   the kernel only if the QCI facility is installed on the host system. It will
+   be turned on automatically for guests started with CPU model zEC12 or newer.
+   The feature is s390-specific and is represented as a parameter of the -cpu
+   option on the QEMU command line:
+
+      qemu-system-s390x -cpu $model,qci=on|off
+
+      Where:
+
+         $model is the CPU model defined for the guest
+
+         qci=on|off indicates whether the QCI facility is installed (on) or not
+                    installed (off). The default for CPU models zEC12 or newer
+                    is qci=on. Turning the QCI facility on makes no sense if it
+                    is not used in conjunction with the
+                    '-device vfio-ap,sysfsdev=$path' option. A warning will be
+                    presented if QCI is turned on and the AP facilities are not
+                    installed.
+
+                    If the QCI facility is turned off, APQNs with an APQI
+                    greater than 15 will not be accessible from the guest.
+
+3. Adjunct Process Facility Test (APFT) facility
+
+   The APFT facility is used by the AP bus running on the guest to test the
+   AP facilities available for a given AP queue. This facility will be enabled
+   by the kernel only if the APFT facility is installed on the host system. It
+   will be turned on automatically for guests started with CPU model zEC12 or
+   newer. The feature is s390-specific and is represented as a parameter of the
+   -cpu option on the QEMU command line:
+
+      qemu-system-s390x -cpu $model,apft=on|off
+
+      Where:
+
+         $model is the CPU model defined for the guest (defaults to the model of
+                the host system if not specified).
+
+         apft=on|off indicates whether the APFT facility is installed (on) or
+                     not installed (off). The default for CPU models zEC12 and
+                     newer is apft=on. Turning the APFT facility on makes no
+                     sense if it is not used in conjunction with the
+                     -device vfio-ap,sysfsdev=$path option. A warning will be
+                     presented if APFT is turned on and the AP facilities are
+                     not installed.
+
+                     It also makes no sense to turn APFT off when used in
+                     conjunction with the vfio-ap device because the APFT
+                     facility is required; the AP bus running on the guest will
+                     not detect CEX4 and newer devices without it. Since only
+                     CEX4 and newer devices are supported for guest usage, no AP
+                     devices can be made accessible to a guest started without
+                     APFT installed.
+
+Example: Configure AP Matrixes for Two Linux Guests:
+===================================================
+Let's now provide an example to illustrate how KVM guests may be given
+access to AP facilities. For this example, we will show how to configure
+two guests such that executing the lszcrypt command on the guests would
+look like this:
+
+Guest1
+------
+CARD.DOMAIN TYPE  MODE       
+------------------------------
+05          CEX5C CCA-Coproc 
+05.0004     CEX5C CCA-Coproc
+05.00ab     CEX5C CCA-Coproc 
+06          CEX5A Accelerator
+06.0004     CEX5A Accelerator
+06.00ab     CEX5C CCA-Coproc 
+
+Guest2
+------
+CARD.DOMAIN TYPE  MODE       
+------------------------------
+05          CEX5A Accelerator
+05.0047     CEX5A Accelerator
+05.00ff     CEX5A Accelerator
+
+These are the steps for configuring the Guest1 and Guest2:
+
+1. The first thing that needs to be done is to secure the AP queues to be
+   used by the two guests so that the host can not access them. This is done
+   by unbinding each AP Queue device from its respective AP driver. In our
+   example, these queues are bound to the cex4queue driver. This would be
+   the sysfs location of these devices:
+
+   /sys/bus/ap
+   --- [drivers]
+   ------ [cex4queue]
+   --------- [05.0004]
+   --------- [05.0047]   --------------------- control_domains
+   --------------------- domains
+   --------- [05.00ab]
+   --------- [05.00ff]
+   --------- [06.0004]
+   --------- [06.00ab]
+   --------- unbind
+
+   To unbind AP queue 05.0004 from the cex4queue device driver:
+
+    echo 05.0004 > unbind
+
+   This must also be done for AP queues 05.00ab, 05.0047, 05.00ff, 06.0004,
+   and 06.00ab.
+
+2. The next step is to reserve the queues for use by the two KVM guests.
+   This is accomplished by binding them to the VFIO AP device driver.
+   This is the sysfs location of the VFIO AP device driver:
+
+   /sys/bus/ap
+   ---[drivers]
+   ------ [vfio_ap]
+   ---------- bind
+
+   To bind queue 05.0004 to the vfio_ap driver:
+
+    echo 05.0004 > bind
+
+   This must also be done for AP queues 05.00ab, 05.0047, 05.00ff, 06.0004,
+   and 06.00ab.
+
+3. Create the mediated devices needed to configure the AP matrixes for the
+   two guests and to provide an interface to the vfio_ap driver for
+   use by the guests:
+
+   /sys/devices/
+   --- [vfio_ap]
+   ------ [matrix] (this is the matrix device)
+   --------- [mdev_supported_types]
+   ------------ [vfio_ap-passthrough] (passthrough mediated matrix device type)
+   --------------- create
+   --------------- [devices]
+
+   To create the mediated devices for the two guests:
+
+    uuidgen > create
+    uuidgen > create
+
+   This will create two mediated devices in the [devices] subdirectory named
+   with the UUID written to the create attribute file. We call them $uuid1
+   and $uuid2:
+
+   /sys/devices/
+   --- [vfio_ap]
+   ------ [matrix]
+   --------- [mdev_supported_types]
+   ------------ [vfio_ap-passthrough]
+   --------------- [devices]
+   ------------------ [$uuid1]
+   --------------------- assign_adapter
+   --------------------- assign_control_domain
+   --------------------- assign_domain
+   --------------------- matrix
+   --------------------- unassign_adapter
+   --------------------- unassign_control_domain
+   --------------------- unassign_domain
+
+   ------------------ [$uuid2]
+   --------------------- assign_adapter
+   --------------------- assign_cTo assign an adapter, the APID of the adapter is written to the
+      file. ontrol_domain
+   --------------------- assign_domain
+   --------------------- matrix
+   --------------------- unassign_adapter
+   --------------------- unassign_control_domain
+   --------------------- unassign_domain
+
+4. The administrator now needs to configure the matrixes for mediated
+   devices $uuid1 (for Guest1) and $uuid2 (for Guest2).
+
+   This is how the matrix is configured for Guest1:
+
+   echo 5 > assign_adapter
+   echo 6 > assign_adapter
+   echo 4 > assign_domain
+   echo 0xab > assign_domain
+
+   By architectural convention, all usage domains - i.e., domains assigned
+   via the assign_domain attribute file - will also be configured in the ADM
+   field of the KVM guest's CRYCB, so there is no need to assign control
+   domains here unless you want to assign control domains that are not
+   assigned as usage domains.
+
+   If a mistake is made configuring an adapter, domain or control domain,
+   you can use the unassign_xxx files to unassign the adapter, domain or
+   control domain.
+
+   To display the matrix configuration for Guest1:
+
+   cat matrix
+
+   This is how the matrix is configured for Guest2:
+
+   echo 5 > assign_adapter
+   echo 0x47 > assign_domain
+   echo 0xff > assign_domain
+
+5. Start Guest1
+
+   /usr/bin/qemu-system-s390x ... -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid1 ...
+
+6. Start Guest2
+
+   /usr/bin/qemu-system-s390x ... -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid2 ...
-- 
1.7.1

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Pierre Morel]

On 16/03/2018 00:24, Tony Krowiak wrote:
> If the CPU model indicates that AP facility is installed on
> the guest (i.e., -cpu xxxx,ap=on), then the expectation is that
> the AP bus running in the guest will initialize; however, if the
> AP instructions are not being interpreted by the firmware, then
> they will be intercepted and routed back to QEMU for handling.
> If a handler is not defined to process the intercepted instruciton,
> then an operation exception will be injected into the
> guest, in which case the AP bus will not initialize.
>
> There are two situations where AP instructions will not be
> interpreted:
>
> 1. The guest is not configured with a vfio-ap device (i.e.,
>     -device vfio-ap,sysfsdev=$path-to-mdev). The realize function for
>     the vfio-ap device enables interpretive execution of AP
>     instructions.
>
> 2. The guest is a second level guest but the first level guest has
>     not enabled interpretive execution.
>
> This patch introduces AP instruction handlers to ensure the AP bus
> module initializes on the guest when the AP facility is installed
> on the guest but AP instructions are not being interpreted. The logic
> incorporated is:
>
> * If the CPU model indicates AP instructions are
>    installed
>
>    * Set the status response code for the instruction to indicate that
>      the APQN contained in the instruction is not valid. This is
>      a valid response because there will be no devices configured for
>      the guest in any of the above scenarios.
>
> * Else return an error from the handler. This will result in an
>    operation being injected into the guest and the AP bus will not
>    initialize on the guest. That is commensurate with how things work
>    today.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>   hw/vfio/ap.c                 |   45 ++++++++++++++++++++++++++++++++++++++++++
>   include/hw/s390x/ap-device.h |    6 +++++
>   target/s390x/kvm.c           |   14 +++++++++++++
>   3 files changed, 65 insertions(+), 0 deletions(-)
>
> diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c
> index b397bb1..88e744d 100644
> --- a/hw/vfio/ap.c
> +++ b/hw/vfio/ap.c
> @@ -148,6 +148,51 @@ static void vfio_ap_unrealize(DeviceState *dev, Error **errp)
>       kvm_s390_set_interpret_ap(0);
>   }
>
> +int ap_device_handle_nqap(S390CPU *cpu)
> +{
> +    CPUS390XState *env = &cpu->env;
> +
> +    if (s390_has_feat(S390_FEAT_AP)) {
> +        env->regs[1] = 0x10000;
> +
> +        return 0;
> +    }
> +
> +    return -EOPNOTSUPP;
> +}
> +
> +int ap_device_handle_dqap(S390CPU *cpu)
> +{
> +    CPUS390XState *env = &cpu->env;
> +
> +    if (s390_has_feat(S390_FEAT_AP)) {
> +        env->regs[1] = 0x10000;
> +
> +        return 0;
> +    }
> +
> +    return -EOPNOTSUPP;
> +}
> +
> +int ap_device_handle_pqap(S390CPU *cpu)
> +{
> +    CPUS390XState *env = &cpu->env;
> +    int fc = 4 & (env->regs[0] >> 24);
> +
> +    /*
> +     * The Query Configuration Information (QCI) function (fc == 4) does not
> +     * set a response code in reg 1, so check for that along with the
> +     * AP feature.
> +     */
> +    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
> +        env->regs[1] = 0x10000;
> +
> +        return 0;
> +    }
> +
> +    return -EOPNOTSUPP;
> +}
> +
>   static Property vfio_ap_properties[] = {
>       DEFINE_PROP_STRING("sysfsdev", VFIOAPDevice, vdev.sysfsdev),
>       DEFINE_PROP_END_OF_LIST(),
> diff --git a/include/hw/s390x/ap-device.h b/include/hw/s390x/ap-device.h
> index 693df90..d45ae38 100644
> --- a/include/hw/s390x/ap-device.h
> +++ b/include/hw/s390x/ap-device.h
> @@ -11,6 +11,8 @@
>   #ifndef HW_S390X_AP_DEVICE_H
>   #define HW_S390X_AP_DEVICE_H
>
> +#include "cpu.h"
> +
>   #define AP_DEVICE_TYPE       "ap-device"
>
>   typedef struct APDevice {
> @@ -35,4 +37,8 @@ static inline APDevice *to_ap_dev(DeviceState *dev)
>   #define AP_DEVICE_CLASS(klass) \
>       OBJECT_CLASS_CHECK(APDeviceClass, (klass), AP_DEVICE_TYPE)
>
> +int ap_device_handle_nqap(S390CPU *cpu);
> +int ap_device_handle_dqap(S390CPU *cpu);
> +int ap_device_handle_pqap(S390CPU *cpu);
> +
>   #endif /* HW_S390X_AP_DEVICE_H */
> diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c
> index 2812e28..a636394 100644
> --- a/target/s390x/kvm.c
> +++ b/target/s390x/kvm.c
> @@ -50,6 +50,7 @@
>   #include "exec/memattrs.h"
>   #include "hw/s390x/s390-virtio-ccw.h"
>   #include "hw/s390x/s390-virtio-hcall.h"
> +#include "hw/s390x/ap-device.h"
>
>   #ifndef DEBUG_KVM
>   #define DEBUG_KVM  0
> @@ -88,6 +89,9 @@
>   #define PRIV_B2_CHSC                    0x5f
>   #define PRIV_B2_SIGA                    0x74
>   #define PRIV_B2_XSCH                    0x76
> +#define PRIV_B2_NQAP                    0xad
> +#define PRIV_B2_DQAP                    0xae
> +#define PRIV_B2_PQAP                    0xaf
>
>   #define PRIV_EB_SQBS                    0x8a
>   #define PRIV_EB_PCISTB                  0xd0
> @@ -1245,6 +1249,16 @@ static int handle_b2(S390CPU *cpu, struct kvm_run *run, uint8_t ipa1)
>       case PRIV_B2_SCLP_CALL:
>           rc = kvm_sclp_service_call(cpu, run, ipbh0);
>           break;
> +    case PRIV_B2_NQAP:
> +        rc = ap_device_handle_nqap(cpu);
> +        break;
> +    case PRIV_B2_DQAP:
> +        rc = ap_device_handle_dqap(cpu);
> +        break;
> +    case PRIV_B2_PQAP:
> +        rc = ap_device_handle_pqap(cpu);
> +        break;
> +        break;

one too much

>       default:
>           rc = -1;
>           DPRINTF("KVM: unhandled PRIV: 0xb2%x\n", ipa1);


-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [Qemu-devel] [PATCH v3 3/7] s390x/cpumodel: Set up CPU model for AP device support

[Pierre Morel]

On 16/03/2018 00:24, Tony Krowiak wrote:
> A new CPU model feature and two new CPU model facilities are
> introduced to support AP devices for a KVM guest.
>
> CPU model features:
>
> 1. The KVM_S390_VM_CPU_FEAT_AP CPU model feature indicates that
>     AP facilities are installed. This feature will be enabled by
>     the kernel only if the AP facilities are installed on the linux
>     host. This feature must be turned on from userspace to access
>     AP devices from the KVM guest. The QEMU command line to turn
>     this feature looks something like this:
>
> 	qemu-system-s390x ... -cpu xxx,ap=on
>
> CPU model facilities:
>
> 1. The S390_FEAT_AP_QUERY_CONFIG_INFO feature indicates the AP Query
>     Configuration Information (QCI) facility is installed. This feature
>     will be enabled by the kernel only if the QCI is installed on
>     the host.
>
> 2. The S390_FEAT_AP_FACILITY_TEST feature indicates that the AP
>     Facility Test (APFT) facility is installed. This feature will
>     be enabled by the kernel only if the APFT facility is installed
>     on the host.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>   target/s390x/cpu_features.c     |    3 +++
>   target/s390x/cpu_features_def.h |    3 +++
>   target/s390x/cpu_models.c       |    2 ++
>   target/s390x/gen-features.c     |    3 +++
>   target/s390x/kvm.c              |    1 +
>   5 files changed, 12 insertions(+), 0 deletions(-)
>
> diff --git a/target/s390x/cpu_features.c b/target/s390x/cpu_features.c
> index a5619f2..1abe987 100644
> --- a/target/s390x/cpu_features.c
> +++ b/target/s390x/cpu_features.c
> @@ -36,8 +36,10 @@ static const S390FeatDef s390_features[] = {
>       FEAT_INIT("srs", S390_FEAT_TYPE_STFL, 9, "Sense-running-status facility"),
>       FEAT_INIT("csske", S390_FEAT_TYPE_STFL, 10, "Conditional-SSKE facility"),
>       FEAT_INIT("ctop", S390_FEAT_TYPE_STFL, 11, "Configuration-topology facility"),
> +    FEAT_INIT("qci", S390_FEAT_TYPE_STFL, 12, "Query AP Configuration facility"),
>       FEAT_INIT("ipter", S390_FEAT_TYPE_STFL, 13, "IPTE-range facility"),
>       FEAT_INIT("nonqks", S390_FEAT_TYPE_STFL, 14, "Nonquiescing key-setting facility"),
> +    FEAT_INIT("apft", S390_FEAT_TYPE_STFL, 15, "Adjunct Processor Facilities Test facility"),
>       FEAT_INIT("etf2", S390_FEAT_TYPE_STFL, 16, "Extended-translation facility 2"),
>       FEAT_INIT("msa-base", S390_FEAT_TYPE_STFL, 17, "Message-security-assist facility (excluding subfunctions)"),
>       FEAT_INIT("ldisp", S390_FEAT_TYPE_STFL, 18, "Long-displacement facility"),
> @@ -125,6 +127,7 @@ static const S390FeatDef s390_features[] = {
>
>       FEAT_INIT("dateh2", S390_FEAT_TYPE_MISC, 0, "DAT-enhancement facility 2"),
>       FEAT_INIT("cmm", S390_FEAT_TYPE_MISC, 0, "Collaborative-memory-management facility"),
> +    FEAT_INIT("ap", S390_FEAT_TYPE_MISC, 0, "AP facilities installed"),

AFAIK you should use  FEAT_INIT_MISC to initialize misc features now.
May be rebase on last qemu ?


>
>       FEAT_INIT("plo-cl", S390_FEAT_TYPE_PLO, 0, "PLO Compare and load (32 bit in general registers)"),
>       FEAT_INIT("plo-clg", S390_FEAT_TYPE_PLO, 1, "PLO Compare and load (64 bit in parameter list)"),
> diff --git a/target/s390x/cpu_features_def.h b/target/s390x/cpu_features_def.h
> index 7c5915c..8998b65 100644
> --- a/target/s390x/cpu_features_def.h
> +++ b/target/s390x/cpu_features_def.h
> @@ -27,8 +27,10 @@ typedef enum {
>       S390_FEAT_SENSE_RUNNING_STATUS,
>       S390_FEAT_CONDITIONAL_SSKE,
>       S390_FEAT_CONFIGURATION_TOPOLOGY,
> +    S390_FEAT_AP_QUERY_CONFIG_INFO,
>       S390_FEAT_IPTE_RANGE,
>       S390_FEAT_NONQ_KEY_SETTING,
> +    S390_FEAT_AP_FACILITIES_TEST,
>       S390_FEAT_EXTENDED_TRANSLATION_2,
>       S390_FEAT_MSA,
>       S390_FEAT_LONG_DISPLACEMENT,
> @@ -118,6 +120,7 @@ typedef enum {
>       /* Misc */
>       S390_FEAT_DAT_ENH_2,
>       S390_FEAT_CMM,
> +    S390_FEAT_AP,
>
>       /* PLO */
>       S390_FEAT_PLO_CL,
> diff --git a/target/s390x/cpu_models.c b/target/s390x/cpu_models.c
> index cf82589..7e2af09 100644
> --- a/target/s390x/cpu_models.c
> +++ b/target/s390x/cpu_models.c
> @@ -769,6 +769,8 @@ static void check_consistency(const S390CPUModel *model)
>           { S390_FEAT_PRNO_TRNG_QRTCR, S390_FEAT_MSA_EXT_5 },
>           { S390_FEAT_PRNO_TRNG, S390_FEAT_MSA_EXT_5 },
>           { S390_FEAT_SIE_KSS, S390_FEAT_SIE_F2 },
> +        { S390_FEAT_AP_QUERY_CONFIG_INFO, S390_FEAT_AP },
> +        { S390_FEAT_AP_FACILITIES_TEST, S390_FEAT_AP },
>       };
>       int i;
>
> diff --git a/target/s390x/gen-features.c b/target/s390x/gen-features.c
> index 0cdbc15..0d5b0f7 100644
> --- a/target/s390x/gen-features.c
> +++ b/target/s390x/gen-features.c
> @@ -447,6 +447,9 @@ static uint16_t full_GEN12_GA1[] = {
>       S390_FEAT_ADAPTER_INT_SUPPRESSION,
>       S390_FEAT_EDAT_2,
>       S390_FEAT_SIDE_EFFECT_ACCESS_ESOP2,
> +    S390_FEAT_AP_QUERY_CONFIG_INFO,
> +    S390_FEAT_AP_FACILITIES_TEST,
> +    S390_FEAT_AP,
>   };
>
>   static uint16_t full_GEN12_GA2[] = {
> diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c
> index fbc887e..33e5ec3 100644
> --- a/target/s390x/kvm.c
> +++ b/target/s390x/kvm.c
> @@ -2178,6 +2178,7 @@ static int kvm_to_feat[][2] = {
>       { KVM_S390_VM_CPU_FEAT_PFMFI, S390_FEAT_SIE_PFMFI},
>       { KVM_S390_VM_CPU_FEAT_SIGPIF, S390_FEAT_SIE_SIGPIF},
>       { KVM_S390_VM_CPU_FEAT_KSS, S390_FEAT_SIE_KSS},
> +    { KVM_S390_VM_CPU_FEAT_AP, S390_FEAT_AP},
>   };
>
>   static int query_cpu_feat(S390FeatBitmap features)


-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [Qemu-devel] [PATCH v3 7/7] s390: doc: detailed specifications for AP virtualization

[Pierre Morel]

On 16/03/2018 00:25, Tony Krowiak wrote:
> This patch provides documentation describing the AP architecture and
> design concepts behind the virtualization of AP devices. It also
> includes an example of how to configure AP devices for exclusive
> use of KVM guests.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>   docs/vfio-ap.txt |  624 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>   1 files changed, 624 insertions(+), 0 deletions(-)
>   create mode 100644 docs/vfio-ap.txt
>
> diff --git a/docs/vfio-ap.txt b/docs/vfio-ap.txt
> new file mode 100644
> index 0000000..54e7523
> --- /dev/null
> +++ b/docs/vfio-ap.txt
> @@ -0,0 +1,624 @@
> +Adjunct Processor (AP) Device
> +=============================
> +
> +Contents:
> +=========
> +* Introduction
> +* AP Architectural Overview
> +* Start Interpretive Execution (SIE) Instruction
> +* AP Matrix Configuration on Linux Host
> +* AP Matrix Configuration for a Linux Guest
> +* Starting a Linux Guest Configured with an AP Matrix
> +* Example: Configure AP Matrices for Two Linux Guests
> +
> +Introduction:
> +============
> +The IBM Adjunct Processor (AP) Cryptographic Facility is comprised
> +of three AP instructions and from 1 to 256 PCIe cryptographic adapter cards.
> +These AP devices provide cryptographic functions to all CPUs assigned to a
> +linux system running in an IBM Z system LPAR.
> +
> +On s390x, AP adapter cards are exposed via the AP bus. This document
> +describes how those cards may be made available to KVM guests using the
> +VFIO mediated device framework.
> +
> +AP Architectural Overview:
> +=========================
> +In order understand the terminology used in the rest of this document, let's
> +start with some definitions:
> +
> +* AP adapter
> +
> +  An AP adapter is an IBM Z adapter card that can perform cryptographic
> +  functions. There can be from 0 to 256 adapters assigned to an LPAR. Adapters
> +  assigned to the LPAR in which a linux host is running will be available to
> +  the linux host. Each adapter is identified by a number from 0 to 255. When
> +  installed, an AP adapter is accessed by AP instructions executed by any CPU.
> +
> +* AP domain
> +
> +  An adapter is partitioned into domains. Each domain can be thought of as
> +  a set of hardware registers for processing AP instructions. An adapter can
> +  hold up to 256 domains. Each domain is identified by a number from 0 to 255.
> +  Domains can be further classified into two types:
> +
> +    * Usage domains are domains that can be accessed directly to process AP
> +      commands
> +
> +    * Control domains are domains that are accessed indirectly by AP
> +      commands sent to a usage domain to control or change the domain, for
> +      example; to set a secure private key for the domain.
> +
> +* AP Queue
> +
> +  An AP queue is the means by which an AP command-request message is sent to an
> +  AP usage domain inside a specific AP. An AP queue is identified by a tuple
> +  comprised of an AP adapter ID (APID) and an AP queue index (APQI). The
> +  APQI corresponds to a given usage domain number within the adapter. This tuple
> +  forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP
> +  instructions include a field containing the APQN to identify the AP queue to
> +  which the AP command-request message is to be sent for processing.
> +
> +* AP Instructions:
> +
> +  There are three AP instructions:
> +
> +  * NQAP: to enqueue an AP command-request message to a queue
> +  * DQAP: to dequeue an AP command-reply message from a queue
> +  * PQAP: to administer the queues
> +
> +Start Interpretive Execution (SIE) Instruction
> +==============================================
> +A KVM guest is started by executing the Start Interpretive Execution (SIE)
> +instruction. The SIE state description is a control block that contains the
> +state information for a KVM guest and is supplied as input to the SIE
> +instruction. The SIE state description contains a field that references
> +a Crypto Control Block (CRYCB). The CRYCB contains three fields to identify the
> +adapters, usage domains and control domains assigned to the KVM guest:
> +
> +* The AP Mask (APM) field is a bit mask that identifies the AP adapters assigned
> +  to the KVM guest. Each bit in the mask, from most significant to least
> +  significant bit, corresponds to an APID from 0-255. If a bit is set, the
> +  corresponding adapter is valid for use by the KVM guest.
> +
> +* The AP Queue Mask (AQM) field is a bit mask identifying the AP queues assigned
> +  to the KVM guest. Each bit in the mask, from most significant to least
> +  significant bit, corresponds to an AP queue index (APQI) from 0-255. If a bit
> +  is set, the corresponding queue is valid for use by the KVM guest.
> +
> +* The AP Domain Mask field is a bit mask that identifies the AP control domains
> +  assigned to the KVM guest. The ADM bit mask controls which domains can be
> +  changed by an AP command-request message sent to a usage domain from the
> +  guest. Each bit in the mask, from least significant to most significant bit,
> +  corresponds to a domain from 0-255. If a bit is set, the corresponding domain
> +  can be modified by an AP command-request message sent to a usage domain
> +  configured for the KVM guest.
> +
> +If you recall from the description of an AP Queue, AP instructions include
> +an APQN to identify the AP adapter and AP queue to which an AP command-request
> +message is to be sent (NQAP and PQAP instructions), or from which a
> +command-reply message is to be received (DQAP instruction). The validity of an
> +APQN is defined by the matrix calculated from the APM and AQM; it is the
> +intersection of all assigned adapter numbers (APM) with all assigned queue
> +indexes (AQM). For example, if adapters 1 and 2 and usage domains 5 and 6 are
> +assigned to a guest, the APQNs (1,5), (1,6), (2,5) and (2,6) will be valid for
> +the guest.
> +
> +The APQNs provide secure key functionality - i.e., a private key is stored on
> +the adapter card for each of its domains - so each APQN must be assigned to at
> +most one guest or the linux host.
> +
> +   Example 1: Valid configuration:
> +   ------------------------------
> +   Guest1: adapters 1,2  domains 5,6
> +   Guest2: adapter  1,2  domain 7
> +
> +   This is valid because both guests have a unique set of APQNs: Guest1 has
> +   APQNs (1,5), (1,6), (2,5) and (2,6); Guest2 has APQNs (1,7) and (2,7).
> +
> +   Example 2: Invalid configuration:
> +   --------------------------------
> +   Guest1: adapters 1,2  domains 5,6
> +   Guest2: adapter  1    domains 6,7
> +
> +   This is an invalid configuration because both guests have access to
> +   APQN (1,6).
> +
> +AP device Configuration on Linux Host:
> +=====================================
> +A linux system is a guest of the LPAR in which it is running and has access to
> +the AP resources configured for the LPAR. The LPAR's AP matrix is
> +configured using the 'Customize/Delete Activation Profiles' dialog from the HMC.
> +This dialog displays the activation profiles configured for the linux system.
> +Selecting the specific activation profile to be edited and clicking the
> +'Customize Profile' button will open the 'Customize Image Profiles' dialog.
> +Selecting the 'Crypto' link in the tree view on the left hand side of the dialog
> +will display the AP matrix configuration in the right hand panel. There, one can
> +assign AP adapters - called Cryptos - and domains to the LPAR. When the linux
> +system is started using this activation profile, it will have access to the
> +matrix of AP adapters and domains configured via the activation profile.
> +
> +When the linux system is started, the AP adapter devices will be connected to
> +the AP bus and the following AP matrix interfaces will be created in sysfs:
> +
> +/sys/bus/ap
> +... [devices]
> +...... xx.yyyy
> +...... ...
> +...... cardxx
> +...... ...
> +
> +Where:
> +    cardxx     is adapter number xx (in hex)
> +    yyyy       is a usage domain number yyyy (in hex)
> +....xx.yyyy    is APQN (xx,yyyy)
> +
> +For example, if AP adapters 5 and 6 and domains 4 and 71 (0x47) are configured
> +for the LPAR, the sysfs representation on the linux system would look like this:
> +
> +/sys/bus/ap
> +... [devices]
> +...... 05.0004
> +...... 05.0047
> +...... 06.0004
> +...... 06.0047
> +...... card05
> +...... card06
> +
> +There will also be AP device drivers created to control each type of AP matrix
> +interface available to the IBM Z system:
> +
> +/sys/bus/ap
> +... [drivers]
> +...... [cex2acard]        for Crypto Express 2/3 accelerator cards
> +...... [cex2aqueue]       for AP queues served by Crypto Express 2/3
> +                          accelerator cards
> +...... [cex4card]         for Crypto Express 4/5/6 accelerator and coprocessor
> +                          cards
> +...... [cex4queue]        for AP queues served by Crypto Express 4/5/6
> +                          accelerator and coprocessor cards
> +...... [pcixcccard]       for Crypto Express 2/3 coprocessor cards
> +...... [pcixccqueue]      for AP queues served by Crypto Express 2/3
> +                          coprocessor cards
> +
> +Links to the AP interfaces controlled by each AP device driver will be created
> +in the device driver's sysfs directory. For example, if AP adapter 5 and domains
> +4 and 71 (0x47) are assigned to the LPAR and adapter 5 is a CEX5 card, the
> +following links will be created in the CEX5 drivers' sysfs directories:
> +
> +/sys/bus/ap
> +... [drivers]
> +...... [cex4card]
> +......... [card05]
> +...... [cex4queue]
> +......... [05.0004]
> +......... [05.0047]
> +
> +AP Matrix Configuration for a Linux Guest:
> +=========================================
> +In order to configure the AP matrix for a guest, the adapters, usage domains
> +and control domains to be used by the guest must be assigned to the guest. This
> +section describes how to configure a guest's AP matrix.
> +
> +The kernel interfaces for configuring an AP matrix for a linux guest are built
> +on the VFIO mediated device framework and are provided by the vfio_ap
> +kernel module. By default, the vfio_ap module is a loadable module, The
> +dependency chain for the vfio_ap module is:
> +* vfio
> +* mdev
> +* vfio_mdev
> +* vfio_ap
> +
> +When installed, the vfio_ap module is initialized. During module initialization,
> +a vfio_ap driver is created and registered with the AP bus creating the
> +following sysfs interfaces:
> +
> + /sys/bus/ap/drivers/
> +...[vfio_ap]
> +...... bind
> +...... unbind
> +
> +The vfio_ap device driver will create a 'matrix' device to hold the APQNs
> +reserved for exclusive use by KVM guests:
> +
> +/sys/devices/
> +... [vfio_ap]
> +......[matrix] symlink to the matrix device directory
> +
> +The vfio_ap device driver serves several purposes:
> +1. Provides an interface for securing APQNs preventing their use by the host
> +   linux system and reserving their use by one or more guests.
> +2. Creates the sysfs interfaces for configuring an AP matrix for a linux guest.
> +
> +Securing APQNs
> +--------------
> +   An APQN is reserved by unbinding an AP queue device AP bus device driver and
> +   binding it to the vfio_ap device driver. For example, suppose we want to
> +   secure APQN (05,0004). Assuming that the AP adapter card 5 is a CEX5
> +   coprocessor card:
> +
> +   echo 05.0004 > /sys/bus/ap/drivers/cex4queue/unbind
> +   echo 05.0004 > /sys/bus/ap/drivers/vfio_ap/bind
> +
> +   This action will store the APQN in the /sys/devices/vfio_ap/matrix device
> +   which makes it available for use by a linux guest.
> +
> +Configuring an AP matrix for a linux guest.
> +------------------------------------------
> +These sysfs interfaces are built on the VFIO mediated device framework. To
> +configure an AP matrix for a guest, a mediated matrix device must first be
> +created for the /sys/devices/vfio_ap/matrix device. The sysfs interfaceAPQI corresponding to
> +for creating a mediated matrix device is in:
> +
> +/sys/devices
> +... [vfio_ap]
> +......[matrix]
> +......... [mdev_supported_types]
> +............ [vfio_ap-passthrough]
> +............... create
> +............... [devices]
> +
> +A mediated AP matrix device is created by writing a UUID to the attribute
> +file named 'create', for example:
> +
> +   uuidgen > create
> +
> +When a mediated AP matrix device is created, a sysfs directory named after
> +the UUID:
> +
> +/sys/devices
> +... [vfio_ap]
> +......[matrix]
> +......... [mdev_supported_types]
> +............ [vfio_ap-passthrough]
> +............... create
> +............... [devices]
> +.................. [$uuid]
> +
> +There will also be three sets of attribute files created in the mediated
> +matrix device's sysfs directory to configure an AP matrix for the
> +KVM guest:
> +
> +/sys/devices
> +... [vfio_ap]
> +......[matrix]
> +......... [mdev_supported_types]
> +............ [vfio_ap-passthrough]
> +............... create
> +............... [devices]
> +.................. [$uuid]
> +..................... assign_adapter
> +..................... assign_control_domain
> +..................... assign_domain
> +..................... matrix
> +..................... unassign_adapter
> +..................... unassign_control_domain
> +..................... unassign_domain
> +
> +assign_adapter
> +   To assign an AP adapter to the mediated matrix device, its APID is written
> +   'assign_adapter' file. This may be done multiple times to assign more than
> +   one adapter. The APID may be specified using conventional semantics
> +   as a decimal, hexidecimal, or octal number. For example, to assign adapters
> +   4, 5 and 16 to mediated matrix device $uuid in decimal, hexidecimal and octal
> +   respectively:
> +
> +       echo 4 > assign_adapter
> +       echo 0x5 > assign_adapter
> +       echo 020
> +
> +unassign_adapter
> +   To unassign an AP adapter, its APID is written to the 'unassign_adapter'
> +   file. This may also be done multiple times to unassign more than one adapter.
> +
> +assign_domain
> +   To assign a usage domain, the APQI corresponding to the domain number is
> +   written into the 'assign_domain' file. This may be done multiple times to
> +   assign more than one usage domain. The APQI may be specified using
> +   conventional semantics as a decimal, hexidecimal, or octal number. For
> +   example, to assign usage domains 4, 8, and 71 to mediated matrix device
> +   $uuid in decimal, hexidecimal and octal respectively:
> +
> +      echo 4 > assign_domain
> +      echo 0x8 > assign_domain
> +      echo 0107 > assign_domain
> +
> +unassign_domain
> +   To unassign a usage domain, the APQI corresponding to the domain number is
> +   written into the 'unassign_domain' file. This may be done multiple times to
> +   unassign more than one usage domain.
> +
> +assign_control_domain
> +   To assign a control domain, the domain number is written into the
> +   'assign_control_domain' file. This may be done multiple times to
> +   assign more than one control domain. The domain number may be specified using
> +   conventional semantics as a decimal, hexidecimal, or octal number. For
> +   example, to assign  control domains 4, 8, and 71 to mediated matrix device
> +   $uuid in decimal, hexidecimal and octal respectively:
> +
> +      echo 4 > assign_domain
> +      echo 0x8 > assign_domain
> +      echo 0107 > assign_domain
> +
> +unassign_control_domain
> +   To unassign a control domain, the domain number is written into the
> +   'unassign_domain' file. This may be done multiple times to unassign more than
> +   one control domain.
> +
> +Notes:
> +* Hot plug/unplug is not currently supported for mediated AP matrix devices,
> +  so the AP matrix resulting from assignment and/or unassignment of AP
> +  adapters, usage domains and control domains to a mediated AP matrix device
> +  while the guest is running will not take affect until the linux guest is
> +  rebooted.
> +* By architectural convention, all usage domains configured for a KVM guest
> +  will also be implicitly assigned as control domains also, to there is no
> +  need to assign control domains that are assigned as usage domains.
> +
> +Starting a Linux Guest Configured with an AP Matrix:
> +===================================================
> +In addition to providing the sysfs interfaces for configuring the AP matrix for
> +a linux guest, a mediated matrix device also acts as a communication pathway
> +between QEMU and the vfio_ap device driver. To gain access to the
> +device driver, the following option must be specified on the QEMU command line:
> +
> +   -device vfio_ap,sysfsdev=$path-to-mdev
> +
> +The sysfsdev parameter specifies the path to the mediated matrix device.
> +There are a number of ways to specify this path:
> +
> +/sys/devices/vfio_ap/matrix/$uuid
> +/sys/bus/mdev/devices/$uuid
> +/sys/bus/mdev/drivers/vfio_mdev/$uuid
> +/sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/devices/$uuid
> +
> +When the linux guest is subsequently started, the guest will open the mediated
> +matrix device's file descriptor to get information about the mediated matrix
> +device. The vfio_ap device driver will update the APM, AQM, and ADM fields in the
> +guest's CRYCB with the adapter, usage domain and control domains assigned to
> +via the mediated matrix device's sysfs attribute files. Programs running on the
> +linux guest will then:
> +
> +1. Have direct access to the APQNs derived from the intersection of the AP
> +   adapter and usage domain numbers specified in the APM and AQM respectively
> +
> +2. Have authorization to process AP commands to change - e.g., store a new
> +   secure key - a control domain identified in an AP instruction sent to a valid
> +   APQN.
> +
> +CPU model features:
> +
> +Three CPU model features are available for controlling guest access to AP
> +facilities:
> +
> +1. AP facilities feature
> +
> +   The AP facilities feature indicates that AP facilities are installed on the
> +   guest. This feature will be enabled by the kernel only if the AP facilities
> +   are installed on the host system. It will turned on automatically for guests
> +   started with CPU model zEC12 or newer. The feature is s390-specific and is
> +   represented as a parameter of the -cpu option on the QEMU command line:
> +
> +      qemu-system-s390x -cpu $model,ap=on|off
> +
> +      Where:
> +
> +         $model is the CPU model defined for the guest (defaults to the model of
> +                the host system if not specified).
> +
> +         ap=on|off indicates whether AP facilities are installed (on) or not
> +                   installed (off). The default for CPU models zEC12 or newer
> +                   is ap=on. AP facilities must be installed when this parameter
> +                   is used in conjunction with -device vfio-ap,sysfsdev=$path or
> +                   the guest will not start.
> +
> +2. Query Configuration Information (QCI) facility
> +
> +   The QCI facility is used by the AP bus running on the guest to query the
> +   configuration of the AP facilities. This facility will be enabled by
> +   the kernel only if the QCI facility is installed on the host system. It will
> +   be turned on automatically for guests started with CPU model zEC12 or newer.
> +   The feature is s390-specific and is represented as a parameter of the -cpu
> +   option on the QEMU command line:
> +
> +      qemu-system-s390x -cpu $model,qci=on|off
> +
> +      Where:
> +
> +         $model is the CPU model defined for the guest
> +
> +         qci=on|off indicates whether the QCI facility is installed (on) or not
> +                    installed (off). The default for CPU models zEC12 or newer
> +                    is qci=on. Turning the QCI facility on makes no sense if it
> +                    is not used in conjunction with the
> +                    '-device vfio-ap,sysfsdev=$path' option. A warning will be
> +                    presented if QCI is turned on and the AP facilities are not
> +                    installed.
> +
> +                    If the QCI facility is turned off, APQNs with an APQI
> +                    greater than 15 will not be accessible from the guest.
> +
> +3. Adjunct Process Facility Test (APFT) facility
> +
> +   The APFT facility is used by the AP bus running on the guest to test the
> +   AP facilities available for a given AP queue. This facility will be enabled
> +   by the kernel only if the APFT facility is installed on the host system. It
> +   will be turned on automatically for guests started with CPU model zEC12 or
> +   newer. The feature is s390-specific and is represented as a parameter of the
> +   -cpu option on the QEMU command line:
> +
> +      qemu-system-s390x -cpu $model,apft=on|off
> +
> +      Where:
> +
> +         $model is the CPU model defined for the guest (defaults to the model of
> +                the host system if not specified).
> +
> +         apft=on|off indicates whether the APFT facility is installed (on) or
> +                     not installed (off). The default for CPU models zEC12 and
> +                     newer is apft=on. Turning the APFT facility on makes no
> +                     sense if it is not used in conjunction with the
> +                     -device vfio-ap,sysfsdev=$path option. A warning will be
> +                     presented if APFT is turned on and the AP facilities are
> +                     not installed.
> +
> +                     It also makes no sense to turn APFT off when used in
> +                     conjunction with the vfio-ap device because the APFT
> +                     facility is required; the AP bus running on the guest will
> +                     not detect CEX4 and newer devices without it. Since only
> +                     CEX4 and newer devices are supported for guest usage, no AP
> +                     devices can be made accessible to a guest started without
> +                     APFT installed.
> +
> +Example: Configure AP Matrixes for Two Linux Guests:
> +===================================================
> +Let's now provide an example to illustrate how KVM guests may be given
> +access to AP facilities. For this example, we will show how to configure
> +two guests such that executing the lszcrypt command on the guests would
> +look like this:
> +
> +Guest1
> +------
> +CARD.DOMAIN TYPE  MODE

> +------------------------------
> +05          CEX5C CCA-Coproc

> +05.0004     CEX5C CCA-Coproc
> +05.00ab     CEX5C CCA-Coproc

> +06          CEX5A Accelerator
> +06.0004     CEX5A Accelerator
> +06.00ab     CEX5C CCA-Coproc
5 whitespace at the end of lines
> +
> +Guest2
> +------
> +CARD.DOMAIN TYPE  MODE
> +------------------------------
> +05          CEX5A Accelerator
> +05.0047     CEX5A Accelerator
> +05.00ff     CEX5A Accelerator
> +
> +These are the steps for configuring the Guest1 and Guest2:
> +
> +1. The first thing that needs to be done is to secure the AP queues to be
> +   used by the two guests so that the host can not access them. This is done
> +   by unbinding each AP Queue device from its respective AP driver. In our
> +   example, these queues are bound to the cex4queue driver. This would be
> +   the sysfs location of these devices:
> +
> +   /sys/bus/ap
> +   --- [drivers]
> +   ------ [cex4queue]
> +   --------- [05.0004]
> +   --------- [05.0047]   --------------------- control_domains
> +   --------------------- domains
> +   --------- [05.00ab]
> +   --------- [05.00ff]
> +   --------- [06.0004]
> +   --------- [06.00ab]
> +   --------- unbind
> +
> +   To unbind AP queue 05.0004 from the cex4queue device driver:
> +
> +    echo 05.0004 > unbind
> +
> +   This must also be done for AP queues 05.00ab, 05.0047, 05.00ff, 06.0004,
> +   and 06.00ab.
> +
> +2. The next step is to reserve the queues for use by the two KVM guests.
> +   This is accomplished by binding them to the VFIO AP device driver.
> +   This is the sysfs location of the VFIO AP device driver:
> +
> +   /sys/bus/ap
> +   ---[drivers]
> +   ------ [vfio_ap]
> +   ---------- bind
> +
> +   To bind queue 05.0004 to the vfio_ap driver:
> +
> +    echo 05.0004 > bind
> +
> +   This must also be done for AP queues 05.00ab, 05.0047, 05.00ff, 06.0004,
> +   and 06.00ab.
> +
> +3. Create the mediated devices needed to configure the AP matrixes for the
> +   two guests and to provide an interface to the vfio_ap driver for
> +   use by the guests:
> +
> +   /sys/devices/
> +   --- [vfio_ap]
> +   ------ [matrix] (this is the matrix device)
> +   --------- [mdev_supported_types]
> +   ------------ [vfio_ap-passthrough] (passthrough mediated matrix device type)
> +   --------------- create
> +   --------------- [devices]
> +
> +   To create the mediated devices for the two guests:
> +
> +    uuidgen > create
> +    uuidgen > create
> +
> +   This will create two mediated devices in the [devices] subdirectory named
> +   with the UUID written to the create attribute file. We call them $uuid1
> +   and $uuid2:
> +
> +   /sys/devices/
> +   --- [vfio_ap]
> +   ------ [matrix]
> +   --------- [mdev_supported_types]
> +   ------------ [vfio_ap-passthrough]
> +   --------------- [devices]
> +   ------------------ [$uuid1]
> +   --------------------- assign_adapter
> +   --------------------- assign_control_domain
> +   --------------------- assign_domain
> +   --------------------- matrix
> +   --------------------- unassign_adapter
> +   --------------------- unassign_control_domain
> +   --------------------- unassign_domain
> +
> +   ------------------ [$uuid2]
> +   --------------------- assign_adapter
> +   --------------------- assign_cTo assign an adapter, the APID of the adapter is written to the
> +      file. ontrol_domain
> +   --------------------- assign_domain
> +   --------------------- matrix
> +   --------------------- unassign_adapter
> +   --------------------- unassign_control_domain
> +   --------------------- unassign_domain
> +
> +4. The administrator now needs to configure the matrixes for mediated
> +   devices $uuid1 (for Guest1) and $uuid2 (for Guest2).
> +
> +   This is how the matrix is configured for Guest1:
> +
> +   echo 5 > assign_adapter
> +   echo 6 > assign_adapter
> +   echo 4 > assign_domain
> +   echo 0xab > assign_domain
> +
> +   By architectural convention, all usage domains - i.e., domains assigned
> +   via the assign_domain attribute file - will also be configured in the ADM
> +   field of the KVM guest's CRYCB, so there is no need to assign control
> +   domains here unless you want to assign control domains that are not
> +   assigned as usage domains.
> +
> +   If a mistake is made configuring an adapter, domain or control domain,
> +   you can use the unassign_xxx files to unassign the adapter, domain or
> +   control domain.
> +
> +   To display the matrix configuration for Guest1:
> +
> +   cat matrix
> +
> +   This is how the matrix is configured for Guest2:
> +
> +   echo 5 > assign_adapter
> +   echo 0x47 > assign_domain
> +   echo 0xff > assign_domain
> +
> +5. Start Guest1
> +
> +   /usr/bin/qemu-system-s390x ... -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid1 ...
> +
> +6. Start Guest2
> +
> +   /usr/bin/qemu-system-s390x ... -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid2 ...


-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [Qemu-devel] [PATCH v3 7/7] s390: doc: detailed specifications for AP virtualization

[Pierre Morel]

On 16/03/2018 00:25, Tony Krowiak wrote:
> This patch provides documentation describing the AP architecture and
> design concepts behind the virtualization of AP devices. It also
> includes an example of how to configure AP devices for exclusive
> use of KVM guests.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>   docs/vfio-ap.txt |  624 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>   1 files changed, 624 insertions(+), 0 deletions(-)
>   create mode 100644 docs/vfio-ap.txt
>
> diff --git a/docs/vfio-ap.txt b/docs/vfio-ap.txt
> new file mode 100644
> index 0000000..54e7523
> --- /dev/null
> +++ b/docs/vfio-ap.txt
> @@ -0,0 +1,624 @@
> +Adjunct Processor (AP) Device
> +=============================
> +
> +Contents:
> +=========
> +* Introduction
> +* AP Architectural Overview
> +* Start Interpretive Execution (SIE) Instruction
> +* AP Matrix Configuration on Linux Host
> +* AP Matrix Configuration for a Linux Guest
> +* Starting a Linux Guest Configured with an AP Matrix
> +* Example: Configure AP Matrices for Two Linux Guests
> +
> +Introduction:
> +============
> +The IBM Adjunct Processor (AP) Cryptographic Facility is comprised
> +of three AP instructions and from 1 to 256 PCIe cryptographic adapter cards.
> +These AP devices provide cryptographic functions to all CPUs assigned to a
> +linux system running in an IBM Z system LPAR.
> +
> +On s390x, AP adapter cards are exposed via the AP bus. This document
> +describes how those cards may be made available to KVM guests using the
> +VFIO mediated device framework.
> +
> +AP Architectural Overview:
> +=========================
> +In order understand the terminology used in the rest of this document, let's
> +start with some definitions:
> +
> +* AP adapter
> +
> +  An AP adapter is an IBM Z adapter card that can perform cryptographic
> +  functions. There can be from 0 to 256 adapters assigned to an LPAR. Adapters
> +  assigned to the LPAR in which a linux host is running will be available to
> +  the linux host. Each adapter is identified by a number from 0 to 255. When
> +  installed, an AP adapter is accessed by AP instructions executed by any CPU.
> +
> +* AP domain
> +
> +  An adapter is partitioned into domains. Each domain can be thought of as
> +  a set of hardware registers for processing AP instructions. An adapter can
> +  hold up to 256 domains. Each domain is identified by a number from 0 to 255.
> +  Domains can be further classified into two types:
> +
> +    * Usage domains are domains that can be accessed directly to process AP
> +      commands
> +
> +    * Control domains are domains that are accessed indirectly by AP
> +      commands sent to a usage domain to control or change the domain, for
> +      example; to set a secure private key for the domain.
> +
> +* AP Queue
> +
> +  An AP queue is the means by which an AP command-request message is sent to an
> +  AP usage domain inside a specific AP. An AP queue is identified by a tuple
> +  comprised of an AP adapter ID (APID) and an AP queue index (APQI). The
> +  APQI corresponds to a given usage domain number within the adapter. This tuple
> +  forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP
> +  instructions include a field containing the APQN to identify the AP queue to
> +  which the AP command-request message is to be sent for processing.
> +
> +* AP Instructions:
> +
> +  There are three AP instructions:
> +
> +  * NQAP: to enqueue an AP command-request message to a queue
> +  * DQAP: to dequeue an AP command-reply message from a queue
> +  * PQAP: to administer the queues
> +
> +Start Interpretive Execution (SIE) Instruction
> +==============================================
> +A KVM guest is started by executing the Start Interpretive Execution (SIE)
> +instruction. The SIE state description is a control block that contains the
> +state information for a KVM guest and is supplied as input to the SIE
> +instruction. The SIE state description contains a field that references
> +a Crypto Control Block (CRYCB). The CRYCB contains three fields to identify the
> +adapters, usage domains and control domains assigned to the KVM guest:
> +
> +* The AP Mask (APM) field is a bit mask that identifies the AP adapters assigned
> +  to the KVM guest. Each bit in the mask, from most significant to least
> +  significant bit, corresponds to an APID from 0-255. If a bit is set, the
> +  corresponding adapter is valid for use by the KVM guest.
> +
> +* The AP Queue Mask (AQM) field is a bit mask identifying the AP queues assigned
> +  to the KVM guest. Each bit in the mask, from most significant to least
> +  significant bit, corresponds to an AP queue index (APQI) from 0-255. If a bit
> +  is set, the corresponding queue is valid for use by the KVM guest.
> +
> +* The AP Domain Mask field is a bit mask that identifies the AP control domains
> +  assigned to the KVM guest. The ADM bit mask controls which domains can be
> +  changed by an AP command-request message sent to a usage domain from the
> +  guest. Each bit in the mask, from least significant to most significant bit,
> +  corresponds to a domain from 0-255. If a bit is set, the corresponding domain
> +  can be modified by an AP command-request message sent to a usage domain
> +  configured for the KVM guest.
> +
> +If you recall from the description of an AP Queue, AP instructions include
> +an APQN to identify the AP adapter and AP queue to which an AP command-request
> +message is to be sent (NQAP and PQAP instructions), or from which a
> +command-reply message is to be received (DQAP instruction). The validity of an
> +APQN is defined by the matrix calculated from the APM and AQM; it is the
> +intersection of all assigned adapter numbers (APM) with all assigned queue
> +indexes (AQM). For example, if adapters 1 and 2 and usage domains 5 and 6 are
> +assigned to a guest, the APQNs (1,5), (1,6), (2,5) and (2,6) will be valid for
> +the guest.
> +
> +The APQNs provide secure key functionality - i.e., a private key is stored on
> +the adapter card for each of its domains - so each APQN must be assigned to at
> +most one guest or the linux host.
> +
> +   Example 1: Valid configuration:
> +   ------------------------------
> +   Guest1: adapters 1,2  domains 5,6
> +   Guest2: adapter  1,2  domain 7
> +
> +   This is valid because both guests have a unique set of APQNs: Guest1 has
> +   APQNs (1,5), (1,6), (2,5) and (2,6); Guest2 has APQNs (1,7) and (2,7).
> +
> +   Example 2: Invalid configuration:
> +   --------------------------------
> +   Guest1: adapters 1,2  domains 5,6
> +   Guest2: adapter  1    domains 6,7
> +
> +   This is an invalid configuration because both guests have access to
> +   APQN (1,6).
> +
> +AP device Configuration on Linux Host:
> +=====================================
> +A linux system is a guest of the LPAR in which it is running and has access to
> +the AP resources configured for the LPAR. The LPAR's AP matrix is
> +configured using the 'Customize/Delete Activation Profiles' dialog from the HMC.
> +This dialog displays the activation profiles configured for the linux system.
> +Selecting the specific activation profile to be edited and clicking the
> +'Customize Profile' button will open the 'Customize Image Profiles' dialog.
> +Selecting the 'Crypto' link in the tree view on the left hand side of the dialog
> +will display the AP matrix configuration in the right hand panel. There, one can
> +assign AP adapters - called Cryptos - and domains to the LPAR. When the linux
> +system is started using this activation profile, it will have access to the
> +matrix of AP adapters and domains configured via the activation profile.
> +
> +When the linux system is started, the AP adapter devices will be connected to
> +the AP bus and the following AP matrix interfaces will be created in sysfs:
> +
> +/sys/bus/ap
> +... [devices]
> +...... xx.yyyy
> +...... ...
> +...... cardxx
> +...... ...
> +
> +Where:
> +    cardxx     is adapter number xx (in hex)
> +    yyyy       is a usage domain number yyyy (in hex)
> +....xx.yyyy    is APQN (xx,yyyy)
> +
> +For example, if AP adapters 5 and 6 and domains 4 and 71 (0x47) are configured
> +for the LPAR, the sysfs representation on the linux system would look like this:
> +
> +/sys/bus/ap
> +... [devices]
> +...... 05.0004
> +...... 05.0047
> +...... 06.0004
> +...... 06.0047
> +...... card05
> +...... card06
> +
> +There will also be AP device drivers created to control each type of AP matrix
> +interface available to the IBM Z system:
> +
> +/sys/bus/ap
> +... [drivers]
> +...... [cex2acard]        for Crypto Express 2/3 accelerator cards
> +...... [cex2aqueue]       for AP queues served by Crypto Express 2/3
> +                          accelerator cards
> +...... [cex4card]         for Crypto Express 4/5/6 accelerator and coprocessor
> +                          cards
> +...... [cex4queue]        for AP queues served by Crypto Express 4/5/6
> +                          accelerator and coprocessor cards
> +...... [pcixcccard]       for Crypto Express 2/3 coprocessor cards
> +...... [pcixccqueue]      for AP queues served by Crypto Express 2/3
> +                          coprocessor cards
> +
> +Links to the AP interfaces controlled by each AP device driver will be created
> +in the device driver's sysfs directory. For example, if AP adapter 5 and domains
> +4 and 71 (0x47) are assigned to the LPAR and adapter 5 is a CEX5 card, the
> +following links will be created in the CEX5 drivers' sysfs directories:
> +
> +/sys/bus/ap
> +... [drivers]
> +...... [cex4card]
> +......... [card05]
> +...... [cex4queue]
> +......... [05.0004]
> +......... [05.0047]
> +
> +AP Matrix Configuration for a Linux Guest:
> +=========================================
> +In order to configure the AP matrix for a guest, the adapters, usage domains
> +and control domains to be used by the guest must be assigned to the guest. This
> +section describes how to configure a guest's AP matrix.
> +
> +The kernel interfaces for configuring an AP matrix for a linux guest are built
> +on the VFIO mediated device framework and are provided by the vfio_ap
> +kernel module. By default, the vfio_ap module is a loadable module, The
> +dependency chain for the vfio_ap module is:
> +* vfio
> +* mdev
> +* vfio_mdev
> +* vfio_ap
> +
> +When installed, the vfio_ap module is initialized. During module initialization,
> +a vfio_ap driver is created and registered with the AP bus creating the
> +following sysfs interfaces:
> +
> + /sys/bus/ap/drivers/
> +...[vfio_ap]
> +...... bind
> +...... unbind
> +
> +The vfio_ap device driver will create a 'matrix' device to hold the APQNs
> +reserved for exclusive use by KVM guests:
> +
> +/sys/devices/
> +... [vfio_ap]
> +......[matrix] symlink to the matrix device directory
> +
> +The vfio_ap device driver serves several purposes:
> +1. Provides an interface for securing APQNs preventing their use by the host
> +   linux system and reserving their use by one or more guests.
> +2. Creates the sysfs interfaces for configuring an AP matrix for a linux guest.
> +
> +Securing APQNs
> +--------------
> +   An APQN is reserved by unbinding an AP queue device AP bus device driver and
> +   binding it to the vfio_ap device driver. For example, suppose we want to
> +   secure APQN (05,0004). Assuming that the AP adapter card 5 is a CEX5
> +   coprocessor card:
> +
> +   echo 05.0004 > /sys/bus/ap/drivers/cex4queue/unbind
> +   echo 05.0004 > /sys/bus/ap/drivers/vfio_ap/bind
> +
> +   This action will store the APQN in the /sys/devices/vfio_ap/matrix device
> +   which makes it available for use by a linux guest.
> +
> +Configuring an AP matrix for a linux guest.
> +------------------------------------------
> +These sysfs interfaces are built on the VFIO mediated device framework. To
> +configure an AP matrix for a guest, a mediated matrix device must first be
> +created for the /sys/devices/vfio_ap/matrix device. The sysfs interfaceAPQI corresponding to
> +for creating a mediated matrix device is in:
> +
> +/sys/devices
> +... [vfio_ap]
> +......[matrix]
> +......... [mdev_supported_types]
> +............ [vfio_ap-passthrough]
> +............... create
> +............... [devices]
> +
> +A mediated AP matrix device is created by writing a UUID to the attribute
> +file named 'create', for example:
> +
> +   uuidgen > create
> +
> +When a mediated AP matrix device is created, a sysfs directory named after
> +the UUID:
> +
> +/sys/devices
> +... [vfio_ap]
> +......[matrix]
> +......... [mdev_supported_types]
> +............ [vfio_ap-passthrough]
> +............... create
> +............... [devices]
> +.................. [$uuid]
> +
> +There will also be three sets of attribute files created in the mediated
> +matrix device's sysfs directory to configure an AP matrix for the
> +KVM guest:
> +
> +/sys/devices
> +... [vfio_ap]
> +......[matrix]
> +......... [mdev_supported_types]
> +............ [vfio_ap-passthrough]
> +............... create
> +............... [devices]
> +.................. [$uuid]
> +..................... assign_adapter
> +..................... assign_control_domain
> +..................... assign_domain
> +..................... matrix
> +..................... unassign_adapter
> +..................... unassign_control_domain
> +..................... unassign_domain
> +
> +assign_adapter
> +   To assign an AP adapter to the mediated matrix device, its APID is written
> +   'assign_adapter' file. This may be done multiple times to assign more than
> +   one adapter. The APID may be specified using conventional semantics
> +   as a decimal, hexidecimal, or octal number. For example, to assign adapters
> +   4, 5 and 16 to mediated matrix device $uuid in decimal, hexidecimal and octal
> +   respectively:
> +
> +       echo 4 > assign_adapter
> +       echo 0x5 > assign_adapter
> +       echo 020
> +
> +unassign_adapter
> +   To unassign an AP adapter, its APID is written to the 'unassign_adapter'
> +   file. This may also be done multiple times to unassign more than one adapter.
> +
> +assign_domain
> +   To assign a usage domain, the APQI corresponding to the domain number is
> +   written into the 'assign_domain' file. This may be done multiple times to
> +   assign more than one usage domain. The APQI may be specified using
> +   conventional semantics as a decimal, hexidecimal, or octal number. For
> +   example, to assign usage domains 4, 8, and 71 to mediated matrix device
> +   $uuid in decimal, hexidecimal and octal respectively:
> +
> +      echo 4 > assign_domain
> +      echo 0x8 > assign_domain
> +      echo 0107 > assign_domain
> +
> +unassign_domain
> +   To unassign a usage domain, the APQI corresponding to the domain number is
> +   written into the 'unassign_domain' file. This may be done multiple times to
> +   unassign more than one usage domain.
> +
> +assign_control_domain
> +   To assign a control domain, the domain number is written into the
> +   'assign_control_domain' file. This may be done multiple times to
> +   assign more than one control domain. The domain number may be specified using
> +   conventional semantics as a decimal, hexidecimal, or octal number. For
> +   example, to assign  control domains 4, 8, and 71 to mediated matrix device
> +   $uuid in decimal, hexidecimal and octal respectively:
> +
> +      echo 4 > assign_domain
> +      echo 0x8 > assign_domain
> +      echo 0107 > assign_domain
> +
> +unassign_control_domain
> +   To unassign a control domain, the domain number is written into the
> +   'unassign_domain' file. This may be done multiple times to unassign more than
> +   one control domain.
> +
> +Notes:
> +* Hot plug/unplug is not currently supported for mediated AP matrix devices,
> +  so the AP matrix resulting from assignment and/or unassignment of AP
> +  adapters, usage domains and control domains to a mediated AP matrix device
> +  while the guest is running will not take affect until the linux guest is
> +  rebooted.
> +* By architectural convention, all usage domains configured for a KVM guest
> +  will also be implicitly assigned as control domains also, to there is no
> +  need to assign control domains that are assigned as usage domains.
> +
> +Starting a Linux Guest Configured with an AP Matrix:
> +===================================================
> +In addition to providing the sysfs interfaces for configuring the AP matrix for
> +a linux guest, a mediated matrix device also acts as a communication pathway
> +between QEMU and the vfio_ap device driver. To gain access to the
> +device driver, the following option must be specified on the QEMU command line:
> +
> +   -device vfio_ap,sysfsdev=$path-to-mdev
> +
> +The sysfsdev parameter specifies the path to the mediated matrix device.
> +There are a number of ways to specify this path:
> +
> +/sys/devices/vfio_ap/matrix/$uuid
> +/sys/bus/mdev/devices/$uuid
> +/sys/bus/mdev/drivers/vfio_mdev/$uuid
> +/sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/devices/$uuid
> +
> +When the linux guest is subsequently started, the guest will open the mediated
> +matrix device's file descriptor to get information about the mediated matrix
> +device. The vfio_ap device driver will update the APM, AQM, and ADM fields in the
> +guest's CRYCB with the adapter, usage domain and control domains assigned to
> +via the mediated matrix device's sysfs attribute files. Programs running on the
> +linux guest will then:
> +
> +1. Have direct access to the APQNs derived from the intersection of the AP
> +   adapter and usage domain numbers specified in the APM and AQM respectively
> +
> +2. Have authorization to process AP commands to change - e.g., store a new
> +   secure key - a control domain identified in an AP instruction sent to a valid
> +   APQN.
> +
> +CPU model features:
> +
> +Three CPU model features are available for controlling guest access to AP
> +facilities:
> +
> +1. AP facilities feature
> +
> +   The AP facilities feature indicates that AP facilities are installed on the
> +   guest. This feature will be enabled by the kernel only if the AP facilities
> +   are installed on the host system. It will turned on automatically for guests
> +   started with CPU model zEC12 or newer. The feature is s390-specific and is
> +   represented as a parameter of the -cpu option on the QEMU command line:
> +
> +      qemu-system-s390x -cpu $model,ap=on|off
> +
> +      Where:
> +
> +         $model is the CPU model defined for the guest (defaults to the model of
> +                the host system if not specified).
> +
> +         ap=on|off indicates whether AP facilities are installed (on) or not
> +                   installed (off). The default for CPU models zEC12 or newer
> +                   is ap=on. AP facilities must be installed when this parameter
> +                   is used in conjunction with -device vfio-ap,sysfsdev=$path or
> +                   the guest will not start.
> +
> +2. Query Configuration Information (QCI) facility
> +
> +   The QCI facility is used by the AP bus running on the guest to query the
> +   configuration of the AP facilities. This facility will be enabled by
> +   the kernel only if the QCI facility is installed on the host system. It will
> +   be turned on automatically for guests started with CPU model zEC12 or newer.
> +   The feature is s390-specific and is represented as a parameter of the -cpu
> +   option on the QEMU command line:
> +
> +      qemu-system-s390x -cpu $model,qci=on|off
> +
> +      Where:
> +
> +         $model is the CPU model defined for the guest
> +
> +         qci=on|off indicates whether the QCI facility is installed (on) or not
> +                    installed (off). The default for CPU models zEC12 or newer
> +                    is qci=on. Turning the QCI facility on makes no sense if it
> +                    is not used in conjunction with the
> +                    '-device vfio-ap,sysfsdev=$path' option. A warning will be
> +                    presented if QCI is turned on and the AP facilities are not
> +                    installed.
> +
> +                    If the QCI facility is turned off, APQNs with an APQI
> +                    greater than 15 will not be accessible from the guest.
> +
> +3. Adjunct Process Facility Test (APFT) facility
> +
> +   The APFT facility is used by the AP bus running on the guest to test the
> +   AP facilities available for a given AP queue. This facility will be enabled
> +   by the kernel only if the APFT facility is installed on the host system. It
> +   will be turned on automatically for guests started with CPU model zEC12 or
> +   newer. The feature is s390-specific and is represented as a parameter of the
> +   -cpu option on the QEMU command line:
> +
> +      qemu-system-s390x -cpu $model,apft=on|off
> +
> +      Where:
> +
> +         $model is the CPU model defined for the guest (defaults to the model of
> +                the host system if not specified).
> +
> +         apft=on|off indicates whether the APFT facility is installed (on) or
> +                     not installed (off). The default for CPU models zEC12 and
> +                     newer is apft=on. Turning the APFT facility on makes no
> +                     sense if it is not used in conjunction with the
> +                     -device vfio-ap,sysfsdev=$path option. A warning will be
> +                     presented if APFT is turned on and the AP facilities are
> +                     not installed.
> +
> +                     It also makes no sense to turn APFT off when used in
> +                     conjunction with the vfio-ap device because the APFT
> +                     facility is required; the AP bus running on the guest will
> +                     not detect CEX4 and newer devices without it. Since only
> +                     CEX4 and newer devices are supported for guest usage, no AP
> +                     devices can be made accessible to a guest started without
> +                     APFT installed.
> +
> +Example: Configure AP Matrixes for Two Linux Guests:
> +===================================================
> +Let's now provide an example to illustrate how KVM guests may be given
> +access to AP facilities. For this example, we will show how to configure
> +two guests such that executing the lszcrypt command on the guests would
> +look like this:
> +
> +Guest1
> +------
> +CARD.DOMAIN TYPE  MODE
> +------------------------------
> +05          CEX5C CCA-Coproc
> +05.0004     CEX5C CCA-Coproc
> +05.00ab     CEX5C CCA-Coproc
> +06          CEX5A Accelerator
> +06.0004     CEX5A Accelerator
> +06.00ab     CEX5C CCA-Coproc
> +

5 whitespace at the end of lines

> +Guest2
> +------
> +CARD.DOMAIN TYPE  MODE
> +------------------------------
> +05          CEX5A Accelerator
> +05.0047     CEX5A Accelerator
> +05.00ff     CEX5A Accelerator
> +
> +These are the steps for configuring the Guest1 and Guest2:
> +
> +1. The first thing that needs to be done is to secure the AP queues to be
> +   used by the two guests so that the host can not access them. This is done
> +   by unbinding each AP Queue device from its respective AP driver. In our
> +   example, these queues are bound to the cex4queue driver. This would be
> +   the sysfs location of these devices:
> +
> +   /sys/bus/ap
> +   --- [drivers]
> +   ------ [cex4queue]
> +   --------- [05.0004]
> +   --------- [05.0047]   --------------------- control_domains
> +   --------------------- domains
> +   --------- [05.00ab]
> +   --------- [05.00ff]
> +   --------- [06.0004]
> +   --------- [06.00ab]
> +   --------- unbind
> +
> +   To unbind AP queue 05.0004 from the cex4queue device driver:
> +
> +    echo 05.0004 > unbind
> +
> +   This must also be done for AP queues 05.00ab, 05.0047, 05.00ff, 06.0004,
> +   and 06.00ab.
> +
> +2. The next step is to reserve the queues for use by the two KVM guests.
> +   This is accomplished by binding them to the VFIO AP device driver.
> +   This is the sysfs location of the VFIO AP device driver:
> +
> +   /sys/bus/ap
> +   ---[drivers]
> +   ------ [vfio_ap]
> +   ---------- bind
> +
> +   To bind queue 05.0004 to the vfio_ap driver:
> +
> +    echo 05.0004 > bind
> +
> +   This must also be done for AP queues 05.00ab, 05.0047, 05.00ff, 06.0004,
> +   and 06.00ab.
> +
> +3. Create the mediated devices needed to configure the AP matrixes for the
> +   two guests and to provide an interface to the vfio_ap driver for
> +   use by the guests:
> +
> +   /sys/devices/
> +   --- [vfio_ap]
> +   ------ [matrix] (this is the matrix device)
> +   --------- [mdev_supported_types]
> +   ------------ [vfio_ap-passthrough] (passthrough mediated matrix device type)
> +   --------------- create
> +   --------------- [devices]
> +
> +   To create the mediated devices for the two guests:
> +
> +    uuidgen > create
> +    uuidgen > create
> +
> +   This will create two mediated devices in the [devices] subdirectory named
> +   with the UUID written to the create attribute file. We call them $uuid1
> +   and $uuid2:
> +
> +   /sys/devices/
> +   --- [vfio_ap]
> +   ------ [matrix]
> +   --------- [mdev_supported_types]
> +   ------------ [vfio_ap-passthrough]
> +   --------------- [devices]
> +   ------------------ [$uuid1]
> +   --------------------- assign_adapter
> +   --------------------- assign_control_domain
> +   --------------------- assign_domain
> +   --------------------- matrix
> +   --------------------- unassign_adapter
> +   --------------------- unassign_control_domain
> +   --------------------- unassign_domain
> +
> +   ------------------ [$uuid2]
> +   --------------------- assign_adapter
> +   --------------------- assign_cTo assign an adapter, the APID of the adapter is written to the
> +      file. ontrol_domain
> +   --------------------- assign_domain
> +   --------------------- matrix
> +   --------------------- unassign_adapter
> +   --------------------- unassign_control_domain
> +   --------------------- unassign_domain
> +
> +4. The administrator now needs to configure the matrixes for mediated
> +   devices $uuid1 (for Guest1) and $uuid2 (for Guest2).
> +
> +   This is how the matrix is configured for Guest1:
> +
> +   echo 5 > assign_adapter
> +   echo 6 > assign_adapter
> +   echo 4 > assign_domain
> +   echo 0xab > assign_domain
> +
> +   By architectural convention, all usage domains - i.e., domains assigned
> +   via the assign_domain attribute file - will also be configured in the ADM
> +   field of the KVM guest's CRYCB, so there is no need to assign control
> +   domains here unless you want to assign control domains that are not
> +   assigned as usage domains.
> +
> +   If a mistake is made configuring an adapter, domain or control domain,
> +   you can use the unassign_xxx files to unassign the adapter, domain or
> +   control domain.
> +
> +   To display the matrix configuration for Guest1:
> +
> +   cat matrix
> +
> +   This is how the matrix is configured for Guest2:
> +
> +   echo 5 > assign_adapter
> +   echo 0x47 > assign_domain
> +   echo 0xff > assign_domain
> +
> +5. Start Guest1
> +
> +   /usr/bin/qemu-system-s390x ... -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid1 ...
> +
> +6. Start Guest2
> +
> +   /usr/bin/qemu-system-s390x ... -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid2 ...


-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [Qemu-devel] [PATCH v3 2/7] s390x/ap: base Adjunct Processor (AP) object

[Pierre Morel]

On 16/03/2018 00:24, Tony Krowiak wrote:
> This patch introduces the base object for an AP device.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>   hw/s390x/Makefile.objs       |    1 +
>   hw/s390x/ap-device.c         |   38 ++++++++++++++++++++++++++++++++++++++
>   include/hw/s390x/ap-device.h |   38 ++++++++++++++++++++++++++++++++++++++
>   3 files changed, 77 insertions(+), 0 deletions(-)
>   create mode 100644 hw/s390x/ap-device.c
>   create mode 100644 include/hw/s390x/ap-device.h
>
> diff --git a/hw/s390x/Makefile.objs b/hw/s390x/Makefile.objs
> index dc704b5..3247a07 100644
> --- a/hw/s390x/Makefile.objs
> +++ b/hw/s390x/Makefile.objs
> @@ -17,3 +17,4 @@ obj-y += s390-stattrib.o
>   obj-$(CONFIG_KVM) += s390-skeys-kvm.o
>   obj-$(CONFIG_KVM) += s390-stattrib-kvm.o
>   obj-y += s390-ccw.o
> +obj-y += ap-device.o
> diff --git a/hw/s390x/ap-device.c b/hw/s390x/ap-device.c
> new file mode 100644
> index 0000000..448f6db
> --- /dev/null
> +++ b/hw/s390x/ap-device.c
> @@ -0,0 +1,38 @@
> +/*
> + * Adjunct Processor (AP) matrix device
> + *
> + * Copyright 2018 IBM Corp.
> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or (at
> + * your option) any later version. See the COPYING file in the top-level
> + * directory.
> + */
> +#include "qemu/osdep.h"
> +#include "qemu/module.h"
> +#include "qapi/error.h"
> +#include "hw/qdev.h"
> +#include "hw/s390x/ap-device.h"
> +
> +static void ap_class_init(ObjectClass *klass, void *data)
> +{
> +    DeviceClass *dc = DEVICE_CLASS(klass);
> +
> +    dc->desc = "AP device class";

Shouldn't we make the AP devices unpluggable ?

> +}
> +
> +static const TypeInfo ap_device_info = {
> +    .name = AP_DEVICE_TYPE,
> +    .parent = TYPE_DEVICE,
> +    .instance_size = sizeof(APDevice),
> +    .class_size = sizeof(APDeviceClass),
> +    .class_init = ap_class_init,
> +    .abstract = true,
> +};
> +
> +static void ap_device_register(void)
> +{
> +    type_register_static(&ap_device_info);
> +}
> +
> +type_init(ap_device_register)
> diff --git a/include/hw/s390x/ap-device.h b/include/hw/s390x/ap-device.h
> new file mode 100644
> index 0000000..693df90
> --- /dev/null
> +++ b/include/hw/s390x/ap-device.h
> @@ -0,0 +1,38 @@
> +/*
> + * Adjunct Processor (AP) matrix device interfaces
> + *
> + * Copyright 2018 IBM Corp.
> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or (at
> + * your option) any later version. See the COPYING file in the top-level
> + * directory.
> + */
> +#ifndef HW_S390X_AP_DEVICE_H
> +#define HW_S390X_AP_DEVICE_H
> +
> +#define AP_DEVICE_TYPE       "ap-device"
> +
> +typedef struct APDevice {
> +    DeviceState parent_obj;
> +} APDevice;
> +
> +typedef struct APDeviceClass {
> +    DeviceClass parent_class;
> +} APDeviceClass;
> +
> +static inline APDevice *to_ap_dev(DeviceState *dev)
> +{
> +    return container_of(dev, APDevice, parent_obj);
> +}
> +
> +#define AP_DEVICE(obj) \
> +    OBJECT_CHECK(APDevice, (obj), AP_DEVICE_TYPE)
> +
> +#define AP_DEVICE_GET_CLASS(obj) \
> +    OBJECT_GET_CLASS(APDeviceClass, (obj), AP_DEVICE_TYPE)
> +
> +#define AP_DEVICE_CLASS(klass) \
> +    OBJECT_CLASS_CHECK(APDeviceClass, (klass), AP_DEVICE_TYPE)
> +
> +#endif /* HW_S390X_AP_DEVICE_H */


-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [Qemu-devel] [PATCH v3 4/7] s390x/kvm: interface to interpret AP instructions

[Pierre Morel]

On 16/03/2018 00:24, Tony Krowiak wrote:
> The VFIO AP device exploits interpretive execution of AP
> instructions (APIE). APIE is enabled by setting a device attribute
> via the KVM_SET_DEVICE_ATTR ioctl.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>   target/s390x/kvm.c       |   16 ++++++++++++++++
>   target/s390x/kvm_s390x.h |    2 ++
>   2 files changed, 18 insertions(+), 0 deletions(-)
>
> diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c
> index 33e5ec3..2812e28 100644
> --- a/target/s390x/kvm.c
> +++ b/target/s390x/kvm.c
> @@ -277,6 +277,22 @@ static void kvm_s390_init_dea_kw(void)
>       }
>   }
>
> +int kvm_s390_set_interpret_ap(uint8_t enable)
> +{
> +    struct kvm_device_attr attribute = {
> +        .group = KVM_S390_VM_CRYPTO,
> +        .attr  = KVM_S390_VM_CRYPTO_INTERPRET_AP,
> +        .addr = 1,
> +    };
> +
> +    if (!kvm_vm_check_attr(kvm_state, KVM_S390_VM_CRYPTO,
> +                           KVM_S390_VM_CRYPTO_INTERPRET_AP)) {

Isn't it enough to have the CPU feature ?
What are expecting here?

> +        return -EOPNOTSUPP;
> +    }
> +
> +    return kvm_vm_ioctl(kvm_state, KVM_SET_DEVICE_ATTR, &attribute);

shouldn't you use the "enable" parameter somewhere?

> +}
> +
>   void kvm_s390_crypto_reset(void)
>   {
>       if (s390_has_feat(S390_FEAT_MSA_EXT_3)) {
> diff --git a/target/s390x/kvm_s390x.h b/target/s390x/kvm_s390x.h
> index 34ee7e7..0d6c6e7 100644
> --- a/target/s390x/kvm_s390x.h
> +++ b/target/s390x/kvm_s390x.h
> @@ -40,4 +40,6 @@ void kvm_s390_crypto_reset(void);
>   void kvm_s390_restart_interrupt(S390CPU *cpu);
>   void kvm_s390_stop_interrupt(S390CPU *cpu);
>
> +int kvm_s390_set_interpret_ap(uint8_t enable);
> +
>   #endif /* KVM_S390X_H */



-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [Qemu-devel] [PATCH v3 4/7] s390x/kvm: interface to interpret AP instructions

[Pierre Morel]

On 16/03/2018 00:24, Tony Krowiak wrote:
> The VFIO AP device exploits interpretive execution of AP
> instructions (APIE). APIE is enabled by setting a device attribute
> via the KVM_SET_DEVICE_ATTR ioctl.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>   target/s390x/kvm.c       |   16 ++++++++++++++++
>   target/s390x/kvm_s390x.h |    2 ++
>   2 files changed, 18 insertions(+), 0 deletions(-)
>
> diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c
> index 33e5ec3..2812e28 100644
> --- a/target/s390x/kvm.c
> +++ b/target/s390x/kvm.c
> @@ -277,6 +277,22 @@ static void kvm_s390_init_dea_kw(void)
>       }
>   }
>
> +int kvm_s390_set_interpret_ap(uint8_t enable)
> +{
> +    struct kvm_device_attr attribute = {
> +        .group = KVM_S390_VM_CRYPTO,
> +        .attr  = KVM_S390_VM_CRYPTO_INTERPRET_AP,
> +        .addr = 1,
> +    };
> +
> +    if (!kvm_vm_check_attr(kvm_state, KVM_S390_VM_CRYPTO,
> +                           KVM_S390_VM_CRYPTO_INTERPRET_AP)) {

Isn't it enough to have the CPU feature ?
What are expecting here?

> +        return -EOPNOTSUPP;
> +    }
> +
> +    return kvm_vm_ioctl(kvm_state, KVM_SET_DEVICE_ATTR, &attribute);

Shouldn't you use the "enable" parameter somewhere?

> +}
> +
>   void kvm_s390_crypto_reset(void)
>   {
>       if (s390_has_feat(S390_FEAT_MSA_EXT_3)) {
> diff --git a/target/s390x/kvm_s390x.h b/target/s390x/kvm_s390x.h
> index 34ee7e7..0d6c6e7 100644
> --- a/target/s390x/kvm_s390x.h
> +++ b/target/s390x/kvm_s390x.h
> @@ -40,4 +40,6 @@ void kvm_s390_crypto_reset(void);
>   void kvm_s390_restart_interrupt(S390CPU *cpu);
>   void kvm_s390_stop_interrupt(S390CPU *cpu);
>
> +int kvm_s390_set_interpret_ap(uint8_t enable);
> +
>   #endif /* KVM_S390X_H */


-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [Qemu-devel] [PATCH v3 2/7] s390x/ap: base Adjunct Processor (AP) object

[Pierre Morel]

On 16/03/2018 00:24, Tony Krowiak wrote:
> This patch introduces the base object for an AP device.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>   hw/s390x/Makefile.objs       |    1 +
>   hw/s390x/ap-device.c         |   38 ++++++++++++++++++++++++++++++++++++++
>   include/hw/s390x/ap-device.h |   38 ++++++++++++++++++++++++++++++++++++++
>   3 files changed, 77 insertions(+), 0 deletions(-)
>   create mode 100644 hw/s390x/ap-device.c
>   create mode 100644 include/hw/s390x/ap-device.h
>
> diff --git a/hw/s390x/Makefile.objs b/hw/s390x/Makefile.objs
> index dc704b5..3247a07 100644
> --- a/hw/s390x/Makefile.objs
> +++ b/hw/s390x/Makefile.objs
> @@ -17,3 +17,4 @@ obj-y += s390-stattrib.o
>   obj-$(CONFIG_KVM) += s390-skeys-kvm.o
>   obj-$(CONFIG_KVM) += s390-stattrib-kvm.o
>   obj-y += s390-ccw.o
> +obj-y += ap-device.o
> diff --git a/hw/s390x/ap-device.c b/hw/s390x/ap-device.c
> new file mode 100644
> index 0000000..448f6db
> --- /dev/null
> +++ b/hw/s390x/ap-device.c
> @@ -0,0 +1,38 @@
> +/*
> + * Adjunct Processor (AP) matrix device
> + *
> + * Copyright 2018 IBM Corp.
> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or (at
> + * your option) any later version. See the COPYING file in the top-level
> + * directory.
> + */
> +#include "qemu/osdep.h"
> +#include "qemu/module.h"
> +#include "qapi/error.h"
> +#include "hw/qdev.h"
> +#include "hw/s390x/ap-device.h"
> +
> +static void ap_class_init(ObjectClass *klass, void *data)
> +{
> +    DeviceClass *dc = DEVICE_CLASS(klass);
> +
> +    dc->desc = "AP device class";

Shouldn't we make the AP devices unpluggable ?

> +}
> +
> +static const TypeInfo ap_device_info = {
> +    .name = AP_DEVICE_TYPE,
> +    .parent = TYPE_DEVICE,
> +    .instance_size = sizeof(APDevice),
> +    .class_size = sizeof(APDeviceClass),
> +    .class_init = ap_class_init,
> +    .abstract = true,
> +};
> +
> +static void ap_device_register(void)
> +{
> +    type_register_static(&ap_device_info);
> +}
> +
> +type_init(ap_device_register)
> diff --git a/include/hw/s390x/ap-device.h b/include/hw/s390x/ap-device.h
> new file mode 100644
> index 0000000..693df90
> --- /dev/null
> +++ b/include/hw/s390x/ap-device.h
> @@ -0,0 +1,38 @@
> +/*
> + * Adjunct Processor (AP) matrix device interfaces
> + *
> + * Copyright 2018 IBM Corp.
> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or (at
> + * your option) any later version. See the COPYING file in the top-level
> + * directory.
> + */
> +#ifndef HW_S390X_AP_DEVICE_H
> +#define HW_S390X_AP_DEVICE_H
> +
> +#define AP_DEVICE_TYPE       "ap-device"
> +
> +typedef struct APDevice {
> +    DeviceState parent_obj;
> +} APDevice;
> +
> +typedef struct APDeviceClass {
> +    DeviceClass parent_class;
> +} APDeviceClass;
> +
> +static inline APDevice *to_ap_dev(DeviceState *dev)
> +{
> +    return container_of(dev, APDevice, parent_obj);
> +}
> +
> +#define AP_DEVICE(obj) \
> +    OBJECT_CHECK(APDevice, (obj), AP_DEVICE_TYPE)
> +
> +#define AP_DEVICE_GET_CLASS(obj) \
> +    OBJECT_GET_CLASS(APDeviceClass, (obj), AP_DEVICE_TYPE)
> +
> +#define AP_DEVICE_CLASS(klass) \
> +    OBJECT_CLASS_CHECK(APDeviceClass, (klass), AP_DEVICE_TYPE)
> +
> +#endif /* HW_S390X_AP_DEVICE_H */


-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [Qemu-devel] [PATCH v3 5/7] s390x/vfio: ap: Introduce VFIO AP device

[Pierre Morel]

On 16/03/2018 00:24, Tony Krowiak wrote:
> Introduces a VFIO based AP device. The device is defined via
> the QEMU command line by specifying:
>
>      -device vfio-ap,sysfsdev=<path-to-mediated-matrix-device>
>
> The mediated matrix device is created by the VFIO AP device
> driver by writing a UUID to a sysfs attribute file (see
> docs/vfio-ap.txt). The mediated matrix device will be named
> after the UUID. Symbolic links to the $uuid are created in
> many places, so the path to the mediated matrix device $uuid
> can be specified in any of the following ways:
>
> /sys/devices/vfio_ap/matrix/$uuid
> /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/devices/$uuid
> /sys/bus/mdev/devices/$uuid
> /sys/bus/mdev/drivers/vfio_mdev/$uuid
>
> When the vfio-ap device is realized, it acquires and opens the
> VFIO iommu group to which the mediated matrix device is
> bound. This causes a VFIO group notification event to be
> signaled. The vfio_ap device driver's group notification
> handler will get called at which time the device driver
> will configure the the AP devices to which the guest will
> be granted access.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>   default-configs/s390x-softmmu.mak |    1 +
>   hw/vfio/Makefile.objs             |    1 +
>   hw/vfio/ap.c                      |  184 +++++++++++++++++++++++++++++++++++++
>   include/hw/vfio/vfio-common.h     |    1 +
>   4 files changed, 187 insertions(+), 0 deletions(-)
>   create mode 100644 hw/vfio/ap.c
>
> diff --git a/default-configs/s390x-softmmu.mak b/default-configs/s390x-softmmu.mak
> index 2f4bfe7..0b784b6 100644
> --- a/default-configs/s390x-softmmu.mak
> +++ b/default-configs/s390x-softmmu.mak
> @@ -9,3 +9,4 @@ CONFIG_S390_FLIC=y
>   CONFIG_S390_FLIC_KVM=$(CONFIG_KVM)
>   CONFIG_VFIO_CCW=$(CONFIG_LINUX)
>   CONFIG_WDT_DIAG288=y
> +CONFIG_VFIO_AP=$(CONFIG_LINUX)
> diff --git a/hw/vfio/Makefile.objs b/hw/vfio/Makefile.objs
> index c3ab909..7300860 100644
> --- a/hw/vfio/Makefile.objs
> +++ b/hw/vfio/Makefile.objs
> @@ -6,4 +6,5 @@ obj-$(CONFIG_SOFTMMU) += platform.o
>   obj-$(CONFIG_VFIO_XGMAC) += calxeda-xgmac.o
>   obj-$(CONFIG_VFIO_AMD_XGBE) += amd-xgbe.o
>   obj-$(CONFIG_SOFTMMU) += spapr.o
> +obj-$(CONFIG_VFIO_AP) += ap.o
>   endif
> diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c
> new file mode 100644
> index 0000000..b397bb1
> --- /dev/null
> +++ b/hw/vfio/ap.c
> @@ -0,0 +1,184 @@
> +/*
> + * VFIO based AP matrix device assignment
> + *
> + * Copyright 2018 IBM Corp.
> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or (at
> + * your option) any later version. See the COPYING file in the top-level
> + * directory.
> + */
> +
> +#include <linux/vfio.h>
> +#include <sys/ioctl.h>
> +#include "qemu/osdep.h"
> +#include "qapi/error.h"
> +#include "hw/sysbus.h"
> +#include "hw/vfio/vfio.h"
> +#include "hw/vfio/vfio-common.h"
> +#include "hw/s390x/ap-device.h"
> +#include "qemu/error-report.h"
> +#include "qemu/queue.h"
> +#include "cpu.h"
> +#include "kvm_s390x.h"
> +
> +#define VFIO_AP_DEVICE_TYPE      "vfio-ap"
> +
> +typedef struct VFIOAPDevice {
> +    APDevice apdev;
> +    VFIODevice vdev;
> +    QTAILQ_ENTRY(VFIOAPDevice) sibling;
> +} VFIOAPDevice;
> +
> +static void vfio_ap_compute_needs_reset(VFIODevice *vdev)
> +{
> +    vdev->needs_reset = false;
> +}
> +
> +/*
> + * We don't need vfio_hot_reset_multi and vfio_eoi operations for
> + * vfio-ap-matrix device now.
> + */
> +struct VFIODeviceOps vfio_ap_ops = {
> +    .vfio_compute_needs_reset = vfio_ap_compute_needs_reset,
> +};
> +
> +static QTAILQ_HEAD(, VFIOAPDevice) vfio_ap_devices =
> +        QTAILQ_HEAD_INITIALIZER(vfio_ap_devices);
> +
> +static void vfio_put_device(VFIOAPDevice *apdev)
> +{
> +    g_free(apdev->vdev.name);
> +    vfio_put_base_device(&apdev->vdev);
> +}
> +
> +static VFIOGroup *vfio_ap_get_group(VFIOAPDevice *vapdev, Error **errp)
> +{
> +    char *tmp, group_path[PATH_MAX];
> +    ssize_t len;
> +    int groupid;
> +
> +    tmp = g_strdup_printf("%s/iommu_group", vapdev->vdev.sysfsdev);
> +    len = readlink(tmp, group_path, sizeof(group_path));
> +    g_free(tmp);
> +
> +    if (len <= 0 || len >= sizeof(group_path)) {
> +        error_setg(errp, "%s: no iommu_group found for %s",
> +                   VFIO_AP_DEVICE_TYPE, vapdev->vdev.sysfsdev);
> +        return NULL;
> +    }
> +
> +    group_path[len] = 0;
> +
> +    if (sscanf(basename(group_path), "%d", &groupid) != 1) {
> +        error_setg(errp, "vfio: failed to read %s", group_path);
> +        return NULL;
> +    }
> +
> +    return vfio_get_group(groupid, &address_space_memory, errp);
> +}
> +
> +static void vfio_ap_realize(DeviceState *dev, Error **errp)
> +{
> +    VFIODevice *vbasedev;
> +    VFIOGroup *vfio_group;
> +    APDevice *apdev = DO_UPCAST(APDevice, parent_obj, dev);
> +    VFIOAPDevice *vapdev = DO_UPCAST(VFIOAPDevice, apdev, apdev);
> +    char *mdevid;
> +    Error *local_err = NULL;
> +    int ret;
> +
> +    if (!s390_has_feat(S390_FEAT_AP)) {
> +        error_setg(&local_err, "AP support not enabled");
> +        goto out_err;
> +    }
> +
> +    ret = kvm_s390_set_interpret_ap(1);

If we have several devices, this is called once per device.
However it is a general switch for the VM.
It looks strange to have this inside the realize callback

> +    if (ret) {
> +        error_setg_errno(&local_err, errno,
> +                         "error setting interpretive execution of AP instructions");
> +        goto out_err;
> +    }
> +
> +    vfio_group = vfio_ap_get_group(vapdev, &local_err);
> +    if (!vfio_group) {
> +        goto out_group_err;
> +    }
> +
> +    vapdev->vdev.ops = &vfio_ap_ops;
> +    vapdev->vdev.type = VFIO_DEVICE_TYPE_AP;
> +    mdevid = basename(vapdev->vdev.sysfsdev);
> +    vapdev->vdev.name = g_strdup_printf("%s", mdevid);
> +    vapdev->vdev.dev = dev;
> +    QLIST_FOREACH(vbasedev, &vfio_group->device_list, next) {
> +        if (strcmp(vbasedev->name, vapdev->vdev.name) == 0) {
> +            error_setg(&local_err,
> +                       "%s: AP device %s has already been realized",
> +                       VFIO_AP_DEVICE_TYPE, vapdev->vdev.name);
> +            goto out_device_err;
> +        }
> +    }
> +
> +    ret = vfio_get_device(vfio_group, mdevid, &vapdev->vdev, &local_err);
> +    if (ret) {
> +        goto out_device_err;
> +    }
> +
> +    QTAILQ_INSERT_TAIL(&vfio_ap_devices, vapdev, sibling);
> +
> +    return;
> +
> +
> +out_device_err:
> +    vfio_put_group(vfio_group);
> +out_group_err:
> +    kvm_s390_set_interpret_ap(0);
> +out_err:
> +    error_propagate(errp, local_err);
> +}
> +
> +static void vfio_ap_unrealize(DeviceState *dev, Error **errp)
> +{
> +    APDevice *apdev = DO_UPCAST(APDevice, parent_obj, dev);
> +    VFIOAPDevice *vapdev = DO_UPCAST(VFIOAPDevice, apdev, apdev);
> +    VFIOGroup *group = vapdev->vdev.group;
> +
> +    vfio_put_device(vapdev);
> +    vfio_put_group(group);
> +    kvm_s390_set_interpret_ap(0);
> +}
> +
> +static Property vfio_ap_properties[] = {
> +    DEFINE_PROP_STRING("sysfsdev", VFIOAPDevice, vdev.sysfsdev),
> +    DEFINE_PROP_END_OF_LIST(),
> +};
> +
> +static const VMStateDescription vfio_ap_vmstate = {
> +    .name = VFIO_AP_DEVICE_TYPE,
> +    .unmigratable = 1,
> +};
> +
> +static void vfio_ap_class_init(ObjectClass *klass, void *data)
> +{
> +    DeviceClass *dc = DEVICE_CLASS(klass);
> +
> +    dc->props = vfio_ap_properties;
> +    dc->vmsd = &vfio_ap_vmstate;
> +    dc->desc = "VFIO-based AP device assignment";
> +    dc->realize = vfio_ap_realize;
> +    dc->unrealize = vfio_ap_unrealize;


Shouldn't we make the VFIO-AP devices unpluggable
in the case the AP device are pluggable?

> +}
> +
> +static const TypeInfo vfio_ap_info = {
> +    .name = VFIO_AP_DEVICE_TYPE,
> +    .parent = AP_DEVICE_TYPE,
> +    .instance_size = sizeof(VFIOAPDevice),
> +    .class_init = vfio_ap_class_init,
> +};
> +
> +static void register_vfio_ap_type(void)
> +{
> +    type_register_static(&vfio_ap_info);
> +}
> +
> +type_init(register_vfio_ap_type)
> diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h
> index f3a2ac9..f1f22d9 100644
> --- a/include/hw/vfio/vfio-common.h
> +++ b/include/hw/vfio/vfio-common.h
> @@ -46,6 +46,7 @@ enum {
>       VFIO_DEVICE_TYPE_PCI = 0,
>       VFIO_DEVICE_TYPE_PLATFORM = 1,
>       VFIO_DEVICE_TYPE_CCW = 2,
> +    VFIO_DEVICE_TYPE_AP = 3,
>   };
>
>   typedef struct VFIOMmap {


-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [Qemu-devel] [PATCH v3 5/7] s390x/vfio: ap: Introduce VFIO AP device

[Halil Pasic]

On 03/16/2018 11:42 AM, Pierre Morel wrote:
> On 16/03/2018 00:24, Tony Krowiak wrote:
>> Introduces a VFIO based AP device. The device is defined via
>> the QEMU command line by specifying:
>>
>>      -device vfio-ap,sysfsdev=<path-to-mediated-matrix-device>
>>
>> The mediated matrix device is created by the VFIO AP device
>> driver by writing a UUID to a sysfs attribute file (see
>> docs/vfio-ap.txt). The mediated matrix device will be named
>> after the UUID. Symbolic links to the $uuid are created in
>> many places, so the path to the mediated matrix device $uuid
>> can be specified in any of the following ways:
>>
>> /sys/devices/vfio_ap/matrix/$uuid
>> /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/devices/$uuid
>> /sys/bus/mdev/devices/$uuid
>> /sys/bus/mdev/drivers/vfio_mdev/$uuid
>>
>> When the vfio-ap device is realized, it acquires and opens the
>> VFIO iommu group to which the mediated matrix device is
>> bound. This causes a VFIO group notification event to be
>> signaled. The vfio_ap device driver's group notification
>> handler will get called at which time the device driver
>> will configure the the AP devices to which the guest will
>> be granted access.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
[..]
>> +static void vfio_ap_realize(DeviceState *dev, Error **errp)
>> +{
>> +    VFIODevice *vbasedev;
>> +    VFIOGroup *vfio_group;
>> +    APDevice *apdev = DO_UPCAST(APDevice, parent_obj, dev);
>> +    VFIOAPDevice *vapdev = DO_UPCAST(VFIOAPDevice, apdev, apdev);
>> +    char *mdevid;
>> +    Error *local_err = NULL;
>> +    int ret;
>> +
>> +    if (!s390_has_feat(S390_FEAT_AP)) {
>> +        error_setg(&local_err, "AP support not enabled");
>> +        goto out_err;
>> +    }
>> +
>> +    ret = kvm_s390_set_interpret_ap(1);
> 
> If we have several devices, this is called once per device.

I don't think having several of these in a single vm makes
any sense. Or does it? IMHO we should make sure there is at
most one device taking care of the crypto pass-through.

> However it is a general switch for the VM.
> It looks strange to have this inside the realize callback
> 

I kind of semi agree.

The previous solution (having this transparent for QEMU) had
benefits. Why did we move away from that again?

Regards,
Halil

Re: [Qemu-devel] [PATCH v3 2/7] s390x/ap: base Adjunct Processor (AP) object

[Tony Krowiak]

On 03/16/2018 06:38 AM, Pierre Morel wrote:
> On 16/03/2018 00:24, Tony Krowiak wrote:
>> This patch introduces the base object for an AP device.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>>   hw/s390x/Makefile.objs       |    1 +
>>   hw/s390x/ap-device.c         |   38 
>> ++++++++++++++++++++++++++++++++++++++
>>   include/hw/s390x/ap-device.h |   38 
>> ++++++++++++++++++++++++++++++++++++++
>>   3 files changed, 77 insertions(+), 0 deletions(-)
>>   create mode 100644 hw/s390x/ap-device.c
>>   create mode 100644 include/hw/s390x/ap-device.h
>>
>> diff --git a/hw/s390x/Makefile.objs b/hw/s390x/Makefile.objs
>> index dc704b5..3247a07 100644
>> --- a/hw/s390x/Makefile.objs
>> +++ b/hw/s390x/Makefile.objs
>> @@ -17,3 +17,4 @@ obj-y += s390-stattrib.o
>>   obj-$(CONFIG_KVM) += s390-skeys-kvm.o
>>   obj-$(CONFIG_KVM) += s390-stattrib-kvm.o
>>   obj-y += s390-ccw.o
>> +obj-y += ap-device.o
>> diff --git a/hw/s390x/ap-device.c b/hw/s390x/ap-device.c
>> new file mode 100644
>> index 0000000..448f6db
>> --- /dev/null
>> +++ b/hw/s390x/ap-device.c
>> @@ -0,0 +1,38 @@
>> +/*
>> + * Adjunct Processor (AP) matrix device
>> + *
>> + * Copyright 2018 IBM Corp.
>> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2 
>> or (at
>> + * your option) any later version. See the COPYING file in the 
>> top-level
>> + * directory.
>> + */
>> +#include "qemu/osdep.h"
>> +#include "qemu/module.h"
>> +#include "qapi/error.h"
>> +#include "hw/qdev.h"
>> +#include "hw/s390x/ap-device.h"
>> +
>> +static void ap_class_init(ObjectClass *klass, void *data)
>> +{
>> +    DeviceClass *dc = DEVICE_CLASS(klass);
>> +
>> +    dc->desc = "AP device class";
>
> Shouldn't we make the AP devices unpluggable ?
Will do.
>
>
>> +}
>> +
>> +static const TypeInfo ap_device_info = {
>> +    .name = AP_DEVICE_TYPE,
>> +    .parent = TYPE_DEVICE,
>> +    .instance_size = sizeof(APDevice),
>> +    .class_size = sizeof(APDeviceClass),
>> +    .class_init = ap_class_init,
>> +    .abstract = true,
>> +};
>> +
>> +static void ap_device_register(void)
>> +{
>> +    type_register_static(&ap_device_info);
>> +}
>> +
>> +type_init(ap_device_register)
>> diff --git a/include/hw/s390x/ap-device.h b/include/hw/s390x/ap-device.h
>> new file mode 100644
>> index 0000000..693df90
>> --- /dev/null
>> +++ b/include/hw/s390x/ap-device.h
>> @@ -0,0 +1,38 @@
>> +/*
>> + * Adjunct Processor (AP) matrix device interfaces
>> + *
>> + * Copyright 2018 IBM Corp.
>> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2 
>> or (at
>> + * your option) any later version. See the COPYING file in the 
>> top-level
>> + * directory.
>> + */
>> +#ifndef HW_S390X_AP_DEVICE_H
>> +#define HW_S390X_AP_DEVICE_H
>> +
>> +#define AP_DEVICE_TYPE       "ap-device"
>> +
>> +typedef struct APDevice {
>> +    DeviceState parent_obj;
>> +} APDevice;
>> +
>> +typedef struct APDeviceClass {
>> +    DeviceClass parent_class;
>> +} APDeviceClass;
>> +
>> +static inline APDevice *to_ap_dev(DeviceState *dev)
>> +{
>> +    return container_of(dev, APDevice, parent_obj);
>> +}
>> +
>> +#define AP_DEVICE(obj) \
>> +    OBJECT_CHECK(APDevice, (obj), AP_DEVICE_TYPE)
>> +
>> +#define AP_DEVICE_GET_CLASS(obj) \
>> +    OBJECT_GET_CLASS(APDeviceClass, (obj), AP_DEVICE_TYPE)
>> +
>> +#define AP_DEVICE_CLASS(klass) \
>> +    OBJECT_CLASS_CHECK(APDeviceClass, (klass), AP_DEVICE_TYPE)
>> +
>> +#endif /* HW_S390X_AP_DEVICE_H */
>
>

Re: [Qemu-devel] [PATCH v3 3/7] s390x/cpumodel: Set up CPU model for AP device support

[Tony Krowiak]

On 03/16/2018 05:36 AM, Pierre Morel wrote:
> On 16/03/2018 00:24, Tony Krowiak wrote:
>> A new CPU model feature and two new CPU model facilities are
>> introduced to support AP devices for a KVM guest.
>>
>> CPU model features:
>>
>> 1. The KVM_S390_VM_CPU_FEAT_AP CPU model feature indicates that
>>     AP facilities are installed. This feature will be enabled by
>>     the kernel only if the AP facilities are installed on the linux
>>     host. This feature must be turned on from userspace to access
>>     AP devices from the KVM guest. The QEMU command line to turn
>>     this feature looks something like this:
>>
>>     qemu-system-s390x ... -cpu xxx,ap=on
>>
>> CPU model facilities:
>>
>> 1. The S390_FEAT_AP_QUERY_CONFIG_INFO feature indicates the AP Query
>>     Configuration Information (QCI) facility is installed. This feature
>>     will be enabled by the kernel only if the QCI is installed on
>>     the host.
>>
>> 2. The S390_FEAT_AP_FACILITY_TEST feature indicates that the AP
>>     Facility Test (APFT) facility is installed. This feature will
>>     be enabled by the kernel only if the APFT facility is installed
>>     on the host.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>>   target/s390x/cpu_features.c     |    3 +++
>>   target/s390x/cpu_features_def.h |    3 +++
>>   target/s390x/cpu_models.c       |    2 ++
>>   target/s390x/gen-features.c     |    3 +++
>>   target/s390x/kvm.c              |    1 +
>>   5 files changed, 12 insertions(+), 0 deletions(-)
>>
>> diff --git a/target/s390x/cpu_features.c b/target/s390x/cpu_features.c
>> index a5619f2..1abe987 100644
>> --- a/target/s390x/cpu_features.c
>> +++ b/target/s390x/cpu_features.c
>> @@ -36,8 +36,10 @@ static const S390FeatDef s390_features[] = {
>>       FEAT_INIT("srs", S390_FEAT_TYPE_STFL, 9, "Sense-running-status 
>> facility"),
>>       FEAT_INIT("csske", S390_FEAT_TYPE_STFL, 10, "Conditional-SSKE 
>> facility"),
>>       FEAT_INIT("ctop", S390_FEAT_TYPE_STFL, 11, 
>> "Configuration-topology facility"),
>> +    FEAT_INIT("qci", S390_FEAT_TYPE_STFL, 12, "Query AP 
>> Configuration facility"),
>>       FEAT_INIT("ipter", S390_FEAT_TYPE_STFL, 13, "IPTE-range 
>> facility"),
>>       FEAT_INIT("nonqks", S390_FEAT_TYPE_STFL, 14, "Nonquiescing 
>> key-setting facility"),
>> +    FEAT_INIT("apft", S390_FEAT_TYPE_STFL, 15, "Adjunct Processor 
>> Facilities Test facility"),
>>       FEAT_INIT("etf2", S390_FEAT_TYPE_STFL, 16, 
>> "Extended-translation facility 2"),
>>       FEAT_INIT("msa-base", S390_FEAT_TYPE_STFL, 17, 
>> "Message-security-assist facility (excluding subfunctions)"),
>>       FEAT_INIT("ldisp", S390_FEAT_TYPE_STFL, 18, "Long-displacement 
>> facility"),
>> @@ -125,6 +127,7 @@ static const S390FeatDef s390_features[] = {
>>
>>       FEAT_INIT("dateh2", S390_FEAT_TYPE_MISC, 0, "DAT-enhancement 
>> facility 2"),
>>       FEAT_INIT("cmm", S390_FEAT_TYPE_MISC, 0, 
>> "Collaborative-memory-management facility"),
>> +    FEAT_INIT("ap", S390_FEAT_TYPE_MISC, 0, "AP facilities installed"),
>
> AFAIK you should use  FEAT_INIT_MISC to initialize misc features now.
> May be rebase on last qemu ?
Yes, I noticed too late that Connie mentioned that in the last round, it 
will be
in the next series.
>
>
>>
>>       FEAT_INIT("plo-cl", S390_FEAT_TYPE_PLO, 0, "PLO Compare and 
>> load (32 bit in general registers)"),
>>       FEAT_INIT("plo-clg", S390_FEAT_TYPE_PLO, 1, "PLO Compare and 
>> load (64 bit in parameter list)"),
>> diff --git a/target/s390x/cpu_features_def.h 
>> b/target/s390x/cpu_features_def.h
>> index 7c5915c..8998b65 100644
>> --- a/target/s390x/cpu_features_def.h
>> +++ b/target/s390x/cpu_features_def.h
>> @@ -27,8 +27,10 @@ typedef enum {
>>       S390_FEAT_SENSE_RUNNING_STATUS,
>>       S390_FEAT_CONDITIONAL_SSKE,
>>       S390_FEAT_CONFIGURATION_TOPOLOGY,
>> +    S390_FEAT_AP_QUERY_CONFIG_INFO,
>>       S390_FEAT_IPTE_RANGE,
>>       S390_FEAT_NONQ_KEY_SETTING,
>> +    S390_FEAT_AP_FACILITIES_TEST,
>>       S390_FEAT_EXTENDED_TRANSLATION_2,
>>       S390_FEAT_MSA,
>>       S390_FEAT_LONG_DISPLACEMENT,
>> @@ -118,6 +120,7 @@ typedef enum {
>>       /* Misc */
>>       S390_FEAT_DAT_ENH_2,
>>       S390_FEAT_CMM,
>> +    S390_FEAT_AP,
>>
>>       /* PLO */
>>       S390_FEAT_PLO_CL,
>> diff --git a/target/s390x/cpu_models.c b/target/s390x/cpu_models.c
>> index cf82589..7e2af09 100644
>> --- a/target/s390x/cpu_models.c
>> +++ b/target/s390x/cpu_models.c
>> @@ -769,6 +769,8 @@ static void check_consistency(const S390CPUModel 
>> *model)
>>           { S390_FEAT_PRNO_TRNG_QRTCR, S390_FEAT_MSA_EXT_5 },
>>           { S390_FEAT_PRNO_TRNG, S390_FEAT_MSA_EXT_5 },
>>           { S390_FEAT_SIE_KSS, S390_FEAT_SIE_F2 },
>> +        { S390_FEAT_AP_QUERY_CONFIG_INFO, S390_FEAT_AP },
>> +        { S390_FEAT_AP_FACILITIES_TEST, S390_FEAT_AP },
>>       };
>>       int i;
>>
>> diff --git a/target/s390x/gen-features.c b/target/s390x/gen-features.c
>> index 0cdbc15..0d5b0f7 100644
>> --- a/target/s390x/gen-features.c
>> +++ b/target/s390x/gen-features.c
>> @@ -447,6 +447,9 @@ static uint16_t full_GEN12_GA1[] = {
>>       S390_FEAT_ADAPTER_INT_SUPPRESSION,
>>       S390_FEAT_EDAT_2,
>>       S390_FEAT_SIDE_EFFECT_ACCESS_ESOP2,
>> +    S390_FEAT_AP_QUERY_CONFIG_INFO,
>> +    S390_FEAT_AP_FACILITIES_TEST,
>> +    S390_FEAT_AP,
>>   };
>>
>>   static uint16_t full_GEN12_GA2[] = {
>> diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c
>> index fbc887e..33e5ec3 100644
>> --- a/target/s390x/kvm.c
>> +++ b/target/s390x/kvm.c
>> @@ -2178,6 +2178,7 @@ static int kvm_to_feat[][2] = {
>>       { KVM_S390_VM_CPU_FEAT_PFMFI, S390_FEAT_SIE_PFMFI},
>>       { KVM_S390_VM_CPU_FEAT_SIGPIF, S390_FEAT_SIE_SIGPIF},
>>       { KVM_S390_VM_CPU_FEAT_KSS, S390_FEAT_SIE_KSS},
>> +    { KVM_S390_VM_CPU_FEAT_AP, S390_FEAT_AP},
>>   };
>>
>>   static int query_cpu_feat(S390FeatBitmap features)
>
>

Re: [Qemu-devel] [PATCH v3 4/7] s390x/kvm: interface to interpret AP instructions

[Tony Krowiak]

On 03/16/2018 06:36 AM, Pierre Morel wrote:
> On 16/03/2018 00:24, Tony Krowiak wrote:
>> The VFIO AP device exploits interpretive execution of AP
>> instructions (APIE). APIE is enabled by setting a device attribute
>> via the KVM_SET_DEVICE_ATTR ioctl.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>>   target/s390x/kvm.c       |   16 ++++++++++++++++
>>   target/s390x/kvm_s390x.h |    2 ++
>>   2 files changed, 18 insertions(+), 0 deletions(-)
>>
>> diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c
>> index 33e5ec3..2812e28 100644
>> --- a/target/s390x/kvm.c
>> +++ b/target/s390x/kvm.c
>> @@ -277,6 +277,22 @@ static void kvm_s390_init_dea_kw(void)
>>       }
>>   }
>>
>> +int kvm_s390_set_interpret_ap(uint8_t enable)
>> +{
>> +    struct kvm_device_attr attribute = {
>> +        .group = KVM_S390_VM_CRYPTO,
>> +        .attr  = KVM_S390_VM_CRYPTO_INTERPRET_AP,
>> +        .addr = 1,
>> +    };
>> +
>> +    if (!kvm_vm_check_attr(kvm_state, KVM_S390_VM_CRYPTO,
>> +                           KVM_S390_VM_CRYPTO_INTERPRET_AP)) {
>
> Isn't it enough to have the CPU feature ?
I don't understand this question within this context. The code above checks
to see whether the KVM_S390_VM_CRYPTO_INTERPRET_AP attribute is supported.
>
> What are expecting here?
I'm expecting that if the KVM_S390_VM_CRYPTO_INTERPRET_AP attribute can
not be set then that is an error condition that should be returned to
the caller.
>
>> +        return -EOPNOTSUPP;
>> +    }
>> +
>> +    return kvm_vm_ioctl(kvm_state, KVM_SET_DEVICE_ATTR, &attribute);
>
> Shouldn't you use the "enable" parameter somewhere?
If attribute.addr is not zero, then that indicates enable. If that is 
objectionable,
I can change it.
>
>> +}
>> +
>>   void kvm_s390_crypto_reset(void)
>>   {
>>       if (s390_has_feat(S390_FEAT_MSA_EXT_3)) {
>> diff --git a/target/s390x/kvm_s390x.h b/target/s390x/kvm_s390x.h
>> index 34ee7e7..0d6c6e7 100644
>> --- a/target/s390x/kvm_s390x.h
>> +++ b/target/s390x/kvm_s390x.h
>> @@ -40,4 +40,6 @@ void kvm_s390_crypto_reset(void);
>>   void kvm_s390_restart_interrupt(S390CPU *cpu);
>>   void kvm_s390_stop_interrupt(S390CPU *cpu);
>>
>> +int kvm_s390_set_interpret_ap(uint8_t enable);
>> +
>>   #endif /* KVM_S390X_H */
>
>

Re: [Qemu-devel] [PATCH v3 5/7] s390x/vfio: ap: Introduce VFIO AP device

[Tony Krowiak]

On 03/16/2018 06:42 AM, Pierre Morel wrote:
> On 16/03/2018 00:24, Tony Krowiak wrote:
>> Introduces a VFIO based AP device. The device is defined via
>> the QEMU command line by specifying:
>>
>>      -device vfio-ap,sysfsdev=<path-to-mediated-matrix-device>
>>
>> The mediated matrix device is created by the VFIO AP device
>> driver by writing a UUID to a sysfs attribute file (see
>> docs/vfio-ap.txt). The mediated matrix device will be named
>> after the UUID. Symbolic links to the $uuid are created in
>> many places, so the path to the mediated matrix device $uuid
>> can be specified in any of the following ways:
>>
>> /sys/devices/vfio_ap/matrix/$uuid
>> /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/devices/$uuid 
>>
>> /sys/bus/mdev/devices/$uuid
>> /sys/bus/mdev/drivers/vfio_mdev/$uuid
>>
>> When the vfio-ap device is realized, it acquires and opens the
>> VFIO iommu group to which the mediated matrix device is
>> bound. This causes a VFIO group notification event to be
>> signaled. The vfio_ap device driver's group notification
>> handler will get called at which time the device driver
>> will configure the the AP devices to which the guest will
>> be granted access.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>>   default-configs/s390x-softmmu.mak |    1 +
>>   hw/vfio/Makefile.objs             |    1 +
>>   hw/vfio/ap.c                      |  184 
>> +++++++++++++++++++++++++++++++++++++
>>   include/hw/vfio/vfio-common.h     |    1 +
>>   4 files changed, 187 insertions(+), 0 deletions(-)
>>   create mode 100644 hw/vfio/ap.c
>>
>> diff --git a/default-configs/s390x-softmmu.mak 
>> b/default-configs/s390x-softmmu.mak
>> index 2f4bfe7..0b784b6 100644
>> --- a/default-configs/s390x-softmmu.mak
>> +++ b/default-configs/s390x-softmmu.mak
>> @@ -9,3 +9,4 @@ CONFIG_S390_FLIC=y
>>   CONFIG_S390_FLIC_KVM=$(CONFIG_KVM)
>>   CONFIG_VFIO_CCW=$(CONFIG_LINUX)
>>   CONFIG_WDT_DIAG288=y
>> +CONFIG_VFIO_AP=$(CONFIG_LINUX)
>> diff --git a/hw/vfio/Makefile.objs b/hw/vfio/Makefile.objs
>> index c3ab909..7300860 100644
>> --- a/hw/vfio/Makefile.objs
>> +++ b/hw/vfio/Makefile.objs
>> @@ -6,4 +6,5 @@ obj-$(CONFIG_SOFTMMU) += platform.o
>>   obj-$(CONFIG_VFIO_XGMAC) += calxeda-xgmac.o
>>   obj-$(CONFIG_VFIO_AMD_XGBE) += amd-xgbe.o
>>   obj-$(CONFIG_SOFTMMU) += spapr.o
>> +obj-$(CONFIG_VFIO_AP) += ap.o
>>   endif
>> diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c
>> new file mode 100644
>> index 0000000..b397bb1
>> --- /dev/null
>> +++ b/hw/vfio/ap.c
>> @@ -0,0 +1,184 @@
>> +/*
>> + * VFIO based AP matrix device assignment
>> + *
>> + * Copyright 2018 IBM Corp.
>> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2 
>> or (at
>> + * your option) any later version. See the COPYING file in the 
>> top-level
>> + * directory.
>> + */
>> +
>> +#include <linux/vfio.h>
>> +#include <sys/ioctl.h>
>> +#include "qemu/osdep.h"
>> +#include "qapi/error.h"
>> +#include "hw/sysbus.h"
>> +#include "hw/vfio/vfio.h"
>> +#include "hw/vfio/vfio-common.h"
>> +#include "hw/s390x/ap-device.h"
>> +#include "qemu/error-report.h"
>> +#include "qemu/queue.h"
>> +#include "cpu.h"
>> +#include "kvm_s390x.h"
>> +
>> +#define VFIO_AP_DEVICE_TYPE      "vfio-ap"
>> +
>> +typedef struct VFIOAPDevice {
>> +    APDevice apdev;
>> +    VFIODevice vdev;
>> +    QTAILQ_ENTRY(VFIOAPDevice) sibling;
>> +} VFIOAPDevice;
>> +
>> +static void vfio_ap_compute_needs_reset(VFIODevice *vdev)
>> +{
>> +    vdev->needs_reset = false;
>> +}
>> +
>> +/*
>> + * We don't need vfio_hot_reset_multi and vfio_eoi operations for
>> + * vfio-ap-matrix device now.
>> + */
>> +struct VFIODeviceOps vfio_ap_ops = {
>> +    .vfio_compute_needs_reset = vfio_ap_compute_needs_reset,
>> +};
>> +
>> +static QTAILQ_HEAD(, VFIOAPDevice) vfio_ap_devices =
>> +        QTAILQ_HEAD_INITIALIZER(vfio_ap_devices);
>> +
>> +static void vfio_put_device(VFIOAPDevice *apdev)
>> +{
>> +    g_free(apdev->vdev.name);
>> +    vfio_put_base_device(&apdev->vdev);
>> +}
>> +
>> +static VFIOGroup *vfio_ap_get_group(VFIOAPDevice *vapdev, Error **errp)
>> +{
>> +    char *tmp, group_path[PATH_MAX];
>> +    ssize_t len;
>> +    int groupid;
>> +
>> +    tmp = g_strdup_printf("%s/iommu_group", vapdev->vdev.sysfsdev);
>> +    len = readlink(tmp, group_path, sizeof(group_path));
>> +    g_free(tmp);
>> +
>> +    if (len <= 0 || len >= sizeof(group_path)) {
>> +        error_setg(errp, "%s: no iommu_group found for %s",
>> +                   VFIO_AP_DEVICE_TYPE, vapdev->vdev.sysfsdev);
>> +        return NULL;
>> +    }
>> +
>> +    group_path[len] = 0;
>> +
>> +    if (sscanf(basename(group_path), "%d", &groupid) != 1) {
>> +        error_setg(errp, "vfio: failed to read %s", group_path);
>> +        return NULL;
>> +    }
>> +
>> +    return vfio_get_group(groupid, &address_space_memory, errp);
>> +}
>> +
>> +static void vfio_ap_realize(DeviceState *dev, Error **errp)
>> +{
>> +    VFIODevice *vbasedev;
>> +    VFIOGroup *vfio_group;
>> +    APDevice *apdev = DO_UPCAST(APDevice, parent_obj, dev);
>> +    VFIOAPDevice *vapdev = DO_UPCAST(VFIOAPDevice, apdev, apdev);
>> +    char *mdevid;
>> +    Error *local_err = NULL;
>> +    int ret;
>> +
>> +    if (!s390_has_feat(S390_FEAT_AP)) {
>> +        error_setg(&local_err, "AP support not enabled");
>> +        goto out_err;
>> +    }
>> +
>> +    ret = kvm_s390_set_interpret_ap(1);
>
> If we have several devices, this is called once per device.
> However it is a general switch for the VM.
> It looks strange to have this inside the realize callback
I've been so focused on the idea that a guest's AP matrix is configured 
via this
device that I hadn't considered the possibility that there is nothing 
precluding
multiple vfio-ap devices being defined for a guest. I agree, this is a VM
switch, but the question is, where does it belong? Whether AP 
instructions are
interpreted or not is dependent upon what type of VFIO driver is being 
used which
is why I put this here. Would it make sense to model this as a -machine 
property
and treat it like we do with key wrapping support?
>
>
>> +    if (ret) {
>> +        error_setg_errno(&local_err, errno,
>> +                         "error setting interpretive execution of AP 
>> instructions");
>> +        goto out_err;
>> +    }
>> +
>> +    vfio_group = vfio_ap_get_group(vapdev, &local_err);
>> +    if (!vfio_group) {
>> +        goto out_group_err;
>> +    }
>> +
>> +    vapdev->vdev.ops = &vfio_ap_ops;
>> +    vapdev->vdev.type = VFIO_DEVICE_TYPE_AP;
>> +    mdevid = basename(vapdev->vdev.sysfsdev);
>> +    vapdev->vdev.name = g_strdup_printf("%s", mdevid);
>> +    vapdev->vdev.dev = dev;
>> +    QLIST_FOREACH(vbasedev, &vfio_group->device_list, next) {
>> +        if (strcmp(vbasedev->name, vapdev->vdev.name) == 0) {
>> +            error_setg(&local_err,
>> +                       "%s: AP device %s has already been realized",
>> +                       VFIO_AP_DEVICE_TYPE, vapdev->vdev.name);
>> +            goto out_device_err;
>> +        }
>> +    }
>> +
>> +    ret = vfio_get_device(vfio_group, mdevid, &vapdev->vdev, 
>> &local_err);
>> +    if (ret) {
>> +        goto out_device_err;
>> +    }
>> +
>> +    QTAILQ_INSERT_TAIL(&vfio_ap_devices, vapdev, sibling);
>> +
>> +    return;
>> +
>> +
>> +out_device_err:
>> +    vfio_put_group(vfio_group);
>> +out_group_err:
>> +    kvm_s390_set_interpret_ap(0);
>> +out_err:
>> +    error_propagate(errp, local_err);
>> +}
>> +
>> +static void vfio_ap_unrealize(DeviceState *dev, Error **errp)
>> +{
>> +    APDevice *apdev = DO_UPCAST(APDevice, parent_obj, dev);
>> +    VFIOAPDevice *vapdev = DO_UPCAST(VFIOAPDevice, apdev, apdev);
>> +    VFIOGroup *group = vapdev->vdev.group;
>> +
>> +    vfio_put_device(vapdev);
>> +    vfio_put_group(group);
>> +    kvm_s390_set_interpret_ap(0);
>> +}
>> +
>> +static Property vfio_ap_properties[] = {
>> +    DEFINE_PROP_STRING("sysfsdev", VFIOAPDevice, vdev.sysfsdev),
>> +    DEFINE_PROP_END_OF_LIST(),
>> +};
>> +
>> +static const VMStateDescription vfio_ap_vmstate = {
>> +    .name = VFIO_AP_DEVICE_TYPE,
>> +    .unmigratable = 1,
>> +};
>> +
>> +static void vfio_ap_class_init(ObjectClass *klass, void *data)
>> +{
>> +    DeviceClass *dc = DEVICE_CLASS(klass);
>> +
>> +    dc->props = vfio_ap_properties;
>> +    dc->vmsd = &vfio_ap_vmstate;
>> +    dc->desc = "VFIO-based AP device assignment";
>> +    dc->realize = vfio_ap_realize;
>> +    dc->unrealize = vfio_ap_unrealize;
>
>
> Shouldn't we make the VFIO-AP devices unpluggable attribute
> in the case the AP device are pluggable?
I'm going to make the AP device unpluggable.
>
>
>> +}
>> +
>> +static const TypeInfo vfio_ap_info = {
>> +    .name = VFIO_AP_DEVICE_TYPE,
>> +    .parent = AP_DEVICE_TYPE,
>> +    .instance_size = sizeof(VFIOAPDevice),
>> +    .class_init = vfio_ap_class_init,
>> +};
>> +
>> +static void register_vfio_ap_type(void)
>> +{
>> +    type_register_static(&vfio_ap_info);
>> +}
>> +
>> +type_init(register_vfio_ap_type)
>> diff --git a/include/hw/vfio/vfio-common.h 
>> b/include/hw/vfio/vfio-common.h
>> index f3a2ac9..f1f22d9 100644
>> --- a/include/hw/vfio/vfio-common.h
>> +++ b/include/hw/vfio/vfio-common.h
>> @@ -46,6 +46,7 @@ enum {
>>       VFIO_DEVICE_TYPE_PCI = 0,
>>       VFIO_DEVICE_TYPE_PLATFORM = 1,
>>       VFIO_DEVICE_TYPE_CCW = 2,
>> +    VFIO_DEVICE_TYPE_AP = 3,
>>   };
>>
>>   typedef struct VFIOMmap {
>
>

Re: [Qemu-devel] [PATCH v3 5/7] s390x/vfio: ap: Introduce VFIO AP device

[Tony Krowiak]

On 03/16/2018 09:22 AM, Halil Pasic wrote:
>
> On 03/16/2018 11:42 AM, Pierre Morel wrote:
>> On 16/03/2018 00:24, Tony Krowiak wrote:
>>> Introduces a VFIO based AP device. The device is defined via
>>> the QEMU command line by specifying:
>>>
>>>       -device vfio-ap,sysfsdev=<path-to-mediated-matrix-device>
>>>
>>> The mediated matrix device is created by the VFIO AP device
>>> driver by writing a UUID to a sysfs attribute file (see
>>> docs/vfio-ap.txt). The mediated matrix device will be named
>>> after the UUID. Symbolic links to the $uuid are created in
>>> many places, so the path to the mediated matrix device $uuid
>>> can be specified in any of the following ways:
>>>
>>> /sys/devices/vfio_ap/matrix/$uuid
>>> /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/devices/$uuid
>>> /sys/bus/mdev/devices/$uuid
>>> /sys/bus/mdev/drivers/vfio_mdev/$uuid
>>>
>>> When the vfio-ap device is realized, it acquires and opens the
>>> VFIO iommu group to which the mediated matrix device is
>>> bound. This causes a VFIO group notification event to be
>>> signaled. The vfio_ap device driver's group notification
>>> handler will get called at which time the device driver
>>> will configure the the AP devices to which the guest will
>>> be granted access.
>>>
>>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>>> ---
> [..]
>>> +static void vfio_ap_realize(DeviceState *dev, Error **errp)
>>> +{
>>> +    VFIODevice *vbasedev;
>>> +    VFIOGroup *vfio_group;
>>> +    APDevice *apdev = DO_UPCAST(APDevice, parent_obj, dev);
>>> +    VFIOAPDevice *vapdev = DO_UPCAST(VFIOAPDevice, apdev, apdev);
>>> +    char *mdevid;
>>> +    Error *local_err = NULL;
>>> +    int ret;
>>> +
>>> +    if (!s390_has_feat(S390_FEAT_AP)) {
>>> +        error_setg(&local_err, "AP support not enabled");
>>> +        goto out_err;
>>> +    }
>>> +
>>> +    ret = kvm_s390_set_interpret_ap(1);
>> If we have several devices, this is called once per device.
> I don't think having several of these in a single vm makes
> any sense. Or does it? IMHO we should make sure there is at
> most one device taking care of the crypto pass-through.
My first thought was that only one vfio-ap device was necessary
to provide the AP matrix to a guest. Allowing additional vfio-ap
devices would definitely be superfluous, so I think it makes sense
to allow at most one vfio-ap device. In that case, setting AP
interpretive execution here makes sense.
>
>> However it is a general switch for the VM.
>> It looks strange to have this inside the realize callback
>>
> I kind of semi agree.
>
> The previous solution (having this transparent for QEMU) had
> benefits. Why did we move away from that again?
This was done to address the multitude of objections and opinions 
related to
how/when/where ECA_APIE is set. Pierre summed it up best with his 
comment "we
need to separate the CPU feature defining 'if AP instructions are 
available'
from the QEMU property defining 'How we provide the instructions'. I 
agree and
this is how I chose to implement that.
>
> Regards,
> Halil

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Tony Krowiak]

On 03/16/2018 04:03 AM, Pierre Morel wrote:
> On 16/03/2018 00:24, Tony Krowiak wrote:
>> If the CPU model indicates that AP facility is installed on
>> the guest (i.e., -cpu xxxx,ap=on), then the expectation is that
>> the AP bus running in the guest will initialize; however, if the
>> AP instructions are not being interpreted by the firmware, then
>> they will be intercepted and routed back to QEMU for handling.
>> If a handler is not defined to process the intercepted instruciton,
>> then an operation exception will be injected into the
>> guest, in which case the AP bus will not initialize.
>>
>> There are two situations where AP instructions will not be
>> interpreted:
>>
>> 1. The guest is not configured with a vfio-ap device (i.e.,
>>     -device vfio-ap,sysfsdev=$path-to-mdev). The realize function for
>>     the vfio-ap device enables interpretive execution of AP
>>     instructions.
>>
>> 2. The guest is a second level guest but the first level guest has
>>     not enabled interpretive execution.
>>
>> This patch introduces AP instruction handlers to ensure the AP bus
>> module initializes on the guest when the AP facility is installed
>> on the guest but AP instructions are not being interpreted. The logic
>> incorporated is:
>>
>> * If the CPU model indicates AP instructions are
>>    installed
>>
>>    * Set the status response code for the instruction to indicate that
>>      the APQN contained in the instruction is not valid. This is
>>      a valid response because there will be no devices configured for
>>      the guest in any of the above scenarios.
>>
>> * Else return an error from the handler. This will result in an
>>    operation being injected into the guest and the AP bus will not
>>    initialize on the guest. That is commensurate with how things work
>>    today.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>>   hw/vfio/ap.c                 |   45 
>> ++++++++++++++++++++++++++++++++++++++++++
>>   include/hw/s390x/ap-device.h |    6 +++++
>>   target/s390x/kvm.c           |   14 +++++++++++++
>>   3 files changed, 65 insertions(+), 0 deletions(-)
>>
>> diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c
>> index b397bb1..88e744d 100644
>> --- a/hw/vfio/ap.c
>> +++ b/hw/vfio/ap.c
>> @@ -148,6 +148,51 @@ static void vfio_ap_unrealize(DeviceState *dev, 
>> Error **errp)
>>       kvm_s390_set_interpret_ap(0);
>>   }
>>
>> +int ap_device_handle_nqap(S390CPU *cpu)
>> +{
>> +    CPUS390XState *env = &cpu->env;
>> +
>> +    if (s390_has_feat(S390_FEAT_AP)) {
>> +        env->regs[1] = 0x10000;
>> +
>> +        return 0;
>> +    }
>> +
>> +    return -EOPNOTSUPP;
>> +}
>> +
>> +int ap_device_handle_dqap(S390CPU *cpu)
>> +{
>> +    CPUS390XState *env = &cpu->env;
>> +
>> +    if (s390_has_feat(S390_FEAT_AP)) {
>> +        env->regs[1] = 0x10000;
>> +
>> +        return 0;
>> +    }
>> +
>> +    return -EOPNOTSUPP;
>> +}
>> +
>> +int ap_device_handle_pqap(S390CPU *cpu)
>> +{
>> +    CPUS390XState *env = &cpu->env;
>> +    int fc = 4 & (env->regs[0] >> 24);
>> +
>> +    /*
>> +     * The Query Configuration Information (QCI) function (fc == 4) 
>> does not
>> +     * set a response code in reg 1, so check for that along with the
>> +     * AP feature.
>> +     */
>> +    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
>> +        env->regs[1] = 0x10000;
>> +
>> +        return 0;
>> +    }
>> +
>> +    return -EOPNOTSUPP;
>> +}
>> +
>>   static Property vfio_ap_properties[] = {
>>       DEFINE_PROP_STRING("sysfsdev", VFIOAPDevice, vdev.sysfsdev),
>>       DEFINE_PROP_END_OF_LIST(),
>> diff --git a/include/hw/s390x/ap-device.h b/include/hw/s390x/ap-device.h
>> index 693df90..d45ae38 100644
>> --- a/include/hw/s390x/ap-device.h
>> +++ b/include/hw/s390x/ap-device.h
>> @@ -11,6 +11,8 @@
>>   #ifndef HW_S390X_AP_DEVICE_H
>>   #define HW_S390X_AP_DEVICE_H
>>
>> +#include "cpu.h"
>> +
>>   #define AP_DEVICE_TYPE       "ap-device"
>>
>>   typedef struct APDevice {
>> @@ -35,4 +37,8 @@ static inline APDevice *to_ap_dev(DeviceState *dev)
>>   #define AP_DEVICE_CLASS(klass) \
>>       OBJECT_CLASS_CHECK(APDeviceClass, (klass), AP_DEVICE_TYPE)
>>
>> +int ap_device_handle_nqap(S390CPU *cpu);
>> +int ap_device_handle_dqap(S390CPU *cpu);
>> +int ap_device_handle_pqap(S390CPU *cpu);
>> +
>>   #endif /* HW_S390X_AP_DEVICE_H */
>> diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c
>> index 2812e28..a636394 100644
>> --- a/target/s390x/kvm.c
>> +++ b/target/s390x/kvm.c
>> @@ -50,6 +50,7 @@
>>   #include "exec/memattrs.h"
>>   #include "hw/s390x/s390-virtio-ccw.h"
>>   #include "hw/s390x/s390-virtio-hcall.h"
>> +#include "hw/s390x/ap-device.h"
>>
>>   #ifndef DEBUG_KVM
>>   #define DEBUG_KVM  0
>> @@ -88,6 +89,9 @@
>>   #define PRIV_B2_CHSC                    0x5f
>>   #define PRIV_B2_SIGA                    0x74
>>   #define PRIV_B2_XSCH                    0x76
>> +#define PRIV_B2_NQAP                    0xad
>> +#define PRIV_B2_DQAP                    0xae
>> +#define PRIV_B2_PQAP                    0xaf
>>
>>   #define PRIV_EB_SQBS                    0x8a
>>   #define PRIV_EB_PCISTB                  0xd0
>> @@ -1245,6 +1249,16 @@ static int handle_b2(S390CPU *cpu, struct 
>> kvm_run *run, uint8_t ipa1)
>>       case PRIV_B2_SCLP_CALL:
>>           rc = kvm_sclp_service_call(cpu, run, ipbh0);
>>           break;
>> +    case PRIV_B2_NQAP:
>> +        rc = ap_device_handle_nqap(cpu);
>> +        break;
>> +    case PRIV_B2_DQAP:
>> +        rc = ap_device_handle_dqap(cpu);
>> +        break;
>> +    case PRIV_B2_PQAP:
>> +        rc = ap_device_handle_pqap(cpu);
>> +        break;
>> +        break;
>
> one too much
Yes there is, how on earth did I miss that? I will fix it.
>
>>       default:
>>           rc = -1;
>>           DPRINTF("KVM: unhandled PRIV: 0xb2%x\n", ipa1);
>
>

Re: [Qemu-devel] [PATCH v3 7/7] s390: doc: detailed specifications for AP virtualization

[Tony Krowiak]

On 03/16/2018 06:03 AM, Pierre Morel wrote:
> On 16/03/2018 00:25, Tony Krowiak wrote:
>> This patch provides documentation describing the AP architecture and
>> design concepts behind the virtualization of AP devices. It also
>> includes an example of how to configure AP devices for exclusive
>> use of KVM guests.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>>   docs/vfio-ap.txt |  624 
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>   1 files changed, 624 insertions(+), 0 deletions(-)
>>   create mode 100644 docs/vfio-ap.txt
>>
>> diff --git a/docs/vfio-ap.txt b/docs/vfio-ap.txt
>> new file mode 100644
>> index 0000000..54e7523
>> --- /dev/null
>> +++ b/docs/vfio-ap.txt
>> @@ -0,0 +1,624 @@
>> +Adjunct Processor (AP) Device
>> +=============================
>> +
>> +Contents:
>> +=========
>> +* Introduction
>> +* AP Architectural Overview
>> +* Start Interpretive Execution (SIE) Instruction
>> +* AP Matrix Configuration on Linux Host
>> +* AP Matrix Configuration for a Linux Guest
>> +* Starting a Linux Guest Configured with an AP Matrix
>> +* Example: Configure AP Matrices for Two Linux Guests
>> +
>> +Introduction:
>> +============
>> +The IBM Adjunct Processor (AP) Cryptographic Facility is comprised
>> +of three AP instructions and from 1 to 256 PCIe cryptographic 
>> adapter cards.
>> +These AP devices provide cryptographic functions to all CPUs 
>> assigned to a
>> +linux system running in an IBM Z system LPAR.
>> +
>> +On s390x, AP adapter cards are exposed via the AP bus. This document
>> +describes how those cards may be made available to KVM guests using the
>> +VFIO mediated device framework.
>> +
>> +AP Architectural Overview:
>> +=========================
>> +In order understand the terminology used in the rest of this 
>> document, let's
>> +start with some definitions:
>> +
>> +* AP adapter
>> +
>> +  An AP adapter is an IBM Z adapter card that can perform cryptographic
>> +  functions. There can be from 0 to 256 adapters assigned to an 
>> LPAR. Adapters
>> +  assigned to the LPAR in which a linux host is running will be 
>> available to
>> +  the linux host. Each adapter is identified by a number from 0 to 
>> 255. When
>> +  installed, an AP adapter is accessed by AP instructions executed 
>> by any CPU.
>> +
>> +* AP domain
>> +
>> +  An adapter is partitioned into domains. Each domain can be thought 
>> of as
>> +  a set of hardware registers for processing AP instructions. An 
>> adapter can
>> +  hold up to 256 domains. Each domain is identified by a number from 
>> 0 to 255.
>> +  Domains can be further classified into two types:
>> +
>> +    * Usage domains are domains that can be accessed directly to 
>> process AP
>> +      commands
>> +
>> +    * Control domains are domains that are accessed indirectly by AP
>> +      commands sent to a usage domain to control or change the 
>> domain, for
>> +      example; to set a secure private key for the domain.
>> +
>> +* AP Queue
>> +
>> +  An AP queue is the means by which an AP command-request message is 
>> sent to an
>> +  AP usage domain inside a specific AP. An AP queue is identified by 
>> a tuple
>> +  comprised of an AP adapter ID (APID) and an AP queue index (APQI). 
>> The
>> +  APQI corresponds to a given usage domain number within the 
>> adapter. This tuple
>> +  forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP
>> +  instructions include a field containing the APQN to identify the 
>> AP queue to
>> +  which the AP command-request message is to be sent for processing.
>> +
>> +* AP Instructions:
>> +
>> +  There are three AP instructions:
>> +
>> +  * NQAP: to enqueue an AP command-request message to a queue
>> +  * DQAP: to dequeue an AP command-reply message from a queue
>> +  * PQAP: to administer the queues
>> +
>> +Start Interpretive Execution (SIE) Instruction
>> +==============================================
>> +A KVM guest is started by executing the Start Interpretive Execution 
>> (SIE)
>> +instruction. The SIE state description is a control block that 
>> contains the
>> +state information for a KVM guest and is supplied as input to the SIE
>> +instruction. The SIE state description contains a field that references
>> +a Crypto Control Block (CRYCB). The CRYCB contains three fields to 
>> identify the
>> +adapters, usage domains and control domains assigned to the KVM guest:
>> +
>> +* The AP Mask (APM) field is a bit mask that identifies the AP 
>> adapters assigned
>> +  to the KVM guest. Each bit in the mask, from most significant to 
>> least
>> +  significant bit, corresponds to an APID from 0-255. If a bit is 
>> set, the
>> +  corresponding adapter is valid for use by the KVM guest.
>> +
>> +* The AP Queue Mask (AQM) field is a bit mask identifying the AP 
>> queues assigned
>> +  to the KVM guest. Each bit in the mask, from most significant to 
>> least
>> +  significant bit, corresponds to an AP queue index (APQI) from 
>> 0-255. If a bit
>> +  is set, the corresponding queue is valid for use by the KVM guest.
>> +
>> +* The AP Domain Mask field is a bit mask that identifies the AP 
>> control domains
>> +  assigned to the KVM guest. The ADM bit mask controls which domains 
>> can be
>> +  changed by an AP command-request message sent to a usage domain 
>> from the
>> +  guest. Each bit in the mask, from least significant to most 
>> significant bit,
>> +  corresponds to a domain from 0-255. If a bit is set, the 
>> corresponding domain
>> +  can be modified by an AP command-request message sent to a usage 
>> domain
>> +  configured for the KVM guest.
>> +
>> +If you recall from the description of an AP Queue, AP instructions 
>> include
>> +an APQN to identify the AP adapter and AP queue to which an AP 
>> command-request
>> +message is to be sent (NQAP and PQAP instructions), or from which a
>> +command-reply message is to be received (DQAP instruction). The 
>> validity of an
>> +APQN is defined by the matrix calculated from the APM and AQM; it is 
>> the
>> +intersection of all assigned adapter numbers (APM) with all assigned 
>> queue
>> +indexes (AQM). For example, if adapters 1 and 2 and usage domains 5 
>> and 6 are
>> +assigned to a guest, the APQNs (1,5), (1,6), (2,5) and (2,6) will be 
>> valid for
>> +the guest.
>> +
>> +The APQNs provide secure key functionality - i.e., a private key is 
>> stored on
>> +the adapter card for each of its domains - so each APQN must be 
>> assigned to at
>> +most one guest or the linux host.
>> +
>> +   Example 1: Valid configuration:
>> +   ------------------------------
>> +   Guest1: adapters 1,2  domains 5,6
>> +   Guest2: adapter  1,2  domain 7
>> +
>> +   This is valid because both guests have a unique set of APQNs: 
>> Guest1 has
>> +   APQNs (1,5), (1,6), (2,5) and (2,6); Guest2 has APQNs (1,7) and 
>> (2,7).
>> +
>> +   Example 2: Invalid configuration:
>> +   --------------------------------
>> +   Guest1: adapters 1,2  domains 5,6
>> +   Guest2: adapter  1    domains 6,7
>> +
>> +   This is an invalid configuration because both guests have access to
>> +   APQN (1,6).
>> +
>> +AP device Configuration on Linux Host:
>> +=====================================
>> +A linux system is a guest of the LPAR in which it is running and has 
>> access to
>> +the AP resources configured for the LPAR. The LPAR's AP matrix is
>> +configured using the 'Customize/Delete Activation Profiles' dialog 
>> from the HMC.
>> +This dialog displays the activation profiles configured for the 
>> linux system.
>> +Selecting the specific activation profile to be edited and clicking the
>> +'Customize Profile' button will open the 'Customize Image Profiles' 
>> dialog.
>> +Selecting the 'Crypto' link in the tree view on the left hand side 
>> of the dialog
>> +will display the AP matrix configuration in the right hand panel. 
>> There, one can
>> +assign AP adapters - called Cryptos - and domains to the LPAR. When 
>> the linux
>> +system is started using this activation profile, it will have access 
>> to the
>> +matrix of AP adapters and domains configured via the activation 
>> profile.
>> +
>> +When the linux system is started, the AP adapter devices will be 
>> connected to
>> +the AP bus and the following AP matrix interfaces will be created in 
>> sysfs:
>> +
>> +/sys/bus/ap
>> +... [devices]
>> +...... xx.yyyy
>> +...... ...
>> +...... cardxx
>> +...... ...
>> +
>> +Where:
>> +    cardxx     is adapter number xx (in hex)
>> +    yyyy       is a usage domain number yyyy (in hex)
>> +....xx.yyyy    is APQN (xx,yyyy)
>> +
>> +For example, if AP adapters 5 and 6 and domains 4 and 71 (0x47) are 
>> configured
>> +for the LPAR, the sysfs representation on the linux system would 
>> look like this:
>> +
>> +/sys/bus/ap
>> +... [devices]
>> +...... 05.0004
>> +...... 05.0047
>> +...... 06.0004
>> +...... 06.0047
>> +...... card05
>> +...... card06
>> +
>> +There will also be AP device drivers created to control each type of 
>> AP matrix
>> +interface available to the IBM Z system:
>> +
>> +/sys/bus/ap
>> +... [drivers]
>> +...... [cex2acard]        for Crypto Express 2/3 accelerator cards
>> +...... [cex2aqueue]       for AP queues served by Crypto Express 2/3
>> +                          accelerator cards
>> +...... [cex4card]         for Crypto Express 4/5/6 accelerator and 
>> coprocessor
>> +                          cards
>> +...... [cex4queue]        for AP queues served by Crypto Express 4/5/6
>> +                          accelerator and coprocessor cards
>> +...... [pcixcccard]       for Crypto Express 2/3 coprocessor cards
>> +...... [pcixccqueue]      for AP queues served by Crypto Express 2/3
>> +                          coprocessor cards
>> +
>> +Links to the AP interfaces controlled by each AP device driver will 
>> be created
>> +in the device driver's sysfs directory. For example, if AP adapter 5 
>> and domains
>> +4 and 71 (0x47) are assigned to the LPAR and adapter 5 is a CEX5 
>> card, the
>> +following links will be created in the CEX5 drivers' sysfs directories:
>> +
>> +/sys/bus/ap
>> +... [drivers]
>> +...... [cex4card]
>> +......... [card05]
>> +...... [cex4queue]
>> +......... [05.0004]
>> +......... [05.0047]
>> +
>> +AP Matrix Configuration for a Linux Guest:
>> +=========================================
>> +In order to configure the AP matrix for a guest, the adapters, usage 
>> domains
>> +and control domains to be used by the guest must be assigned to the 
>> guest. This
>> +section describes how to configure a guest's AP matrix.
>> +
>> +The kernel interfaces for configuring an AP matrix for a linux guest 
>> are built
>> +on the VFIO mediated device framework and are provided by the vfio_ap
>> +kernel module. By default, the vfio_ap module is a loadable module, The
>> +dependency chain for the vfio_ap module is:
>> +* vfio
>> +* mdev
>> +* vfio_mdev
>> +* vfio_ap
>> +
>> +When installed, the vfio_ap module is initialized. During module 
>> initialization,
>> +a vfio_ap driver is created and registered with the AP bus creating the
>> +following sysfs interfaces:
>> +
>> + /sys/bus/ap/drivers/
>> +...[vfio_ap]
>> +...... bind
>> +...... unbind
>> +
>> +The vfio_ap device driver will create a 'matrix' device to hold the 
>> APQNs
>> +reserved for exclusive use by KVM guests:
>> +
>> +/sys/devices/
>> +... [vfio_ap]
>> +......[matrix] symlink to the matrix device directory
>> +
>> +The vfio_ap device driver serves several purposes:
>> +1. Provides an interface for securing APQNs preventing their use by 
>> the host
>> +   linux system and reserving their use by one or more guests.
>> +2. Creates the sysfs interfaces for configuring an AP matrix for a 
>> linux guest.
>> +
>> +Securing APQNs
>> +--------------
>> +   An APQN is reserved by unbinding an AP queue device AP bus device 
>> driver and
>> +   binding it to the vfio_ap device driver. For example, suppose we 
>> want to
>> +   secure APQN (05,0004). Assuming that the AP adapter card 5 is a CEX5
>> +   coprocessor card:
>> +
>> +   echo 05.0004 > /sys/bus/ap/drivers/cex4queue/unbind
>> +   echo 05.0004 > /sys/bus/ap/drivers/vfio_ap/bind
>> +
>> +   This action will store the APQN in the 
>> /sys/devices/vfio_ap/matrix device
>> +   which makes it available for use by a linux guest.
>> +
>> +Configuring an AP matrix for a linux guest.
>> +------------------------------------------
>> +These sysfs interfaces are built on the VFIO mediated device 
>> framework. To
>> +configure an AP matrix for a guest, a mediated matrix device must 
>> first be
>> +created for the /sys/devices/vfio_ap/matrix device. The sysfs 
>> interfaceAPQI corresponding to
>> +for creating a mediated matrix device is in:
>> +
>> +/sys/devices
>> +... [vfio_ap]
>> +......[matrix]
>> +......... [mdev_supported_types]
>> +............ [vfio_ap-passthrough]
>> +............... create
>> +............... [devices]
>> +
>> +A mediated AP matrix device is created by writing a UUID to the 
>> attribute
>> +file named 'create', for example:
>> +
>> +   uuidgen > create
>> +
>> +When a mediated AP matrix device is created, a sysfs directory named 
>> after
>> +the UUID:
>> +
>> +/sys/devices
>> +... [vfio_ap]
>> +......[matrix]
>> +......... [mdev_supported_types]
>> +............ [vfio_ap-passthrough]
>> +............... create
>> +............... [devices]
>> +.................. [$uuid]
>> +
>> +There will also be three sets of attribute files created in the 
>> mediated
>> +matrix device's sysfs directory to configure an AP matrix for the
>> +KVM guest:
>> +
>> +/sys/devices
>> +... [vfio_ap]
>> +......[matrix]
>> +......... [mdev_supported_types]
>> +............ [vfio_ap-passthrough]
>> +............... create
>> +............... [devices]
>> +.................. [$uuid]
>> +..................... assign_adapter
>> +..................... assign_control_domain
>> +..................... assign_domain
>> +..................... matrix
>> +..................... unassign_adapter
>> +..................... unassign_control_domain
>> +..................... unassign_domain
>> +
>> +assign_adapter
>> +   To assign an AP adapter to the mediated matrix device, its APID 
>> is written
>> +   'assign_adapter' file. This may be done multiple times to assign 
>> more than
>> +   one adapter. The APID may be specified using conventional semantics
>> +   as a decimal, hexidecimal, or octal number. For example, to 
>> assign adapters
>> +   4, 5 and 16 to mediated matrix device $uuid in decimal, 
>> hexidecimal and octal
>> +   respectively:
>> +
>> +       echo 4 > assign_adapter
>> +       echo 0x5 > assign_adapter
>> +       echo 020
>> +
>> +unassign_adapter
>> +   To unassign an AP adapter, its APID is written to the 
>> 'unassign_adapter'
>> +   file. This may also be done multiple times to unassign more than 
>> one adapter.
>> +
>> +assign_domain
>> +   To assign a usage domain, the APQI corresponding to the domain 
>> number is
>> +   written into the 'assign_domain' file. This may be done multiple 
>> times to
>> +   assign more than one usage domain. The APQI may be specified using
>> +   conventional semantics as a decimal, hexidecimal, or octal 
>> number. For
>> +   example, to assign usage domains 4, 8, and 71 to mediated matrix 
>> device
>> +   $uuid in decimal, hexidecimal and octal respectively:
>> +
>> +      echo 4 > assign_domain
>> +      echo 0x8 > assign_domain
>> +      echo 0107 > assign_domain
>> +
>> +unassign_domain
>> +   To unassign a usage domain, the APQI corresponding to the domain 
>> number is
>> +   written into the 'unassign_domain' file. This may be done 
>> multiple times to
>> +   unassign more than one usage domain.
>> +
>> +assign_control_domain
>> +   To assign a control domain, the domain number is written into the
>> +   'assign_control_domain' file. This may be done multiple times to
>> +   assign more than one control domain. The domain number may be 
>> specified using
>> +   conventional semantics as a decimal, hexidecimal, or octal 
>> number. For
>> +   example, to assign  control domains 4, 8, and 71 to mediated 
>> matrix device
>> +   $uuid in decimal, hexidecimal and octal respectively:
>> +
>> +      echo 4 > assign_domain
>> +      echo 0x8 > assign_domain
>> +      echo 0107 > assign_domain
>> +
>> +unassign_control_domain
>> +   To unassign a control domain, the domain number is written into the
>> +   'unassign_domain' file. This may be done multiple times to 
>> unassign more than
>> +   one control domain.
>> +
>> +Notes:
>> +* Hot plug/unplug is not currently supported for mediated AP matrix 
>> devices,
>> +  so the AP matrix resulting from assignment and/or unassignment of AP
>> +  adapters, usage domains and control domains to a mediated AP 
>> matrix device
>> +  while the guest is running will not take affect until the linux 
>> guest is
>> +  rebooted.
>> +* By architectural convention, all usage domains configured for a 
>> KVM guest
>> +  will also be implicitly assigned as control domains also, to there 
>> is no
>> +  need to assign control domains that are assigned as usage domains.
>> +
>> +Starting a Linux Guest Configured with an AP Matrix:
>> +===================================================
>> +In addition to providing the sysfs interfaces for configuring the AP 
>> matrix for
>> +a linux guest, a mediated matrix device also acts as a communication 
>> pathway
>> +between QEMU and the vfio_ap device driver. To gain access to the
>> +device driver, the following option must be specified on the QEMU 
>> command line:
>> +
>> +   -device vfio_ap,sysfsdev=$path-to-mdev
>> +
>> +The sysfsdev parameter specifies the path to the mediated matrix 
>> device.
>> +There are a number of ways to specify this path:
>> +
>> +/sys/devices/vfio_ap/matrix/$uuid
>> +/sys/bus/mdev/devices/$uuid
>> +/sys/bus/mdev/drivers/vfio_mdev/$uuid
>> +/sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/devices/$uuid 
>>
>> +
>> +When the linux guest is subsequently started, the guest will open 
>> the mediated
>> +matrix device's file descriptor to get information about the 
>> mediated matrix
>> +device. The vfio_ap device driver will update the APM, AQM, and ADM 
>> fields in the
>> +guest's CRYCB with the adapter, usage domain and control domains 
>> assigned to
>> +via the mediated matrix device's sysfs attribute files. Programs 
>> running on the
>> +linux guest will then:
>> +
>> +1. Have direct access to the APQNs derived from the intersection of 
>> the AP
>> +   adapter and usage domain numbers specified in the APM and AQM 
>> respectively
>> +
>> +2. Have authorization to process AP commands to change - e.g., store 
>> a new
>> +   secure key - a control domain identified in an AP instruction 
>> sent to a valid
>> +   APQN.
>> +
>> +CPU model features:
>> +
>> +Three CPU model features are available for controlling guest access 
>> to AP
>> +facilities:
>> +
>> +1. AP facilities feature
>> +
>> +   The AP facilities feature indicates that AP facilities are 
>> installed on the
>> +   guest. This feature will be enabled by the kernel only if the AP 
>> facilities
>> +   are installed on the host system. It will turned on automatically 
>> for guests
>> +   started with CPU model zEC12 or newer. The feature is 
>> s390-specific and is
>> +   represented as a parameter of the -cpu option on the QEMU command 
>> line:
>> +
>> +      qemu-system-s390x -cpu $model,ap=on|off
>> +
>> +      Where:
>> +
>> +         $model is the CPU model defined for the guest (defaults to 
>> the model of
>> +                the host system if not specified).
>> +
>> +         ap=on|off indicates whether AP facilities are installed 
>> (on) or not
>> +                   installed (off). The default for CPU models zEC12 
>> or newer
>> +                   is ap=on. AP facilities must be installed when 
>> this parameter
>> +                   is used in conjunction with -device 
>> vfio-ap,sysfsdev=$path or
>> +                   the guest will not start.
>> +
>> +2. Query Configuration Information (QCI) facility
>> +
>> +   The QCI facility is used by the AP bus running on the guest to 
>> query the
>> +   configuration of the AP facilities. This facility will be enabled by
>> +   the kernel only if the QCI facility is installed on the host 
>> system. It will
>> +   be turned on automatically for guests started with CPU model 
>> zEC12 or newer.
>> +   The feature is s390-specific and is represented as a parameter of 
>> the -cpu
>> +   option on the QEMU command line:
>> +
>> +      qemu-system-s390x -cpu $model,qci=on|off
>> +
>> +      Where:
>> +
>> +         $model is the CPU model defined for the guest
>> +
>> +         qci=on|off indicates whether the QCI facility is installed 
>> (on) or not
>> +                    installed (off). The default for CPU models 
>> zEC12 or newer
>> +                    is qci=on. Turning the QCI facility on makes no 
>> sense if it
>> +                    is not used in conjunction with the
>> +                    '-device vfio-ap,sysfsdev=$path' option. A 
>> warning will be
>> +                    presented if QCI is turned on and the AP 
>> facilities are not
>> +                    installed.
>> +
>> +                    If the QCI facility is turned off, APQNs with an 
>> APQI
>> +                    greater than 15 will not be accessible from the 
>> guest.
>> +
>> +3. Adjunct Process Facility Test (APFT) facility
>> +
>> +   The APFT facility is used by the AP bus running on the guest to 
>> test the
>> +   AP facilities available for a given AP queue. This facility will 
>> be enabled
>> +   by the kernel only if the APFT facility is installed on the host 
>> system. It
>> +   will be turned on automatically for guests started with CPU model 
>> zEC12 or
>> +   newer. The feature is s390-specific and is represented as a 
>> parameter of the
>> +   -cpu option on the QEMU command line:
>> +
>> +      qemu-system-s390x -cpu $model,apft=on|off
>> +
>> +      Where:
>> +
>> +         $model is the CPU model defined for the guest (defaults to 
>> the model of
>> +                the host system if not specified).
>> +
>> +         apft=on|off indicates whether the APFT facility is 
>> installed (on) or
>> +                     not installed (off). The default for CPU models 
>> zEC12 and
>> +                     newer is apft=on. Turning the APFT facility on 
>> makes no
>> +                     sense if it is not used in conjunction with the
>> +                     -device vfio-ap,sysfsdev=$path option. A 
>> warning will be
>> +                     presented if APFT is turned on and the AP 
>> facilities are
>> +                     not installed.
>> +
>> +                     It also makes no sense to turn APFT off when 
>> used in
>> +                     conjunction with the vfio-ap device because the 
>> APFT
>> +                     facility is required; the AP bus running on the 
>> guest will
>> +                     not detect CEX4 and newer devices without it. 
>> Since only
>> +                     CEX4 and newer devices are supported for guest 
>> usage, no AP
>> +                     devices can be made accessible to a guest 
>> started without
>> +                     APFT installed.
>> +
>> +Example: Configure AP Matrixes for Two Linux Guests:
>> +===================================================
>> +Let's now provide an example to illustrate how KVM guests may be given
>> +access to AP facilities. For this example, we will show how to 
>> configure
>> +two guests such that executing the lszcrypt command on the guests would
>> +look like this:
>> +
>> +Guest1
>> +------
>> +CARD.DOMAIN TYPE  MODE
>> +------------------------------
>> +05          CEX5C CCA-Coproc
>> +05.0004     CEX5C CCA-Coproc
>> +05.00ab     CEX5C CCA-Coproc
>> +06          CEX5A Accelerator
>> +06.0004     CEX5A Accelerator
>> +06.00ab     CEX5C CCA-Coproc
>> +
>
> 5 whitespace at the end of lines
hmmm .... either checkpatch didn't catch it or I didn't see the error 
message,.
in either case, I'll fix it
>
>
>> +Guest2
>> +------
>> +CARD.DOMAIN TYPE  MODE
>> +------------------------------
>> +05          CEX5A Accelerator
>> +05.0047     CEX5A Accelerator
>> +05.00ff     CEX5A Accelerator
>> +
>> +These are the steps for configuring the Guest1 and Guest2:
>> +
>> +1. The first thing that needs to be done is to secure the AP queues 
>> to be
>> +   used by the two guests so that the host can not access them. This 
>> is done
>> +   by unbinding each AP Queue device from its respective AP driver. 
>> In our
>> +   example, these queues are bound to the cex4queue driver. This 
>> would be
>> +   the sysfs location of these devices:
>> +
>> +   /sys/bus/ap
>> +   --- [drivers]
>> +   ------ [cex4queue]
>> +   --------- [05.0004]
>> +   --------- [05.0047]   --------------------- control_domains
>> +   --------------------- domains
>> +   --------- [05.00ab]
>> +   --------- [05.00ff]
>> +   --------- [06.0004]
>> +   --------- [06.00ab]
>> +   --------- unbind
>> +
>> +   To unbind AP queue 05.0004 from the cex4queue device driver:
>> +
>> +    echo 05.0004 > unbind
>> +
>> +   This must also be done for AP queues 05.00ab, 05.0047, 05.00ff, 
>> 06.0004,
>> +   and 06.00ab.
>> +
>> +2. The next step is to reserve the queues for use by the two KVM 
>> guests.
>> +   This is accomplished by binding them to the VFIO AP device driver.
>> +   This is the sysfs location of the VFIO AP device driver:
>> +
>> +   /sys/bus/ap
>> +   ---[drivers]
>> +   ------ [vfio_ap]
>> +   ---------- bind
>> +
>> +   To bind queue 05.0004 to the vfio_ap driver:
>> +
>> +    echo 05.0004 > bind
>> +
>> +   This must also be done for AP queues 05.00ab, 05.0047, 05.00ff, 
>> 06.0004,
>> +   and 06.00ab.
>> +
>> +3. Create the mediated devices needed to configure the AP matrixes 
>> for the
>> +   two guests and to provide an interface to the vfio_ap driver for
>> +   use by the guests:
>> +
>> +   /sys/devices/
>> +   --- [vfio_ap]
>> +   ------ [matrix] (this is the matrix device)
>> +   --------- [mdev_supported_types]
>> +   ------------ [vfio_ap-passthrough] (passthrough mediated matrix 
>> device type)
>> +   --------------- create
>> +   --------------- [devices]
>> +
>> +   To create the mediated devices for the two guests:
>> +
>> +    uuidgen > create
>> +    uuidgen > create
>> +
>> +   This will create two mediated devices in the [devices] 
>> subdirectory named
>> +   with the UUID written to the create attribute file. We call them 
>> $uuid1
>> +   and $uuid2:
>> +
>> +   /sys/devices/
>> +   --- [vfio_ap]
>> +   ------ [matrix]
>> +   --------- [mdev_supported_types]
>> +   ------------ [vfio_ap-passthrough]
>> +   --------------- [devices]
>> +   ------------------ [$uuid1]
>> +   --------------------- assign_adapter
>> +   --------------------- assign_control_domain
>> +   --------------------- assign_domain
>> +   --------------------- matrix
>> +   --------------------- unassign_adapter
>> +   --------------------- unassign_control_domain
>> +   --------------------- unassign_domain
>> +
>> +   ------------------ [$uuid2]
>> +   --------------------- assign_adapter
>> +   --------------------- assign_cTo assign an adapter, the APID of 
>> the adapter is written to the
>> +      file. ontrol_domain
>> +   --------------------- assign_domain
>> +   --------------------- matrix
>> +   --------------------- unassign_adapter
>> +   --------------------- unassign_control_domain
>> +   --------------------- unassign_domain
>> +
>> +4. The administrator now needs to configure the matrixes for mediated
>> +   devices $uuid1 (for Guest1) and $uuid2 (for Guest2).
>> +
>> +   This is how the matrix is configured for Guest1:
>> +
>> +   echo 5 > assign_adapter
>> +   echo 6 > assign_adapter
>> +   echo 4 > assign_domain
>> +   echo 0xab > assign_domain
>> +
>> +   By architectural convention, all usage domains - i.e., domains 
>> assigned
>> +   via the assign_domain attribute file - will also be configured in 
>> the ADM
>> +   field of the KVM guest's CRYCB, so there is no need to assign 
>> control
>> +   domains here unless you want to assign control domains that are not
>> +   assigned as usage domains.
>> +
>> +   If a mistake is made configuring an adapter, domain or control 
>> domain,
>> +   you can use the unassign_xxx files to unassign the adapter, 
>> domain or
>> +   control domain.
>> +
>> +   To display the matrix configuration for Guest1:
>> +
>> +   cat matrix
>> +
>> +   This is how the matrix is configured for Guest2:
>> +
>> +   echo 5 > assign_adapter
>> +   echo 0x47 > assign_domain
>> +   echo 0xff > assign_domain
>> +
>> +5. Start Guest1
>> +
>> +   /usr/bin/qemu-system-s390x ... -device 
>> vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid1 ...
>> +
>> +6. Start Guest2
>> +
>> +   /usr/bin/qemu-system-s390x ... -device 
>> vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid2 ...
>
>

Re: [Qemu-devel] [PATCH v3 5/7] s390x/vfio: ap: Introduce VFIO AP device

[Halil Pasic]

On 03/16/2018 04:29 PM, Tony Krowiak wrote:
>>> However it is a general switch for the VM.
>>> It looks strange to have this inside the realize callback
>>>
>> I kind of semi agree.
>>
>> The previous solution (having this transparent for QEMU) had
>> benefits. Why did we move away from that again?
> This was done to address the multitude of objections and opinions related to
> how/when/where ECA_APIE is set. Pierre summed it up best with his comment "we
> need to separate the CPU feature defining 'if AP instructions are available'
> from the QEMU property defining 'How we provide the instructions'. I agree and
> this is how I chose to implement that.

It's even more separated if the one is controlled via a
userspace facing interface (cpu models) and the other
is controlled via an in-kernel interface (e.g. like
done previously in vfio open AFAIR).

I can not accept this answer. I don't recall the multitude
and I think I've showed that the 'need to separate' argument
does not work.

Regards,
Halil

Re: [Qemu-devel] [PATCH v3 5/7] s390x/vfio: ap: Introduce VFIO AP device

[Tony Krowiak]

On 03/16/2018 11:36 AM, Halil Pasic wrote:
>
> On 03/16/2018 04:29 PM, Tony Krowiak wrote:
>>>> However it is a general switch for the VM.
>>>> It looks strange to have this inside the realize callback
>>>>
>>> I kind of semi agree.
>>>
>>> The previous solution (having this transparent for QEMU) had
>>> benefits. Why did we move away from that again?
>> This was done to address the multitude of objections and opinions related to
>> how/when/where ECA_APIE is set. Pierre summed it up best with his comment "we
>> need to separate the CPU feature defining 'if AP instructions are available'
>> from the QEMU property defining 'How we provide the instructions'. I agree and
>> this is how I chose to implement that.
> It's even more separated if the one is controlled via a
> userspace facing interface (cpu models) and the other
> is controlled via an in-kernel interface (e.g. like
> done previously in vfio open AFAIR).
There were objections to that.
>
> I can not accept this answer. I don't recall the multitude
> and I think I've showed that the 'need to separate' argument
> does not work.
I suppose it depends upon how you define multitude. I counted at
at least 11 - actually more - review comments related to the
CPU model feature and setting ECA_APIE. I suggest you go back
and look at the thread.

Where did you show that the 'need to separate' argument does
not work?
>
> Regards,
> Halil

Re: [Qemu-devel] [PATCH v3 5/7] s390x/vfio: ap: Introduce VFIO AP device

[Halil Pasic]

On 03/16/2018 04:53 PM, Tony Krowiak wrote:
> On 03/16/2018 11:36 AM, Halil Pasic wrote:
>>
>> On 03/16/2018 04:29 PM, Tony Krowiak wrote:
>>>>> However it is a general switch for the VM.
>>>>> It looks strange to have this inside the realize callback
>>>>>
>>>> I kind of semi agree.
>>>>
>>>> The previous solution (having this transparent for QEMU) had
>>>> benefits. Why did we move away from that again?
>>> This was done to address the multitude of objections and opinions related to
>>> how/when/where ECA_APIE is set. Pierre summed it up best with his comment "we
>>> need to separate the CPU feature defining 'if AP instructions are available'
>>> from the QEMU property defining 'How we provide the instructions'. I agree and
>>> this is how I chose to implement that.
>> It's even more separated if the one is controlled via a
>> userspace facing interface (cpu models) and the other
>> is controlled via an in-kernel interface (e.g. like
>> done previously in vfio open AFAIR).
> There were objections to that.

AFAIR we agreed that keeping that interface is OK, but I may be wrong.
You being more specific that 'there were objections'  and 'multitude
of objections and opinions' would have been appreciated. But never
mind.

AFAIU Pierre is nagging you about this in the corresponding kernel
thread. I will let Pierre drive this discussion.

>>
>> I can not accept this answer. I don't recall the multitude
>> and I think I've showed that the 'need to separate' argument
>> does not work.
> I suppose it depends upon how you define multitude. I counted at
> at least 11 - actually more - review comments related to the
> CPU model feature and setting ECA_APIE. I suggest you go back
> and look at the thread.
> 
> Where did you show that the 'need to separate' argument does
> not work?

You answered to that part with 'There were objections to that'.
As I've said before never mind.

Halil

Re: [Qemu-devel] [PATCH v3 4/7] s390x/kvm: interface to interpret AP instructions

[Tony Krowiak]

On 03/15/2018 07:24 PM, Tony Krowiak wrote:
> The VFIO AP device exploits interpretive execution of AP
> instructions (APIE). APIE is enabled by setting a device attribute
> via the KVM_SET_DEVICE_ATTR ioctl.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>   target/s390x/kvm.c       |   16 ++++++++++++++++
>   target/s390x/kvm_s390x.h |    2 ++
>   2 files changed, 18 insertions(+), 0 deletions(-)
>
> diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c
> index 33e5ec3..2812e28 100644
> --- a/target/s390x/kvm.c
> +++ b/target/s390x/kvm.c
> @@ -277,6 +277,22 @@ static void kvm_s390_init_dea_kw(void)
>       }
>   }
>
> +int kvm_s390_set_interpret_ap(uint8_t enable)
> +{
> +    struct kvm_device_attr attribute = {
> +        .group = KVM_S390_VM_CRYPTO,
> +        .attr  = KVM_S390_VM_CRYPTO_INTERPRET_AP,
> +        .addr = 1,
> +    };
> +
> +    if (!kvm_vm_check_attr(kvm_state, KVM_S390_VM_CRYPTO,
> +                           KVM_S390_VM_CRYPTO_INTERPRET_AP)) {
> +        return -EOPNOTSUPP;
> +    }
> +
> +    return kvm_vm_ioctl(kvm_state, KVM_SET_DEVICE_ATTR, &attribute);
> +}
I proposed removing the KVM_S390_VM_CRYPTO_INTERPRET_AP device attribute 
from the kernel in
Message ID <1347ed2e-7bdb-e455-971a-cf60899e3c19@linux.vnet.ibm.com> on 
the kernel mailing list
and proposed setting interpretive execution for AP instructions via the 
mdev open callback. That
would eliminate the need for this patch.
> +
>   void kvm_s390_crypto_reset(void)
>   {
>       if (s390_has_feat(S390_FEAT_MSA_EXT_3)) {
> diff --git a/target/s390x/kvm_s390x.h b/target/s390x/kvm_s390x.h
> index 34ee7e7..0d6c6e7 100644
> --- a/target/s390x/kvm_s390x.h
> +++ b/target/s390x/kvm_s390x.h
> @@ -40,4 +40,6 @@ void kvm_s390_crypto_reset(void);
>   void kvm_s390_restart_interrupt(S390CPU *cpu);
>   void kvm_s390_stop_interrupt(S390CPU *cpu);
>
> +int kvm_s390_set_interpret_ap(uint8_t enable);
> +
>   #endif /* KVM_S390X_H */

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[David Hildenbrand]

On 16.03.2018 00:24, Tony Krowiak wrote:
> If the CPU model indicates that AP facility is installed on
> the guest (i.e., -cpu xxxx,ap=on), then the expectation is that
> the AP bus running in the guest will initialize; however, if the
> AP instructions are not being interpreted by the firmware, then
> they will be intercepted and routed back to QEMU for handling.
> If a handler is not defined to process the intercepted instruciton,
> then an operation exception will be injected into the
> guest, in which case the AP bus will not initialize.
> 
> There are two situations where AP instructions will not be
> interpreted:
> 
> 1. The guest is not configured with a vfio-ap device (i.e.,
>    -device vfio-ap,sysfsdev=$path-to-mdev). The realize function for
>    the vfio-ap device enables interpretive execution of AP
>    instructions.
> 
> 2. The guest is a second level guest but the first level guest has
>    not enabled interpretive execution.
> 
> This patch introduces AP instruction handlers to ensure the AP bus
> module initializes on the guest when the AP facility is installed
> on the guest but AP instructions are not being interpreted. The logic
> incorporated is:
> 
> * If the CPU model indicates AP instructions are
>   installed
> 
>   * Set the status response code for the instruction to indicate that
>     the APQN contained in the instruction is not valid. This is
>     a valid response because there will be no devices configured for
>     the guest in any of the above scenarios.
> 
> * Else return an error from the handler. This will result in an
>   operation being injected into the guest and the AP bus will not

s/operation/operation exception/

>   initialize on the guest. That is commensurate with how things work
>   today.
> 
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>  hw/vfio/ap.c                 |   45 ++++++++++++++++++++++++++++++++++++++++++
>  include/hw/s390x/ap-device.h |    6 +++++
>  target/s390x/kvm.c           |   14 +++++++++++++
>  3 files changed, 65 insertions(+), 0 deletions(-)
> 
> diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c
> index b397bb1..88e744d 100644
> --- a/hw/vfio/ap.c
> +++ b/hw/vfio/ap.c
> @@ -148,6 +148,51 @@ static void vfio_ap_unrealize(DeviceState *dev, Error **errp)
>      kvm_s390_set_interpret_ap(0);
>  }
>  

I think you are missing cpu_synchronize_state(CPU(cpu)) in all of these
functions.

> +int ap_device_handle_nqap(S390CPU *cpu)
> +{
> +    CPUS390XState *env = &cpu->env;
> +
> +    if (s390_has_feat(S390_FEAT_AP)) {
> +        env->regs[1] = 0x10000;
> +
> +        return 0;
> +    }
> +
> +    return -EOPNOTSUPP;
> +}
> +
> +int ap_device_handle_dqap(S390CPU *cpu)
> +{
> +    CPUS390XState *env = &cpu->env;
> +
> +    if (s390_has_feat(S390_FEAT_AP)) {
> +        env->regs[1] = 0x10000;
> +
> +        return 0;
> +    }
> +
> +    return -EOPNOTSUPP;
> +}
> +
> +int ap_device_handle_pqap(S390CPU *cpu)
> +{
> +    CPUS390XState *env = &cpu->env;
> +    int fc = 4 & (env->regs[0] >> 24);
> +
> +    /*
> +     * The Query Configuration Information (QCI) function (fc == 4) does not
> +     * set a response code in reg 1, so check for that along with the
> +     * AP feature.
> +     */
> +    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
> +        env->regs[1] = 0x10000;
> +
> +        return 0;
> +    }

This would imply an operation exception in case fc==4, which sounds very
wrong.

Hard to review without access to documentation. (hard to understand why
such stuff is to be kept confidential for decades)

> +
> +    return -EOPNOTSUPP;
> +}
> +
>  static Property vfio_ap_properties[] = {
>      DEFINE_PROP_STRING("sysfsdev", VFIOAPDevice, vdev.sysfsdev),
>      DEFINE_PROP_END_OF_LIST(),
> diff --git a/include/hw/s390x/ap-device.h b/include/hw/s390x/ap-device.h
> index 693df90..d45ae38 100644
> --- a/include/hw/s390x/ap-device.h
> +++ b/include/hw/s390x/ap-device.h
> @@ -11,6 +11,8 @@
>  #ifndef HW_S390X_AP_DEVICE_H
>  #define HW_S390X_AP_DEVICE_H
>  


-- 

Thanks,

David / dhildenb

Re: [Qemu-devel] [PATCH v3 4/7] s390x/kvm: interface to interpret AP instructions

[David Hildenbrand]

On 16.03.2018 00:24, Tony Krowiak wrote:
> The VFIO AP device exploits interpretive execution of AP
> instructions (APIE). APIE is enabled by setting a device attribute
> via the KVM_SET_DEVICE_ATTR ioctl.
> 
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>  target/s390x/kvm.c       |   16 ++++++++++++++++
>  target/s390x/kvm_s390x.h |    2 ++
>  2 files changed, 18 insertions(+), 0 deletions(-)
> 
> diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c
> index 33e5ec3..2812e28 100644
> --- a/target/s390x/kvm.c
> +++ b/target/s390x/kvm.c
> @@ -277,6 +277,22 @@ static void kvm_s390_init_dea_kw(void)
>      }
>  }
>  
> +int kvm_s390_set_interpret_ap(uint8_t enable)
> +{
> +    struct kvm_device_attr attribute = {
> +        .group = KVM_S390_VM_CRYPTO,
> +        .attr  = KVM_S390_VM_CRYPTO_INTERPRET_AP,
> +        .addr = 1,
> +    };
> +
> +    if (!kvm_vm_check_attr(kvm_state, KVM_S390_VM_CRYPTO,
> +                           KVM_S390_VM_CRYPTO_INTERPRET_AP)) {
> +        return -EOPNOTSUPP;
> +    }
> +
> +    return kvm_vm_ioctl(kvm_state, KVM_SET_DEVICE_ATTR, &attribute);
> +}
> +
>  void kvm_s390_crypto_reset(void)
>  {
>      if (s390_has_feat(S390_FEAT_MSA_EXT_3)) {
> diff --git a/target/s390x/kvm_s390x.h b/target/s390x/kvm_s390x.h
> index 34ee7e7..0d6c6e7 100644
> --- a/target/s390x/kvm_s390x.h
> +++ b/target/s390x/kvm_s390x.h
> @@ -40,4 +40,6 @@ void kvm_s390_crypto_reset(void);
>  void kvm_s390_restart_interrupt(S390CPU *cpu);
>  void kvm_s390_stop_interrupt(S390CPU *cpu);
>  
> +int kvm_s390_set_interpret_ap(uint8_t enable);
> +
>  #endif /* KVM_S390X_H */
> 

Wonder if a capability - like we use e.g. for SIGP user space
interpretation - would be a better fit.

We can provide the AP feature to the guest in case:
- KVM_S390_VM_CPU_FEAT_AP ("interpretation support") is available
- KVM_S390_VM_CRYPTO_INTERPRET_AP ("interception support") is available

I am missing the second check in your code. (for now you only rely on
KVM_S390_VM_CPU_FEAT_AP)

I think you have to change the order of the patches so they work also
properly when bisectin.

-- 

Thanks,

David / dhildenb

Re: [Qemu-devel] [qemu-s390x] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[David Hildenbrand]

On 26.03.2018 10:32, David Hildenbrand wrote:
> On 16.03.2018 00:24, Tony Krowiak wrote:
>> If the CPU model indicates that AP facility is installed on
>> the guest (i.e., -cpu xxxx,ap=on), then the expectation is that
>> the AP bus running in the guest will initialize; however, if the
>> AP instructions are not being interpreted by the firmware, then
>> they will be intercepted and routed back to QEMU for handling.
>> If a handler is not defined to process the intercepted instruciton,
>> then an operation exception will be injected into the
>> guest, in which case the AP bus will not initialize.
>>
>> There are two situations where AP instructions will not be
>> interpreted:
>>
>> 1. The guest is not configured with a vfio-ap device (i.e.,
>>    -device vfio-ap,sysfsdev=$path-to-mdev). The realize function for
>>    the vfio-ap device enables interpretive execution of AP
>>    instructions.
>>
>> 2. The guest is a second level guest but the first level guest has
>>    not enabled interpretive execution.
>>
>> This patch introduces AP instruction handlers to ensure the AP bus
>> module initializes on the guest when the AP facility is installed
>> on the guest but AP instructions are not being interpreted. The logic
>> incorporated is:
>>
>> * If the CPU model indicates AP instructions are
>>   installed
>>
>>   * Set the status response code for the instruction to indicate that
>>     the APQN contained in the instruction is not valid. This is
>>     a valid response because there will be no devices configured for
>>     the guest in any of the above scenarios.
>>
>> * Else return an error from the handler. This will result in an
>>   operation being injected into the guest and the AP bus will not
> 
> s/operation/operation exception/
> 
>>   initialize on the guest. That is commensurate with how things work
>>   today.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>>  hw/vfio/ap.c                 |   45 ++++++++++++++++++++++++++++++++++++++++++
>>  include/hw/s390x/ap-device.h |    6 +++++
>>  target/s390x/kvm.c           |   14 +++++++++++++
>>  3 files changed, 65 insertions(+), 0 deletions(-)
>>
>> diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c
>> index b397bb1..88e744d 100644
>> --- a/hw/vfio/ap.c
>> +++ b/hw/vfio/ap.c
>> @@ -148,6 +148,51 @@ static void vfio_ap_unrealize(DeviceState *dev, Error **errp)
>>      kvm_s390_set_interpret_ap(0);
>>  }
>>  
> 
> I think you are missing cpu_synchronize_state(CPU(cpu)) in all of these
> functions.
> 

No you don't. There is just a duplicate cpu_synchronize_state(CPU(cpu))
in kvm_sclp_service_call() which confused me.


-- 

Thanks,

David / dhildenb

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Pierre Morel]

On 26/03/2018 10:32, David Hildenbrand wrote:
> On 16.03.2018 00:24, Tony Krowiak wrote:
>> If the CPU model indicates that AP facility is installed on
>> the guest (i.e., -cpu xxxx,ap=on), then the expectation is that
>> the AP bus running in the guest will initialize; however, if the
>> AP instructions are not being interpreted by the firmware, then
>> they will be intercepted and routed back to QEMU for handling.
>> If a handler is not defined to process the intercepted instruciton,
>> t
...snip...
>
>> +int ap_device_handle_nqap(S390CPU *cpu)
>> +{
>> +    CPUS390XState *env = &cpu->env;
>> +
>> +    if (s390_has_feat(S390_FEAT_AP)) {
>> +        env->regs[1] = 0x10000;
>> +
>> +        return 0;
>> +    }
>> +
>> +    return -EOPNOTSUPP;
>> +}
>> +
>> +int ap_device_handle_dqap(S390CPU *cpu)
>> +{
>> +    CPUS390XState *env = &cpu->env;
>> +
>> +    if (s390_has_feat(S390_FEAT_AP)) {
>> +        env->regs[1] = 0x10000;
>> +
>> +        return 0;
>> +    }
>> +
>> +    return -EOPNOTSUPP;
>> +}
>> +
>> +int ap_device_handle_pqap(S390CPU *cpu)
>> +{
>> +    CPUS390XState *env = &cpu->env;
>> +    int fc = 4 & (env->regs[0] >> 24);
>> +
>> +    /*
>> +     * The Query Configuration Information (QCI) function (fc == 4) does not
>> +     * set a response code in reg 1, so check for that along with the
>> +     * AP feature.
>> +     */
>> +    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
>> +        env->regs[1] = 0x10000;
>> +
>> +        return 0;
>> +    }
> This would imply an operation exception in case fc==4, which sounds very
> wrong.

It depends but I think that the S390_FEAT_AP_QUERY_CONFIG_INFO must be 
tested
to know what to answer.
If the feature is there, QCI must be answered correctly.

there are also some error situations to handle in all three functions.


>
> Hard to review without access to documentation. (hard to understand why
> such stuff is to be kept confidential for decades)
>
>> +
>> +    return -EOPNOTSUPP;
>> +}
>> +
>>   static Property vfio_ap_properties[] = {
>>       DEFINE_PROP_STRING("sysfsdev", VFIOAPDevice, vdev.sysfsdev),
>>       DEFINE_PROP_END_OF_LIST(),
>> diff --git a/include/hw/s390x/ap-device.h b/include/hw/s390x/ap-device.h
>> index 693df90..d45ae38 100644
>> --- a/include/hw/s390x/ap-device.h
>> +++ b/include/hw/s390x/ap-device.h
>> @@ -11,6 +11,8 @@
>>   #ifndef HW_S390X_AP_DEVICE_H
>>   #define HW_S390X_AP_DEVICE_H
>>   
>

-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Halil Pasic]

On 03/26/2018 11:03 AM, Pierre Morel wrote:
>>> +int ap_device_handle_pqap(S390CPU *cpu)
>>> +{
>>> +    CPUS390XState *env = &cpu->env;
>>> +    int fc = 4 & (env->regs[0] >> 24);
>>> +
>>> +    /*
>>> +     * The Query Configuration Information (QCI) function (fc == 4) does not
>>> +     * set a response code in reg 1, so check for that along with the
>>> +     * AP feature.
>>> +     */
>>> +    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
>>> +        env->regs[1] = 0x10000;
>>> +
>>> +        return 0;
>>> +    }
>> This would imply an operation exception in case fc==4, which sounds very
>> wrong.

Yes, operation exception is a wrong response under the condition
(fc == 4) && s390_has_feat(S390_FEAT_AP).

@David:
FYI Tony is likely to respond after Wednesday as he is on vacation right
now.

> 
> It depends but I think that the S390_FEAT_AP_QUERY_CONFIG_INFO must be tested
> to know what to answer.
> If the feature is there, QCI must be answered correctly.
> 
> there are also some error situations to handle in all three functions.
>

I agree. 
 
Regards,
Halil

Re: [Qemu-devel] [PATCH v3 5/7] s390x/vfio: ap: Introduce VFIO AP device

[Cornelia Huck]

On Fri, 16 Mar 2018 14:22:52 +0100
Halil Pasic <pasic@linux.vnet.ibm.com> wrote:

> On 03/16/2018 11:42 AM, Pierre Morel wrote:
> > On 16/03/2018 00:24, Tony Krowiak wrote:  
> >> Introduces a VFIO based AP device. The device is defined via
> >> the QEMU command line by specifying:
> >>
> >>      -device vfio-ap,sysfsdev=<path-to-mediated-matrix-device>
> >>
> >> The mediated matrix device is created by the VFIO AP device
> >> driver by writing a UUID to a sysfs attribute file (see
> >> docs/vfio-ap.txt). The mediated matrix device will be named
> >> after the UUID. Symbolic links to the $uuid are created in
> >> many places, so the path to the mediated matrix device $uuid
> >> can be specified in any of the following ways:
> >>
> >> /sys/devices/vfio_ap/matrix/$uuid
> >> /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/devices/$uuid
> >> /sys/bus/mdev/devices/$uuid
> >> /sys/bus/mdev/drivers/vfio_mdev/$uuid
> >>
> >> When the vfio-ap device is realized, it acquires and opens the
> >> VFIO iommu group to which the mediated matrix device is
> >> bound. This causes a VFIO group notification event to be
> >> signaled. The vfio_ap device driver's group notification
> >> handler will get called at which time the device driver
> >> will configure the the AP devices to which the guest will
> >> be granted access.
> >>
> >> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> >> ---  
> [..]
> >> +static void vfio_ap_realize(DeviceState *dev, Error **errp)
> >> +{
> >> +    VFIODevice *vbasedev;
> >> +    VFIOGroup *vfio_group;
> >> +    APDevice *apdev = DO_UPCAST(APDevice, parent_obj, dev);
> >> +    VFIOAPDevice *vapdev = DO_UPCAST(VFIOAPDevice, apdev, apdev);
> >> +    char *mdevid;
> >> +    Error *local_err = NULL;
> >> +    int ret;
> >> +
> >> +    if (!s390_has_feat(S390_FEAT_AP)) {
> >> +        error_setg(&local_err, "AP support not enabled");
> >> +        goto out_err;
> >> +    }
> >> +
> >> +    ret = kvm_s390_set_interpret_ap(1);  
> > 
> > If we have several devices, this is called once per device.  
> 
> I don't think having several of these in a single vm makes
> any sense. Or does it? IMHO we should make sure there is at
> most one device taking care of the crypto pass-through.

Yes, I think we should fence off adding a second device in the realize
function (probably by checking a global variable?)

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Tony Krowiak]

On 03/26/2018 04:32 AM, David Hildenbrand wrote:
> On 16.03.2018 00:24, Tony Krowiak wrote:
>> If the CPU model indicates that AP facility is installed on
>> the guest (i.e., -cpu xxxx,ap=on), then the expectation is that
>> the AP bus running in the guest will initialize; however, if the
>> AP instructions are not being interpreted by the firmware, then
>> they will be intercepted and routed back to QEMU for handling.
>> If a handler is not defined to process the intercepted instruciton,
>> then an operation exception will be injected into the
>> guest, in which case the AP bus will not initialize.
>>
>> There are two situations where AP instructions will not be
>> interpreted:
>>
>> 1. The guest is not configured with a vfio-ap device (i.e.,
>>     -device vfio-ap,sysfsdev=$path-to-mdev). The realize function for
>>     the vfio-ap device enables interpretive execution of AP
>>     instructions.
>>
>> 2. The guest is a second level guest but the first level guest has
>>     not enabled interpretive execution.
>>
>> This patch introduces AP instruction handlers to ensure the AP bus
>> module initializes on the guest when the AP facility is installed
>> on the guest but AP instructions are not being interpreted. The logic
>> incorporated is:
>>
>> * If the CPU model indicates AP instructions are
>>    installed
>>
>>    * Set the status response code for the instruction to indicate that
>>      the APQN contained in the instruction is not valid. This is
>>      a valid response because there will be no devices configured for
>>      the guest in any of the above scenarios.
>>
>> * Else return an error from the handler. This will result in an
>>    operation being injected into the guest and the AP bus will not
> s/operation/operation exception/
thanks
>
>>    initialize on the guest. That is commensurate with how things work
>>    today.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>>   hw/vfio/ap.c                 |   45 ++++++++++++++++++++++++++++++++++++++++++
>>   include/hw/s390x/ap-device.h |    6 +++++
>>   target/s390x/kvm.c           |   14 +++++++++++++
>>   3 files changed, 65 insertions(+), 0 deletions(-)
>>
>> diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c
>> index b397bb1..88e744d 100644
>> --- a/hw/vfio/ap.c
>> +++ b/hw/vfio/ap.c
>> @@ -148,6 +148,51 @@ static void vfio_ap_unrealize(DeviceState *dev, Error **errp)
>>       kvm_s390_set_interpret_ap(0);
>>   }
>>   
> I think you are missing cpu_synchronize_state(CPU(cpu)) in all of these
> functions.
I see you negated this in your next response.
>
>> +int ap_device_handle_nqap(S390CPU *cpu)
>> +{
>> +    CPUS390XState *env = &cpu->env;
>> +
>> +    if (s390_has_feat(S390_FEAT_AP)) {
>> +        env->regs[1] = 0x10000;
>> +
>> +        return 0;
>> +    }
>> +
>> +    return -EOPNOTSUPP;
>> +}
>> +
>> +int ap_device_handle_dqap(S390CPU *cpu)
>> +{
>> +    CPUS390XState *env = &cpu->env;
>> +
>> +    if (s390_has_feat(S390_FEAT_AP)) {
>> +        env->regs[1] = 0x10000;
>> +
>> +        return 0;
>> +    }
>> +
>> +    return -EOPNOTSUPP;
>> +}
>> +
>> +int ap_device_handle_pqap(S390CPU *cpu)
>> +{
>> +    CPUS390XState *env = &cpu->env;
>> +    int fc = 4 & (env->regs[0] >> 24);
>> +
>> +    /*
>> +     * The Query Configuration Information (QCI) function (fc == 4) does not
>> +     * set a response code in reg 1, so check for that along with the
>> +     * AP feature.
>> +     */
>> +    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
>> +        env->regs[1] = 0x10000;
>> +
>> +        return 0;
>> +    }
> This would imply an operation exception in case fc==4, which sounds very
> wrong.
Take a look at my response to Pierre for an answer to this one.
>
> Hard to review without access to documentation. (hard to understand why
> such stuff is to be kept confidential for decades)
You'll have to talk to the powers that be at IBM about that one :)
>
>> +
>> +    return -EOPNOTSUPP;
>> +}
>> +
>>   static Property vfio_ap_properties[] = {
>>       DEFINE_PROP_STRING("sysfsdev", VFIOAPDevice, vdev.sysfsdev),
>>       DEFINE_PROP_END_OF_LIST(),
>> diff --git a/include/hw/s390x/ap-device.h b/include/hw/s390x/ap-device.h
>> index 693df90..d45ae38 100644
>> --- a/include/hw/s390x/ap-device.h
>> +++ b/include/hw/s390x/ap-device.h
>> @@ -11,6 +11,8 @@
>>   #ifndef HW_S390X_AP_DEVICE_H
>>   #define HW_S390X_AP_DEVICE_H
>>   
>

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Tony Krowiak]

On 03/26/2018 05:03 AM, Pierre Morel wrote:
> On 26/03/2018 10:32, David Hildenbrand wrote:
>> On 16.03.2018 00:24, Tony Krowiak wrote:
>>> If the CPU model indicates that AP facility is installed on
>>> the guest (i.e., -cpu xxxx,ap=on), then the expectation is that
>>> the AP bus running in the guest will initialize; however, if the
>>> AP instructions are not being interpreted by the firmware, then
>>> they will be intercepted and routed back to QEMU for handling.
>>> If a handler is not defined to process the intercepted instruciton,
>>> t
> ...snip...
>>
>>> +int ap_device_handle_nqap(S390CPU *cpu)
>>> +{
>>> +    CPUS390XState *env = &cpu->env;
>>> +
>>> +    if (s390_has_feat(S390_FEAT_AP)) {
>>> +        env->regs[1] = 0x10000;
>>> +
>>> +        return 0;
>>> +    }
>>> +
>>> +    return -EOPNOTSUPP;
>>> +}
>>> +
>>> +int ap_device_handle_dqap(S390CPU *cpu)
>>> +{
>>> +    CPUS390XState *env = &cpu->env;
>>> +
>>> +    if (s390_has_feat(S390_FEAT_AP)) {
>>> +        env->regs[1] = 0x10000;
>>> +
>>> +        return 0;
>>> +    }
>>> +
>>> +    return -EOPNOTSUPP;
>>> +}
>>> +
>>> +int ap_device_handle_pqap(S390CPU *cpu)
>>> +{
>>> +    CPUS390XState *env = &cpu->env;
>>> +    int fc = 4 & (env->regs[0] >> 24);
>>> +
>>> +    /*
>>> +     * The Query Configuration Information (QCI) function (fc == 4) 
>>> does not
>>> +     * set a response code in reg 1, so check for that along with the
>>> +     * AP feature.
>>> +     */
>>> +    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
>>> +        env->regs[1] = 0x10000;
>>> +
>>> +        return 0;
>>> +    }
>> This would imply an operation exception in case fc==4, which sounds very
>> wrong.
>
> It depends but I think that the S390_FEAT_AP_QUERY_CONFIG_INFO must be 
> tested
> to know what to answer.
> If the feature is there, QCI must be answered correctly.
This is an interesting proposition which raises several issues that will 
need to
be addressed. The only time the PQAP(QCI) instruction is intercepted is 
when:
* A vfio-ap device is NOT defined for the guest because the vfio_ap 
device driver
   will set ECA.28 and the PQAP(QCI) instruction will be interpreted
* STFLE.12 is set for the guest

You say that the QCI must be answered correctly, but what is the correct 
response?
If we return the matrix - i.e., APM, ADM and AQM - configured via the 
mediated
matrix device's sysfs attributes files, then if any APQNs are defined in 
the matrix,
we will have to handle *ALL* AP instructions by executing them on behalf 
of the
guest. I suppose we could return an empty matrix in which case the AP 
bus will come
up without any devices on the guest, but what is the expectation of an 
admin who
deliberately configures the mediated matrix device? Should we forego 
handling interception
of AP instructions and consider a guest configuration that turns on 
S390_FEAT_AP but
does not define a vfio-ap device to be erroneous and terminate starting 
of the guest?
Anybody have any thoughts?
>
>
> there are also some error situations to handle in all three functions.
To what error situations are you referring?
>
>
>
>>
>> Hard to review without access to documentation. (hard to understand why
>> such stuff is to be kept confidential for decades)
>>
>>> +
>>> +    return -EOPNOTSUPP;
>>> +}
>>> +
>>>   static Property vfio_ap_properties[] = {
>>>       DEFINE_PROP_STRING("sysfsdev", VFIOAPDevice, vdev.sysfsdev),
>>>       DEFINE_PROP_END_OF_LIST(),
>>> diff --git a/include/hw/s390x/ap-device.h 
>>> b/include/hw/s390x/ap-device.h
>>> index 693df90..d45ae38 100644
>>> --- a/include/hw/s390x/ap-device.h
>>> +++ b/include/hw/s390x/ap-device.h
>>> @@ -11,6 +11,8 @@
>>>   #ifndef HW_S390X_AP_DEVICE_H
>>>   #define HW_S390X_AP_DEVICE_H
>>
>

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Tony Krowiak]

On 03/26/2018 08:01 AM, Halil Pasic wrote:
>
> On 03/26/2018 11:03 AM, Pierre Morel wrote:
>>>> +int ap_device_handle_pqap(S390CPU *cpu)
>>>> +{
>>>> +    CPUS390XState *env = &cpu->env;
>>>> +    int fc = 4 & (env->regs[0] >> 24);
>>>> +
>>>> +    /*
>>>> +     * The Query Configuration Information (QCI) function (fc == 4) does not
>>>> +     * set a response code in reg 1, so check for that along with the
>>>> +     * AP feature.
>>>> +     */
>>>> +    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
>>>> +        env->regs[1] = 0x10000;
>>>> +
>>>> +        return 0;
>>>> +    }
>>> This would imply an operation exception in case fc==4, which sounds very
>>> wrong.
> Yes, operation exception is a wrong response under the condition
> (fc == 4) && s390_has_feat(S390_FEAT_AP).
See my response to Pierre:
Message ID: <0b957a5c-1a87-7952-292d-f65052bb6c5a@linux.vnet.ibm.com>
>
> @David:
> FYI Tony is likely to respond after Wednesday as he is on vacation right
> now.
>
>> It depends but I think that the S390_FEAT_AP_QUERY_CONFIG_INFO must be tested
>> to know what to answer.
>> If the feature is there, QCI must be answered correctly.
>>
>> there are also some error situations to handle in all three functions.
>>
> I agree.
>
> Regards,
> Halil

Re: [Qemu-devel] [PATCH v3 7/7] s390: doc: detailed specifications for AP virtualization

[Tony Krowiak]

On 03/16/2018 06:03 AM, Pierre Morel wrote:
> On 16/03/2018 00:25, Tony Krowiak wrote:
>> This patch provides documentation describing the AP architecture and
>> design concepts behind the virtualization of AP devices. It also
>> includes an example of how to configure AP devices for exclusive
>> use of KVM guests.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>>   docs/vfio-ap.txt |  624 
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>   1 files changed, 624 insertions(+), 0 deletions(-)
>>   create mode 100644 docs/vfio-ap.txt
>>
>> diff --git a/docs/vfio-ap.txt b/docs/vfio-ap.txt
>> new file mode 100644
>> index 0000000..54e7523
>> --- /dev/null
>> +++ b/docs/vfio-ap.txt
>> @@ -0,0 +1,624 @@
>> +Adjunct Processor (AP) Device
>> +=============================
>> +
>> +Contents:
>> +=========
>> +* Introduction
>> +* AP Architectural Overview
>> +* Start Interpretive Execution (SIE) Instruction
>> +* AP Matrix Configuration on Linux Host
>> +* AP Matrix Configuration for a Linux Guest
>> +* Starting a Linux Guest Configured with an AP Matrix
>> +* Example: Configure AP Matrices for Two Linux Guests
>> +
>> +Introduction:
>> +============
>> +The IBM Adjunct Processor (AP) Cryptographic Facility is comprised
>> +of three AP instructions and from 1 to 256 PCIe cryptographic 
>> adapter cards.
>> +These AP devices provide cryptographic functions to all CPUs 
>> assigned to a
>> +linux system running in an IBM Z system LPAR.
>> +
>> +On s390x, AP adapter cards are exposed via the AP bus. This document
>> +describes how those cards may be made available to KVM guests using the
>> +VFIO mediated device framework.
>> +
>> +AP Architectural Overview:
>> +=========================
>> +In order understand the terminology used in the rest of this 
>> document, let's
>> +start with some definitions:
>> +
>> +* AP adapter
>> +
>> +  An AP adapter is an IBM Z adapter card that can perform cryptographic
>> +  functions. There can be from 0 to 256 adapters assigned to an 
>> LPAR. Adapters
>> +  assigned to the LPAR in which a linux host is running will be 
>> available to
>> +  the linux host. Each adapter is identified by a number from 0 to 
>> 255. When
>> +  installed, an AP adapter is accessed by AP instructions executed 
>> by any CPU.
>> +
>> +* AP domain
>> +
>> +  An adapter is partitioned into domains. Each domain can be thought 
>> of as
>> +  a set of hardware registers for processing AP instructions. An 
>> adapter can
>> +  hold up to 256 domains. Each domain is identified by a number from 
>> 0 to 255.
>> +  Domains can be further classified into two types:
>> +
>> +    * Usage domains are domains that can be accessed directly to 
>> process AP
>> +      commands
>> +
>> +    * Control domains are domains that are accessed indirectly by AP
>> +      commands sent to a usage domain to control or change the 
>> domain, for
>> +      example; to set a secure private key for the domain.
>> +
>> +* AP Queue
>> +
>> +  An AP queue is the means by which an AP command-request message is 
>> sent to an
>> +  AP usage domain inside a specific AP. An AP queue is identified by 
>> a tuple
>> +  comprised of an AP adapter ID (APID) and an AP queue index (APQI). 
>> The
>> +  APQI corresponds to a given usage domain number within the 
>> adapter. This tuple
>> +  forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP
>> +  instructions include a field containing the APQN to identify the 
>> AP queue to
>> +  which the AP command-request message is to be sent for processing.
>> +
>> +* AP Instructions:
>> +
>> +  There are three AP instructions:
>> +
>> +  * NQAP: to enqueue an AP command-request message to a queue
>> +  * DQAP: to dequeue an AP command-reply message from a queue
>> +  * PQAP: to administer the queues
>> +
>> +Start Interpretive Execution (SIE) Instruction
>> +==============================================
>> +A KVM guest is started by executing the Start Interpretive Execution 
>> (SIE)
>> +instruction. The SIE state description is a control block that 
>> contains the
>> +state information for a KVM guest and is supplied as input to the SIE
>> +instruction. The SIE state description contains a field that references
>> +a Crypto Control Block (CRYCB). The CRYCB contains three fields to 
>> identify the
>> +adapters, usage domains and control domains assigned to the KVM guest:
>> +
>> +* The AP Mask (APM) field is a bit mask that identifies the AP 
>> adapters assigned
>> +  to the KVM guest. Each bit in the mask, from most significant to 
>> least
>> +  significant bit, corresponds to an APID from 0-255. If a bit is 
>> set, the
>> +  corresponding adapter is valid for use by the KVM guest.
>> +
>> +* The AP Queue Mask (AQM) field is a bit mask identifying the AP 
>> queues assigned
>> +  to the KVM guest. Each bit in the mask, from most significant to 
>> least
>> +  significant bit, corresponds to an AP queue index (APQI) from 
>> 0-255. If a bit
>> +  is set, the corresponding queue is valid for use by the KVM guest.
>> +
>> +* The AP Domain Mask field is a bit mask that identifies the AP 
>> control domains
>> +  assigned to the KVM guest. The ADM bit mask controls which domains 
>> can be
>> +  changed by an AP command-request message sent to a usage domain 
>> from the
>> +  guest. Each bit in the mask, from least significant to most 
>> significant bit,
>> +  corresponds to a domain from 0-255. If a bit is set, the 
>> corresponding domain
>> +  can be modified by an AP command-request message sent to a usage 
>> domain
>> +  configured for the KVM guest.
>> +
>> +If you recall from the description of an AP Queue, AP instructions 
>> include
>> +an APQN to identify the AP adapter and AP queue to which an AP 
>> command-request
>> +message is to be sent (NQAP and PQAP instructions), or from which a
>> +command-reply message is to be received (DQAP instruction). The 
>> validity of an
>> +APQN is defined by the matrix calculated from the APM and AQM; it is 
>> the
>> +intersection of all assigned adapter numbers (APM) with all assigned 
>> queue
>> +indexes (AQM). For example, if adapters 1 and 2 and usage domains 5 
>> and 6 are
>> +assigned to a guest, the APQNs (1,5), (1,6), (2,5) and (2,6) will be 
>> valid for
>> +the guest.
>> +
>> +The APQNs provide secure key functionality - i.e., a private key is 
>> stored on
>> +the adapter card for each of its domains - so each APQN must be 
>> assigned to at
>> +most one guest or the linux host.
>> +
>> +   Example 1: Valid configuration:
>> +   ------------------------------
>> +   Guest1: adapters 1,2  domains 5,6
>> +   Guest2: adapter  1,2  domain 7
>> +
>> +   This is valid because both guests have a unique set of APQNs: 
>> Guest1 has
>> +   APQNs (1,5), (1,6), (2,5) and (2,6); Guest2 has APQNs (1,7) and 
>> (2,7).
>> +
>> +   Example 2: Invalid configuration:
>> +   --------------------------------
>> +   Guest1: adapters 1,2  domains 5,6
>> +   Guest2: adapter  1    domains 6,7
>> +
>> +   This is an invalid configuration because both guests have access to
>> +   APQN (1,6).
>> +
>> +AP device Configuration on Linux Host:
>> +=====================================
>> +A linux system is a guest of the LPAR in which it is running and has 
>> access to
>> +the AP resources configured for the LPAR. The LPAR's AP matrix is
>> +configured using the 'Customize/Delete Activation Profiles' dialog 
>> from the HMC.
>> +This dialog displays the activation profiles configured for the 
>> linux system.
>> +Selecting the specific activation profile to be edited and clicking the
>> +'Customize Profile' button will open the 'Customize Image Profiles' 
>> dialog.
>> +Selecting the 'Crypto' link in the tree view on the left hand side 
>> of the dialog
>> +will display the AP matrix configuration in the right hand panel. 
>> There, one can
>> +assign AP adapters - called Cryptos - and domains to the LPAR. When 
>> the linux
>> +system is started using this activation profile, it will have access 
>> to the
>> +matrix of AP adapters and domains configured via the activation 
>> profile.
>> +
>> +When the linux system is started, the AP adapter devices will be 
>> connected to
>> +the AP bus and the following AP matrix interfaces will be created in 
>> sysfs:
>> +
>> +/sys/bus/ap
>> +... [devices]
>> +...... xx.yyyy
>> +...... ...
>> +...... cardxx
>> +...... ...
>> +
>> +Where:
>> +    cardxx     is adapter number xx (in hex)
>> +    yyyy       is a usage domain number yyyy (in hex)
>> +....xx.yyyy    is APQN (xx,yyyy)
>> +
>> +For example, if AP adapters 5 and 6 and domains 4 and 71 (0x47) are 
>> configured
>> +for the LPAR, the sysfs representation on the linux system would 
>> look like this:
>> +
>> +/sys/bus/ap
>> +... [devices]
>> +...... 05.0004
>> +...... 05.0047
>> +...... 06.0004
>> +...... 06.0047
>> +...... card05
>> +...... card06
>> +
>> +There will also be AP device drivers created to control each type of 
>> AP matrix
>> +interface available to the IBM Z system:
>> +
>> +/sys/bus/ap
>> +... [drivers]
>> +...... [cex2acard]        for Crypto Express 2/3 accelerator cards
>> +...... [cex2aqueue]       for AP queues served by Crypto Express 2/3
>> +                          accelerator cards
>> +...... [cex4card]         for Crypto Express 4/5/6 accelerator and 
>> coprocessor
>> +                          cards
>> +...... [cex4queue]        for AP queues served by Crypto Express 4/5/6
>> +                          accelerator and coprocessor cards
>> +...... [pcixcccard]       for Crypto Express 2/3 coprocessor cards
>> +...... [pcixccqueue]      for AP queues served by Crypto Express 2/3
>> +                          coprocessor cards
>> +
>> +Links to the AP interfaces controlled by each AP device driver will 
>> be created
>> +in the device driver's sysfs directory. For example, if AP adapter 5 
>> and domains
>> +4 and 71 (0x47) are assigned to the LPAR and adapter 5 is a CEX5 
>> card, the
>> +following links will be created in the CEX5 drivers' sysfs directories:
>> +
>> +/sys/bus/ap
>> +... [drivers]
>> +...... [cex4card]
>> +......... [card05]
>> +...... [cex4queue]
>> +......... [05.0004]
>> +......... [05.0047]
>> +
>> +AP Matrix Configuration for a Linux Guest:
>> +=========================================
>> +In order to configure the AP matrix for a guest, the adapters, usage 
>> domains
>> +and control domains to be used by the guest must be assigned to the 
>> guest. This
>> +section describes how to configure a guest's AP matrix.
>> +
>> +The kernel interfaces for configuring an AP matrix for a linux guest 
>> are built
>> +on the VFIO mediated device framework and are provided by the vfio_ap
>> +kernel module. By default, the vfio_ap module is a loadable module, The
>> +dependency chain for the vfio_ap module is:
>> +* vfio
>> +* mdev
>> +* vfio_mdev
>> +* vfio_ap
>> +
>> +When installed, the vfio_ap module is initialized. During module 
>> initialization,
>> +a vfio_ap driver is created and registered with the AP bus creating the
>> +following sysfs interfaces:
>> +
>> + /sys/bus/ap/drivers/
>> +...[vfio_ap]
>> +...... bind
>> +...... unbind
>> +
>> +The vfio_ap device driver will create a 'matrix' device to hold the 
>> APQNs
>> +reserved for exclusive use by KVM guests:
>> +
>> +/sys/devices/
>> +... [vfio_ap]
>> +......[matrix] symlink to the matrix device directory
>> +
>> +The vfio_ap device driver serves several purposes:
>> +1. Provides an interface for securing APQNs preventing their use by 
>> the host
>> +   linux system and reserving their use by one or more guests.
>> +2. Creates the sysfs interfaces for configuring an AP matrix for a 
>> linux guest.
>> +
>> +Securing APQNs
>> +--------------
>> +   An APQN is reserved by unbinding an AP queue device AP bus device 
>> driver and
>> +   binding it to the vfio_ap device driver. For example, suppose we 
>> want to
>> +   secure APQN (05,0004). Assuming that the AP adapter card 5 is a CEX5
>> +   coprocessor card:
>> +
>> +   echo 05.0004 > /sys/bus/ap/drivers/cex4queue/unbind
>> +   echo 05.0004 > /sys/bus/ap/drivers/vfio_ap/bind
>> +
>> +   This action will store the APQN in the 
>> /sys/devices/vfio_ap/matrix device
>> +   which makes it available for use by a linux guest.
>> +
>> +Configuring an AP matrix for a linux guest.
>> +------------------------------------------
>> +These sysfs interfaces are built on the VFIO mediated device 
>> framework. To
>> +configure an AP matrix for a guest, a mediated matrix device must 
>> first be
>> +created for the /sys/devices/vfio_ap/matrix device. The sysfs 
>> interfaceAPQI corresponding to
>> +for creating a mediated matrix device is in:
>> +
>> +/sys/devices
>> +... [vfio_ap]
>> +......[matrix]
>> +......... [mdev_supported_types]
>> +............ [vfio_ap-passthrough]
>> +............... create
>> +............... [devices]
>> +
>> +A mediated AP matrix device is created by writing a UUID to the 
>> attribute
>> +file named 'create', for example:
>> +
>> +   uuidgen > create
>> +
>> +When a mediated AP matrix device is created, a sysfs directory named 
>> after
>> +the UUID:
>> +
>> +/sys/devices
>> +... [vfio_ap]
>> +......[matrix]
>> +......... [mdev_supported_types]
>> +............ [vfio_ap-passthrough]
>> +............... create
>> +............... [devices]
>> +.................. [$uuid]
>> +
>> +There will also be three sets of attribute files created in the 
>> mediated
>> +matrix device's sysfs directory to configure an AP matrix for the
>> +KVM guest:
>> +
>> +/sys/devices
>> +... [vfio_ap]
>> +......[matrix]
>> +......... [mdev_supported_types]
>> +............ [vfio_ap-passthrough]
>> +............... create
>> +............... [devices]
>> +.................. [$uuid]
>> +..................... assign_adapter
>> +..................... assign_control_domain
>> +..................... assign_domain
>> +..................... matrix
>> +..................... unassign_adapter
>> +..................... unassign_control_domain
>> +..................... unassign_domain
>> +
>> +assign_adapter
>> +   To assign an AP adapter to the mediated matrix device, its APID 
>> is written
>> +   'assign_adapter' file. This may be done multiple times to assign 
>> more than
>> +   one adapter. The APID may be specified using conventional semantics
>> +   as a decimal, hexidecimal, or octal number. For example, to 
>> assign adapters
>> +   4, 5 and 16 to mediated matrix device $uuid in decimal, 
>> hexidecimal and octal
>> +   respectively:
>> +
>> +       echo 4 > assign_adapter
>> +       echo 0x5 > assign_adapter
>> +       echo 020
>> +
>> +unassign_adapter
>> +   To unassign an AP adapter, its APID is written to the 
>> 'unassign_adapter'
>> +   file. This may also be done multiple times to unassign more than 
>> one adapter.
>> +
>> +assign_domain
>> +   To assign a usage domain, the APQI corresponding to the domain 
>> number is
>> +   written into the 'assign_domain' file. This may be done multiple 
>> times to
>> +   assign more than one usage domain. The APQI may be specified using
>> +   conventional semantics as a decimal, hexidecimal, or octal 
>> number. For
>> +   example, to assign usage domains 4, 8, and 71 to mediated matrix 
>> device
>> +   $uuid in decimal, hexidecimal and octal respectively:
>> +
>> +      echo 4 > assign_domain
>> +      echo 0x8 > assign_domain
>> +      echo 0107 > assign_domain
>> +
>> +unassign_domain
>> +   To unassign a usage domain, the APQI corresponding to the domain 
>> number is
>> +   written into the 'unassign_domain' file. This may be done 
>> multiple times to
>> +   unassign more than one usage domain.
>> +
>> +assign_control_domain
>> +   To assign a control domain, the domain number is written into the
>> +   'assign_control_domain' file. This may be done multiple times to
>> +   assign more than one control domain. The domain number may be 
>> specified using
>> +   conventional semantics as a decimal, hexidecimal, or octal 
>> number. For
>> +   example, to assign  control domains 4, 8, and 71 to mediated 
>> matrix device
>> +   $uuid in decimal, hexidecimal and octal respectively:
>> +
>> +      echo 4 > assign_domain
>> +      echo 0x8 > assign_domain
>> +      echo 0107 > assign_domain
>> +
>> +unassign_control_domain
>> +   To unassign a control domain, the domain number is written into the
>> +   'unassign_domain' file. This may be done multiple times to 
>> unassign more than
>> +   one control domain.
>> +
>> +Notes:
>> +* Hot plug/unplug is not currently supported for mediated AP matrix 
>> devices,
>> +  so the AP matrix resulting from assignment and/or unassignment of AP
>> +  adapters, usage domains and control domains to a mediated AP 
>> matrix device
>> +  while the guest is running will not take affect until the linux 
>> guest is
>> +  rebooted.
>> +* By architectural convention, all usage domains configured for a 
>> KVM guest
>> +  will also be implicitly assigned as control domains also, to there 
>> is no
>> +  need to assign control domains that are assigned as usage domains.
>> +
>> +Starting a Linux Guest Configured with an AP Matrix:
>> +===================================================
>> +In addition to providing the sysfs interfaces for configuring the AP 
>> matrix for
>> +a linux guest, a mediated matrix device also acts as a communication 
>> pathway
>> +between QEMU and the vfio_ap device driver. To gain access to the
>> +device driver, the following option must be specified on the QEMU 
>> command line:
>> +
>> +   -device vfio_ap,sysfsdev=$path-to-mdev
>> +
>> +The sysfsdev parameter specifies the path to the mediated matrix 
>> device.
>> +There are a number of ways to specify this path:
>> +
>> +/sys/devices/vfio_ap/matrix/$uuid
>> +/sys/bus/mdev/devices/$uuid
>> +/sys/bus/mdev/drivers/vfio_mdev/$uuid
>> +/sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/devices/$uuid 
>>
>> +
>> +When the linux guest is subsequently started, the guest will open 
>> the mediated
>> +matrix device's file descriptor to get information about the 
>> mediated matrix
>> +device. The vfio_ap device driver will update the APM, AQM, and ADM 
>> fields in the
>> +guest's CRYCB with the adapter, usage domain and control domains 
>> assigned to
>> +via the mediated matrix device's sysfs attribute files. Programs 
>> running on the
>> +linux guest will then:
>> +
>> +1. Have direct access to the APQNs derived from the intersection of 
>> the AP
>> +   adapter and usage domain numbers specified in the APM and AQM 
>> respectively
>> +
>> +2. Have authorization to process AP commands to change - e.g., store 
>> a new
>> +   secure key - a control domain identified in an AP instruction 
>> sent to a valid
>> +   APQN.
>> +
>> +CPU model features:
>> +
>> +Three CPU model features are available for controlling guest access 
>> to AP
>> +facilities:
>> +
>> +1. AP facilities feature
>> +
>> +   The AP facilities feature indicates that AP facilities are 
>> installed on the
>> +   guest. This feature will be enabled by the kernel only if the AP 
>> facilities
>> +   are installed on the host system. It will turned on automatically 
>> for guests
>> +   started with CPU model zEC12 or newer. The feature is 
>> s390-specific and is
>> +   represented as a parameter of the -cpu option on the QEMU command 
>> line:
>> +
>> +      qemu-system-s390x -cpu $model,ap=on|off
>> +
>> +      Where:
>> +
>> +         $model is the CPU model defined for the guest (defaults to 
>> the model of
>> +                the host system if not specified).
>> +
>> +         ap=on|off indicates whether AP facilities are installed 
>> (on) or not
>> +                   installed (off). The default for CPU models zEC12 
>> or newer
>> +                   is ap=on. AP facilities must be installed when 
>> this parameter
>> +                   is used in conjunction with -device 
>> vfio-ap,sysfsdev=$path or
>> +                   the guest will not start.
>> +
>> +2. Query Configuration Information (QCI) facility
>> +
>> +   The QCI facility is used by the AP bus running on the guest to 
>> query the
>> +   configuration of the AP facilities. This facility will be enabled by
>> +   the kernel only if the QCI facility is installed on the host 
>> system. It will
>> +   be turned on automatically for guests started with CPU model 
>> zEC12 or newer.
>> +   The feature is s390-specific and is represented as a parameter of 
>> the -cpu
>> +   option on the QEMU command line:
>> +
>> +      qemu-system-s390x -cpu $model,qci=on|off
>> +
>> +      Where:
>> +
>> +         $model is the CPU model defined for the guest
>> +
>> +         qci=on|off indicates whether the QCI facility is installed 
>> (on) or not
>> +                    installed (off). The default for CPU models 
>> zEC12 or newer
>> +                    is qci=on. Turning the QCI facility on makes no 
>> sense if it
>> +                    is not used in conjunction with the
>> +                    '-device vfio-ap,sysfsdev=$path' option. A 
>> warning will be
>> +                    presented if QCI is turned on and the AP 
>> facilities are not
>> +                    installed.
>> +
>> +                    If the QCI facility is turned off, APQNs with an 
>> APQI
>> +                    greater than 15 will not be accessible from the 
>> guest.
>> +
>> +3. Adjunct Process Facility Test (APFT) facility
>> +
>> +   The APFT facility is used by the AP bus running on the guest to 
>> test the
>> +   AP facilities available for a given AP queue. This facility will 
>> be enabled
>> +   by the kernel only if the APFT facility is installed on the host 
>> system. It
>> +   will be turned on automatically for guests started with CPU model 
>> zEC12 or
>> +   newer. The feature is s390-specific and is represented as a 
>> parameter of the
>> +   -cpu option on the QEMU command line:
>> +
>> +      qemu-system-s390x -cpu $model,apft=on|off
>> +
>> +      Where:
>> +
>> +         $model is the CPU model defined for the guest (defaults to 
>> the model of
>> +                the host system if not specified).
>> +
>> +         apft=on|off indicates whether the APFT facility is 
>> installed (on) or
>> +                     not installed (off). The default for CPU models 
>> zEC12 and
>> +                     newer is apft=on. Turning the APFT facility on 
>> makes no
>> +                     sense if it is not used in conjunction with the
>> +                     -device vfio-ap,sysfsdev=$path option. A 
>> warning will be
>> +                     presented if APFT is turned on and the AP 
>> facilities are
>> +                     not installed.
>> +
>> +                     It also makes no sense to turn APFT off when 
>> used in
>> +                     conjunction with the vfio-ap device because the 
>> APFT
>> +                     facility is required; the AP bus running on the 
>> guest will
>> +                     not detect CEX4 and newer devices without it. 
>> Since only
>> +                     CEX4 and newer devices are supported for guest 
>> usage, no AP
>> +                     devices can be made accessible to a guest 
>> started without
>> +                     APFT installed.
>> +
>> +Example: Configure AP Matrixes for Two Linux Guests:
>> +===================================================
>> +Let's now provide an example to illustrate how KVM guests may be given
>> +access to AP facilities. For this example, we will show how to 
>> configure
>> +two guests such that executing the lszcrypt command on the guests would
>> +look like this:
>> +
>> +Guest1
>> +------
>> +CARD.DOMAIN TYPE  MODE
>> +------------------------------
>> +05          CEX5C CCA-Coproc
>> +05.0004     CEX5C CCA-Coproc
>> +05.00ab     CEX5C CCA-Coproc
>> +06          CEX5A Accelerator
>> +06.0004     CEX5A Accelerator
>> +06.00ab     CEX5C CCA-Coproc
>> +
>
> 5 whitespace at the end of lines
Right you are! I'll fix 'em.
>
>> +Guest2
>> +------
>> +CARD.DOMAIN TYPE  MODE
>> +------------------------------
>> +05          CEX5A Accelerator
>> +05.0047     CEX5A Accelerator
>> +05.00ff     CEX5A Accelerator
>> +
>> +These are the steps for configuring the Guest1 and Guest2:
>> +
>> +1. The first thing that needs to be done is to secure the AP queues 
>> to be
>> +   used by the two guests so that the host can not access them. This 
>> is done
>> +   by unbinding each AP Queue device from its respective AP driver. 
>> In our
>> +   example, these queues are bound to the cex4queue driver. This 
>> would be
>> +   the sysfs location of these devices:
>> +
>> +   /sys/bus/ap
>> +   --- [drivers]
>> +   ------ [cex4queue]
>> +   --------- [05.0004]
>> +   --------- [05.0047]   --------------------- control_domains
>> +   --------------------- domains
>> +   --------- [05.00ab]
>> +   --------- [05.00ff]
>> +   --------- [06.0004]
>> +   --------- [06.00ab]
>> +   --------- unbind
>> +
>> +   To unbind AP queue 05.0004 from the cex4queue device driver:
>> +
>> +    echo 05.0004 > unbind
>> +
>> +   This must also be done for AP queues 05.00ab, 05.0047, 05.00ff, 
>> 06.0004,
>> +   and 06.00ab.
>> +
>> +2. The next step is to reserve the queues for use by the two KVM 
>> guests.
>> +   This is accomplished by binding them to the VFIO AP device driver.
>> +   This is the sysfs location of the VFIO AP device driver:
>> +
>> +   /sys/bus/ap
>> +   ---[drivers]
>> +   ------ [vfio_ap]
>> +   ---------- bind
>> +
>> +   To bind queue 05.0004 to the vfio_ap driver:
>> +
>> +    echo 05.0004 > bind
>> +
>> +   This must also be done for AP queues 05.00ab, 05.0047, 05.00ff, 
>> 06.0004,
>> +   and 06.00ab.
>> +
>> +3. Create the mediated devices needed to configure the AP matrixes 
>> for the
>> +   two guests and to provide an interface to the vfio_ap driver for
>> +   use by the guests:
>> +
>> +   /sys/devices/
>> +   --- [vfio_ap]
>> +   ------ [matrix] (this is the matrix device)
>> +   --------- [mdev_supported_types]
>> +   ------------ [vfio_ap-passthrough] (passthrough mediated matrix 
>> device type)
>> +   --------------- create
>> +   --------------- [devices]
>> +
>> +   To create the mediated devices for the two guests:
>> +
>> +    uuidgen > create
>> +    uuidgen > create
>> +
>> +   This will create two mediated devices in the [devices] 
>> subdirectory named
>> +   with the UUID written to the create attribute file. We call them 
>> $uuid1
>> +   and $uuid2:
>> +
>> +   /sys/devices/
>> +   --- [vfio_ap]
>> +   ------ [matrix]
>> +   --------- [mdev_supported_types]
>> +   ------------ [vfio_ap-passthrough]
>> +   --------------- [devices]
>> +   ------------------ [$uuid1]
>> +   --------------------- assign_adapter
>> +   --------------------- assign_control_domain
>> +   --------------------- assign_domain
>> +   --------------------- matrix
>> +   --------------------- unassign_adapter
>> +   --------------------- unassign_control_domain
>> +   --------------------- unassign_domain
>> +
>> +   ------------------ [$uuid2]
>> +   --------------------- assign_adapter
>> +   --------------------- assign_cTo assign an adapter, the APID of 
>> the adapter is written to the
>> +      file. ontrol_domain
>> +   --------------------- assign_domain
>> +   --------------------- matrix
>> +   --------------------- unassign_adapter
>> +   --------------------- unassign_control_domain
>> +   --------------------- unassign_domain
>> +
>> +4. The administrator now needs to configure the matrixes for mediated
>> +   devices $uuid1 (for Guest1) and $uuid2 (for Guest2).
>> +
>> +   This is how the matrix is configured for Guest1:
>> +
>> +   echo 5 > assign_adapter
>> +   echo 6 > assign_adapter
>> +   echo 4 > assign_domain
>> +   echo 0xab > assign_domain
>> +
>> +   By architectural convention, all usage domains - i.e., domains 
>> assigned
>> +   via the assign_domain attribute file - will also be configured in 
>> the ADM
>> +   field of the KVM guest's CRYCB, so there is no need to assign 
>> control
>> +   domains here unless you want to assign control domains that are not
>> +   assigned as usage domains.
>> +
>> +   If a mistake is made configuring an adapter, domain or control 
>> domain,
>> +   you can use the unassign_xxx files to unassign the adapter, 
>> domain or
>> +   control domain.
>> +
>> +   To display the matrix configuration for Guest1:
>> +
>> +   cat matrix
>> +
>> +   This is how the matrix is configured for Guest2:
>> +
>> +   echo 5 > assign_adapter
>> +   echo 0x47 > assign_domain
>> +   echo 0xff > assign_domain
>> +
>> +5. Start Guest1
>> +
>> +   /usr/bin/qemu-system-s390x ... -device 
>> vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid1 ...
>> +
>> +6. Start Guest2
>> +
>> +   /usr/bin/qemu-system-s390x ... -device 
>> vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid2 ...
>
>

Re: [Qemu-devel] [PATCH v3 5/7] s390x/vfio: ap: Introduce VFIO AP device

[Tony Krowiak]

On 03/27/2018 08:02 AM, Cornelia Huck wrote:
> On Fri, 16 Mar 2018 14:22:52 +0100
> Halil Pasic <pasic@linux.vnet.ibm.com> wrote:
>
>> On 03/16/2018 11:42 AM, Pierre Morel wrote:
>>> On 16/03/2018 00:24, Tony Krowiak wrote:
>>>> Introduces a VFIO based AP device. The device is defined via
>>>> the QEMU command line by specifying:
>>>>
>>>>       -device vfio-ap,sysfsdev=<path-to-mediated-matrix-device>
>>>>
>>>> The mediated matrix device is created by the VFIO AP device
>>>> driver by writing a UUID to a sysfs attribute file (see
>>>> docs/vfio-ap.txt). The mediated matrix device will be named
>>>> after the UUID. Symbolic links to the $uuid are created in
>>>> many places, so the path to the mediated matrix device $uuid
>>>> can be specified in any of the following ways:
>>>>
>>>> /sys/devices/vfio_ap/matrix/$uuid
>>>> /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/devices/$uuid
>>>> /sys/bus/mdev/devices/$uuid
>>>> /sys/bus/mdev/drivers/vfio_mdev/$uuid
>>>>
>>>> When the vfio-ap device is realized, it acquires and opens the
>>>> VFIO iommu group to which the mediated matrix device is
>>>> bound. This causes a VFIO group notification event to be
>>>> signaled. The vfio_ap device driver's group notification
>>>> handler will get called at which time the device driver
>>>> will configure the the AP devices to which the guest will
>>>> be granted access.
>>>>
>>>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>>>> ---
>> [..]
>>>> +static void vfio_ap_realize(DeviceState *dev, Error **errp)
>>>> +{
>>>> +    VFIODevice *vbasedev;
>>>> +    VFIOGroup *vfio_group;
>>>> +    APDevice *apdev = DO_UPCAST(APDevice, parent_obj, dev);
>>>> +    VFIOAPDevice *vapdev = DO_UPCAST(VFIOAPDevice, apdev, apdev);
>>>> +    char *mdevid;
>>>> +    Error *local_err = NULL;
>>>> +    int ret;
>>>> +
>>>> +    if (!s390_has_feat(S390_FEAT_AP)) {
>>>> +        error_setg(&local_err, "AP support not enabled");
>>>> +        goto out_err;
>>>> +    }
>>>> +
>>>> +    ret = kvm_s390_set_interpret_ap(1);
>>> If we have several devices, this is called once per device.
>> I don't think having several of these in a single vm makes
>> any sense. Or does it? IMHO we should make sure there is at
>> most one device taking care of the crypto pass-through.
> Yes, I think we should fence off adding a second device in the realize
> function (probably by checking a global variable?)
I agree, will do.
>

Re: [Qemu-devel] [PATCH v3 4/7] s390x/kvm: interface to interpret AP instructions

[Tony Krowiak]

On 03/26/2018 04:38 AM, David Hildenbrand wrote:
> On 16.03.2018 00:24, Tony Krowiak wrote:
>> The VFIO AP device exploits interpretive execution of AP
>> instructions (APIE). APIE is enabled by setting a device attribute
>> via the KVM_SET_DEVICE_ATTR ioctl.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>>   target/s390x/kvm.c       |   16 ++++++++++++++++
>>   target/s390x/kvm_s390x.h |    2 ++
>>   2 files changed, 18 insertions(+), 0 deletions(-)
>>
>> diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c
>> index 33e5ec3..2812e28 100644
>> --- a/target/s390x/kvm.c
>> +++ b/target/s390x/kvm.c
>> @@ -277,6 +277,22 @@ static void kvm_s390_init_dea_kw(void)
>>       }
>>   }
>>   
>> +int kvm_s390_set_interpret_ap(uint8_t enable)
>> +{
>> +    struct kvm_device_attr attribute = {
>> +        .group = KVM_S390_VM_CRYPTO,
>> +        .attr  = KVM_S390_VM_CRYPTO_INTERPRET_AP,
>> +        .addr = 1,
>> +    };
>> +
>> +    if (!kvm_vm_check_attr(kvm_state, KVM_S390_VM_CRYPTO,
>> +                           KVM_S390_VM_CRYPTO_INTERPRET_AP)) {
>> +        return -EOPNOTSUPP;
>> +    }
>> +
>> +    return kvm_vm_ioctl(kvm_state, KVM_SET_DEVICE_ATTR, &attribute);
>> +}
>> +
>>   void kvm_s390_crypto_reset(void)
>>   {
>>       if (s390_has_feat(S390_FEAT_MSA_EXT_3)) {
>> diff --git a/target/s390x/kvm_s390x.h b/target/s390x/kvm_s390x.h
>> index 34ee7e7..0d6c6e7 100644
>> --- a/target/s390x/kvm_s390x.h
>> +++ b/target/s390x/kvm_s390x.h
>> @@ -40,4 +40,6 @@ void kvm_s390_crypto_reset(void);
>>   void kvm_s390_restart_interrupt(S390CPU *cpu);
>>   void kvm_s390_stop_interrupt(S390CPU *cpu);
>>   
>> +int kvm_s390_set_interpret_ap(uint8_t enable);
>> +
>>   #endif /* KVM_S390X_H */
>>
> Wonder if a capability - like we use e.g. for SIGP user space
> interpretation - would be a better fit.
>
> We can provide the AP feature to the guest in case:
> - KVM_S390_VM_CPU_FEAT_AP ("interpretation support") is available
> - KVM_S390_VM_CRYPTO_INTERPRET_AP ("interception support") is available
I don't think we need this. I have found a way to set ECA_APIE from the
vfio_ap device driver, so there is no need to do it from the guest. Stay
tuned to this station for v4 of the patch series.
>
> I am missing the second check in your code. (for now you only rely on
> KVM_S390_VM_CPU_FEAT_AP)
For what else are you suggesting we need to check?
>
> I think you have to change the order of the patches so they work also
> properly when bisectin.
I'll take a look at it.
>

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Cornelia Huck]

On Mon, 2 Apr 2018 12:36:27 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> On 03/26/2018 05:03 AM, Pierre Morel wrote:
> > On 26/03/2018 10:32, David Hildenbrand wrote:  
> >> On 16.03.2018 00:24, Tony Krowiak wrote:  

> >>> +    /*
> >>> +     * The Query Configuration Information (QCI) function (fc == 4) 
> >>> does not
> >>> +     * set a response code in reg 1, so check for that along with the
> >>> +     * AP feature.
> >>> +     */
> >>> +    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
> >>> +        env->regs[1] = 0x10000;
> >>> +
> >>> +        return 0;
> >>> +    }  
> >> This would imply an operation exception in case fc==4, which sounds very
> >> wrong.  
> >
> > It depends but I think that the S390_FEAT_AP_QUERY_CONFIG_INFO must be 
> > tested
> > to know what to answer.
> > If the feature is there, QCI must be answered correctly.  
> This is an interesting proposition which raises several issues that will 
> need to
> be addressed. The only time the PQAP(QCI) instruction is intercepted is 
> when:
> * A vfio-ap device is NOT defined for the guest because the vfio_ap 
> device driver
>    will set ECA.28 and the PQAP(QCI) instruction will be interpreted
> * STFLE.12 is set for the guest
> 
> You say that the QCI must be answered correctly, but what is the correct 
> response?
> If we return the matrix - i.e., APM, ADM and AQM - configured via the 
> mediated
> matrix device's sysfs attributes files, then if any APQNs are defined in 
> the matrix,
> we will have to handle *ALL* AP instructions by executing them on behalf 
> of the
> guest. I suppose we could return an empty matrix in which case the AP 
> bus will come
> up without any devices on the guest, but what is the expectation of an 
> admin who
> deliberately configures the mediated matrix device? Should we forego 
> handling interception
> of AP instructions and consider a guest configuration that turns on 
> S390_FEAT_AP but
> does not define a vfio-ap device to be erroneous and terminate starting 
> of the guest?
> Anybody have any thoughts?

Hard to really give good advice without access to the documentation, but:
- If we tell the guest that the feature is available, but it does not
  get any cards to use, returning an empty matrix makes the most sense
  to me.
- I would not tie starting the guest to the presence of a vfio-ap
  device. Having the feature available in theory but without any
  devices actually being usable by the guest does not really sound
  wrong (can we hotplug this later?)

Re: [Qemu-devel] [PATCH v3 5/7] s390x/vfio: ap: Introduce VFIO AP device

[Tony Krowiak]

On 04/02/2018 01:05 PM, Tony Krowiak wrote:
> On 03/27/2018 08:02 AM, Cornelia Huck wrote:
>> On Fri, 16 Mar 2018 14:22:52 +0100
>> Halil Pasic <pasic@linux.vnet.ibm.com> wrote:
>>
>>> On 03/16/2018 11:42 AM, Pierre Morel wrote:
>>>> On 16/03/2018 00:24, Tony Krowiak wrote:
>>>>> Introduces a VFIO based AP device. The device is defined via
>>>>> the QEMU command line by specifying:
>>>>>
>>>>>       -device vfio-ap,sysfsdev=<path-to-mediated-matrix-device>
>>>>>
>>>>> The mediated matrix device is created by the VFIO AP device
>>>>> driver by writing a UUID to a sysfs attribute file (see
>>>>> docs/vfio-ap.txt). The mediated matrix device will be named
>>>>> after the UUID. Symbolic links to the $uuid are created in
>>>>> many places, so the path to the mediated matrix device $uuid
>>>>> can be specified in any of the following ways:
>>>>>
>>>>> /sys/devices/vfio_ap/matrix/$uuid
>>>>> /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/devices/$uuid 
>>>>>
>>>>> /sys/bus/mdev/devices/$uuid
>>>>> /sys/bus/mdev/drivers/vfio_mdev/$uuid
>>>>>
>>>>> When the vfio-ap device is realized, it acquires and opens the
>>>>> VFIO iommu group to which the mediated matrix device is
>>>>> bound. This causes a VFIO group notification event to be
>>>>> signaled. The vfio_ap device driver's group notification
>>>>> handler will get called at which time the device driver
>>>>> will configure the the AP devices to which the guest will
>>>>> be granted access.
>>>>>
>>>>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>>>>> ---
>>> [..]
>>>>> +static void vfio_ap_realize(DeviceState *dev, Error **errp)
>>>>> +{
>>>>> +    VFIODevice *vbasedev;
>>>>> +    VFIOGroup *vfio_group;
>>>>> +    APDevice *apdev = DO_UPCAST(APDevice, parent_obj, dev);
>>>>> +    VFIOAPDevice *vapdev = DO_UPCAST(VFIOAPDevice, apdev, apdev);
>>>>> +    char *mdevid;
>>>>> +    Error *local_err = NULL;
>>>>> +    int ret;
>>>>> +
>>>>> +    if (!s390_has_feat(S390_FEAT_AP)) {
>>>>> +        error_setg(&local_err, "AP support not enabled");
>>>>> +        goto out_err;
>>>>> +    }
>>>>> +
>>>>> +    ret = kvm_s390_set_interpret_ap(1);
>>>> If we have several devices, this is called once per device.
>>> I don't think having several of these in a single vm makes
>>> any sense. Or does it? IMHO we should make sure there is at
>>> most one device taking care of the crypto pass-through.
>> Yes, I think we should fence off adding a second device in the realize
>> function (probably by checking a global variable?)
> I agree, will do.
In light of comments Connie made in Message ID: 
<20180403113619.54ff1e18.cohuck@redhat.com>, this
raises the question: What about hotplug? If we allow only one vfio_ap 
device for a guest, then
there will be no hot plugging of vfio_ap devices except for the case 
where no vfio_ap device is
defined for the guest at startup. The question then becomes how do we 
configure (hot plug?)
additional queues for guest usage? I will have to give this some 
additional thought before
committing to a solution here.
>
>>
>

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Pierre Morel]

On 03/04/2018 11:36, Cornelia Huck wrote:
> On Mon, 2 Apr 2018 12:36:27 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> On 03/26/2018 05:03 AM, Pierre Morel wrote:
>>> On 26/03/2018 10:32, David Hildenbrand wrote:
>>>> On 16.03.2018 00:24, Tony Krowiak wrote:
>>>>> +    /*
>>>>> +     * The Query Configuration Information (QCI) function (fc == 4)
>>>>> does not
>>>>> +     * set a response code in reg 1, so check for that along with the
>>>>> +     * AP feature.
>>>>> +     */
>>>>> +    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
>>>>> +        env->regs[1] = 0x10000;
>>>>> +
>>>>> +        return 0;
>>>>> +    }
>>>> This would imply an operation exception in case fc==4, which sounds very
>>>> wrong.
>>> It depends but I think that the S390_FEAT_AP_QUERY_CONFIG_INFO must be
>>> tested
>>> to know what to answer.
>>> If the feature is there, QCI must be answered correctly.
>> This is an interesting proposition which raises several issues that will
>> need to
>> be addressed. The only time the PQAP(QCI) instruction is intercepted is
>> when:
>> * A vfio-ap device is NOT defined for the guest because the vfio_ap
>> device driver
>>     will set ECA.28 and the PQAP(QCI) instruction will be interpreted
>> * STFLE.12 is set for the guest
>>
>> You say that the QCI must be answered correctly, but what is the correct
>> response?
>> If we return the matrix - i.e., APM, ADM and AQM - configured via the
>> mediated
>> matrix device's sysfs attributes files, then if any APQNs are defined in
>> the matrix,
>> we will have to handle *ALL* AP instructions by executing them on behalf
>> of the
>> guest. I suppose we could return an empty matrix in which case the AP
>> bus will come
>> up without any devices on the guest, but what is the expectation of an
>> admin who
>> deliberately configures the mediated matrix device? Should we forego
>> handling interception
>> of AP instructions and consider a guest configuration that turns on
>> S390_FEAT_AP but
>> does not define a vfio-ap device to be erroneous and terminate starting
>> of the guest?
>> Anybody have any thoughts?
> Hard to really give good advice without access to the documentation, but:
> - If we tell the guest that the feature is available, but it does not
>    get any cards to use, returning an empty matrix makes the most sense
>    to me.

+1

> - I would not tie starting the guest to the presence of a vfio-ap
>    device. Having the feature available in theory but without any
>    devices actually being usable by the guest does not really sound
>    wrong (can we hotplug this later?)
>
+1


-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Pierre Morel]

On 02/04/2018 18:36, Tony Krowiak wrote:
> On 03/26/2018 05:03 AM, Pierre Morel wrote:
>> On 26/03/2018 10:32, David Hildenbrand wrote:
>>> On 16.03.2018 00:24, Tony Krowiak wrote:
>>>> If the CPU model indicates that AP facility is installed on
>>>> the guest (i.e., -cpu xxxx,ap=on), then the expectation is that
>>>> the AP bus running in the guest will initialize; however, if the
>>>> AP instructions are not being interpreted by the firmware, then
>>>> they will be intercepted and routed back to QEMU for handling.
>>>> If a handler is not defined to process the intercepted instruciton,
>>>> t
>> ...snip...
>>>
>>>> +int ap_device_handle_nqap(S390CPU *cpu)
>>>> +{
>>>> +    CPUS390XState *env = &cpu->env;
>>>> +
>>>> +    if (s390_has_feat(S390_FEAT_AP)) {
>>>> +        env->regs[1] = 0x10000;
>>>> +
>>>> +        return 0;
>>>> +    }
>>>> +
>>>> +    return -EOPNOTSUPP;
>>>> +}
>>>> +
>>>> +int ap_device_handle_dqap(S390CPU *cpu)
>>>> +{
>>>> +    CPUS390XState *env = &cpu->env;
>>>> +
>>>> +    if (s390_has_feat(S390_FEAT_AP)) {
>>>> +        env->regs[1] = 0x10000;
>>>> +
>>>> +        return 0;
>>>> +    }
>>>> +
>>>> +    return -EOPNOTSUPP;
>>>> +}
>>>> +
>>>> +int ap_device_handle_pqap(S390CPU *cpu)
>>>> +{
>>>> +    CPUS390XState *env = &cpu->env;
>>>> +    int fc = 4 & (env->regs[0] >> 24);
>>>> +
>>>> +    /*
>>>> +     * The Query Configuration Information (QCI) function (fc == 
>>>> 4) does not
>>>> +     * set a response code in reg 1, so check for that along with the
>>>> +     * AP feature.
>>>> +     */
>>>> +    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
>>>> +        env->regs[1] = 0x10000;
>>>> +
>>>> +        return 0;
>>>> +    }
>>> This would imply an operation exception in case fc==4, which sounds 
>>> very
>>> wrong.
>>
>> It depends but I think that the S390_FEAT_AP_QUERY_CONFIG_INFO must 
>> be tested
>> to know what to answer.
>> If the feature is there, QCI must be answered correctly.
> This is an interesting proposition which raises several issues that 
> will need to
> be addressed. The only time the PQAP(QCI) instruction is intercepted 
> is when:
> * A vfio-ap device is NOT defined for the guest because the vfio_ap 
> device driver
>   will set ECA.28 and the PQAP(QCI) instruction will be interpreted
> * STFLE.12 is set for the guest
>
> You say that the QCI must be answered correctly, but what is the 
> correct response?
> If we return the matrix - i.e., APM, ADM and AQM - configured via the 
> mediated
> matrix device's sysfs attributes files, then if any APQNs are defined 
> in the matrix,
> we will have to handle *ALL* AP instructions by executing them on 
> behalf of the
> guest. I suppose we could return an empty matrix in which case the AP 
> bus will come
> up without any devices on the guest, but what is the expectation of an 
> admin who
> deliberately configures the mediated matrix device? Should we forego 
> handling interception
> of AP instructions and consider a guest configuration that turns on 
> S390_FEAT_AP but
> does not define a vfio-ap device to be erroneous and terminate 
> starting of the guest?
> Anybody have any thoughts?
>>
>>
>> there are also some error situations to handle in all three functions.
> To what error situations are you referring?

program exceptions, like access, privilege or specification exceptions.


>>
>>
>>
>>>
>>> Hard to review without access to documentation. (hard to understand why
>>> such stuff is to be kept confidential for decades)
>>>
>>>> +
>>>> +    return -EOPNOTSUPP;
>>>> +}
>>>> +
>>>>   static Property vfio_ap_properties[] = {
>>>>       DEFINE_PROP_STRING("sysfsdev", VFIOAPDevice, vdev.sysfsdev),
>>>>       DEFINE_PROP_END_OF_LIST(),
>>>> diff --git a/include/hw/s390x/ap-device.h 
>>>> b/include/hw/s390x/ap-device.h
>>>> index 693df90..d45ae38 100644
>>>> --- a/include/hw/s390x/ap-device.h
>>>> +++ b/include/hw/s390x/ap-device.h
>>>> @@ -11,6 +11,8 @@
>>>>   #ifndef HW_S390X_AP_DEVICE_H
>>>>   #define HW_S390X_AP_DEVICE_H
>>>
>>
>
>

-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Tony Krowiak]

On 04/04/2018 07:09 AM, Pierre Morel wrote:
> On 02/04/2018 18:36, Tony Krowiak wrote:
>> On 03/26/2018 05:03 AM, Pierre Morel wrote:
>>> On 26/03/2018 10:32, David Hildenbrand wrote:
>>>> On 16.03.2018 00:24, Tony Krowiak wrote:
>>>>> If the CPU model indicates that AP facility is installed on
>>>>> the guest (i.e., -cpu xxxx,ap=on), then the expectation is that
>>>>> the AP bus running in the guest will initialize; however, if the
>>>>> AP instructions are not being interpreted by the firmware, then
>>>>> they will be intercepted and routed back to QEMU for handling.
>>>>> If a handler is not defined to process the intercepted instruciton,
>>>>> t
>>> ...snip...
>>>>
>>>>> +int ap_device_handle_nqap(S390CPU *cpu)
>>>>> +{
>>>>> +    CPUS390XState *env = &cpu->env;
>>>>> +
>>>>> +    if (s390_has_feat(S390_FEAT_AP)) {
>>>>> +        env->regs[1] = 0x10000;
>>>>> +
>>>>> +        return 0;
>>>>> +    }
>>>>> +
>>>>> +    return -EOPNOTSUPP;
>>>>> +}
>>>>> +
>>>>> +int ap_device_handle_dqap(S390CPU *cpu)
>>>>> +{
>>>>> +    CPUS390XState *env = &cpu->env;
>>>>> +
>>>>> +    if (s390_has_feat(S390_FEAT_AP)) {
>>>>> +        env->regs[1] = 0x10000;
>>>>> +
>>>>> +        return 0;
>>>>> +    }
>>>>> +
>>>>> +    return -EOPNOTSUPP;
>>>>> +}
>>>>> +
>>>>> +int ap_device_handle_pqap(S390CPU *cpu)
>>>>> +{
>>>>> +    CPUS390XState *env = &cpu->env;
>>>>> +    int fc = 4 & (env->regs[0] >> 24);
>>>>> +
>>>>> +    /*
>>>>> +     * The Query Configuration Information (QCI) function (fc == 
>>>>> 4) does not
>>>>> +     * set a response code in reg 1, so check for that along with 
>>>>> the
>>>>> +     * AP feature.
>>>>> +     */
>>>>> +    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
>>>>> +        env->regs[1] = 0x10000;
>>>>> +
>>>>> +        return 0;
>>>>> +    }
>>>> This would imply an operation exception in case fc==4, which sounds 
>>>> very
>>>> wrong.
>>>
>>> It depends but I think that the S390_FEAT_AP_QUERY_CONFIG_INFO must 
>>> be tested
>>> to know what to answer.
>>> If the feature is there, QCI must be answered correctly.
>> This is an interesting proposition which raises several issues that 
>> will need to
>> be addressed. The only time the PQAP(QCI) instruction is intercepted 
>> is when:
>> * A vfio-ap device is NOT defined for the guest because the vfio_ap 
>> device driver
>>   will set ECA.28 and the PQAP(QCI) instruction will be interpreted
>> * STFLE.12 is set for the guest
>>
>> You say that the QCI must be answered correctly, but what is the 
>> correct response?
>> If we return the matrix - i.e., APM, ADM and AQM - configured via the 
>> mediated
>> matrix device's sysfs attributes files, then if any APQNs are defined 
>> in the matrix,
>> we will have to handle *ALL* AP instructions by executing them on 
>> behalf of the
>> guest. I suppose we could return an empty matrix in which case the AP 
>> bus will come
>> up without any devices on the guest, but what is the expectation of 
>> an admin who
>> deliberately configures the mediated matrix device? Should we forego 
>> handling interception
>> of AP instructions and consider a guest configuration that turns on 
>> S390_FEAT_AP but
>> does not define a vfio-ap device to be erroneous and terminate 
>> starting of the guest?
>> Anybody have any thoughts?
>>>
>>>
>>> there are also some error situations to handle in all three functions.
>> To what error situations are you referring?
>
> program exceptions, like access, privilege or specification exceptions.
I'm sorry, I still don't know what you are referring to, can you be more 
specific?
>
>
>
>>>
>>>
>>>
>>>>
>>>> Hard to review without access to documentation. (hard to understand 
>>>> why
>>>> such stuff is to be kept confidential for decades)
>>>>
>>>>> +
>>>>> +    return -EOPNOTSUPP;
>>>>> +}
>>>>> +
>>>>>   static Property vfio_ap_properties[] = {
>>>>>       DEFINE_PROP_STRING("sysfsdev", VFIOAPDevice, vdev.sysfsdev),
>>>>>       DEFINE_PROP_END_OF_LIST(),
>>>>> diff --git a/include/hw/s390x/ap-device.h 
>>>>> b/include/hw/s390x/ap-device.h
>>>>> index 693df90..d45ae38 100644
>>>>> --- a/include/hw/s390x/ap-device.h
>>>>> +++ b/include/hw/s390x/ap-device.h
>>>>> @@ -11,6 +11,8 @@
>>>>>   #ifndef HW_S390X_AP_DEVICE_H
>>>>>   #define HW_S390X_AP_DEVICE_H
>>>>
>>>
>>
>>
>

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Tony Krowiak]

On 04/04/2018 07:09 AM, Pierre Morel wrote:
> On 02/04/2018 18:36, Tony Krowiak wrote:
>> On 03/26/2018 05:03 AM, Pierre Morel wrote:
>>> On 26/03/2018 10:32, David Hildenbrand wrote:
>>>> On 16.03.2018 00:24, Tony Krowiak wrote:
>>>>> If the CPU model indicates that AP facility is installed on
>>>>> the guest (i.e., -cpu xxxx,ap=on), then the expectation is that
>>>>> the AP bus running in the guest will initialize; however, if the
>>>>> AP instructions are not being interpreted by the firmware, then
>>>>> they will be intercepted and routed back to QEMU for handling.
>>>>> If a handler is not defined to process the intercepted instruciton,
>>>>> t
>>> ...snip...
>>>>
>>>>> +int ap_device_handle_nqap(S390CPU *cpu)
>>>>> +{
>>>>> +    CPUS390XState *env = &cpu->env;
>>>>> +
>>>>> +    if (s390_has_feat(S390_FEAT_AP)) {
>>>>> +        env->regs[1] = 0x10000;
>>>>> +
>>>>> +        return 0;
>>>>> +    }
>>>>> +
>>>>> +    return -EOPNOTSUPP;
>>>>> +}
>>>>> +
>>>>> +int ap_device_handle_dqap(S390CPU *cpu)
>>>>> +{
>>>>> +    CPUS390XState *env = &cpu->env;
>>>>> +
>>>>> +    if (s390_has_feat(S390_FEAT_AP)) {
>>>>> +        env->regs[1] = 0x10000;
>>>>> +
>>>>> +        return 0;
>>>>> +    }
>>>>> +
>>>>> +    return -EOPNOTSUPP;
>>>>> +}
>>>>> +
>>>>> +int ap_device_handle_pqap(S390CPU *cpu)
>>>>> +{
>>>>> +    CPUS390XState *env = &cpu->env;
>>>>> +    int fc = 4 & (env->regs[0] >> 24);
>>>>> +
>>>>> +    /*
>>>>> +     * The Query Configuration Information (QCI) function (fc == 
>>>>> 4) does not
>>>>> +     * set a response code in reg 1, so check for that along with 
>>>>> the
>>>>> +     * AP feature.
>>>>> +     */
>>>>> +    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
>>>>> +        env->regs[1] = 0x10000;
>>>>> +
>>>>> +        return 0;
>>>>> +    }
>>>> This would imply an operation exception in case fc==4, which sounds 
>>>> very
>>>> wrong.
>>>
>>> It depends but I think that the S390_FEAT_AP_QUERY_CONFIG_INFO must 
>>> be tested
>>> to know what to answer.
>>> If the feature is there, QCI must be answered correctly.
>> This is an interesting proposition which raises several issues that 
>> will need to
>> be addressed. The only time the PQAP(QCI) instruction is intercepted 
>> is when:
>> * A vfio-ap device is NOT defined for the guest because the vfio_ap 
>> device driver
>>   will set ECA.28 and the PQAP(QCI) instruction will be interpreted
>> * STFLE.12 is set for the guest
>>
>> You say that the QCI must be answered correctly, but what is the 
>> correct response?
>> If we return the matrix - i.e., APM, ADM and AQM - configured via the 
>> mediated
>> matrix device's sysfs attributes files, then if any APQNs are defined 
>> in the matrix,
>> we will have to handle *ALL* AP instructions by executing them on 
>> behalf of the
>> guest. I suppose we could return an empty matrix in which case the AP 
>> bus will come
>> up without any devices on the guest, but what is the expectation of 
>> an admin who
>> deliberately configures the mediated matrix device? Should we forego 
>> handling interception
>> of AP instructions and consider a guest configuration that turns on 
>> S390_FEAT_AP but
>> does not define a vfio-ap device to be erroneous and terminate 
>> starting of the guest?
>> Anybody have any thoughts?
>>>
>>>
>>> there are also some error situations to handle in all three functions.
>> To what error situations are you referring?
>
> program exceptions, like access, privilege or specification exceptions.
Are you suggesting that these handlers need to verify that the instruction
specification is correct? For example, the NQAP instruction - NQAP r1,r2 -
expects an even-odd pair of general registers in the r1 and r2 and must
designate an even register; otherwise a specification exception is
recognized. Are you saying that the handler for NQAP must verify this
requirement and inject a specification exception if it is not met? If not,
then can you provide a specific example of what you are talking about?
>
>
>
>>>
>>>
>>>
>>>>
>>>> Hard to review without access to documentation. (hard to understand 
>>>> why
>>>> such stuff is to be kept confidential for decades)
>>>>
>>>>> +
>>>>> +    return -EOPNOTSUPP;
>>>>> +}
>>>>> +
>>>>>   static Property vfio_ap_properties[] = {
>>>>>       DEFINE_PROP_STRING("sysfsdev", VFIOAPDevice, vdev.sysfsdev),
>>>>>       DEFINE_PROP_END_OF_LIST(),
>>>>> diff --git a/include/hw/s390x/ap-device.h 
>>>>> b/include/hw/s390x/ap-device.h
>>>>> index 693df90..d45ae38 100644
>>>>> --- a/include/hw/s390x/ap-device.h
>>>>> +++ b/include/hw/s390x/ap-device.h
>>>>> @@ -11,6 +11,8 @@
>>>>>   #ifndef HW_S390X_AP_DEVICE_H
>>>>>   #define HW_S390X_AP_DEVICE_H
>>>>
>>>
>>
>>
>

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Pierre Morel]

On 04/04/2018 14:59, Tony Krowiak wrote:
> On 04/04/2018 07:09 AM, Pierre Morel wrote:
>> On 02/04/2018 18:36, Tony Krowiak wrote:
>>> On 03/26/2018 05:03 AM, Pierre Morel wrote:
>>>> On 26/03/2018 10:32, David Hildenbrand wrote:
>>>>> On 16.03.2018 00:24, Tony Krowiak wrote:
>>>>>> If the CPU model indicates that AP facility is installed on
>>>>>> the guest (i.e., -cpu xxxx,ap=on), then the expectation is that
>>>>>> the AP bus running in the guest will initialize; however, if the
>>>>>> AP instructions are not being interpreted by the firmware, then
>>>>>> they will be intercepted and routed back to QEMU for handling.
>>>>>> If a handler is not defined to process the intercepted instruciton,
>>>>>> t
>>>> ...snip...
>>>>>
>>>>>> +int ap_device_handle_nqap(S390CPU *cpu)
>>>>>> +{
>>>>>> +    CPUS390XState *env = &cpu->env;
>>>>>> +
>>>>>> +    if (s390_has_feat(S390_FEAT_AP)) {
>>>>>> +        env->regs[1] = 0x10000;
>>>>>> +
>>>>>> +        return 0;
>>>>>> +    }
>>>>>> +
>>>>>> +    return -EOPNOTSUPP;
>>>>>> +}
>>>>>> +
>>>>>> +int ap_device_handle_dqap(S390CPU *cpu)
>>>>>> +{
>>>>>> +    CPUS390XState *env = &cpu->env;
>>>>>> +
>>>>>> +    if (s390_has_feat(S390_FEAT_AP)) {
>>>>>> +        env->regs[1] = 0x10000;
>>>>>> +
>>>>>> +        return 0;
>>>>>> +    }
>>>>>> +
>>>>>> +    return -EOPNOTSUPP;
>>>>>> +}
>>>>>> +
>>>>>> +int ap_device_handle_pqap(S390CPU *cpu)
>>>>>> +{
>>>>>> +    CPUS390XState *env = &cpu->env;
>>>>>> +    int fc = 4 & (env->regs[0] >> 24);
>>>>>> +
>>>>>> +    /*
>>>>>> +     * The Query Configuration Information (QCI) function (fc == 
>>>>>> 4) does not
>>>>>> +     * set a response code in reg 1, so check for that along 
>>>>>> with the
>>>>>> +     * AP feature.
>>>>>> +     */
>>>>>> +    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
>>>>>> +        env->regs[1] = 0x10000;
>>>>>> +
>>>>>> +        return 0;
>>>>>> +    }
>>>>> This would imply an operation exception in case fc==4, which 
>>>>> sounds very
>>>>> wrong.
>>>>
>>>> It depends but I think that the S390_FEAT_AP_QUERY_CONFIG_INFO must 
>>>> be tested
>>>> to know what to answer.
>>>> If the feature is there, QCI must be answered correctly.
>>> This is an interesting proposition which raises several issues that 
>>> will need to
>>> be addressed. The only time the PQAP(QCI) instruction is intercepted 
>>> is when:
>>> * A vfio-ap device is NOT defined for the guest because the vfio_ap 
>>> device driver
>>>   will set ECA.28 and the PQAP(QCI) instruction will be interpreted
>>> * STFLE.12 is set for the guest
>>>
>>> You say that the QCI must be answered correctly, but what is the 
>>> correct response?
>>> If we return the matrix - i.e., APM, ADM and AQM - configured via 
>>> the mediated
>>> matrix device's sysfs attributes files, then if any APQNs are 
>>> defined in the matrix,
>>> we will have to handle *ALL* AP instructions by executing them on 
>>> behalf of the
>>> guest. I suppose we could return an empty matrix in which case the 
>>> AP bus will come
>>> up without any devices on the guest, but what is the expectation of 
>>> an admin who
>>> deliberately configures the mediated matrix device? Should we forego 
>>> handling interception
>>> of AP instructions and consider a guest configuration that turns on 
>>> S390_FEAT_AP but
>>> does not define a vfio-ap device to be erroneous and terminate 
>>> starting of the guest?
>>> Anybody have any thoughts?
>>>>
>>>>
>>>> there are also some error situations to handle in all three functions.
>>> To what error situations are you referring?
>>
>> program exceptions, like access, privilege or specification exceptions.
> I'm sorry, I still don't know what you are referring to, can you be 
> more specific?

You do not handle the program exceptions.
The xQAP are priviledged instruction, you must test that it is called 
from the kernel
with
     if (env->psw.mask & PSW_MASK_PSTATE) {
         s390_program_interrupt(env, PGM_PRIVILEGED, 4, ra);
         return 0;
     }

Specification exceptions are recognized if the R1 or R2 specifies a odd 
number register

...etc. more in the documentation.

Since to know the queue you are working on you need the content of the 
registers
you must handle the exceptions.




>>
>>
>>
>>>>
>>>>
>>>>
>>>>>
>>>>> Hard to review without access to documentation. (hard to 
>>>>> understand why
>>>>> such stuff is to be kept confidential for decades)
>>>>>
>>>>>> +
>>>>>> +    return -EOPNOTSUPP;
>>>>>> +}
>>>>>> +
>>>>>>   static Property vfio_ap_properties[] = {
>>>>>>       DEFINE_PROP_STRING("sysfsdev", VFIOAPDevice, vdev.sysfsdev),
>>>>>>       DEFINE_PROP_END_OF_LIST(),
>>>>>> diff --git a/include/hw/s390x/ap-device.h 
>>>>>> b/include/hw/s390x/ap-device.h
>>>>>> index 693df90..d45ae38 100644
>>>>>> --- a/include/hw/s390x/ap-device.h
>>>>>> +++ b/include/hw/s390x/ap-device.h
>>>>>> @@ -11,6 +11,8 @@
>>>>>>   #ifndef HW_S390X_AP_DEVICE_H
>>>>>>   #define HW_S390X_AP_DEVICE_H
>>>>>
>>>>
>>>
>>>
>>
>

-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Tony Krowiak]

On 04/03/2018 05:36 AM, Cornelia Huck wrote:
> On Mon, 2 Apr 2018 12:36:27 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> On 03/26/2018 05:03 AM, Pierre Morel wrote:
>>> On 26/03/2018 10:32, David Hildenbrand wrote:
>>>> On 16.03.2018 00:24, Tony Krowiak wrote:
>>>>> +    /*
>>>>> +     * The Query Configuration Information (QCI) function (fc == 4)
>>>>> does not
>>>>> +     * set a response code in reg 1, so check for that along with the
>>>>> +     * AP feature.
>>>>> +     */
>>>>> +    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
>>>>> +        env->regs[1] = 0x10000;
>>>>> +
>>>>> +        return 0;
>>>>> +    }
>>>> This would imply an operation exception in case fc==4, which sounds very
>>>> wrong.
>>> It depends but I think that the S390_FEAT_AP_QUERY_CONFIG_INFO must be
>>> tested
>>> to know what to answer.
>>> If the feature is there, QCI must be answered correctly.
>> This is an interesting proposition which raises several issues that will
>> need to
>> be addressed. The only time the PQAP(QCI) instruction is intercepted is
>> when:
>> * A vfio-ap device is NOT defined for the guest because the vfio_ap
>> device driver
>>     will set ECA.28 and the PQAP(QCI) instruction will be interpreted
>> * STFLE.12 is set for the guest
>>
>> You say that the QCI must be answered correctly, but what is the correct
>> response?
>> If we return the matrix - i.e., APM, ADM and AQM - configured via the
>> mediated
>> matrix device's sysfs attributes files, then if any APQNs are defined in
>> the matrix,
>> we will have to handle *ALL* AP instructions by executing them on behalf
>> of the
>> guest. I suppose we could return an empty matrix in which case the AP
>> bus will come
>> up without any devices on the guest, but what is the expectation of an
>> admin who
>> deliberately configures the mediated matrix device? Should we forego
>> handling interception
>> of AP instructions and consider a guest configuration that turns on
>> S390_FEAT_AP but
>> does not define a vfio-ap device to be erroneous and terminate starting
>> of the guest?
>> Anybody have any thoughts?
> Hard to really give good advice without access to the documentation, but:
> - If we tell the guest that the feature is available, but it does not
>    get any cards to use, returning an empty matrix makes the most sense
>    to me.
After some thought, I am inclined to agree with this. If the no vfio-ap 
device
is defined for the guest, then anything configured for the mediated matrix
device is not relevant and the guest would therefore have no AP devices.
> - I would not tie starting the guest to the presence of a vfio-ap
>    device. Having the feature available in theory but without any
>    devices actually being usable by the guest does not really sound
>    wrong (can we hotplug this later?)
>

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Pierre Morel]

On 04/04/2018 15:33, Tony Krowiak wrote:
> On 04/04/2018 07:09 AM, Pierre Morel wrote:
>> On 02/04/2018 18:36, Tony Krowiak wrote:
>>> On 03/26/2018 05:03 AM, Pierre Morel wrote:
>>>> On 26/03/2018 10:32, David Hildenbrand wrote:
>>>>> On 16.03.2018 00:24, Tony Krowiak wrote:
>>>>>> If the CPU model indicates that AP facility is installed on
>>>>>> the guest (i.e., -cpu xxxx,ap=on), then the expectation is that
>>>>>> the AP bus running in the guest will initialize; however, if the
>>>>>> AP instructions are not being interpreted by the firmware, then
>>>>>> they will be intercepted and routed back to QEMU for handling.
>>>>>> If a handler is not defined to process the intercepted instruciton,
>>>>>> t
>>>> ...snip...
>>>>>
>>>>>> +int ap_device_handle_nqap(S390CPU *cpu)
>>>>>> +{
>>>>>> +    CPUS390XState *env = &cpu->env;
>>>>>> +
>>>>>> +    if (s390_has_feat(S390_FEAT_AP)) {
>>>>>> +        env->regs[1] = 0x10000;
>>>>>> +
>>>>>> +        return 0;
>>>>>> +    }
>>>>>> +
>>>>>> +    return -EOPNOTSUPP;
>>>>>> +}
>>>>>> +
>>>>>> +int ap_device_handle_dqap(S390CPU *cpu)
>>>>>> +{
>>>>>> +    CPUS390XState *env = &cpu->env;
>>>>>> +
>>>>>> +    if (s390_has_feat(S390_FEAT_AP)) {
>>>>>> +        env->regs[1] = 0x10000;
>>>>>> +
>>>>>> +        return 0;
>>>>>> +    }
>>>>>> +
>>>>>> +    return -EOPNOTSUPP;
>>>>>> +}
>>>>>> +
>>>>>> +int ap_device_handle_pqap(S390CPU *cpu)
>>>>>> +{
>>>>>> +    CPUS390XState *env = &cpu->env;
>>>>>> +    int fc = 4 & (env->regs[0] >> 24);
>>>>>> +
>>>>>> +    /*
>>>>>> +     * The Query Configuration Information (QCI) function (fc == 
>>>>>> 4) does not
>>>>>> +     * set a response code in reg 1, so check for that along 
>>>>>> with the
>>>>>> +     * AP feature.
>>>>>> +     */
>>>>>> +    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
>>>>>> +        env->regs[1] = 0x10000;
>>>>>> +
>>>>>> +        return 0;
>>>>>> +    }
>>>>> This would imply an operation exception in case fc==4, which 
>>>>> sounds very
>>>>> wrong.
>>>>
>>>> It depends but I think that the S390_FEAT_AP_QUERY_CONFIG_INFO must 
>>>> be tested
>>>> to know what to answer.
>>>> If the feature is there, QCI must be answered correctly.
>>> This is an interesting proposition which raises several issues that 
>>> will need to
>>> be addressed. The only time the PQAP(QCI) instruction is intercepted 
>>> is when:
>>> * A vfio-ap device is NOT defined for the guest because the vfio_ap 
>>> device driver
>>>   will set ECA.28 and the PQAP(QCI) instruction will be interpreted
>>> * STFLE.12 is set for the guest
>>>
>>> You say that the QCI must be answered correctly, but what is the 
>>> correct response?
>>> If we return the matrix - i.e., APM, ADM and AQM - configured via 
>>> the mediated
>>> matrix device's sysfs attributes files, then if any APQNs are 
>>> defined in the matrix,
>>> we will have to handle *ALL* AP instructions by executing them on 
>>> behalf of the
>>> guest. I suppose we could return an empty matrix in which case the 
>>> AP bus will come
>>> up without any devices on the guest, but what is the expectation of 
>>> an admin who
>>> deliberately configures the mediated matrix device? Should we forego 
>>> handling interception
>>> of AP instructions and consider a guest configuration that turns on 
>>> S390_FEAT_AP but
>>> does not define a vfio-ap device to be erroneous and terminate 
>>> starting of the guest?
>>> Anybody have any thoughts?
>>>>
>>>>
>>>> there are also some error situations to handle in all three functions.
>>> To what error situations are you referring?
>>
>> program exceptions, like access, privilege or specification exceptions.
> Are you suggesting that these handlers need to verify that the 
> instruction
> specification is correct? For example, the NQAP instruction - NQAP 
> r1,r2 -
> expects an even-odd pair of general registers in the r1 and r2 and must
> designate an even register; otherwise a specification exception is
> recognized. Are you saying that the handler for NQAP must verify this
> requirement and inject a specification exception if it is not met? If 
> not,
> then can you provide a specific example of what you are talking about?

Yes I mean this.

But one other thing.
Returning code 0x01 seems wrong for me, this is right if no AP card
are in the system but I do not thing that it is what we mean.
I think we should return code 0x03 stating that the AP card is not in 
configured state

The difference is that we can get out of a not-configured state with a 
SCLP command but
we can not recover from an un-existing APQN.


>>
>>
>>
>>>>
>>>>
>>>>
>>>>>
>>>>> Hard to review without access to documentation. (hard to 
>>>>> understand why
>>>>> such stuff is to be kept confidential for decades)
>>>>>
>>>>>> +
>>>>>> +    return -EOPNOTSUPP;
>>>>>> +}
>>>>>> +
>>>>>>   static Property vfio_ap_properties[] = {
>>>>>>       DEFINE_PROP_STRING("sysfsdev", VFIOAPDevice, vdev.sysfsdev),
>>>>>>       DEFINE_PROP_END_OF_LIST(),
>>>>>> diff --git a/include/hw/s390x/ap-device.h 
>>>>>> b/include/hw/s390x/ap-device.h
>>>>>> index 693df90..d45ae38 100644
>>>>>> --- a/include/hw/s390x/ap-device.h
>>>>>> +++ b/include/hw/s390x/ap-device.h
>>>>>> @@ -11,6 +11,8 @@
>>>>>>   #ifndef HW_S390X_AP_DEVICE_H
>>>>>>   #define HW_S390X_AP_DEVICE_H
>>>>>
>>>>
>>>
>>>
>>
>

-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Tony Krowiak]

On 04/04/2018 09:43 AM, Pierre Morel wrote:
> On 04/04/2018 15:33, Tony Krowiak wrote:
>> On 04/04/2018 07:09 AM, Pierre Morel wrote:
>>> On 02/04/2018 18:36, Tony Krowiak wrote:
>>>> On 03/26/2018 05:03 AM, Pierre Morel wrote:
>>>>> On 26/03/2018 10:32, David Hildenbrand wrote:
>>>>>> On 16.03.2018 00:24, Tony Krowiak wrote:
>>>>>>> If the CPU model indicates that AP facility is installed on
>>>>>>> the guest (i.e., -cpu xxxx,ap=on), then the expectation is that
>>>>>>> the AP bus running in the guest will initialize; however, if the
>>>>>>> AP instructions are not being interpreted by the firmware, then
>>>>>>> they will be intercepted and routed back to QEMU for handling.
>>>>>>> If a handler is not defined to process the intercepted instruciton,
>>>>>>> t
>>>>> ...snip...
>>>>>>
>>>>>>> +int ap_device_handle_nqap(S390CPU *cpu)
>>>>>>> +{
>>>>>>> +    CPUS390XState *env = &cpu->env;
>>>>>>> +
>>>>>>> +    if (s390_has_feat(S390_FEAT_AP)) {
>>>>>>> +        env->regs[1] = 0x10000;
>>>>>>> +
>>>>>>> +        return 0;
>>>>>>> +    }
>>>>>>> +
>>>>>>> +    return -EOPNOTSUPP;
>>>>>>> +}
>>>>>>> +
>>>>>>> +int ap_device_handle_dqap(S390CPU *cpu)
>>>>>>> +{
>>>>>>> +    CPUS390XState *env = &cpu->env;
>>>>>>> +
>>>>>>> +    if (s390_has_feat(S390_FEAT_AP)) {
>>>>>>> +        env->regs[1] = 0x10000;
>>>>>>> +
>>>>>>> +        return 0;
>>>>>>> +    }
>>>>>>> +
>>>>>>> +    return -EOPNOTSUPP;
>>>>>>> +}
>>>>>>> +
>>>>>>> +int ap_device_handle_pqap(S390CPU *cpu)
>>>>>>> +{
>>>>>>> +    CPUS390XState *env = &cpu->env;
>>>>>>> +    int fc = 4 & (env->regs[0] >> 24);
>>>>>>> +
>>>>>>> +    /*
>>>>>>> +     * The Query Configuration Information (QCI) function (fc 
>>>>>>> == 4) does not
>>>>>>> +     * set a response code in reg 1, so check for that along 
>>>>>>> with the
>>>>>>> +     * AP feature.
>>>>>>> +     */
>>>>>>> +    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
>>>>>>> +        env->regs[1] = 0x10000;
>>>>>>> +
>>>>>>> +        return 0;
>>>>>>> +    }
>>>>>> This would imply an operation exception in case fc==4, which 
>>>>>> sounds very
>>>>>> wrong.
>>>>>
>>>>> It depends but I think that the S390_FEAT_AP_QUERY_CONFIG_INFO 
>>>>> must be tested
>>>>> to know what to answer.
>>>>> If the feature is there, QCI must be answered correctly.
>>>> This is an interesting proposition which raises several issues that 
>>>> will need to
>>>> be addressed. The only time the PQAP(QCI) instruction is 
>>>> intercepted is when:
>>>> * A vfio-ap device is NOT defined for the guest because the vfio_ap 
>>>> device driver
>>>>   will set ECA.28 and the PQAP(QCI) instruction will be interpreted
>>>> * STFLE.12 is set for the guest
>>>>
>>>> You say that the QCI must be answered correctly, but what is the 
>>>> correct response?
>>>> If we return the matrix - i.e., APM, ADM and AQM - configured via 
>>>> the mediated
>>>> matrix device's sysfs attributes files, then if any APQNs are 
>>>> defined in the matrix,
>>>> we will have to handle *ALL* AP instructions by executing them on 
>>>> behalf of the
>>>> guest. I suppose we could return an empty matrix in which case the 
>>>> AP bus will come
>>>> up without any devices on the guest, but what is the expectation of 
>>>> an admin who
>>>> deliberately configures the mediated matrix device? Should we 
>>>> forego handling interception
>>>> of AP instructions and consider a guest configuration that turns on 
>>>> S390_FEAT_AP but
>>>> does not define a vfio-ap device to be erroneous and terminate 
>>>> starting of the guest?
>>>> Anybody have any thoughts?
>>>>>
>>>>>
>>>>> there are also some error situations to handle in all three 
>>>>> functions.
>>>> To what error situations are you referring?
>>>
>>> program exceptions, like access, privilege or specification exceptions.
>> Are you suggesting that these handlers need to verify that the 
>> instruction
>> specification is correct? For example, the NQAP instruction - NQAP 
>> r1,r2 -
>> expects an even-odd pair of general registers in the r1 and r2 and must
>> designate an even register; otherwise a specification exception is
>> recognized. Are you saying that the handler for NQAP must verify this
>> requirement and inject a specification exception if it is not met? If 
>> not,
>> then can you provide a specific example of what you are talking about?
>
> Yes I mean this.
>
> But one other thing.
> Returning code 0x01 seems wrong for me, this is right if no AP card
> are in the system but I do not thing that it is what we mean.
I disagree .... for the guest, this is exactly what we mean. In my 
opinion, assigning
AP devices to the mediated matrix device is like assigning AP devices to 
an LPAR.
If no vfio-ap device is defined for the guest, then it is effectively 
the same as
starting a linux host in an LPAR without any AP devices assigned to it.
>
> I think we should return code 0x03 stating that the AP card is not in 
> configured state

>
>
> The difference is that we can get out of a not-configured state with a 
> SCLP command but
> we can not recover from an un-existing APQN.
Remember, this state will occur only when the AP feature of the CPU 
model is turned on for
the guest and no vfio-ap device is defined, so there is no configuration 
to recover for
the guest because no AP devices will be configured for the guest.
>
>
>
>>>
>>>
>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Hard to review without access to documentation. (hard to 
>>>>>> understand why
>>>>>> such stuff is to be kept confidential for decades)
>>>>>>
>>>>>>> +
>>>>>>> +    return -EOPNOTSUPP;
>>>>>>> +}
>>>>>>> +
>>>>>>>   static Property vfio_ap_properties[] = {
>>>>>>>       DEFINE_PROP_STRING("sysfsdev", VFIOAPDevice, vdev.sysfsdev),
>>>>>>>       DEFINE_PROP_END_OF_LIST(),
>>>>>>> diff --git a/include/hw/s390x/ap-device.h 
>>>>>>> b/include/hw/s390x/ap-device.h
>>>>>>> index 693df90..d45ae38 100644
>>>>>>> --- a/include/hw/s390x/ap-device.h
>>>>>>> +++ b/include/hw/s390x/ap-device.h
>>>>>>> @@ -11,6 +11,8 @@
>>>>>>>   #ifndef HW_S390X_AP_DEVICE_H
>>>>>>>   #define HW_S390X_AP_DEVICE_H
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Halil Pasic]

On 04/04/2018 10:12 PM, Tony Krowiak wrote:
> On 04/04/2018 09:43 AM, Pierre Morel wrote:
>> On 04/04/2018 15:33, Tony Krowiak wrote:
>>> On 04/04/2018 07:09 AM, Pierre Morel wrote:
>>>> On 02/04/2018 18:36, Tony Krowiak wrote:
>>>>> On 03/26/2018 05:03 AM, Pierre Morel wrote:
>>>>>> On 26/03/2018 10:32, David Hildenbrand wrote:
>>>>>>> On 16.03.2018 00:24, Tony Krowiak wrote:
>>>>>>>> If the CPU model indicates that AP facility is installed on
>>>>>>>> the guest (i.e., -cpu xxxx,ap=on), then the expectation is that
>>>>>>>> the AP bus running in the guest will initialize; however, if the
>>>>>>>> AP instructions are not being interpreted by the firmware, then
>>>>>>>> they will be intercepted and routed back to QEMU for handling.
>>>>>>>> If a handler is not defined to process the intercepted instruciton,
>>>>>>>> t
>>>>>> ...snip...
>>>>>>>
>>>>>>>> +int ap_device_handle_nqap(S390CPU *cpu)
>>>>>>>> +{
>>>>>>>> +    CPUS390XState *env = &cpu->env;
>>>>>>>> +
>>>>>>>> +    if (s390_has_feat(S390_FEAT_AP)) {
>>>>>>>> +        env->regs[1] = 0x10000;
>>>>>>>> +
>>>>>>>> +        return 0;
>>>>>>>> +    }
>>>>>>>> +
>>>>>>>> +    return -EOPNOTSUPP;
>>>>>>>> +}
>>>>>>>> +
>>>>>>>> +int ap_device_handle_dqap(S390CPU *cpu)
>>>>>>>> +{
>>>>>>>> +    CPUS390XState *env = &cpu->env;
>>>>>>>> +
>>>>>>>> +    if (s390_has_feat(S390_FEAT_AP)) {
>>>>>>>> +        env->regs[1] = 0x10000;
>>>>>>>> +
>>>>>>>> +        return 0;
>>>>>>>> +    }
>>>>>>>> +
>>>>>>>> +    return -EOPNOTSUPP;
>>>>>>>> +}
>>>>>>>> +
>>>>>>>> +int ap_device_handle_pqap(S390CPU *cpu)
>>>>>>>> +{
>>>>>>>> +    CPUS390XState *env = &cpu->env;
>>>>>>>> +    int fc = 4 & (env->regs[0] >> 24);
>>>>>>>> +
>>>>>>>> +    /*
>>>>>>>> +     * The Query Configuration Information (QCI) function (fc == 4) does not
>>>>>>>> +     * set a response code in reg 1, so check for that along with the
>>>>>>>> +     * AP feature.
>>>>>>>> +     */
>>>>>>>> +    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
>>>>>>>> +        env->regs[1] = 0x10000;
>>>>>>>> +
>>>>>>>> +        return 0;
>>>>>>>> +    }
>>>>>>> This would imply an operation exception in case fc==4, which sounds very
>>>>>>> wrong.
>>>>>>
>>>>>> It depends but I think that the S390_FEAT_AP_QUERY_CONFIG_INFO must be tested
>>>>>> to know what to answer.
>>>>>> If the feature is there, QCI must be answered correctly.
>>>>> This is an interesting proposition which raises several issues that will need to
>>>>> be addressed. The only time the PQAP(QCI) instruction is intercepted is when:
>>>>> * A vfio-ap device is NOT defined for the guest because the vfio_ap device driver
>>>>>   will set ECA.28 and the PQAP(QCI) instruction will be interpreted
>>>>> * STFLE.12 is set for the guest
>>>>>
>>>>> You say that the QCI must be answered correctly, but what is the correct response?
>>>>> If we return the matrix - i.e., APM, ADM and AQM - configured via the mediated
>>>>> matrix device's sysfs attributes files, then if any APQNs are defined in the matrix,
>>>>> we will have to handle *ALL* AP instructions by executing them on behalf of the
>>>>> guest. I suppose we could return an empty matrix in which case the AP bus will come
>>>>> up without any devices on the guest, but what is the expectation of an admin who
>>>>> deliberately configures the mediated matrix device? Should we forego handling interception
>>>>> of AP instructions and consider a guest configuration that turns on S390_FEAT_AP but
>>>>> does not define a vfio-ap device to be erroneous and terminate starting of the guest?
>>>>> Anybody have any thoughts?
>>>>>>
>>>>>>
>>>>>> there are also some error situations to handle in all three functions.
>>>>> To what error situations are you referring?
>>>>
>>>> program exceptions, like access, privilege or specification exceptions.
>>> Are you suggesting that these handlers need to verify that the instruction
>>> specification is correct? For example, the NQAP instruction - NQAP r1,r2 -
>>> expects an even-odd pair of general registers in the r1 and r2 and must
>>> designate an even register; otherwise a specification exception is
>>> recognized. Are you saying that the handler for NQAP must verify this
>>> requirement and inject a specification exception if it is not met? If not,
>>> then can you provide a specific example of what you are talking about?
>>
>> Yes I mean this.
>>
>> But one other thing.
>> Returning code 0x01 seems wrong for me, this is right if no AP card
>> are in the system but I do not thing that it is what we mean.
> I disagree .... for the guest, this is exactly what we mean. In my opinion, assigning
> AP devices to the mediated matrix device is like assigning AP devices to an LPAR.
> If no vfio-ap device is defined for the guest, then it is effectively the same as
> starting a linux host in an LPAR without any AP devices assigned to it.
>>
>> I think we should return code 0x03 stating that the AP card is not in configured state
> 
>>
>>
>> The difference is that we can get out of a not-configured state with a SCLP command but
>> we can not recover from an un-existing APQN.
> Remember, this state will occur only when the AP feature of the CPU model is turned on for
> the guest and no vfio-ap device is defined, so there is no configuration to recover for
> the guest because no AP devices will be configured for the guest.

Spoke with Pierre and we agreed that the response code 01 is what
we need to do.

Regards,
Halil

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Tony Krowiak]

On 04/03/2018 05:36 AM, Cornelia Huck wrote:
> On Mon, 2 Apr 2018 12:36:27 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> On 03/26/2018 05:03 AM, Pierre Morel wrote:
>>> On 26/03/2018 10:32, David Hildenbrand wrote:
>>>> On 16.03.2018 00:24, Tony Krowiak wrote:
>>>>> +    /*
>>>>> +     * The Query Configuration Information (QCI) function (fc == 4)
>>>>> does not
>>>>> +     * set a response code in reg 1, so check for that along with the
>>>>> +     * AP feature.
>>>>> +     */
>>>>> +    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
>>>>> +        env->regs[1] = 0x10000;
>>>>> +
>>>>> +        return 0;
>>>>> +    }
>>>> This would imply an operation exception in case fc==4, which sounds very
>>>> wrong.
>>> It depends but I think that the S390_FEAT_AP_QUERY_CONFIG_INFO must be
>>> tested
>>> to know what to answer.
>>> If the feature is there, QCI must be answered correctly.
>> This is an interesting proposition which raises several issues that will
>> need to
>> be addressed. The only time the PQAP(QCI) instruction is intercepted is
>> when:
>> * A vfio-ap device is NOT defined for the guest because the vfio_ap
>> device driver
>>     will set ECA.28 and the PQAP(QCI) instruction will be interpreted
>> * STFLE.12 is set for the guest
>>
>> You say that the QCI must be answered correctly, but what is the correct
>> response?
>> If we return the matrix - i.e., APM, ADM and AQM - configured via the
>> mediated
>> matrix device's sysfs attributes files, then if any APQNs are defined in
>> the matrix,
>> we will have to handle *ALL* AP instructions by executing them on behalf
>> of the
>> guest. I suppose we could return an empty matrix in which case the AP
>> bus will come
>> up without any devices on the guest, but what is the expectation of an
>> admin who
>> deliberately configures the mediated matrix device? Should we forego
>> handling interception
>> of AP instructions and consider a guest configuration that turns on
>> S390_FEAT_AP but
>> does not define a vfio-ap device to be erroneous and terminate starting
>> of the guest?
>> Anybody have any thoughts?
> Hard to really give good advice without access to the documentation, but:
> - If we tell the guest that the feature is available, but it does not
>    get any cards to use, returning an empty matrix makes the most sense
>    to me.
> - I would not tie starting the guest to the presence of a vfio-ap
>    device. Having the feature available in theory but without any
>    devices actually being usable by the guest does not really sound
>    wrong (can we hotplug this later?)
For this phase of development, it is my opinion that introducing AP 
instruction
interception handlers is superfluous for the following reasons:

1. Interception handling was introduced solely to ensure an operation 
exception would
    not be injected into the guest when CPU model feature for AP (i.e., 
ap=on)
    is specified but a VFIO AP device (i.e., -device 
vfio-ap,sysfsdev=$path)
    is not.

2. The implementation of guest dedicated crypto adapters uses AP 
instruction
    interpretation to virtualize AP devices for a guest. As such, the NQAP,
    DQAP and most variants of the PQAP instructions will not be
    intercepted.

3. Hot plugging AP devices is not being supported for this phase of 
development.

It is my opinion that introducing these interception handlers at this 
time is
unnecessary and risks opening a can of worms that would be
better dealt with in a subsequent phase. For that reason and the reasons 
stated
above, I think the better option is to terminate starting the guest if the
CPU model feature for AP is enabled but an AP device is not defined for the
guest. This restriction, of course, will be removed when hot plug is 
implemented
in a subsequent development phase.
>

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Halil Pasic]

On 04/05/2018 06:38 PM, Tony Krowiak wrote:
>> Hard to really give good advice without access to the documentation, but:
>> - If we tell the guest that the feature is available, but it does not
>>    get any cards to use, returning an empty matrix makes the most sense
>>    to me.
>> - I would not tie starting the guest to the presence of a vfio-ap
>>    device. Having the feature available in theory but without any
>>    devices actually being usable by the guest does not really sound
>>    wrong (can we hotplug this later?)
> For this phase of development, it is my opinion that introducing AP instruction
> interception handlers is superfluous for the following reasons:
> 
> 1. Interception handling was introduced solely to ensure an operation exception would
>    not be injected into the guest when CPU model feature for AP (i.e., ap=on)
>    is specified but a VFIO AP device (i.e., -device vfio-ap,sysfsdev=$path)
>    is not.
> 
> 2. The implementation of guest dedicated crypto adapters uses AP instruction
>    interpretation to virtualize AP devices for a guest. As such, the NQAP,
>    DQAP and most variants of the PQAP instructions will not be
>    intercepted.
> 
> 3. Hot plugging AP devices is not being supported for this phase of development.
> 
> It is my opinion that introducing these interception handlers at this time is
> unnecessary and risks opening a can of worms that would be
> better dealt with in a subsequent phase. For that reason and the reasons stated
> above, I think the better option is to terminate starting the guest if the
> CPU model feature for AP is enabled but an AP device is not defined for the
> guest. This restriction, of course, will be removed when hot plug is implemented
> in a subsequent development phase.

I second that! I agree that having ap instructions but not having the
possibility to actually do AP crypto is probably not what the user wants.
Preventing such a guest form starting (with a nice message) sounds reasonable
to me.

I agree with Connie, the approach 'hold the line' (until future hotplugs)
is the most reasonable thing to do *in the long run*. But I think it's better
to limit ourselves to the simplest case for now, I don't see any problems
with doing the hotplug support later.

Regards,
Halil

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Cornelia Huck]

On Thu, 5 Apr 2018 19:17:47 +0200
Halil Pasic <pasic@linux.vnet.ibm.com> wrote:

> On 04/05/2018 06:38 PM, Tony Krowiak wrote:
> >> Hard to really give good advice without access to the documentation, but:
> >> - If we tell the guest that the feature is available, but it does not
> >>    get any cards to use, returning an empty matrix makes the most sense
> >>    to me.
> >> - I would not tie starting the guest to the presence of a vfio-ap
> >>    device. Having the feature available in theory but without any
> >>    devices actually being usable by the guest does not really sound
> >>    wrong (can we hotplug this later?)  
> > For this phase of development, it is my opinion that introducing AP instruction
> > interception handlers is superfluous for the following reasons:
> > 
> > 1. Interception handling was introduced solely to ensure an operation exception would
> >    not be injected into the guest when CPU model feature for AP (i.e., ap=on)
> >    is specified but a VFIO AP device (i.e., -device vfio-ap,sysfsdev=$path)
> >    is not.
> > 
> > 2. The implementation of guest dedicated crypto adapters uses AP instruction
> >    interpretation to virtualize AP devices for a guest. As such, the NQAP,
> >    DQAP and most variants of the PQAP instructions will not be
> >    intercepted.
> > 
> > 3. Hot plugging AP devices is not being supported for this phase of development.
> > 
> > It is my opinion that introducing these interception handlers at this time is
> > unnecessary and risks opening a can of worms that would be
> > better dealt with in a subsequent phase. For that reason and the reasons stated
> > above, I think the better option is to terminate starting the guest if the
> > CPU model feature for AP is enabled but an AP device is not defined for the
> > guest. This restriction, of course, will be removed when hot plug is implemented
> > in a subsequent development phase.  
> 
> I second that! I agree that having ap instructions but not having the
> possibility to actually do AP crypto is probably not what the user wants.
> Preventing such a guest form starting (with a nice message) sounds reasonable
> to me.

One problem I have with that is that it feels backwards to me.

The situation "you cannot add this device unless $FEATURE is present"
is quite common and thus easily understood. Now, this would introduce
the situation "you cannot present $FEATURE unless this device is also
present, and that right at the start". I'm not sure how you are
supposed to correlate a cpu feature with the existence of a device.

> I agree with Connie, the approach 'hold the line' (until future hotplugs)
> is the most reasonable thing to do *in the long run*. But I think it's better
> to limit ourselves to the simplest case for now, I don't see any problems
> with doing the hotplug support later.

Yes, having to add handlers that add very little benefit sucks, I
agree. But I fear if we add the "feature needs device" dependency, we
open another can of worms, including the question what happens if we
want to support hotplug in the future (I'm not altogether sure how to
handle the whole checking from qemu).

Making sure that we have both the feature and the device when using a
management software (e.g. libvirt) makes a lot of sense (and is
probably also easier to implement), but it won't help us with the issue
of the interception handlers, unfortunately.

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[David Hildenbrand]

On 06.04.2018 10:40, Cornelia Huck wrote:
> On Thu, 5 Apr 2018 19:17:47 +0200
> Halil Pasic <pasic@linux.vnet.ibm.com> wrote:
> 
>> On 04/05/2018 06:38 PM, Tony Krowiak wrote:
>>>> Hard to really give good advice without access to the documentation, but:
>>>> - If we tell the guest that the feature is available, but it does not
>>>>    get any cards to use, returning an empty matrix makes the most sense
>>>>    to me.
>>>> - I would not tie starting the guest to the presence of a vfio-ap
>>>>    device. Having the feature available in theory but without any
>>>>    devices actually being usable by the guest does not really sound
>>>>    wrong (can we hotplug this later?)  
>>> For this phase of development, it is my opinion that introducing AP instruction
>>> interception handlers is superfluous for the following reasons:
>>>
>>> 1. Interception handling was introduced solely to ensure an operation exception would
>>>    not be injected into the guest when CPU model feature for AP (i.e., ap=on)
>>>    is specified but a VFIO AP device (i.e., -device vfio-ap,sysfsdev=$path)
>>>    is not.
>>>
>>> 2. The implementation of guest dedicated crypto adapters uses AP instruction
>>>    interpretation to virtualize AP devices for a guest. As such, the NQAP,
>>>    DQAP and most variants of the PQAP instructions will not be
>>>    intercepted.
>>>
>>> 3. Hot plugging AP devices is not being supported for this phase of development.
>>>
>>> It is my opinion that introducing these interception handlers at this time is
>>> unnecessary and risks opening a can of worms that would be
>>> better dealt with in a subsequent phase. For that reason and the reasons stated
>>> above, I think the better option is to terminate starting the guest if the
>>> CPU model feature for AP is enabled but an AP device is not defined for the
>>> guest. This restriction, of course, will be removed when hot plug is implemented
>>> in a subsequent development phase.  
>>
>> I second that! I agree that having ap instructions but not having the
>> possibility to actually do AP crypto is probably not what the user wants.
>> Preventing such a guest form starting (with a nice message) sounds reasonable
>> to me.
> 
> One problem I have with that is that it feels backwards to me.
> 
> The situation "you cannot add this device unless $FEATURE is present"
> is quite common and thus easily understood. Now, this would introduce
> the situation "you cannot present $FEATURE unless this device is also
> present, and that right at the start". I'm not sure how you are
> supposed to correlate a cpu feature with the existence of a device.

I agree. Don't make things harder than they are. This smells like "cpu
feature can only be provided if another magical QEMU command line option
is present". Don't do that.

Is it really that hard to implement a very simple interception handler
that says t all instructions "yeah, I'm alive, but no, nothing to see here".

> 
>> I agree with Connie, the approach 'hold the line' (until future hotplugs)
>> is the most reasonable thing to do *in the long run*. But I think it's better
>> to limit ourselves to the simplest case for now, I don't see any problems
>> with doing the hotplug support later.
> 
> Yes, having to add handlers that add very little benefit sucks, I
> agree. But I fear if we add the "feature needs device" dependency, we
> open another can of worms, including the question what happens if we
> want to support hotplug in the future (I'm not altogether sure how to
> handle the whole checking from qemu).
> 
> Making sure that we have both the feature and the device when using a
> management software (e.g. libvirt) makes a lot of sense (and is
> probably also easier to implement), but it won't help us with the issue
> of the interception handlers, unfortunately.
> 

I agree.


-- 

Thanks,

David / dhildenb

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Halil Pasic]

On 04/06/2018 11:11 AM, David Hildenbrand wrote:
> On 06.04.2018 10:40, Cornelia Huck wrote:
>> On Thu, 5 Apr 2018 19:17:47 +0200
>> Halil Pasic <pasic@linux.vnet.ibm.com> wrote:
>>
>>> On 04/05/2018 06:38 PM, Tony Krowiak wrote:
>>>>> Hard to really give good advice without access to the documentation, but:
>>>>> - If we tell the guest that the feature is available, but it does not
>>>>>    get any cards to use, returning an empty matrix makes the most sense
>>>>>    to me.
>>>>> - I would not tie starting the guest to the presence of a vfio-ap
>>>>>    device. Having the feature available in theory but without any
>>>>>    devices actually being usable by the guest does not really sound
>>>>>    wrong (can we hotplug this later?)  
>>>> For this phase of development, it is my opinion that introducing AP instruction
>>>> interception handlers is superfluous for the following reasons:
>>>>
>>>> 1. Interception handling was introduced solely to ensure an operation exception would
>>>>    not be injected into the guest when CPU model feature for AP (i.e., ap=on)
>>>>    is specified but a VFIO AP device (i.e., -device vfio-ap,sysfsdev=$path)
>>>>    is not.
>>>>
>>>> 2. The implementation of guest dedicated crypto adapters uses AP instruction
>>>>    interpretation to virtualize AP devices for a guest. As such, the NQAP,
>>>>    DQAP and most variants of the PQAP instructions will not be
>>>>    intercepted.
>>>>
>>>> 3. Hot plugging AP devices is not being supported for this phase of development.
>>>>
>>>> It is my opinion that introducing these interception handlers at this time is
>>>> unnecessary and risks opening a can of worms that would be
>>>> better dealt with in a subsequent phase. For that reason and the reasons stated
>>>> above, I think the better option is to terminate starting the guest if the
>>>> CPU model feature for AP is enabled but an AP device is not defined for the
>>>> guest. This restriction, of course, will be removed when hot plug is implemented
>>>> in a subsequent development phase.  
>>>
>>> I second that! I agree that having ap instructions but not having the
>>> possibility to actually do AP crypto is probably not what the user wants.
>>> Preventing such a guest form starting (with a nice message) sounds reasonable
>>> to me.
>>
>> One problem I have with that is that it feels backwards to me.
>>
>> The situation "you cannot add this device unless $FEATURE is present"
>> is quite common and thus easily understood. Now, this would introduce
>> the situation "you cannot present $FEATURE unless this device is also
>> present, and that right at the start". I'm not sure how you are
>> supposed to correlate a cpu feature with the existence of a device.


I think it can be done straightforward and with less LOC than interception
and emulation of 'nothing to see here' requires.

> 
> I agree. Don't make things harder than they are. This smells like "cpu
> feature can only be provided if another magical QEMU command line option
> is present". Don't do that.

Yes it is conceptually ugly. I'm 100% with you. That's why it should go
away soon. From the practicality perspective however I would even argue that it's
helpful to the user: tells 'oops you have forgotten something'. IMHO
it's a shortcut of type make the problem smaller. Regarding what is
harder and what is easier: the author is probably the most fit to decide
that. If it is harder, it makes no sense, as this is all about cutting
corners.

> 
> Is it really that hard to implement a very simple interception handler
> that says t all instructions "yeah, I'm alive, but no, nothing to see here".
> 

I find it somewhat difficult to reason about what is static and what is
dynamic in the AP architecture. To put something together that seems to
work should be relatively easy. I could even say, I hope Tony tested the
no device case with v3 and it apparently seemed to work -- as I don't see
any does not work disclaimer. But getting all the stuff correct is IMHO
a bit more of a challenge.

>>
>>> I agree with Connie, the approach 'hold the line' (until future hotplugs)
>>> is the most reasonable thing to do *in the long run*. But I think it's better
>>> to limit ourselves to the simplest case for now, I don't see any problems
>>> with doing the hotplug support later.
>>
>> Yes, having to add handlers that add very little benefit sucks, I
>> agree. But I fear if we add the "feature needs device" dependency, we
>> open another can of worms, including the question what happens if we
>> want to support hotplug in the future (I'm not altogether sure how to
>> handle the whole checking from qemu).
>>

We do want hotplug in the future AFAIK. The idea was to just remove the
limitation when everything is in place.

Regarding the implementation, the idea was to use 
qemu_add_machine_init_done_notifier and  only catch both of the following true
* the cpu model has ap=on
* on the ap bus (I think it would be nice to have a bus with max_dev = 1)
  there is no device

When hotplug becomes available for vfio-ap we would just remove that code.
For me it seemed legit. We have a precedence for not really complete stuff
with vfio-ccw. So if it was OK to defer the stuff that was deferred there,
I think, ap=on suddenly working without the strange device should be OK too.

>> Making sure that we have both the feature and the device when using a
>> management software (e.g. libvirt) makes a lot of sense (and is
>> probably also easier to implement), but it won't help us with the issue
>> of the interception handlers, unfortunately.
>>

I disagree. Conceptually hotplug of vfio-ap is perfectly legit. So doing
something like this in management software sounds wrong. The idea here
is cutting corners in order to have something that works reasonably well
but with a couple of well defined limitations sooner.

Regards,
Halil

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Halil Pasic]

On 04/06/2018 02:09 PM, Halil Pasic wrote:
> Yes it is conceptually ugly. I'm 100% with you. That's why it should go
> away soon. From the practicality perspective however I would even argue that it's
> helpful to the user: tells 'oops you have forgotten something'. IMHO
> it's a shortcut of type make the problem smaller. Regarding what is
> harder and what is easier: the author is probably the most fit to decide
> that. If it is harder, it makes no sense, as this is all about cutting
> corners.

I've just realized, I have overlooked something. And that is using
what libvirt calls host-model and host-passthrough mode. There the
user does not explicitly ask for ap=on. So the user would get slapped
in the face by this 'needs vfio-ap device' message (AFAIU) after
upgrading stuff (without even knowing that AP was added) which is
extremely ugly! I need to think about this some more.

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Daniel P. Berrangé]

On Fri, Apr 06, 2018 at 02:32:49PM +0200, Halil Pasic wrote:
> 
> 
> On 04/06/2018 02:09 PM, Halil Pasic wrote:
> > Yes it is conceptually ugly. I'm 100% with you. That's why it should go
> > away soon. From the practicality perspective however I would even argue that it's
> > helpful to the user: tells 'oops you have forgotten something'. IMHO
> > it's a shortcut of type make the problem smaller. Regarding what is
> > harder and what is easier: the author is probably the most fit to decide
> > that. If it is harder, it makes no sense, as this is all about cutting
> > corners.
> 
> I've just realized, I have overlooked something. And that is using
> what libvirt calls host-model and host-passthrough mode. There the
> user does not explicitly ask for ap=on. So the user would get slapped
> in the face by this 'needs vfio-ap device' message (AFAIU) after
> upgrading stuff (without even knowing that AP was added) which is
> extremely ugly! I need to think about this some more.

Typically in this kind of scenario, enablement of the new feature is tied
to QEMU machine type. IOW, existing machine types should not get the new
feature, only the very latest machine type. That way existing guests are
not exposed to it when upgrading QEMU, unless they also change their machine
type.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Pierre Morel]

On 16/03/2018 00:24, Tony Krowiak wrote:
> If the CPU model indicates that AP facility is installed on
> the guest (i.e., -cpu xxxx,ap=on), then the expectation is that
> the AP bus running in the guest will initialize; however, if the
> AP instructions are not being interpreted by the firmware, then
> they will be intercepted and routed back to QEMU for handling.
> If a handler is not defined to process the intercepted instruciton,
> then an operation exception will be injected into the
> guest, in which case the AP bus will not initialize.
>
> There are two situations where AP instructions will not be
> interpreted:
>
> 1. The guest is not configured with a vfio-ap device (i.e.,
>     -device vfio-ap,sysfsdev=$path-to-mdev). The realize function for
>     the vfio-ap device enables interpretive execution of AP
>     instructions.
>
> 2. The guest is a second level guest but the first level guest has
>     not enabled interpretive execution.
>
> This patch introduces AP instruction handlers to ensure the AP bus
> module initializes on the guest when the AP facility is installed
> on the guest but AP instructions are not being interpreted. The logic
> incorporated is:
>
> * If the CPU model indicates AP instructions are
>    installed
>
>    * Set the status response code for the instruction to indicate that
>      the APQN contained in the instruction is not valid. This is
>      a valid response because there will be no devices configured for
>      the guest in any of the above scenarios.
>
> * Else return an error from the handler. This will result in an
>    operation being injected into the guest and the AP bus will not
>    initialize on the guest. That is commensurate with how things work
>    today.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>   hw/vfio/ap.c                 |   45 ++++++++++++++++++++++++++++++++++++++++++
>   include/hw/s390x/ap-device.h |    6 +++++
>   target/s390x/kvm.c           |   14 +++++++++++++
>   3 files changed, 65 insertions(+), 0 deletions(-)
>
> diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c
> index b397bb1..88e744d 100644
> --- a/hw/vfio/ap.c
> +++ b/hw/vfio/ap.c
> @@ -148,6 +148,51 @@ static void vfio_ap_unrealize(DeviceState *dev, Error **errp)
>       kvm_s390_set_interpret_ap(0);
>   }
>
> +int ap_device_handle_nqap(S390CPU *cpu)
> +{
> +    CPUS390XState *env = &cpu->env;
> +
> +    if (s390_has_feat(S390_FEAT_AP)) {
> +        env->regs[1] = 0x10000;
> +
> +        return 0;
> +    }
> +
> +    return -EOPNOTSUPP;
> +}
> +
> +int ap_device_handle_dqap(S390CPU *cpu)
> +{
> +    CPUS390XState *env = &cpu->env;
> +
> +    if (s390_has_feat(S390_FEAT_AP)) {
> +        env->regs[1] = 0x10000;
> +
> +        return 0;
> +    }
> +
> +    return -EOPNOTSUPP;
> +}
> +
> +int ap_device_handle_pqap(S390CPU *cpu)
> +{
> +    CPUS390XState *env = &cpu->env;
> +    int fc = 4 & (env->regs[0] >> 24);
> +
> +    /*
> +     * The Query Configuration Information (QCI) function (fc == 4) does not
> +     * set a response code in reg 1, so check for that along with the
> +     * AP feature.
> +     */

APFT-t must be taken care of, depending on APQN and APFT feature:

If APFT feature is enabled it should report the right information for 
the existing AP
If it is not enabled it should report OPERATION_EXCEPTION

> +    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
> +        env->regs[1] = 0x10000;
> +
> +        return 0;
> +    }
> +
> +    return -EOPNOTSUPP;
> +}
> +
>   static Property vfio_ap_properties[] = {
>       DEFINE_PROP_STRING("sysfsdev", VFIOAPDevice, vdev.sysfsdev),
>       DEFINE_PROP_END_OF_LIST(),
> diff --git a/include/hw/s390x/ap-device.h b/include/hw/s390x/ap-device.h
> index 693df90..d45ae38 100644
> --- a/include/hw/s390x/ap-device.h
> +++ b/include/hw/s390x/ap-device.h
> @@ -11,6 +11,8 @@
>   #ifndef HW_S390X_AP_DEVICE_H
>   #define HW_S390X_AP_DEVICE_H
>
> +#include "cpu.h"
> +
>   #define AP_DEVICE_TYPE       "ap-device"
>
>   typedef struct APDevice {
> @@ -35,4 +37,8 @@ static inline APDevice *to_ap_dev(DeviceState *dev)
>   #define AP_DEVICE_CLASS(klass) \
>       OBJECT_CLASS_CHECK(APDeviceClass, (klass), AP_DEVICE_TYPE)
>
> +int ap_device_handle_nqap(S390CPU *cpu);
> +int ap_device_handle_dqap(S390CPU *cpu);
> +int ap_device_handle_pqap(S390CPU *cpu);
> +
>   #endif /* HW_S390X_AP_DEVICE_H */
> diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c
> index 2812e28..a636394 100644
> --- a/target/s390x/kvm.c
> +++ b/target/s390x/kvm.c
> @@ -50,6 +50,7 @@
>   #include "exec/memattrs.h"
>   #include "hw/s390x/s390-virtio-ccw.h"
>   #include "hw/s390x/s390-virtio-hcall.h"
> +#include "hw/s390x/ap-device.h"
>
>   #ifndef DEBUG_KVM
>   #define DEBUG_KVM  0
> @@ -88,6 +89,9 @@
>   #define PRIV_B2_CHSC                    0x5f
>   #define PRIV_B2_SIGA                    0x74
>   #define PRIV_B2_XSCH                    0x76
> +#define PRIV_B2_NQAP                    0xad
> +#define PRIV_B2_DQAP                    0xae
> +#define PRIV_B2_PQAP                    0xaf
>
>   #define PRIV_EB_SQBS                    0x8a
>   #define PRIV_EB_PCISTB                  0xd0
> @@ -1245,6 +1249,16 @@ static int handle_b2(S390CPU *cpu, struct kvm_run *run, uint8_t ipa1)
>       case PRIV_B2_SCLP_CALL:
>           rc = kvm_sclp_service_call(cpu, run, ipbh0);
>           break;
> +    case PRIV_B2_NQAP:
> +        rc = ap_device_handle_nqap(cpu);
> +        break;
> +    case PRIV_B2_DQAP:
> +        rc = ap_device_handle_dqap(cpu);
> +        break;
> +    case PRIV_B2_PQAP:
> +        rc = ap_device_handle_pqap(cpu);
> +        break;
> +        break;
>       default:
>           rc = -1;
>           DPRINTF("KVM: unhandled PRIV: 0xb2%x\n", ipa1);


-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Pierre Morel]

On 06/04/2018 16:08, Pierre Morel wrote:
> On 16/03/2018 00:24, Tony Krowiak wrote:
>> If the CPU model indicates that AP facility is installed on
>> the guest (i.e., -cpu xxxx,ap=on), then the expectation is that
>> the AP bus running in the guest will initialize; however, if the
>> AP instructions are not being interpreted by the firmware, then
>> they will be intercepted and routed back to QEMU for handling.
>> If a handler is not defined to process the intercepted instruciton,
>> then an operation exception will be injected into the
>> guest, in which case the AP bus will not initialize.
>>
>> There are two situations where AP instructions will not be
>> interpreted:
>>
>> 1. The guest is not configured with a vfio-ap device (i.e.,
>>     -device vfio-ap,sysfsdev=$path-to-mdev). The realize function for
>>     the vfio-ap device enables interpretive execution of AP
>>     instructions.
>>
>> 2. The guest is a second level guest but the first level guest has
>>     not enabled interpretive execution.
>>
>> This patch introduces AP instruction handlers to ensure the AP bus
>> module initializes on the guest when the AP facility is installed
>> on the guest but AP instructions are not being interpreted. The logic
>> incorporated is:
>>
>> * If the CPU model indicates AP instructions are
>>    installed
>>
>>    * Set the status response code for the instruction to indicate that
>>      the APQN contained in the instruction is not valid. This is
>>      a valid response because there will be no devices configured for
>>      the guest in any of the above scenarios.
>>
>> * Else return an error from the handler. This will result in an
>>    operation being injected into the guest and the AP bus will not
>>    initialize on the guest. That is commensurate with how things work
>>    today.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>>   hw/vfio/ap.c                 |   45 
>> ++++++++++++++++++++++++++++++++++++++++++
>>   include/hw/s390x/ap-device.h |    6 +++++
>>   target/s390x/kvm.c           |   14 +++++++++++++
>>   3 files changed, 65 insertions(+), 0 deletions(-)
>>
>> diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c
>> index b397bb1..88e744d 100644
>> --- a/hw/vfio/ap.c
>> +++ b/hw/vfio/ap.c
>> @@ -148,6 +148,51 @@ static void vfio_ap_unrealize(DeviceState *dev, 
>> Error **errp)
>>       kvm_s390_set_interpret_ap(0);
>>   }
>>
>> +int ap_device_handle_nqap(S390CPU *cpu)
>> +{
>> +    CPUS390XState *env = &cpu->env;
>> +
>> +    if (s390_has_feat(S390_FEAT_AP)) {
>> +        env->regs[1] = 0x10000;
>> +
>> +        return 0;
>> +    }
>> +
>> +    return -EOPNOTSUPP;
>> +}
>> +
>> +int ap_device_handle_dqap(S390CPU *cpu)
>> +{
>> +    CPUS390XState *env = &cpu->env;
>> +
>> +    if (s390_has_feat(S390_FEAT_AP)) {
>> +        env->regs[1] = 0x10000;
>> +
>> +        return 0;
>> +    }
>> +
>> +    return -EOPNOTSUPP;
>> +}
>> +
>> +int ap_device_handle_pqap(S390CPU *cpu)
>> +{
>> +    CPUS390XState *env = &cpu->env;
>> +    int fc = 4 & (env->regs[0] >> 24);
>> +
>> +    /*
>> +     * The Query Configuration Information (QCI) function (fc == 4) 
>> does not
>> +     * set a response code in reg 1, so check for that along with the
>> +     * AP feature.
>> +     */
>
> APFT-t must be taken care of, depending on APQN and APFT feature:
>
> If APFT feature is enabled it should report the right information for 
> the existing AP
> If it is not enabled it should report OPERATION_EXCEPTION

hum, sorry, only if interception of TAPQ-t is enabled and it is not.
so forget it.
Sorry for the noise.

>
>> +    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
>> +        env->regs[1] = 0x10000;
>> +
>> +        return 0;
>> +    }
>> +
>> +    return -EOPNOTSUPP;
>> +}
>> +
>>   static Property vfio_ap_properties[] = {
>>       DEFINE_PROP_STRING("sysfsdev", VFIOAPDevice, vdev.sysfsdev),
>>       DEFINE_PROP_END_OF_LIST(),
>> diff --git a/include/hw/s390x/ap-device.h b/include/hw/s390x/ap-device.h
>> index 693df90..d45ae38 100644
>> --- a/include/hw/s390x/ap-device.h
>> +++ b/include/hw/s390x/ap-device.h
>> @@ -11,6 +11,8 @@
>>   #ifndef HW_S390X_AP_DEVICE_H
>>   #define HW_S390X_AP_DEVICE_H
>>
>> +#include "cpu.h"
>> +
>>   #define AP_DEVICE_TYPE       "ap-device"
>>
>>   typedef struct APDevice {
>> @@ -35,4 +37,8 @@ static inline APDevice *to_ap_dev(DeviceState *dev)
>>   #define AP_DEVICE_CLASS(klass) \
>>       OBJECT_CLASS_CHECK(APDeviceClass, (klass), AP_DEVICE_TYPE)
>>
>> +int ap_device_handle_nqap(S390CPU *cpu);
>> +int ap_device_handle_dqap(S390CPU *cpu);
>> +int ap_device_handle_pqap(S390CPU *cpu);
>> +
>>   #endif /* HW_S390X_AP_DEVICE_H */
>> diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c
>> index 2812e28..a636394 100644
>> --- a/target/s390x/kvm.c
>> +++ b/target/s390x/kvm.c
>> @@ -50,6 +50,7 @@
>>   #include "exec/memattrs.h"
>>   #include "hw/s390x/s390-virtio-ccw.h"
>>   #include "hw/s390x/s390-virtio-hcall.h"
>> +#include "hw/s390x/ap-device.h"
>>
>>   #ifndef DEBUG_KVM
>>   #define DEBUG_KVM  0
>> @@ -88,6 +89,9 @@
>>   #define PRIV_B2_CHSC                    0x5f
>>   #define PRIV_B2_SIGA                    0x74
>>   #define PRIV_B2_XSCH                    0x76
>> +#define PRIV_B2_NQAP                    0xad
>> +#define PRIV_B2_DQAP                    0xae
>> +#define PRIV_B2_PQAP                    0xaf
>>
>>   #define PRIV_EB_SQBS                    0x8a
>>   #define PRIV_EB_PCISTB                  0xd0
>> @@ -1245,6 +1249,16 @@ static int handle_b2(S390CPU *cpu, struct 
>> kvm_run *run, uint8_t ipa1)
>>       case PRIV_B2_SCLP_CALL:
>>           rc = kvm_sclp_service_call(cpu, run, ipbh0);
>>           break;
>> +    case PRIV_B2_NQAP:
>> +        rc = ap_device_handle_nqap(cpu);
>> +        break;
>> +    case PRIV_B2_DQAP:
>> +        rc = ap_device_handle_dqap(cpu);
>> +        break;
>> +    case PRIV_B2_PQAP:
>> +        rc = ap_device_handle_pqap(cpu);
>> +        break;
>> +        break;
>>       default:
>>           rc = -1;
>>           DPRINTF("KVM: unhandled PRIV: 0xb2%x\n", ipa1);
>
>

-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [Qemu-devel] [PATCH v3 3/7] s390x/cpumodel: Set up CPU model for AP device support

[Pierre Morel]

On 16/03/2018 00:24, Tony Krowiak wrote:
> A new CPU model feature and two new CPU model facilities are
> introduced to support AP devices for a KVM guest.
>
> CPU model features:
>
> 1. The KVM_S390_VM_CPU_FEAT_AP CPU model feature indicates that
>     AP facilities are installed. This feature will be enabled by
>     the kernel only if the AP facilities are installed on the linux
>     host. This feature must be turned on from userspace to access
>     AP devices from the KVM guest. The QEMU command line to turn
>     this feature looks something like this:
>
> 	qemu-system-s390x ... -cpu xxx,ap=on
>
> CPU model facilities:
>
> 1. The S390_FEAT_AP_QUERY_CONFIG_INFO feature indicates the AP Query
>     Configuration Information (QCI) facility is installed. This feature
>     will be enabled by the kernel only if the QCI is installed on
>     the host.
>
> 2. The S390_FEAT_AP_FACILITY_TEST feature indicates that the AP
>     Facility Test (APFT) facility is installed. This feature will
>     be enabled by the kernel only if the APFT facility is installed
>     on the host.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>   target/s390x/cpu_features.c     |    3 +++
>   target/s390x/cpu_features_def.h |    3 +++
>   target/s390x/cpu_models.c       |    2 ++
>   target/s390x/gen-features.c     |    3 +++
>   target/s390x/kvm.c              |    1 +
>   5 files changed, 12 insertions(+), 0 deletions(-)
>
> diff --git a/target/s390x/cpu_features.c b/target/s390x/cpu_features.c
> index a5619f2..1abe987 100644
> --- a/target/s390x/cpu_features.c
> +++ b/target/s390x/cpu_features.c
> @@ -36,8 +36,10 @@ static const S390FeatDef s390_features[] = {
>       FEAT_INIT("srs", S390_FEAT_TYPE_STFL, 9, "Sense-running-status facility"),
>       FEAT_INIT("csske", S390_FEAT_TYPE_STFL, 10, "Conditional-SSKE facility"),
>       FEAT_INIT("ctop", S390_FEAT_TYPE_STFL, 11, "Configuration-topology facility"),
> +    FEAT_INIT("qci", S390_FEAT_TYPE_STFL, 12, "Query AP Configuration facility"),
>       FEAT_INIT("ipter", S390_FEAT_TYPE_STFL, 13, "IPTE-range facility"),
>       FEAT_INIT("nonqks", S390_FEAT_TYPE_STFL, 14, "Nonquiescing key-setting facility"),
> +    FEAT_INIT("apft", S390_FEAT_TYPE_STFL, 15, "Adjunct Processor Facilities Test facility"),
>       FEAT_INIT("etf2", S390_FEAT_TYPE_STFL, 16, "Extended-translation facility 2"),
>       FEAT_INIT("msa-base", S390_FEAT_TYPE_STFL, 17, "Message-security-assist facility (excluding subfunctions)"),
>       FEAT_INIT("ldisp", S390_FEAT_TYPE_STFL, 18, "Long-displacement facility"),
> @@ -125,6 +127,7 @@ static const S390FeatDef s390_features[] = {
>
>       FEAT_INIT("dateh2", S390_FEAT_TYPE_MISC, 0, "DAT-enhancement facility 2"),
>       FEAT_INIT("cmm", S390_FEAT_TYPE_MISC, 0, "Collaborative-memory-management facility"),
> +    FEAT_INIT("ap", S390_FEAT_TYPE_MISC, 0, "AP facilities installed"),
>
>       FEAT_INIT("plo-cl", S390_FEAT_TYPE_PLO, 0, "PLO Compare and load (32 bit in general registers)"),
>       FEAT_INIT("plo-clg", S390_FEAT_TYPE_PLO, 1, "PLO Compare and load (64 bit in parameter list)"),
> diff --git a/target/s390x/cpu_features_def.h b/target/s390x/cpu_features_def.h
> index 7c5915c..8998b65 100644
> --- a/target/s390x/cpu_features_def.h
> +++ b/target/s390x/cpu_features_def.h
> @@ -27,8 +27,10 @@ typedef enum {
>       S390_FEAT_SENSE_RUNNING_STATUS,
>       S390_FEAT_CONDITIONAL_SSKE,
>       S390_FEAT_CONFIGURATION_TOPOLOGY,
> +    S390_FEAT_AP_QUERY_CONFIG_INFO,
>       S390_FEAT_IPTE_RANGE,
>       S390_FEAT_NONQ_KEY_SETTING,
> +    S390_FEAT_AP_FACILITIES_TEST,
>       S390_FEAT_EXTENDED_TRANSLATION_2,
>       S390_FEAT_MSA,
>       S390_FEAT_LONG_DISPLACEMENT,
> @@ -118,6 +120,7 @@ typedef enum {
>       /* Misc */
>       S390_FEAT_DAT_ENH_2,
>       S390_FEAT_CMM,
> +    S390_FEAT_AP,
>
>       /* PLO */
>       S390_FEAT_PLO_CL,
> diff --git a/target/s390x/cpu_models.c b/target/s390x/cpu_models.c
> index cf82589..7e2af09 100644
> --- a/target/s390x/cpu_models.c
> +++ b/target/s390x/cpu_models.c
> @@ -769,6 +769,8 @@ static void check_consistency(const S390CPUModel *model)
>           { S390_FEAT_PRNO_TRNG_QRTCR, S390_FEAT_MSA_EXT_5 },
>           { S390_FEAT_PRNO_TRNG, S390_FEAT_MSA_EXT_5 },
>           { S390_FEAT_SIE_KSS, S390_FEAT_SIE_F2 },
> +        { S390_FEAT_AP_QUERY_CONFIG_INFO, S390_FEAT_AP },
> +        { S390_FEAT_AP_FACILITIES_TEST, S390_FEAT_AP },
>       };
>       int i;
>
> diff --git a/target/s390x/gen-features.c b/target/s390x/gen-features.c
> index 0cdbc15..0d5b0f7 100644
> --- a/target/s390x/gen-features.c
> +++ b/target/s390x/gen-features.c
> @@ -447,6 +447,9 @@ static uint16_t full_GEN12_GA1[] = {
>       S390_FEAT_ADAPTER_INT_SUPPRESSION,
>       S390_FEAT_EDAT_2,
>       S390_FEAT_SIDE_EFFECT_ACCESS_ESOP2,
> +    S390_FEAT_AP_QUERY_CONFIG_INFO,

Placing QCI feature in the full feature for z13 has for effect that:

1)  it must be on the qemu command line.
Do we really want this?
QCI is always available when AP is available in the real hardware
before z13.

2) AP exist since z10 and QCI since EC12 AFAIK
Do we enable both only beginning with z13?

> +    S390_FEAT_AP_FACILITIES_TEST,
> +    S390_FEAT_AP,
>   };
>
>   static uint16_t full_GEN12_GA2[] = {
> diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c
> index fbc887e..33e5ec3 100644
> --- a/target/s390x/kvm.c
> +++ b/target/s390x/kvm.c
> @@ -2178,6 +2178,7 @@ static int kvm_to_feat[][2] = {
>       { KVM_S390_VM_CPU_FEAT_PFMFI, S390_FEAT_SIE_PFMFI},
>       { KVM_S390_VM_CPU_FEAT_SIGPIF, S390_FEAT_SIE_SIGPIF},
>       { KVM_S390_VM_CPU_FEAT_KSS, S390_FEAT_SIE_KSS},
> +    { KVM_S390_VM_CPU_FEAT_AP, S390_FEAT_AP},
>   };
>
>   static int query_cpu_feat(S390FeatBitmap features)


-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Halil Pasic]

On 04/05/2018 06:38 PM, Tony Krowiak wrote:
> On 04/03/2018 05:36 AM, Cornelia Huck wrote:
>> On Mon, 2 Apr 2018 12:36:27 -0400
>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>
>>> On 03/26/2018 05:03 AM, Pierre Morel wrote:
>>>> On 26/03/2018 10:32, David Hildenbrand wrote:
>>>>> On 16.03.2018 00:24, Tony Krowiak wrote:
>>>>>> +    /*
>>>>>> +     * The Query Configuration Information (QCI) function (fc == 4)
>>>>>> does not
>>>>>> +     * set a response code in reg 1, so check for that along with the
>>>>>> +     * AP feature.
>>>>>> +     */
>>>>>> +    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
>>>>>> +        env->regs[1] = 0x10000;
>>>>>> +
>>>>>> +        return 0;
>>>>>> +    }
>>>>> This would imply an operation exception in case fc==4, which sounds very
>>>>> wrong.
>>>> It depends but I think that the S390_FEAT_AP_QUERY_CONFIG_INFO must be
>>>> tested
>>>> to know what to answer.
>>>> If the feature is there, QCI must be answered correctly.
>>> This is an interesting proposition which raises several issues that will
>>> need to
>>> be addressed. The only time the PQAP(QCI) instruction is intercepted is
>>> when:
>>> * A vfio-ap device is NOT defined for the guest because the vfio_ap
>>> device driver
>>>     will set ECA.28 and the PQAP(QCI) instruction will be interpreted
>>> * STFLE.12 is set for the guest
>>>
>>> You say that the QCI must be answered correctly, but what is the correct
>>> response?
>>> If we return the matrix - i.e., APM, ADM and AQM - configured via the
>>> mediated
>>> matrix device's sysfs attributes files, then if any APQNs are defined in
>>> the matrix,
>>> we will have to handle *ALL* AP instructions by executing them on behalf
>>> of the
>>> guest. I suppose we could return an empty matrix in which case the AP
>>> bus will come
>>> up without any devices on the guest, but what is the expectation of an
>>> admin who
>>> deliberately configures the mediated matrix device? Should we forego
>>> handling interception
>>> of AP instructions and consider a guest configuration that turns on
>>> S390_FEAT_AP but
>>> does not define a vfio-ap device to be erroneous and terminate starting
>>> of the guest?
>>> Anybody have any thoughts?
>> Hard to really give good advice without access to the documentation, but:
>> - If we tell the guest that the feature is available, but it does not
>>    get any cards to use, returning an empty matrix makes the most sense
>>    to me.
>> - I would not tie starting the guest to the presence of a vfio-ap
>>    device. Having the feature available in theory but without any
>>    devices actually being usable by the guest does not really sound
>>    wrong (can we hotplug this later?)
> For this phase of development, it is my opinion that introducing AP instruction
> interception handlers is superfluous for the following reasons:
> 
> 1. Interception handling was introduced solely to ensure an operation exception would
>    not be injected into the guest when CPU model feature for AP (i.e., ap=on)
>    is specified but a VFIO AP device (i.e., -device vfio-ap,sysfsdev=$path)
>    is not.

We can kind of (i.e. modulo EECA.28) ensure this in a different fashion I think. How
about proclaiming a 'has ap instructions, but nothing to see here' in the
SIE interpreted flavor (ECA.28 set) the default way of having ap instructions
under KVM. This should be easily accomplished with an all zero CRYCB and eca.28
set. The for the guest to actually get real work done with AP we would
still require some sort of driver to either provide a non-zero matrix by
altering the CRYCB or unsettling ECA.28 and doing the intercepted flavor.

Please notice, the cpu facility ap would still keep it's semantic
'has ap instructions' (opposed to 'has ap instructions implemented in
SIE interpreted flavor). And give us all the flexibility.

Yet implementing what we want to have in absence of a driver would become
much easier (under the assumption that ECA.28 equals EECA.28).

How about this?

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Cornelia Huck]

On Fri, 6 Apr 2018 18:07:56 +0200
Halil Pasic <pasic@linux.vnet.ibm.com> wrote:

> On 04/05/2018 06:38 PM, Tony Krowiak wrote:
> > On 04/03/2018 05:36 AM, Cornelia Huck wrote:  
> >> On Mon, 2 Apr 2018 12:36:27 -0400
> >> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> >>  
> >>> On 03/26/2018 05:03 AM, Pierre Morel wrote:  
> >>>> On 26/03/2018 10:32, David Hildenbrand wrote:  
> >>>>> On 16.03.2018 00:24, Tony Krowiak wrote:  
> >>>>>> +    /*
> >>>>>> +     * The Query Configuration Information (QCI) function (fc == 4)
> >>>>>> does not
> >>>>>> +     * set a response code in reg 1, so check for that along with the
> >>>>>> +     * AP feature.
> >>>>>> +     */
> >>>>>> +    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
> >>>>>> +        env->regs[1] = 0x10000;
> >>>>>> +
> >>>>>> +        return 0;
> >>>>>> +    }  
> >>>>> This would imply an operation exception in case fc==4, which sounds very
> >>>>> wrong.  
> >>>> It depends but I think that the S390_FEAT_AP_QUERY_CONFIG_INFO must be
> >>>> tested
> >>>> to know what to answer.
> >>>> If the feature is there, QCI must be answered correctly.  
> >>> This is an interesting proposition which raises several issues that will
> >>> need to
> >>> be addressed. The only time the PQAP(QCI) instruction is intercepted is
> >>> when:
> >>> * A vfio-ap device is NOT defined for the guest because the vfio_ap
> >>> device driver
> >>>     will set ECA.28 and the PQAP(QCI) instruction will be interpreted
> >>> * STFLE.12 is set for the guest
> >>>
> >>> You say that the QCI must be answered correctly, but what is the correct
> >>> response?
> >>> If we return the matrix - i.e., APM, ADM and AQM - configured via the
> >>> mediated
> >>> matrix device's sysfs attributes files, then if any APQNs are defined in
> >>> the matrix,
> >>> we will have to handle *ALL* AP instructions by executing them on behalf
> >>> of the
> >>> guest. I suppose we could return an empty matrix in which case the AP
> >>> bus will come
> >>> up without any devices on the guest, but what is the expectation of an
> >>> admin who
> >>> deliberately configures the mediated matrix device? Should we forego
> >>> handling interception
> >>> of AP instructions and consider a guest configuration that turns on
> >>> S390_FEAT_AP but
> >>> does not define a vfio-ap device to be erroneous and terminate starting
> >>> of the guest?
> >>> Anybody have any thoughts?  
> >> Hard to really give good advice without access to the documentation, but:
> >> - If we tell the guest that the feature is available, but it does not
> >>    get any cards to use, returning an empty matrix makes the most sense
> >>    to me.
> >> - I would not tie starting the guest to the presence of a vfio-ap
> >>    device. Having the feature available in theory but without any
> >>    devices actually being usable by the guest does not really sound
> >>    wrong (can we hotplug this later?)  
> > For this phase of development, it is my opinion that introducing AP instruction
> > interception handlers is superfluous for the following reasons:
> > 
> > 1. Interception handling was introduced solely to ensure an operation exception would
> >    not be injected into the guest when CPU model feature for AP (i.e., ap=on)
> >    is specified but a VFIO AP device (i.e., -device vfio-ap,sysfsdev=$path)
> >    is not.  
> 
> We can kind of (i.e. modulo EECA.28) ensure this in a different fashion I think. How
> about proclaiming a 'has ap instructions, but nothing to see here' in the
> SIE interpreted flavor (ECA.28 set) the default way of having ap instructions
> under KVM. This should be easily accomplished with an all zero CRYCB and eca.28
> set. The for the guest to actually get real work done with AP we would
> still require some sort of driver to either provide a non-zero matrix by
> altering the CRYCB or unsettling ECA.28 and doing the intercepted flavor.
> 
> Please notice, the cpu facility ap would still keep it's semantic
> 'has ap instructions' (opposed to 'has ap instructions implemented in
> SIE interpreted flavor). And give us all the flexibility.
> 
> Yet implementing what we want to have in absence of a driver would become
> much easier (under the assumption that ECA.28 equals EECA.28).
> 
> How about this?

Unfortunately, this is really hard to follow without the AR... let me
summarize it to check whether I got the gist of it :)

- If the "ap" cpu feature is specified, set a bit that indicates "hey,
  we basically have have AP support" and create the basics, but don't
  enable actual SIE handling. This means the guest gets exceptions from
  the SIE already and we don't need to emulate them.
- Actually enable the missing pieces if a vfio device is created. This
  would enable processing by the SIE, and we would not need to do
  emulation, either (for most of it, IIRC).

I may be all wrong, though... can we at least have a translation of
ECA.28 and EECA.28 (the "ap is there" bit and the "ap instructions are
interpreted" bit?)

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Halil Pasic]

On 04/09/2018 11:32 AM, Cornelia Huck wrote:
>> We can kind of (i.e. modulo EECA.28) ensure this in a different fashion I think. How
>> about proclaiming a 'has ap instructions, but nothing to see here' in the
>> SIE interpreted flavor (ECA.28 set) the default way of having ap instructions
>> under KVM. This should be easily accomplished with an all zero CRYCB and eca.28
>> set. The for the guest to actually get real work done with AP we would
>> still require some sort of driver to either provide a non-zero matrix by
>> altering the CRYCB or unsettling ECA.28 and doing the intercepted flavor.
>>
>> Please notice, the cpu facility ap would still keep it's semantic
>> 'has ap instructions' (opposed to 'has ap instructions implemented in
>> SIE interpreted flavor). And give us all the flexibility.
>>
>> Yet implementing what we want to have in absence of a driver would become
>> much easier (under the assumption that ECA.28 equals EECA.28).
>>
>> How about this?
> Unfortunately, this is really hard to follow without the AR... let me
> summarize it to check whether I got the gist of it :)
> 
> - If the "ap" cpu feature is specified, set a bit that indicates "hey,
>   we basically have have AP support" and create the basics, but don't
>   enable actual SIE handling. This means the guest gets exceptions from
>   the SIE already and we don't need to emulate them.

Kind of. The bit is ECA.28 and tells SIE 'hey SIE shall interpret ap
instructions for the guest (as specified)'. Then SIE has an SD satellite
called CRYCB that contains the which ap resources is the guest authorized
to use. These are the masks. If we set each mask to all zero, we will
effectively accomplish 'hey,we basically have have AP support but no
resources at the moment'. So, right, we don't have to emulate that.

I don't know what do you mean by exceptions. For most legit requests the
SIE should say APQN invalid, with QCI being a notable exception. But
of course SIE would inject program exceptions (access, specification,
and privileged operation) accordingly I guess.


In short, the SIE would do what we are trying to emulate in this patch.

> - Actually enable the missing pieces if a vfio device is created. This
>   would enable processing by the SIE, and we would not need to do
>   emulation, either (for most of it, IIRC).

Yes. It would actually assign resources to the guest. That would enable
doing real work with the AP instructions.

> 
> I may be all wrong, though... can we at least have a translation of
> ECA.28 and EECA.28 (the "ap is there" bit and the "ap instructions are
> interpreted" bit?)
> 

I think we have a misunderstanding here. I will wait for Tony. Maybe
he can understand this better or explain in more accessible way.

Regards,
Halil

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Cornelia Huck]

On Mon, 9 Apr 2018 12:37:42 +0200
Halil Pasic <pasic@linux.vnet.ibm.com> wrote:

> On 04/09/2018 11:32 AM, Cornelia Huck wrote:
> >> We can kind of (i.e. modulo EECA.28) ensure this in a different fashion I think. How
> >> about proclaiming a 'has ap instructions, but nothing to see here' in the
> >> SIE interpreted flavor (ECA.28 set) the default way of having ap instructions
> >> under KVM. This should be easily accomplished with an all zero CRYCB and eca.28
> >> set. The for the guest to actually get real work done with AP we would
> >> still require some sort of driver to either provide a non-zero matrix by
> >> altering the CRYCB or unsettling ECA.28 and doing the intercepted flavor.
> >>
> >> Please notice, the cpu facility ap would still keep it's semantic
> >> 'has ap instructions' (opposed to 'has ap instructions implemented in
> >> SIE interpreted flavor). And give us all the flexibility.
> >>
> >> Yet implementing what we want to have in absence of a driver would become
> >> much easier (under the assumption that ECA.28 equals EECA.28).
> >>
> >> How about this?  
> > Unfortunately, this is really hard to follow without the AR... let me
> > summarize it to check whether I got the gist of it :)
> > 
> > - If the "ap" cpu feature is specified, set a bit that indicates "hey,
> >   we basically have have AP support" and create the basics, but don't
> >   enable actual SIE handling. This means the guest gets exceptions from
> >   the SIE already and we don't need to emulate them.  
> 
> Kind of. The bit is ECA.28 and tells SIE 'hey SIE shall interpret ap
> instructions for the guest (as specified)'. Then SIE has an SD satellite
> called CRYCB that contains the which ap resources is the guest authorized
> to use. These are the masks. If we set each mask to all zero, we will
> effectively accomplish 'hey,we basically have have AP support but no
> resources at the moment'. So, right, we don't have to emulate that.
> 
> I don't know what do you mean by exceptions. For most legit requests the
> SIE should say APQN invalid, with QCI being a notable exception. But
> of course SIE would inject program exceptions (access, specification,
> and privileged operation) accordingly I guess.

I meant "emulate exceptions"...

> 
> 
> In short, the SIE would do what we are trying to emulate in this patch.

...so yes, exactly that.

> 
> > - Actually enable the missing pieces if a vfio device is created. This
> >   would enable processing by the SIE, and we would not need to do
> >   emulation, either (for most of it, IIRC).  
> 
> Yes. It would actually assign resources to the guest. That would enable
> doing real work with the AP instructions.

Ok.

> 
> > 
> > I may be all wrong, though... can we at least have a translation of
> > ECA.28 and EECA.28 (the "ap is there" bit and the "ap instructions are
> > interpreted" bit?)
> >   
> 
> I think we have a misunderstanding here. I will wait for Tony. Maybe
> he can understand this better or explain in more accessible way.

From what I get from your explanation, this approach sounds like a good
way forward. But let's wait for Tony.

Re: [Qemu-devel] [PATCH v3 3/7] s390x/cpumodel: Set up CPU model for AP device support

[Tony Krowiak]

On 04/06/2018 10:51 AM, Pierre Morel wrote:
> On 16/03/2018 00:24, Tony Krowiak wrote:
>> A new CPU model feature and two new CPU model facilities are
>> introduced to support AP devices for a KVM guest.
>>
>> CPU model features:
>>
>> 1. The KVM_S390_VM_CPU_FEAT_AP CPU model feature indicates that
>>     AP facilities are installed. This feature will be enabled by
>>     the kernel only if the AP facilities are installed on the linux
>>     host. This feature must be turned on from userspace to access
>>     AP devices from the KVM guest. The QEMU command line to turn
>>     this feature looks something like this:
>>
>>     qemu-system-s390x ... -cpu xxx,ap=on
>>
>> CPU model facilities:
>>
>> 1. The S390_FEAT_AP_QUERY_CONFIG_INFO feature indicates the AP Query
>>     Configuration Information (QCI) facility is installed. This feature
>>     will be enabled by the kernel only if the QCI is installed on
>>     the host.
>>
>> 2. The S390_FEAT_AP_FACILITY_TEST feature indicates that the AP
>>     Facility Test (APFT) facility is installed. This feature will
>>     be enabled by the kernel only if the APFT facility is installed
>>     on the host.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>>   target/s390x/cpu_features.c     |    3 +++
>>   target/s390x/cpu_features_def.h |    3 +++
>>   target/s390x/cpu_models.c       |    2 ++
>>   target/s390x/gen-features.c     |    3 +++
>>   target/s390x/kvm.c              |    1 +
>>   5 files changed, 12 insertions(+), 0 deletions(-)
>>
>> diff --git a/target/s390x/cpu_features.c b/target/s390x/cpu_features.c
>> index a5619f2..1abe987 100644
>> --- a/target/s390x/cpu_features.c
>> +++ b/target/s390x/cpu_features.c
>> @@ -36,8 +36,10 @@ static const S390FeatDef s390_features[] = {
>>       FEAT_INIT("srs", S390_FEAT_TYPE_STFL, 9, "Sense-running-status 
>> facility"),
>>       FEAT_INIT("csske", S390_FEAT_TYPE_STFL, 10, "Conditional-SSKE 
>> facility"),
>>       FEAT_INIT("ctop", S390_FEAT_TYPE_STFL, 11, 
>> "Configuration-topology facility"),
>> +    FEAT_INIT("qci", S390_FEAT_TYPE_STFL, 12, "Query AP 
>> Configuration facility"),
>>       FEAT_INIT("ipter", S390_FEAT_TYPE_STFL, 13, "IPTE-range 
>> facility"),
>>       FEAT_INIT("nonqks", S390_FEAT_TYPE_STFL, 14, "Nonquiescing 
>> key-setting facility"),
>> +    FEAT_INIT("apft", S390_FEAT_TYPE_STFL, 15, "Adjunct Processor 
>> Facilities Test facility"),
>>       FEAT_INIT("etf2", S390_FEAT_TYPE_STFL, 16, 
>> "Extended-translation facility 2"),
>>       FEAT_INIT("msa-base", S390_FEAT_TYPE_STFL, 17, 
>> "Message-security-assist facility (excluding subfunctions)"),
>>       FEAT_INIT("ldisp", S390_FEAT_TYPE_STFL, 18, "Long-displacement 
>> facility"),
>> @@ -125,6 +127,7 @@ static const S390FeatDef s390_features[] = {
>>
>>       FEAT_INIT("dateh2", S390_FEAT_TYPE_MISC, 0, "DAT-enhancement 
>> facility 2"),
>>       FEAT_INIT("cmm", S390_FEAT_TYPE_MISC, 0, 
>> "Collaborative-memory-management facility"),
>> +    FEAT_INIT("ap", S390_FEAT_TYPE_MISC, 0, "AP facilities installed"),
>>
>>       FEAT_INIT("plo-cl", S390_FEAT_TYPE_PLO, 0, "PLO Compare and 
>> load (32 bit in general registers)"),
>>       FEAT_INIT("plo-clg", S390_FEAT_TYPE_PLO, 1, "PLO Compare and 
>> load (64 bit in parameter list)"),
>> diff --git a/target/s390x/cpu_features_def.h 
>> b/target/s390x/cpu_features_def.h
>> index 7c5915c..8998b65 100644
>> --- a/target/s390x/cpu_features_def.h
>> +++ b/target/s390x/cpu_features_def.h
>> @@ -27,8 +27,10 @@ typedef enum {
>>       S390_FEAT_SENSE_RUNNING_STATUS,
>>       S390_FEAT_CONDITIONAL_SSKE,
>>       S390_FEAT_CONFIGURATION_TOPOLOGY,
>> +    S390_FEAT_AP_QUERY_CONFIG_INFO,
>>       S390_FEAT_IPTE_RANGE,
>>       S390_FEAT_NONQ_KEY_SETTING,
>> +    S390_FEAT_AP_FACILITIES_TEST,
>>       S390_FEAT_EXTENDED_TRANSLATION_2,
>>       S390_FEAT_MSA,
>>       S390_FEAT_LONG_DISPLACEMENT,
>> @@ -118,6 +120,7 @@ typedef enum {
>>       /* Misc */
>>       S390_FEAT_DAT_ENH_2,
>>       S390_FEAT_CMM,
>> +    S390_FEAT_AP,
>>
>>       /* PLO */
>>       S390_FEAT_PLO_CL,
>> diff --git a/target/s390x/cpu_models.c b/target/s390x/cpu_models.c
>> index cf82589..7e2af09 100644
>> --- a/target/s390x/cpu_models.c
>> +++ b/target/s390x/cpu_models.c
>> @@ -769,6 +769,8 @@ static void check_consistency(const S390CPUModel 
>> *model)
>>           { S390_FEAT_PRNO_TRNG_QRTCR, S390_FEAT_MSA_EXT_5 },
>>           { S390_FEAT_PRNO_TRNG, S390_FEAT_MSA_EXT_5 },
>>           { S390_FEAT_SIE_KSS, S390_FEAT_SIE_F2 },
>> +        { S390_FEAT_AP_QUERY_CONFIG_INFO, S390_FEAT_AP },
>> +        { S390_FEAT_AP_FACILITIES_TEST, S390_FEAT_AP },
>>       };
>>       int i;
>>
>> diff --git a/target/s390x/gen-features.c b/target/s390x/gen-features.c
>> index 0cdbc15..0d5b0f7 100644
>> --- a/target/s390x/gen-features.c
>> +++ b/target/s390x/gen-features.c
>> @@ -447,6 +447,9 @@ static uint16_t full_GEN12_GA1[] = {
>>       S390_FEAT_ADAPTER_INT_SUPPRESSION,
>>       S390_FEAT_EDAT_2,
>>       S390_FEAT_SIDE_EFFECT_ACCESS_ESOP2,
>> +    S390_FEAT_AP_QUERY_CONFIG_INFO,
>
> Placing QCI feature in the full feature for z13 has for effect that:
>
> 1)  it must be on the qemu command line.
> Do we really want this?
> QCI is always available when AP is available in the real hardware
> before z13.

Are you suggesting we always set STFLE.12 for the guest whenever
AP is available and get rid of this feature? This feature maps
to what the architecture requires, i.e., that STFLE.12 indicates
whether QCI is available. I think we should keep it. This was
discussed and decided upon during v1 review.

>
>
> 2) AP exist since z10 and QCI since EC12 AFAIK

This seems to conflict with what you stated above concerning QCI
availability.

> Do we enable both only beginning with z13?

If I am not mistaken, GEN12_GA1 is EC12 .... or is that not true?
I queried the team architects and there was unanimous agreement
that AP virtualization should start with EC12 regardless of when
it was available in the real hardware.

>
>> +    S390_FEAT_AP_FACILITIES_TEST,
>> +    S390_FEAT_AP,
>>   };
>>
>>   static uint16_t full_GEN12_GA2[] = {
>> diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c
>> index fbc887e..33e5ec3 100644
>> --- a/target/s390x/kvm.c
>> +++ b/target/s390x/kvm.c
>> @@ -2178,6 +2178,7 @@ static int kvm_to_feat[][2] = {
>>       { KVM_S390_VM_CPU_FEAT_PFMFI, S390_FEAT_SIE_PFMFI},
>>       { KVM_S390_VM_CPU_FEAT_SIGPIF, S390_FEAT_SIE_SIGPIF},
>>       { KVM_S390_VM_CPU_FEAT_KSS, S390_FEAT_SIE_KSS},
>> +    { KVM_S390_VM_CPU_FEAT_AP, S390_FEAT_AP},
>>   };
>>
>>   static int query_cpu_feat(S390FeatBitmap features)
>
>

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Tony Krowiak]

On 04/09/2018 06:51 AM, Cornelia Huck wrote:
> On Mon, 9 Apr 2018 12:37:42 +0200
> Halil Pasic <pasic@linux.vnet.ibm.com> wrote:
>
>> On 04/09/2018 11:32 AM, Cornelia Huck wrote:
>>>> We can kind of (i.e. modulo EECA.28) ensure this in a different fashion I think. How
>>>> about proclaiming a 'has ap instructions, but nothing to see here' in the
>>>> SIE interpreted flavor (ECA.28 set) the default way of having ap instructions
>>>> under KVM. This should be easily accomplished with an all zero CRYCB and eca.28
>>>> set. The for the guest to actually get real work done with AP we would
>>>> still require some sort of driver to either provide a non-zero matrix by
>>>> altering the CRYCB or unsettling ECA.28 and doing the intercepted flavor.
>>>>
>>>> Please notice, the cpu facility ap would still keep it's semantic
>>>> 'has ap instructions' (opposed to 'has ap instructions implemented in
>>>> SIE interpreted flavor). And give us all the flexibility.
>>>>
>>>> Yet implementing what we want to have in absence of a driver would become
>>>> much easier (under the assumption that ECA.28 equals EECA.28).
>>>>
>>>> How about this?
>>> Unfortunately, this is really hard to follow without the AR... let me
>>> summarize it to check whether I got the gist of it :)
>>>
>>> - If the "ap" cpu feature is specified, set a bit that indicates "hey,
>>>    we basically have have AP support" and create the basics, but don't
>>>    enable actual SIE handling. This means the guest gets exceptions from
>>>    the SIE already and we don't need to emulate them.
>> Kind of. The bit is ECA.28 and tells SIE 'hey SIE shall interpret ap
>> instructions for the guest (as specified)'. Then SIE has an SD satellite
>> called CRYCB that contains the which ap resources is the guest authorized
>> to use. These are the masks. If we set each mask to all zero, we will
>> effectively accomplish 'hey,we basically have have AP support but no
>> resources at the moment'. So, right, we don't have to emulate that.
>>
>> I don't know what do you mean by exceptions. For most legit requests the
>> SIE should say APQN invalid, with QCI being a notable exception. But
>> of course SIE would inject program exceptions (access, specification,
>> and privileged operation) accordingly I guess.
> I meant "emulate exceptions"...
>
>>
>> In short, the SIE would do what we are trying to emulate in this patch.
> ...so yes, exactly that.
>
>>> - Actually enable the missing pieces if a vfio device is created. This
>>>    would enable processing by the SIE, and we would not need to do
>>>    emulation, either (for most of it, IIRC).
>> Yes. It would actually assign resources to the guest. That would enable
>> doing real work with the AP instructions.
> Ok.
>
>>> I may be all wrong, though... can we at least have a translation of
>>> ECA.28 and EECA.28 (the "ap is there" bit and the "ap instructions are
>>> interpreted" bit?)
>>>    
>> I think we have a misunderstanding here. I will wait for Tony. Maybe
>> he can understand this better or explain in more accessible way.
>  From what I get from your explanation, this approach sounds like a good
> way forward. But let's wait for Tony.

I agree, this solves the problem with specifying installing AP facilities
on the guest (-cpu xxx,ap=on) and not configuring a vfio-ap device
(-device vfio-ap,sysfsdev=$path-to-mediated-device).

To recap:
The realize() function for the vfio-ap device opens the mediated matrix
device file descriptor to set up the communication pathway with the vfio_ap
kernel driver. When the fd is opened, the vfio_ap driver sets ECA.28 for 
the
guest to instruct SIE to interpret AP instructions. The driver also
configures the AP matrix for the guest in its SIE state description. 
Consequently,
if a vfio-ap device is not configured for the guest, but AP facilities are
installed, all AP instructions will be intercepted and routed to QEMU. 
If there
are no interception handlers for the AP instructions, QEMU injects an 
operation
exception into the guest. This results in initialization of the AP bus 
on the
guest to terminate. This patch was intended to circumvent that problem.

With Halil's suggestion, there is no need to provide these handlers. If 
ECA.28
is set for the guest by default when the AP facilities are installed, 
then the AP
instructions will be interpreted and the AP bus will get initialized on the
guest. Since there is no vfio-ap device to provide the AP matrix 
configuration
for the guest, the AP bus will not detect any devices, but that's okay. AP
instructions targeting at an APQN will execute successfully and set a 
response
code in the status block returned from the instruction indicating the
APQN is invalid .... but there will be no exception unless there is truly
an exception condition caused by the execution of the instruction.

>

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Halil Pasic]

On 04/11/2018 03:20 PM, Tony Krowiak wrote:
>>>> I may be all wrong, though... can we at least have a translation of
>>>> ECA.28 and EECA.28 (the "ap is there" bit and the "ap instructions are
>>>> interpreted" bit?)
>>>>    
>>> I think we have a misunderstanding here. I will wait for Tony. Maybe
>>> he can understand this better or explain in more accessible way.
>>  From what I get from your explanation, this approach sounds like a good
>> way forward. But let's wait for Tony.
> 
> I agree, this solves the problem with specifying installing AP facilities
> on the guest (-cpu xxx,ap=on) and not configuring a vfio-ap device
> (-device vfio-ap,sysfsdev=$path-to-mediated-device).
> 
> To recap:
> The realize() function for the vfio-ap device opens the mediated matrix
> device file descriptor to set up the communication pathway with the vfio_ap
> kernel driver. When the fd is opened, the vfio_ap driver sets ECA.28 for the
> guest to instruct SIE to interpret AP instructions. The driver also
> configures the AP matrix for the guest in its SIE state description. Consequently,
> if a vfio-ap device is not configured for the guest, but AP facilities are
> installed, all AP instructions will be intercepted and routed to QEMU. If there
> are no interception handlers for the AP instructions, QEMU injects an operation
> exception into the guest. This results in initialization of the AP bus on the
> guest to terminate. This patch was intended to circumvent that problem.
> 
> With Halil's suggestion, there is no need to provide these handlers. If ECA.28
> is set for the guest by default when the AP facilities are installed, then the AP
> instructions will be interpreted and the AP bus will get initialized on the
> guest. Since there is no vfio-ap device to provide the AP matrix configuration
> for the guest, the AP bus will not detect any devices, but that's okay. AP
> instructions targeting at an APQN will execute successfully and set a response
> code in the status block returned from the instruction indicating the
> APQN is invalid .... but there will be no exception unless there is truly
> an exception condition caused by the execution of the instruction.

That's exactly the idea, but it will only work out this way if the effective
ECA.28 bit (i.e. EECA.28) is also set.

Could you please comment the first paragraph quoted in this email?

Halil

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Tony Krowiak]

On 04/09/2018 05:32 AM, Cornelia Huck wrote:
> On Fri, 6 Apr 2018 18:07:56 +0200
> Halil Pasic <pasic@linux.vnet.ibm.com> wrote:
>
>> On 04/05/2018 06:38 PM, Tony Krowiak wrote:
>>> On 04/03/2018 05:36 AM, Cornelia Huck wrote:
>>>> On Mon, 2 Apr 2018 12:36:27 -0400
>>>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>>>   
>>>>> On 03/26/2018 05:03 AM, Pierre Morel wrote:
>>>>>> On 26/03/2018 10:32, David Hildenbrand wrote:
>>>>>>> On 16.03.2018 00:24, Tony Krowiak wrote:
>>>>>>>> +    /*
>>>>>>>> +     * The Query Configuration Information (QCI) function (fc == 4)
>>>>>>>> does not
>>>>>>>> +     * set a response code in reg 1, so check for that along with the
>>>>>>>> +     * AP feature.
>>>>>>>> +     */
>>>>>>>> +    if ((fc != 4) && s390_has_feat(S390_FEAT_AP)) {
>>>>>>>> +        env->regs[1] = 0x10000;
>>>>>>>> +
>>>>>>>> +        return 0;
>>>>>>>> +    }
>>>>>>> This would imply an operation exception in case fc==4, which sounds very
>>>>>>> wrong.
>>>>>> It depends but I think that the S390_FEAT_AP_QUERY_CONFIG_INFO must be
>>>>>> tested
>>>>>> to know what to answer.
>>>>>> If the feature is there, QCI must be answered correctly.
>>>>> This is an interesting proposition which raises several issues that will
>>>>> need to
>>>>> be addressed. The only time the PQAP(QCI) instruction is intercepted is
>>>>> when:
>>>>> * A vfio-ap device is NOT defined for the guest because the vfio_ap
>>>>> device driver
>>>>>      will set ECA.28 and the PQAP(QCI) instruction will be interpreted
>>>>> * STFLE.12 is set for the guest
>>>>>
>>>>> You say that the QCI must be answered correctly, but what is the correct
>>>>> response?
>>>>> If we return the matrix - i.e., APM, ADM and AQM - configured via the
>>>>> mediated
>>>>> matrix device's sysfs attributes files, then if any APQNs are defined in
>>>>> the matrix,
>>>>> we will have to handle *ALL* AP instructions by executing them on behalf
>>>>> of the
>>>>> guest. I suppose we could return an empty matrix in which case the AP
>>>>> bus will come
>>>>> up without any devices on the guest, but what is the expectation of an
>>>>> admin who
>>>>> deliberately configures the mediated matrix device? Should we forego
>>>>> handling interception
>>>>> of AP instructions and consider a guest configuration that turns on
>>>>> S390_FEAT_AP but
>>>>> does not define a vfio-ap device to be erroneous and terminate starting
>>>>> of the guest?
>>>>> Anybody have any thoughts?
>>>> Hard to really give good advice without access to the documentation, but:
>>>> - If we tell the guest that the feature is available, but it does not
>>>>     get any cards to use, returning an empty matrix makes the most sense
>>>>     to me.
>>>> - I would not tie starting the guest to the presence of a vfio-ap
>>>>     device. Having the feature available in theory but without any
>>>>     devices actually being usable by the guest does not really sound
>>>>     wrong (can we hotplug this later?)
>>> For this phase of development, it is my opinion that introducing AP instruction
>>> interception handlers is superfluous for the following reasons:
>>>
>>> 1. Interception handling was introduced solely to ensure an operation exception would
>>>     not be injected into the guest when CPU model feature for AP (i.e., ap=on)
>>>     is specified but a VFIO AP device (i.e., -device vfio-ap,sysfsdev=$path)
>>>     is not.
>> We can kind of (i.e. modulo EECA.28) ensure this in a different fashion I think. How
>> about proclaiming a 'has ap instructions, but nothing to see here' in the
>> SIE interpreted flavor (ECA.28 set) the default way of having ap instructions
>> under KVM. This should be easily accomplished with an all zero CRYCB and eca.28
>> set. The for the guest to actually get real work done with AP we would
>> still require some sort of driver to either provide a non-zero matrix by
>> altering the CRYCB or unsettling ECA.28 and doing the intercepted flavor.
>>
>> Please notice, the cpu facility ap would still keep it's semantic
>> 'has ap instructions' (opposed to 'has ap instructions implemented in
>> SIE interpreted flavor). And give us all the flexibility.
>>
>> Yet implementing what we want to have in absence of a driver would become
>> much easier (under the assumption that ECA.28 equals EECA.28).
>>
>> How about this?
> Unfortunately, this is really hard to follow without the AR... let me
> summarize it to check whether I got the gist of it :)
>
> - If the "ap" cpu feature is specified, set a bit that indicates "hey,
>    we basically have have AP support" and create the basics, but don't
>    enable actual SIE handling. This means the guest gets exceptions from
>    the SIE already and we don't need to emulate them.
> - Actually enable the missing pieces if a vfio device is created. This
>    would enable processing by the SIE, and we would not need to do
>    emulation, either (for most of it, IIRC).
>
> I may be all wrong, though... can we at least have a translation of
> ECA.28 and EECA.28 (the "ap is there" bit and the "ap instructions are
> interpreted" bit?)
I am not sure what you are asking here, but I'll attempt to answer the
question I think you are asking.

The ap=on|off flag indicates that AP instructions are installed on the 
guest.
This feature is enabled by the kernel only if AP instructions are installed
on the host. Since there is no facilities bit to query, this is determined
by attempting to execute an AP instruction using an exception table. If 
there
is an exception, it is assumed that the AP instructions are not installed.

The ECA.28 bit in the SIE state description indicates whether AP 
instructions
are interpreted. For level 1 guests, the ECA.28 bit specified in the SIE
state description is used directly. For guest level 2 guests, the value is
calculated by doing a logical AND of the guest level 1 ECA.28 bit and the
guest level 2 ECA.28 bit. This value is known by the term Effective
Execution Control A bit 28, or EECA.28. To the best of my knowledge, - 
as well
as verified empirically, ECA.28 for the linux host (i.e., guest level 1) is
set by default, so EECA.28 will effectively be whatever value is 
specified by
ECA.28 in the level 2 guest's SIE state description. This will not be the
case for guest level 3 when we implement VSIE support.
>

Re: [Qemu-devel] [PATCH v3 6/7] s390x/kvm: handle AP instruction interception

[Tony Krowiak]

On 04/11/2018 09:50 AM, Halil Pasic wrote:
>
> On 04/11/2018 03:20 PM, Tony Krowiak wrote:
>>>>> I may be all wrong, though... can we at least have a translation of
>>>>> ECA.28 and EECA.28 (the "ap is there" bit and the "ap instructions are
>>>>> interpreted" bit?)
>>>>>     
>>>> I think we have a misunderstanding here. I will wait for Tony. Maybe
>>>> he can understand this better or explain in more accessible way.
>>>   From what I get from your explanation, this approach sounds like a good
>>> way forward. But let's wait for Tony.
>> I agree, this solves the problem with specifying installing AP facilities
>> on the guest (-cpu xxx,ap=on) and not configuring a vfio-ap device
>> (-device vfio-ap,sysfsdev=$path-to-mediated-device).
>>
>> To recap:
>> The realize() function for the vfio-ap device opens the mediated matrix
>> device file descriptor to set up the communication pathway with the vfio_ap
>> kernel driver. When the fd is opened, the vfio_ap driver sets ECA.28 for the
>> guest to instruct SIE to interpret AP instructions. The driver also
>> configures the AP matrix for the guest in its SIE state description. Consequently,
>> if a vfio-ap device is not configured for the guest, but AP facilities are
>> installed, all AP instructions will be intercepted and routed to QEMU. If there
>> are no interception handlers for the AP instructions, QEMU injects an operation
>> exception into the guest. This results in initialization of the AP bus on the
>> guest to terminate. This patch was intended to circumvent that problem.
>>
>> With Halil's suggestion, there is no need to provide these handlers. If ECA.28
>> is set for the guest by default when the AP facilities are installed, then the AP
>> instructions will be interpreted and the AP bus will get initialized on the
>> guest. Since there is no vfio-ap device to provide the AP matrix configuration
>> for the guest, the AP bus will not detect any devices, but that's okay. AP
>> instructions targeting at an APQN will execute successfully and set a response
>> code in the status block returned from the instruction indicating the
>> APQN is invalid .... but there will be no exception unless there is truly
>> an exception condition caused by the execution of the instruction.
> That's exactly the idea, but it will only work out this way if the effective
> ECA.28 bit (i.e. EECA.28) is also set.
>
> Could you please comment the first paragraph quoted in this email?
See my response to Connie's comment in Message ID 
<20180409113244.380a32c4.cohuck@redhat.com>.
My response is in Message ID 
<17b4ab9b-a8c7-37ec-bae4-c57b146ba0b8@linux.vnet.ibm.com>.
>
> Halil