* [PATCH v2 00/15] s390: vfio-ap: guest dedicated crypto adapters
@ 2018-02-27 14:27 Tony Krowiak
2018-02-27 14:27 ` [PATCH v2 01/15] KVM: s390: refactor crypto initialization Tony Krowiak
` (15 more replies)
0 siblings, 16 replies; 56+ messages in thread
From: Tony Krowiak @ 2018-02-27 14:27 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen, Tony Krowiak
On s390, we have cryptographic coprocessor cards, which are modeled on
Linux as devices on the AP bus. Each card can be partitioned into domains
which can be thought of as a set of hardware registers for processing
crypto commands. Crypto commands are sent to a specific domain within a
card is via a queue which is identified as a (card,domain) tuple. We model
this something like the following (assuming we have access to cards 3 and
4 and domains 1 and 2):
AP -> card3 -> queue (3,1)
-> queue (3,2)
-> card4 -> queue (4,1)
-> queue (4,2)
If we want to virtualize this, we can use a feature provided by the
hardware. We basically attach a satellite control block to our main
hardware virtualization control block and the hardware takes care of
most of the rest.
For this control block, we don't specify explicit tuples, but a list of
cards and a list of domains. The guest will get access to the cross
product.
Because of this, we need to take care that the lists provided to
different guests don't overlap; i.e., we need to enforce sane
configurations. Otherwise, one guest may get access to things like
secret keys for another guest.
The idea of this patch set is to introduce a new device, the matrix
device. This matrix device hangs off a different root and acts as the
parent node for mdev devices.
If you now want to give the tuples (4,1) and (4,2), you need to do the
following:
- Unbind the (4,1) and (4,2) tuples from their ap bus driver.
- Bind the (4,1) and (4,2) tuples to the vfio_ap driver.
- Create the mediated device.
- Assign card 4 and domains 1 and 2 to the mediated device
QEMU will now simply consume the mediated device and things should work.
For a complete description of the architecture and concepts underlying the
design, see the Documentation/s390/vfio-ap.txt file included with this
patch set.
v1 => v2 Change log:
===================
* Added documentation vfio-ap.txt
* Renamed vfio_ap_matrix module and device driver to vfio_ap
* Use device core device list instead of maintaining list of matrix
devices in driver
* Added VSIE support for AP
* Create matrix device before registering VFIO AP device driver with the
AP bus
* Renamed the following files in drivers/s390/crypto:
* vfio_ap_matrix.drv -> vfio_ap_drv
* vfio_ap_matrix_private.h -> vfio_ap_private.h
* vfio_ap_matrix_ops.c -> vfio_ap_ops.c
* arch/s390/include/asm/kvm/ap-matrix-config.h
* Renamed to kvm-ap.h
* Changed the data type of the bit mask fields for the matrix structure
to unsigned long and create them with DECLARE_BITMAP
* Changed #define prefixes from AP_MATRIX to KVM_AP
* Changed function and structure prefixes from ap_matrix to kvm_ap
* Added function interface to check if AP Extended Addressing (APXA)
facility is installedCRYCB_FORMAT_MASK
* Added function interface to get the maximum ID for AP mask type
* Added function interface to set the AP execution mode
* arch/s390/kvm/ap-matrix-config.c
* Renamed to kvm-ap.c
* Changed function prefixes from ap_matrix to kvm_ap
* Added function to check if AP Extended Addressing (APXA) facility is
installed
* Added function to get the maximum ID for AP mask type
* Added function to set the AP execution mode
* Added a boolean parameter to the functions that retrieve the APM, AQM
and ADM bit mask fields from the CRYCB. If true, then the function
will clear the bits in the mask before returning a reference to it
* Added validation to verify that APM, AQM and ADM bits that are set do
not exceed the maximum ID value allowed
*
* arch/s390/include/asm/kvm_host.h
* Changed defined for ECA_AP to ECA_APIE - interpretive execution mode
* Added a flag to struct kvm_s390_crypto to indicate whether the
KVM_S390_VM_CPU_FEAT_AP CPU model feature for AP facilities is set
* Added two CPU facilities features to set STFLE.12 and STFLE.15
* arch/s390/kvm/kvm-s390.c
* Added initialization for new KVM_S390_VM_CPU_FEAT_AP CPU model feature
* Removed kvm_s390_apxa_installed() function
* Changed call to kvm_s390_apxa_installed() which has been removed to a
call to new kvm_ap_apxa_installed() function.
* Added code to kvm_s390_vcpu_crypto_setup() to set the new CPU model
feature flag in the kvm_s390_crypto structure
* Added CRYCB_FORMAT_MASK to mask CRYCBD
* arch/s390/tools/gen_facilities.c
* Added STFLE.12 and STFLE.15 to struct facility _def
* drivers/s390/crypto/vfio_ap_matrix_private.h
* Changed name of file to vfio_ap.private.h
* Changed #define prefixes from VFIO_AP_MATRIX to VFIO_AP
* struct ap_matrix: removed list fields and locks
* struct vfio_ap_queue: removed list field
* Renamed functions ap_matrix_mdev_register and ap_matrix_mdev_unregister
to vfio_ap_mdev_register and vfio_ap_mdev_unregister respectively
* drivers/s390/crypto/vfio_ap_matrix_drv.c
* Renamed file to drivers/s390/crypto/vfio_ap_drv.c
* Changed all #define, structure and function prefixes to vfio_ap
* probe function
* Changed root device name for the matrix device to vfio_ap:
i.e., /sys/devices/vfio_ap/matrix
* No longer storing the AP queue device in a list, it is retrievable
via the device core
* Removed unnecessary check whether matrix device exists
* Store the vfio_ap_queue structure in the private field of the
ap_queue structure rather than using list interface
* remove function
* Retrieve vfio_ap_queue structure from the struct ap_queue private
data rather than from a list
* Removed unnecessary check
* drivers/s390/crypto/vfio_ap_matrix_ops.c
* Renamed file to vfio_ap_ops.c
* Changed #define prefixes from AP_MATRIX to VFIO_AP
* Changed function name prefixes from ap_matrix to vfio_ap
* Removed ioctl to configure the CRYCB
* create function
* Removed ap_matrix_mdev_find_by_uuid() function - function is provided
by mdev core
* Removed available_instances verification, provided by mdev core
* Removed check to see if mediated device exists, handled by mdev core
* notifier function
* Configuring matrix here instead of via ioctl
* Set interpretive execution mode for all VCPUs
* Removed R/O attributes to display adapters and domains
* Added an R/O attribute to display the matrix
* assign_control_domain mdev attribute:
* Removed check to see if the domain is installed on the linux host
* Added check to verify the control domain ID does not exceed the max
value
* assign_adapter mdev attribute:
* Added check to verify the adapter ID does not exceed the max
value
* If any APQNs configured for the mediated matrix device that
have an APID matching the adapter ID being assigned are not
bound to the vfio_ap device driver then it is assumed that the APQN
is bound to another driver and assignment will fail
* assign_domain mdev attribute:
* Added check to verify the domain ID does not exceed the max
value
* If any APQNs configured for the mediated matrix device that
have an APQI matching the domain ID being assigned are not
bound to the vfio_ap device driver then it is assumed that the APQN
is bound to another driver and assignment will fail
* tools/arch/s390/include/uapi/asm/kvm.h
* removed KVM_S390_VM_CPU_FEAT_AP feature definition
Tony Krowiak (15):
KVM: s390: refactor crypto initialization
s390: vsie: implement AP support for second level guest
s390: zcrypt: externalize AP instructions available function
KVM: s390: CPU model support for AP virtualization
s390: vfio-ap: base implementation of VFIO AP device driver
s390: vfio-ap: register matrix device with VFIO mdev framework
KVM: s390: Interfaces to configure/deconfigure guest's AP matrix
KVM: s390: interface to enable AP execution mode
s390: vfio-ap: sysfs interfaces to configure adapters
s390: vfio-ap: sysfs interfaces to configure domains
s390: vfio-ap: sysfs interfaces to configure control domains
s390: vfio-ap: sysfs interface to view matrix mdev matrix
KVM: s390: Configure the guest's CRYCB
s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl
s390: doc: detailed specifications for AP virtualization
Documentation/s390/vfio-ap.txt | 514 ++++++++++++++++++
MAINTAINERS | 14 +
arch/s390/Kconfig | 8 +
arch/s390/configs/default_defconfig | 3 +
arch/s390/configs/gcov_defconfig | 3 +
arch/s390/configs/performance_defconfig | 3 +
arch/s390/defconfig | 3 +
arch/s390/include/asm/ap.h | 7 +
arch/s390/include/asm/kvm-ap.h | 59 +++
arch/s390/include/asm/kvm_host.h | 2 +
arch/s390/include/uapi/asm/kvm.h | 1 +
arch/s390/kvm/Makefile | 2 +-
arch/s390/kvm/kvm-ap.c | 336 ++++++++++++
arch/s390/kvm/kvm-s390.c | 66 +--
arch/s390/kvm/kvm-s390.h | 1 +
arch/s390/kvm/vsie.c | 71 +++-
arch/s390/tools/gen_facilities.c | 2 +
drivers/s390/crypto/Makefile | 4 +
drivers/s390/crypto/ap_bus.c | 6 +
drivers/s390/crypto/vfio_ap_drv.c | 143 +++++
drivers/s390/crypto/vfio_ap_ops.c | 868 +++++++++++++++++++++++++++++++
drivers/s390/crypto/vfio_ap_private.h | 45 ++
include/uapi/linux/vfio.h | 2 +
23 files changed, 2096 insertions(+), 67 deletions(-)
create mode 100644 Documentation/s390/vfio-ap.txt
create mode 100644 arch/s390/include/asm/kvm-ap.h
create mode 100644 arch/s390/kvm/kvm-ap.c
create mode 100644 drivers/s390/crypto/vfio_ap_drv.c
create mode 100644 drivers/s390/crypto/vfio_ap_ops.c
create mode 100644 drivers/s390/crypto/vfio_ap_private.h
^ permalink raw reply [flat|nested] 56+ messages in thread
* [PATCH v2 01/15] KVM: s390: refactor crypto initialization
2018-02-27 14:27 [PATCH v2 00/15] s390: vfio-ap: guest dedicated crypto adapters Tony Krowiak
@ 2018-02-27 14:27 ` Tony Krowiak
2018-02-28 17:37 ` Cornelia Huck
2018-02-27 14:28 ` [PATCH v2 02/15] s390: vsie: implement AP support for second level guest Tony Krowiak
` (14 subsequent siblings)
15 siblings, 1 reply; 56+ messages in thread
From: Tony Krowiak @ 2018-02-27 14:27 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen, Tony Krowiak
The crypto control block designation (CRYCBD) is a 32-bit
field in the KVM guest's SIE state description. The
contents of bits 1-28 of this field, with three zero bits
appended on the right, designate the host real 31-bit
address of a crypto control block (CRYCB). Bits 30-31
specify the format of the CRYCB. In the current
implementation, the address of the CRYCB is stored in
the CRYCBD only if the Message-Security-Assist extension
3 (MSA3) facility is installed. Virtualization of AP
facilities, however, requires that a CRYCB of the
appropriate format be made available to SIE regardless
of whether MSA3 is installed or not.
This patch introduces a new compilation unit to provide
all interfaces related to configuration of AP facilities.
Let's start by moving the function for setting the CRYCB
format from arch/s390/kvm/kvm-s390 to this new AP
configuration interface.
Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
MAINTAINERS | 10 ++++++
arch/s390/include/asm/kvm-ap.h | 16 ++++++++++
arch/s390/include/asm/kvm_host.h | 1 +
arch/s390/kvm/Makefile | 2 +-
arch/s390/kvm/kvm-ap.c | 47 ++++++++++++++++++++++++++++
arch/s390/kvm/kvm-s390.c | 62 +++++---------------------------------
6 files changed, 83 insertions(+), 55 deletions(-)
create mode 100644 arch/s390/include/asm/kvm-ap.h
create mode 100644 arch/s390/kvm/kvm-ap.c
diff --git a/MAINTAINERS b/MAINTAINERS
index 0ec5881..4acf7c2 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -11875,6 +11875,16 @@ W: http://www.ibm.com/developerworks/linux/linux390/
S: Supported
F: drivers/s390/crypto/
+S390 VFIO AP DRIVER
+M: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
+M: Christian BornTraeger <borntraeger@de.ibm.com>
+M: Martin Schwidefsky <schwidefsky@de.ibm.com>
+L: linux-s390@vger.kernel.org
+W: http://www.ibm.com/developerworks/linux/linux390/
+S: Supported
+F: arch/s390/include/asm/kvm/kvm-ap.h
+F: arch/s390/kvm/kvm-ap.c
+
S390 ZFCP DRIVER
M: Steffen Maier <maier@linux.vnet.ibm.com>
M: Benjamin Block <bblock@linux.vnet.ibm.com>
diff --git a/arch/s390/include/asm/kvm-ap.h b/arch/s390/include/asm/kvm-ap.h
new file mode 100644
index 0000000..4e43117
--- /dev/null
+++ b/arch/s390/include/asm/kvm-ap.h
@@ -0,0 +1,16 @@
+/*
+ * Adjunct Processor (AP) configuration management for KVM guests
+ *
+ * Copyright IBM Corp. 2017
+ *
+ * Author(s): Tony Krowiak <akrowia@linux.vnet.ibm.com>
+ */
+
+#ifndef _ASM_KVM_AP
+#define _ASM_KVM_AP
+#include <linux/types.h>
+#include <linux/kvm_host.h>
+
+void kvm_ap_set_crycb_format(struct kvm *kvm, __u32 *crycbd);
+
+#endif /* _ASM_KVM_AP */
diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index 27918b1..a4c77d3 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -257,6 +257,7 @@ struct kvm_s390_sie_block {
__u8 reservedf0[12]; /* 0x00f0 */
#define CRYCB_FORMAT1 0x00000001
#define CRYCB_FORMAT2 0x00000003
+#define CRYCB_FORMAT_MASK 0x00000003
__u32 crycbd; /* 0x00fc */
__u64 gcr[16]; /* 0x0100 */
__u64 gbea; /* 0x0180 */
diff --git a/arch/s390/kvm/Makefile b/arch/s390/kvm/Makefile
index 05ee90a..1876bfe 100644
--- a/arch/s390/kvm/Makefile
+++ b/arch/s390/kvm/Makefile
@@ -9,6 +9,6 @@ common-objs = $(KVM)/kvm_main.o $(KVM)/eventfd.o $(KVM)/async_pf.o $(KVM)/irqch
ccflags-y := -Ivirt/kvm -Iarch/s390/kvm
kvm-objs := $(common-objs) kvm-s390.o intercept.o interrupt.o priv.o sigp.o
-kvm-objs += diag.o gaccess.o guestdbg.o vsie.o
+kvm-objs += diag.o gaccess.o guestdbg.o vsie.o kvm-ap.o
obj-$(CONFIG_KVM) += kvm.o
diff --git a/arch/s390/kvm/kvm-ap.c b/arch/s390/kvm/kvm-ap.c
new file mode 100644
index 0000000..5305f4c
--- /dev/null
+++ b/arch/s390/kvm/kvm-ap.c
@@ -0,0 +1,47 @@
+/*
+ * Adjunct Processor (AP) configuration management for KVM guests
+ *
+ * Copyright IBM Corp. 2017
+ *
+ * Author(s): Tony Krowiak <akrowia@linux.vnet.ibm.com>
+ */
+
+#include <asm/kvm-ap.h>
+#include <asm/ap.h>
+
+#include "kvm-s390.h"
+
+static int kvm_ap_apxa_installed(void)
+{
+ int ret;
+ struct ap_config_info config;
+
+ ret = ap_query_configuration(&config);
+ if (ret)
+ return 0;
+
+ return (config.apxa == 1);
+}
+
+/**
+ * kvm_ap_set_crycb_format
+ *
+ * Set the CRYCB format in the CRYCBD for the KVM guest.
+ *
+ * @kvm: the KVM guest
+ * @crycbd: the CRYCB descriptor
+ */
+void kvm_ap_set_crycb_format(struct kvm *kvm, __u32 *crycbd)
+{
+ *crycbd = (__u32)(unsigned long) kvm->arch.crypto.crycb;
+
+ *crycbd &= ~(CRYCB_FORMAT_MASK);
+
+ /* If the MSAX3 is installed */
+ if (test_kvm_facility(kvm, 76)) {
+ if (kvm_ap_apxa_installed())
+ *crycbd |= CRYCB_FORMAT2;
+ else
+ *crycbd |= CRYCB_FORMAT1;
+ }
+}
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 5f5a4cb..de1e299 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -40,6 +40,8 @@
#include <asm/sclp.h>
#include <asm/cpacf.h>
#include <asm/timex.h>
+#include <asm/ap.h>
+#include <asm/kvm-ap.h>
#include "kvm-s390.h"
#include "gaccess.h"
@@ -1853,55 +1855,6 @@ long kvm_arch_vm_ioctl(struct file *filp,
return r;
}
-static int kvm_s390_query_ap_config(u8 *config)
-{
- u32 fcn_code = 0x04000000UL;
- u32 cc = 0;
-
- memset(config, 0, 128);
- asm volatile(
- "lgr 0,%1\n"
- "lgr 2,%2\n"
- ".long 0xb2af0000\n" /* PQAP(QCI) */
- "0: ipm %0\n"
- "srl %0,28\n"
- "1:\n"
- EX_TABLE(0b, 1b)
- : "+r" (cc)
- : "r" (fcn_code), "r" (config)
- : "cc", "0", "2", "memory"
- );
-
- return cc;
-}
-
-static int kvm_s390_apxa_installed(void)
-{
- u8 config[128];
- int cc;
-
- if (test_facility(12)) {
- cc = kvm_s390_query_ap_config(config);
-
- if (cc)
- pr_err("PQAP(QCI) failed with cc=%d", cc);
- else
- return config[0] & 0x40;
- }
-
- return 0;
-}
-
-static void kvm_s390_set_crycb_format(struct kvm *kvm)
-{
- kvm->arch.crypto.crycbd = (__u32)(unsigned long) kvm->arch.crypto.crycb;
-
- if (kvm_s390_apxa_installed())
- kvm->arch.crypto.crycbd |= CRYCB_FORMAT2;
- else
- kvm->arch.crypto.crycbd |= CRYCB_FORMAT1;
-}
-
static u64 kvm_s390_get_initial_cpuid(void)
{
struct cpuid cpuid;
@@ -1913,12 +1866,13 @@ static u64 kvm_s390_get_initial_cpuid(void)
static void kvm_s390_crypto_init(struct kvm *kvm)
{
+ kvm->arch.crypto.crycb = &kvm->arch.sie_page2->crycb;
+ kvm->arch.crypto.crycbd = (__u32)(unsigned long) kvm->arch.crypto.crycb;
+ kvm_ap_set_crycb_format(kvm, &kvm->arch.crypto.crycbd);
+
if (!test_kvm_facility(kvm, 76))
return;
- kvm->arch.crypto.crycb = &kvm->arch.sie_page2->crycb;
- kvm_s390_set_crycb_format(kvm);
-
/* Enable AES/DEA protected key functions by default */
kvm->arch.crypto.aes_kw = 1;
kvm->arch.crypto.dea_kw = 1;
@@ -2446,6 +2400,8 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)
static void kvm_s390_vcpu_crypto_setup(struct kvm_vcpu *vcpu)
{
+ vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
+
if (!test_kvm_facility(vcpu->kvm, 76))
return;
@@ -2455,8 +2411,6 @@ static void kvm_s390_vcpu_crypto_setup(struct kvm_vcpu *vcpu)
vcpu->arch.sie_block->ecb3 |= ECB3_AES;
if (vcpu->kvm->arch.crypto.dea_kw)
vcpu->arch.sie_block->ecb3 |= ECB3_DEA;
-
- vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
}
void kvm_s390_vcpu_unsetup_cmma(struct kvm_vcpu *vcpu)
--
1.7.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 02/15] s390: vsie: implement AP support for second level guest
2018-02-27 14:27 [PATCH v2 00/15] s390: vfio-ap: guest dedicated crypto adapters Tony Krowiak
2018-02-27 14:27 ` [PATCH v2 01/15] KVM: s390: refactor crypto initialization Tony Krowiak
@ 2018-02-27 14:28 ` Tony Krowiak
2018-02-28 9:39 ` David Hildenbrand
2018-02-27 14:28 ` [PATCH v2 03/15] s390: zcrypt: externalize AP instructions available function Tony Krowiak
` (13 subsequent siblings)
15 siblings, 1 reply; 56+ messages in thread
From: Tony Krowiak @ 2018-02-27 14:28 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen, Tony Krowiak
Set effective masks and CRYCB format in the shadow copy of the
guest level 2 CRYCB.
Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
arch/s390/include/asm/kvm-ap.h | 2 +
arch/s390/kvm/kvm-ap.c | 5 +++
arch/s390/kvm/vsie.c | 71 +++++++++++++++++++++++++++++++++-------
3 files changed, 66 insertions(+), 12 deletions(-)
diff --git a/arch/s390/include/asm/kvm-ap.h b/arch/s390/include/asm/kvm-ap.h
index 4e43117..ef749e7 100644
--- a/arch/s390/include/asm/kvm-ap.h
+++ b/arch/s390/include/asm/kvm-ap.h
@@ -13,4 +13,6 @@
void kvm_ap_set_crycb_format(struct kvm *kvm, __u32 *crycbd);
+int kvm_ap_get_crycb_format(struct kvm *kvm);
+
#endif /* _ASM_KVM_AP */
diff --git a/arch/s390/kvm/kvm-ap.c b/arch/s390/kvm/kvm-ap.c
index 5305f4c..bafe63b 100644
--- a/arch/s390/kvm/kvm-ap.c
+++ b/arch/s390/kvm/kvm-ap.c
@@ -11,6 +11,11 @@
#include "kvm-s390.h"
+int kvm_ap_get_crycb_format(struct kvm *kvm)
+{
+ return kvm->arch.crypto.crycbd & CRYCB_FORMAT_MASK;
+}
+
static int kvm_ap_apxa_installed(void)
{
int ret;
diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c
index 8961e39..93076ba 100644
--- a/arch/s390/kvm/vsie.c
+++ b/arch/s390/kvm/vsie.c
@@ -18,6 +18,7 @@
#include <asm/sclp.h>
#include <asm/nmi.h>
#include <asm/dis.h>
+#include <asm/kvm-ap.h>
#include "kvm-s390.h"
#include "gaccess.h"
@@ -137,12 +138,56 @@ static int prepare_cpuflags(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
}
/*
+ * Set up the effective masks for the shadow copy of the crycb. The effective
+ * masks for guest 3 are set by performing a logical bitwise AND of the guest 3
+ * masks with the guest 2 masks.
+ */
+static void set_crycb_emasks(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
+{
+ int fmt = kvm_ap_get_crycb_format(vcpu->kvm);
+ unsigned long *mask1, *mask2;
+
+ switch (fmt) {
+ case CRYCB_FORMAT1:
+ case CRYCB_FORMAT2:
+ mask1 = (unsigned long *)vsie_page->crycb.apcb1.apm;
+ mask2 = (unsigned long *)
+ vcpu->kvm->arch.crypto.crycb->apcb1.apm;
+ bitmap_and(mask1, mask1, mask2, APCB1_MASK_SIZE);
+
+ mask1 = (unsigned long *)vsie_page->crycb.apcb1.aqm;
+ mask2 = (unsigned long *)
+ vcpu->kvm->arch.crypto.crycb->apcb1.aqm;
+ bitmap_and(mask1, mask1, mask2, APCB1_MASK_SIZE);
+
+ mask1 = (unsigned long *)vsie_page->crycb.apcb1.adm;
+ mask2 = (unsigned long *)
+ vcpu->kvm->arch.crypto.crycb->apcb1.adm;
+ bitmap_and(mask1, mask1, mask2, APCB1_MASK_SIZE);
+ break;
+ default:
+ mask1 = (unsigned long *)vsie_page->crycb.apcb0.apm;
+ mask2 = (unsigned long *)
+ vcpu->kvm->arch.crypto.crycb->apcb1.apm;
+ bitmap_and(mask1, mask1, mask2, APCB0_MASK_SIZE);
+
+ mask1 = (unsigned long *)vsie_page->crycb.apcb0.aqm;
+ mask2 = (unsigned long *)
+ vcpu->kvm->arch.crypto.crycb->apcb1.aqm;
+ bitmap_and(mask1, mask1, mask2, APCB0_MASK_SIZE);
+
+ mask1 = (unsigned long *)vsie_page->crycb.apcb0.adm;
+ mask2 = (unsigned long *)
+ vcpu->kvm->arch.crypto.crycb->apcb1.adm;
+ bitmap_and(mask1, mask1, mask2, APCB0_MASK_SIZE);
+ break;
+ }
+}
+
+/*
* Create a shadow copy of the crycb block and setup key wrapping, if
* requested for guest 3 and enabled for guest 2.
*
- * We only accept format-1 (no AP in g2), but convert it into format-2
- * There is nothing to do for format-0.
- *
* Returns: - 0 if shadowed or nothing to do
* - > 0 if control has to be given to guest 2
*/
@@ -155,9 +200,17 @@ static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
unsigned long *b1, *b2;
u8 ecb3_flags;
- scb_s->crycbd = 0;
- if (!(crycbd_o & vcpu->arch.sie_block->crycbd & CRYCB_FORMAT1))
- return 0;
+ scb_s->crycbd = ((__u32)(__u64) &vsie_page->crycb);
+ kvm_ap_set_crycb_format(vcpu->kvm, &scb_s->crycbd);
+
+ /* copy the crycb */
+ if (read_guest_real(vcpu, crycb_addr, &vsie_page->crycb,
+ sizeof(vsie_page->crycb)))
+ return set_validity_icpt(scb_s, 0x0035U);
+
+ /* set up the effective masks */
+ set_crycb_emasks(vcpu, vsie_page);
+
/* format-1 is supported with message-security-assist extension 3 */
if (!test_kvm_facility(vcpu->kvm, 76))
return 0;
@@ -172,13 +225,7 @@ static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
else if (!crycb_addr)
return set_validity_icpt(scb_s, 0x0039U);
- /* copy only the wrapping keys */
- if (read_guest_real(vcpu, crycb_addr + 72, &vsie_page->crycb, 56))
- return set_validity_icpt(scb_s, 0x0035U);
-
scb_s->ecb3 |= ecb3_flags;
- scb_s->crycbd = ((__u32)(__u64) &vsie_page->crycb) | CRYCB_FORMAT1 |
- CRYCB_FORMAT2;
/* xor both blocks in one run */
b1 = (unsigned long *) vsie_page->crycb.dea_wrapping_key_mask;
--
1.7.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 03/15] s390: zcrypt: externalize AP instructions available function
2018-02-27 14:27 [PATCH v2 00/15] s390: vfio-ap: guest dedicated crypto adapters Tony Krowiak
2018-02-27 14:27 ` [PATCH v2 01/15] KVM: s390: refactor crypto initialization Tony Krowiak
2018-02-27 14:28 ` [PATCH v2 02/15] s390: vsie: implement AP support for second level guest Tony Krowiak
@ 2018-02-27 14:28 ` Tony Krowiak
2018-02-28 11:40 ` Harald Freudenberger
2018-02-28 17:41 ` Cornelia Huck
2018-02-27 14:28 ` [PATCH v2 04/15] KVM: s390: CPU model support for AP virtualization Tony Krowiak
` (12 subsequent siblings)
15 siblings, 2 replies; 56+ messages in thread
From: Tony Krowiak @ 2018-02-27 14:28 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen, Tony Krowiak
If the AP instructions are not available on the linux host, then
AP devices can not be interpreted by the SIE. The AP bus has a
function it uses to determine if the AP instructions are
available. This patch provides a new function that wraps the
AP bus's function to externalize it for use by KVM.
Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
Reviewed-by: Pierre Morel <pmorel@linux.vnet.ibm.com>
---
arch/s390/include/asm/ap.h | 7 +++++++
drivers/s390/crypto/ap_bus.c | 6 ++++++
2 files changed, 13 insertions(+), 0 deletions(-)
diff --git a/arch/s390/include/asm/ap.h b/arch/s390/include/asm/ap.h
index cfce683..1df6b59 100644
--- a/arch/s390/include/asm/ap.h
+++ b/arch/s390/include/asm/ap.h
@@ -120,4 +120,11 @@ struct ap_queue_status ap_queue_irq_ctrl(ap_qid_t qid,
struct ap_qirq_ctrl qirqctrl,
void *ind);
+/**
+ * ap_instructions_installed() - Tests whether AP instructions are installed
+ *
+ * Returns 1 if the AP instructions are installed, otherwise; returns 0
+ */
+int ap_instructions_installed(void);
+
#endif /* _ASM_S390_AP_H_ */
diff --git a/drivers/s390/crypto/ap_bus.c b/drivers/s390/crypto/ap_bus.c
index 48d55dc..089b1cf 100644
--- a/drivers/s390/crypto/ap_bus.c
+++ b/drivers/s390/crypto/ap_bus.c
@@ -211,6 +211,12 @@ int ap_query_configuration(struct ap_config_info *info)
}
EXPORT_SYMBOL(ap_query_configuration);
+int ap_instructions_installed(void)
+{
+ return (ap_instructions_available() == 0);
+}
+EXPORT_SYMBOL(ap_instructions_installed);
+
/**
* ap_init_configuration(): Allocate and query configuration array.
*/
--
1.7.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 04/15] KVM: s390: CPU model support for AP virtualization
2018-02-27 14:27 [PATCH v2 00/15] s390: vfio-ap: guest dedicated crypto adapters Tony Krowiak
` (2 preceding siblings ...)
2018-02-27 14:28 ` [PATCH v2 03/15] s390: zcrypt: externalize AP instructions available function Tony Krowiak
@ 2018-02-27 14:28 ` Tony Krowiak
2018-02-28 9:48 ` David Hildenbrand
2018-02-27 14:28 ` [PATCH v2 05/15] s390: vfio-ap: base implementation of VFIO AP device driver Tony Krowiak
` (11 subsequent siblings)
15 siblings, 1 reply; 56+ messages in thread
From: Tony Krowiak @ 2018-02-27 14:28 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen, Tony Krowiak
Introduces a new CPU model feature and two CPU model
facilities to support AP virtualization for KVM guests.
CPU model feature:
The KVM_S390_VM_CPU_FEAT_AP feature indicates that the
AP facilities are installed on the KVM guest. This
feature will be enabled by the kernel only if the AP
facilities are installed on the linux host. This feature
must be specifically turned on for the KVM guest from
userspace to allow guest access to AP devices installed
on the linux host.
CPU model facilities:
1. AP Query Configuration Information (QCI) facility is installed.
This is indicated by setting facilities bit 12 for
the guest. The kernel will not enable this facility
for the guest if it is not set on the host. This facility
must not be set by userspace if the KVM_S390_VM_CPU_FEAT_AP
feature is not installed.
2. AP Facilities Test facility (APFT) is installed.
This is indicated by setting facilities bit 15 for
the guest. The kernel will not enable this facility for
the guest if it is not set on the host. This facility
must not be set by userspace if the KVM_S390_VM_CPU_FEAT_AP
feature is not installed.
Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com>
Reviewed-by: Halil Pasic <pasic@linux.vnet.ibm.com>
Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
arch/s390/include/uapi/asm/kvm.h | 1 +
arch/s390/kvm/kvm-s390.c | 4 ++++
arch/s390/tools/gen_facilities.c | 2 ++
3 files changed, 7 insertions(+), 0 deletions(-)
diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h
index 4cdaa55..a580dec 100644
--- a/arch/s390/include/uapi/asm/kvm.h
+++ b/arch/s390/include/uapi/asm/kvm.h
@@ -130,6 +130,7 @@ struct kvm_s390_vm_cpu_machine {
#define KVM_S390_VM_CPU_FEAT_PFMFI 11
#define KVM_S390_VM_CPU_FEAT_SIGPIF 12
#define KVM_S390_VM_CPU_FEAT_KSS 13
+#define KVM_S390_VM_CPU_FEAT_AP 14
struct kvm_s390_vm_cpu_feat {
__u64 feat[16];
};
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index de1e299..c68ca86 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -347,6 +347,10 @@ static void kvm_s390_cpu_feat_init(void)
if (MACHINE_HAS_ESOP)
allow_cpu_feat(KVM_S390_VM_CPU_FEAT_ESOP);
+
+ if (ap_instructions_installed()) /* AP instructions installed on host */
+ allow_cpu_feat(KVM_S390_VM_CPU_FEAT_AP);
+
/*
* We need SIE support, ESOP (PROT_READ protection for gmap_shadow),
* 64bit SCAO (SCA passthrough) and IDTE (for gmap_shadow unshadowing).
diff --git a/arch/s390/tools/gen_facilities.c b/arch/s390/tools/gen_facilities.c
index 90a8c9e..a52290b 100644
--- a/arch/s390/tools/gen_facilities.c
+++ b/arch/s390/tools/gen_facilities.c
@@ -106,6 +106,8 @@ struct facility_def {
.name = "FACILITIES_KVM_CPUMODEL",
.bits = (int[]){
+ 12, /* AP Query Configuration Information */
+ 15, /* AP Facilities Test */
-1 /* END */
}
},
--
1.7.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 05/15] s390: vfio-ap: base implementation of VFIO AP device driver
2018-02-27 14:27 [PATCH v2 00/15] s390: vfio-ap: guest dedicated crypto adapters Tony Krowiak
` (3 preceding siblings ...)
2018-02-27 14:28 ` [PATCH v2 04/15] KVM: s390: CPU model support for AP virtualization Tony Krowiak
@ 2018-02-27 14:28 ` Tony Krowiak
2018-02-28 15:33 ` Pierre Morel
2018-02-27 14:28 ` [PATCH v2 06/15] s390: vfio-ap: register matrix device with VFIO mdev framework Tony Krowiak
` (10 subsequent siblings)
15 siblings, 1 reply; 56+ messages in thread
From: Tony Krowiak @ 2018-02-27 14:28 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen, Tony Krowiak
Introduces a new AP device driver. This device driver
is built on the VFIO mediated device framework. The framework
provides sysfs interfaces that facilitate passthrough
access by guests to devices installed on the linux host.
The VFIO AP device driver will serve two purposes:
1. Provide the interfaces to reserve AP devices for exclusive
use by KVM guests. This is accomplished by unbinding the
devices to be reserved for guest usage from the default AP
device driver and binding them to the VFIO AP device driver.
2. Implements the functions, callbacks and sysfs attribute
interfaces required to create one or more VFIO mediated
devices each of which will be used to configure the AP
matrix for a guest and serve as a file descriptor
for facilitating communication between QEMU and the
VFIO AP device driver.
When the VFIO AP device driver is initialized:
* It registers with the AP bus for control of type 10 (CEX4
and newer) AP queue devices. The probe and remove callbacks
will be provided to support the binding/unbinding of
AP queue devices to/from the VFIO AP device driver.
* Creates a /sys/devices/vfio-ap/matrix device to hold
the APQNs of the AP devices bound to the VFIO
AP device driver and serves as the parent of the
mediated devices created for each guest.
Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
MAINTAINERS | 2 +
arch/s390/Kconfig | 8 ++
arch/s390/configs/default_defconfig | 3 +
arch/s390/configs/gcov_defconfig | 3 +
arch/s390/configs/performance_defconfig | 3 +
arch/s390/defconfig | 3 +
drivers/s390/crypto/Makefile | 4 +
drivers/s390/crypto/vfio_ap_drv.c | 134 +++++++++++++++++++++++++++++++
drivers/s390/crypto/vfio_ap_private.h | 20 +++++
include/uapi/linux/vfio.h | 2 +
10 files changed, 182 insertions(+), 0 deletions(-)
create mode 100644 drivers/s390/crypto/vfio_ap_drv.c
create mode 100644 drivers/s390/crypto/vfio_ap_private.h
diff --git a/MAINTAINERS b/MAINTAINERS
index 4acf7c2..a2f232d 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -11884,6 +11884,8 @@ W: http://www.ibm.com/developerworks/linux/linux390/
S: Supported
F: arch/s390/include/asm/kvm/kvm-ap.h
F: arch/s390/kvm/kvm-ap.c
+F: drivers/s390/crypto/vfio_ap_drv.c
+F: drivers/s390/crypto/vfio_ap_private.h
S390 ZFCP DRIVER
M: Steffen Maier <maier@linux.vnet.ibm.com>
diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
index cbe1d97..9f23caf 100644
--- a/arch/s390/Kconfig
+++ b/arch/s390/Kconfig
@@ -771,6 +771,14 @@ config VFIO_CCW
To compile this driver as a module, choose M here: the
module will be called vfio_ccw.
+config VFIO_AP
+ def_tristate m
+ prompt "Support for virtual Adjunct Processor device interface"
+ depends on ZCRYPT && VFIO_MDEV_DEVICE
+ help
+ driver grants access to Adjunct Processor (AP) devices
+ via the VFIO mediated device interface.
+
endmenu
menu "Dump support"
diff --git a/arch/s390/configs/default_defconfig b/arch/s390/configs/default_defconfig
index 5af8458..40fa3f6 100644
--- a/arch/s390/configs/default_defconfig
+++ b/arch/s390/configs/default_defconfig
@@ -719,3 +719,6 @@ CONFIG_APPLDATA_BASE=y
CONFIG_KVM=m
CONFIG_KVM_S390_UCONTROL=y
CONFIG_VHOST_NET=m
+VFIO_MDEV=m
+VFIO_MDEV_DEVICE=m
+CONFIG_VFIO_AP=m
diff --git a/arch/s390/configs/gcov_defconfig b/arch/s390/configs/gcov_defconfig
index d52eafe..377d40f 100644
--- a/arch/s390/configs/gcov_defconfig
+++ b/arch/s390/configs/gcov_defconfig
@@ -659,3 +659,6 @@ CONFIG_APPLDATA_BASE=y
CONFIG_KVM=m
CONFIG_KVM_S390_UCONTROL=y
CONFIG_VHOST_NET=m
+VFIO_MDEV=m
+VFIO_MDEV_DEVICE=m
+CONFIG_VFIO_AP=m
diff --git a/arch/s390/configs/performance_defconfig b/arch/s390/configs/performance_defconfig
index 20ed149..f0c52df 100644
--- a/arch/s390/configs/performance_defconfig
+++ b/arch/s390/configs/performance_defconfig
@@ -657,3 +657,6 @@ CONFIG_APPLDATA_BASE=y
CONFIG_KVM=m
CONFIG_KVM_S390_UCONTROL=y
CONFIG_VHOST_NET=m
+VFIO_MDEV=m
+VFIO_MDEV_DEVICE=m
+CONFIG_VFIO_AP=m
diff --git a/arch/s390/defconfig b/arch/s390/defconfig
index 46a3178..0996eb7 100644
--- a/arch/s390/defconfig
+++ b/arch/s390/defconfig
@@ -239,3 +239,6 @@ CONFIG_CRC7=m
# CONFIG_XZ_DEC_ARMTHUMB is not set
# CONFIG_XZ_DEC_SPARC is not set
CONFIG_CMM=m
+VFIO_MDEV=m
+VFIO_MDEV_DEVICE=m
+CONFIG_VFIO_AP=m
diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile
index b59af54..48e466e 100644
--- a/drivers/s390/crypto/Makefile
+++ b/drivers/s390/crypto/Makefile
@@ -15,3 +15,7 @@ obj-$(CONFIG_ZCRYPT) += zcrypt_pcixcc.o zcrypt_cex2a.o zcrypt_cex4.o
# pkey kernel module
pkey-objs := pkey_api.o
obj-$(CONFIG_PKEY) += pkey.o
+
+# adjunct processor matrix
+vfio_ap-objs := vfio_ap_drv.o
+obj-$(CONFIG_VFIO_AP) += vfio_ap.o
diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c
new file mode 100644
index 0000000..8bb72af
--- /dev/null
+++ b/drivers/s390/crypto/vfio_ap_drv.c
@@ -0,0 +1,134 @@
+/*
+ * VFIO based AP device driver
+ *
+ * Copyright IBM Corp. 2017
+ *
+ * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
+ */
+
+#include <linux/module.h>
+#include <linux/mod_devicetable.h>
+#include <linux/slab.h>
+
+#include "ap_bus.h"
+#include "vfio_ap_private.h"
+
+#define VFIO_AP_ROOT_NAME "vfio_ap"
+#define VFIO_AP_DEV_TYPE_NAME "ap_matrix"
+#define VFIO_AP_DEV_NAME "matrix"
+
+MODULE_AUTHOR("IBM Corporation");
+MODULE_DESCRIPTION("VFIO AP device driver, Copyright IBM Corp. 2017");
+MODULE_LICENSE("GPL v2");
+
+static struct device *vfio_ap_root_device;
+
+static struct ap_driver vfio_ap_drv;
+
+static struct ap_matrix *ap_matrix;
+
+static struct device_type vfio_ap_dev_type = {
+ .name = VFIO_AP_DEV_TYPE_NAME,
+};
+
+/* Only type 10 adapters (CEX4 and later) are supported
+ * by the AP matrix device driver
+ */
+static struct ap_device_id ap_queue_ids[] = {
+ { .dev_type = AP_DEVICE_TYPE_CEX4,
+ .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
+ { .dev_type = AP_DEVICE_TYPE_CEX5,
+ .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
+ { .dev_type = AP_DEVICE_TYPE_CEX6,
+ .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
+ { /* end of sibling */ },
+};
+
+MODULE_DEVICE_TABLE(vfio_ap, ap_queue_ids);
+
+static int vfio_ap_queue_dev_probe(struct ap_device *apdev)
+{
+ return 0;
+}
+
+static void vfio_ap_matrix_dev_release(struct device *dev)
+{
+ struct ap_matrix *ap_matrix = dev_get_drvdata(dev);
+
+ kfree(ap_matrix);
+}
+
+static int vfio_ap_matrix_dev_create(void)
+{
+ int ret;
+
+ vfio_ap_root_device = root_device_register(VFIO_AP_ROOT_NAME);
+
+ ret = PTR_ERR_OR_ZERO(vfio_ap_root_device);
+ if (ret)
+ goto done;
+
+ ap_matrix = kzalloc(sizeof(*ap_matrix), GFP_KERNEL);
+ if (!ap_matrix) {
+ ret = -ENOMEM;
+ goto matrix_alloc_err;
+ }
+
+ ap_matrix->device.type = &vfio_ap_dev_type;
+ dev_set_name(&ap_matrix->device, "%s", VFIO_AP_DEV_NAME);
+ ap_matrix->device.parent = vfio_ap_root_device;
+ ap_matrix->device.release = vfio_ap_matrix_dev_release;
+ ap_matrix->device.driver = &vfio_ap_drv.driver;
+
+ ret = device_register(&ap_matrix->device);
+ if (ret)
+ goto matrix_reg_err;
+
+ goto done;
+
+matrix_reg_err:
+ put_device(&ap_matrix->device);
+ kfree(ap_matrix);
+
+matrix_alloc_err:
+ root_device_unregister(vfio_ap_root_device);
+
+done:
+ return ret;
+}
+
+static void vfio_ap_matrix_dev_destroy(struct ap_matrix *ap_matrix)
+{
+ device_unregister(&ap_matrix->device);
+ root_device_unregister(vfio_ap_root_device);
+}
+
+int __init vfio_ap_init(void)
+{
+ int ret;
+
+ ret = vfio_ap_matrix_dev_create();
+ if (ret)
+ return ret;
+
+ memset(&vfio_ap_drv, 0, sizeof(vfio_ap_drv));
+ vfio_ap_drv.probe = vfio_ap_queue_dev_probe;
+ vfio_ap_drv.ids = ap_queue_ids;
+
+ ret = ap_driver_register(&vfio_ap_drv, THIS_MODULE, VFIO_AP_DRV_NAME);
+ if (ret) {
+ vfio_ap_matrix_dev_destroy(ap_matrix);
+ return ret;
+ }
+
+ return 0;
+}
+
+void __exit vfio_ap_exit(void)
+{
+ ap_driver_unregister(&vfio_ap_drv);
+ vfio_ap_matrix_dev_destroy(ap_matrix);
+}
+
+module_init(vfio_ap_init);
+module_exit(vfio_ap_exit);
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
new file mode 100644
index 0000000..3505947
--- /dev/null
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -0,0 +1,20 @@
+/*
+ * Private data and functions for adjunct processor VFIO matrix driver.
+ *
+ * Copyright IBM Corp. 2017
+ * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
+ */
+
+#ifndef _VFIO_AP_PRIVATE_H_
+#define _VFIO_AP_PRIVATE_H_
+
+#include <linux/types.h>
+
+#define VFIO_AP_MODULE_NAME "vfio_ap"
+#define VFIO_AP_DRV_NAME "vfio_ap"
+
+struct ap_matrix {
+ struct device device;
+};
+
+#endif /* _VFIO_AP_PRIVATE_H_ */
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index e3301db..cf2a5e9 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -200,6 +200,7 @@ struct vfio_device_info {
#define VFIO_DEVICE_FLAGS_PLATFORM (1 << 2) /* vfio-platform device */
#define VFIO_DEVICE_FLAGS_AMBA (1 << 3) /* vfio-amba device */
#define VFIO_DEVICE_FLAGS_CCW (1 << 4) /* vfio-ccw device */
+#define VFIO_DEVICE_FLAGS_AP (1 << 5) /* vfio-ap device */
__u32 num_regions; /* Max region index + 1 */
__u32 num_irqs; /* Max IRQ index + 1 */
};
@@ -215,6 +216,7 @@ struct vfio_device_info {
#define VFIO_DEVICE_API_PLATFORM_STRING "vfio-platform"
#define VFIO_DEVICE_API_AMBA_STRING "vfio-amba"
#define VFIO_DEVICE_API_CCW_STRING "vfio-ccw"
+#define VFIO_DEVICE_API_AP_STRING "vfio-ap"
/**
* VFIO_DEVICE_GET_REGION_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 8,
--
1.7.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 06/15] s390: vfio-ap: register matrix device with VFIO mdev framework
2018-02-27 14:27 [PATCH v2 00/15] s390: vfio-ap: guest dedicated crypto adapters Tony Krowiak
` (4 preceding siblings ...)
2018-02-27 14:28 ` [PATCH v2 05/15] s390: vfio-ap: base implementation of VFIO AP device driver Tony Krowiak
@ 2018-02-27 14:28 ` Tony Krowiak
2018-02-27 14:28 ` [PATCH v2 07/15] KVM: s390: Interfaces to configure/deconfigure guest's AP matrix Tony Krowiak
` (9 subsequent siblings)
15 siblings, 0 replies; 56+ messages in thread
From: Tony Krowiak @ 2018-02-27 14:28 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen, Tony Krowiak
Registers the matrix device created by the VFIO AP device
driver with the VFIO mediated device framework.
Registering the matrix device will create the sysfs
structures needed to create mediated matrix devices
each of which will be used to configure the AP matrix
for a guest and connect it to the VFIO AP device driver.
Registering the matrix device with the VFIO mediated device
framework will create the following sysfs structures:
/sys/devices/vfio_ap
... [matrix]
...... [mdev_supported_types]
......... [vfio_ap-passthrough]
............ create
To create a mediated device for the AP matrix device, write a UUID
to the create file:
uuidgen > create
A symbolic link to the mediated device's directory will be created in the
devices subdirectory named after the generated $uuid:
/sys/devices/vfio_ap
... [matrix]
...... [mdev_supported_types]
......... [vfio_ap-passthrough]
............ [devices]
............... [$uuid]
Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
MAINTAINERS | 1 +
drivers/s390/crypto/Makefile | 2 +-
drivers/s390/crypto/vfio_ap_drv.c | 9 +++
drivers/s390/crypto/vfio_ap_ops.c | 105 +++++++++++++++++++++++++++++++++
drivers/s390/crypto/vfio_ap_private.h | 18 ++++++
5 files changed, 134 insertions(+), 1 deletions(-)
create mode 100644 drivers/s390/crypto/vfio_ap_ops.c
diff --git a/MAINTAINERS b/MAINTAINERS
index a2f232d..fb9e0d4 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -11886,6 +11886,7 @@ F: arch/s390/include/asm/kvm/kvm-ap.h
F: arch/s390/kvm/kvm-ap.c
F: drivers/s390/crypto/vfio_ap_drv.c
F: drivers/s390/crypto/vfio_ap_private.h
+F: drivers/s390/crypto/vfio_ap_ops.c
S390 ZFCP DRIVER
M: Steffen Maier <maier@linux.vnet.ibm.com>
diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile
index 48e466e..8d36b05 100644
--- a/drivers/s390/crypto/Makefile
+++ b/drivers/s390/crypto/Makefile
@@ -17,5 +17,5 @@ pkey-objs := pkey_api.o
obj-$(CONFIG_PKEY) += pkey.o
# adjunct processor matrix
-vfio_ap-objs := vfio_ap_drv.o
+vfio_ap-objs := vfio_ap_drv.o vfio_ap_ops.o
obj-$(CONFIG_VFIO_AP) += vfio_ap.o
diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c
index 8bb72af..f45f83b 100644
--- a/drivers/s390/crypto/vfio_ap_drv.c
+++ b/drivers/s390/crypto/vfio_ap_drv.c
@@ -121,11 +121,20 @@ int __init vfio_ap_init(void)
return ret;
}
+ ret = vfio_ap_mdev_register(ap_matrix);
+ if (ret) {
+ ap_driver_unregister(&vfio_ap_drv);
+ vfio_ap_matrix_dev_destroy(ap_matrix);
+
+ return ret;
+ }
+
return 0;
}
void __exit vfio_ap_exit(void)
{
+ vfio_ap_mdev_unregister(ap_matrix);
ap_driver_unregister(&vfio_ap_drv);
vfio_ap_matrix_dev_destroy(ap_matrix);
}
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
new file mode 100644
index 0000000..4292a5e
--- /dev/null
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -0,0 +1,105 @@
+/*
+ * Adjunct processor matrix VFIO device driver callbacks.
+ *
+ * Copyright IBM Corp. 2017
+ * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
+ *
+ */
+#include <linux/string.h>
+#include <linux/vfio.h>
+#include <linux/device.h>
+#include <linux/list.h>
+#include <linux/ctype.h>
+
+#include "vfio_ap_private.h"
+
+#define VFOP_AP_MDEV_TYPE_HWVIRT "passthrough"
+#define VFIO_AP_MDEV_NAME_HWVIRT "VFIO AP Passthrough Device"
+
+static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev)
+{
+ struct ap_matrix *ap_matrix = to_ap_matrix(mdev_parent_dev(mdev));
+
+ ap_matrix->available_instances--;
+
+ return 0;
+}
+
+static int vfio_ap_mdev_remove(struct mdev_device *mdev)
+{
+ struct ap_matrix *ap_matrix = to_ap_matrix(mdev_parent_dev(mdev));
+
+ ap_matrix->available_instances++;
+
+ return 0;
+}
+
+static ssize_t name_show(struct kobject *kobj, struct device *dev, char *buf)
+{
+ return sprintf(buf, "%s\n", VFIO_AP_MDEV_NAME_HWVIRT);
+}
+
+MDEV_TYPE_ATTR_RO(name);
+
+static ssize_t available_instances_show(struct kobject *kobj,
+ struct device *dev, char *buf)
+{
+ struct ap_matrix *ap_matrix;
+
+ ap_matrix = to_ap_matrix(dev);
+
+ return sprintf(buf, "%d\n", ap_matrix->available_instances);
+}
+
+MDEV_TYPE_ATTR_RO(available_instances);
+
+static ssize_t device_api_show(struct kobject *kobj, struct device *dev,
+ char *buf)
+{
+ return sprintf(buf, "%s\n", VFIO_DEVICE_API_AP_STRING);
+}
+
+MDEV_TYPE_ATTR_RO(device_api);
+
+static struct attribute *vfio_ap_mdev_type_attrs[] = {
+ &mdev_type_attr_name.attr,
+ &mdev_type_attr_device_api.attr,
+ &mdev_type_attr_available_instances.attr,
+ NULL,
+};
+
+static struct attribute_group vfio_ap_mdev_hwvirt_type_group = {
+ .name = VFOP_AP_MDEV_TYPE_HWVIRT,
+ .attrs = vfio_ap_mdev_type_attrs,
+};
+
+static struct attribute_group *vfio_ap_mdev_type_groups[] = {
+ &vfio_ap_mdev_hwvirt_type_group,
+ NULL,
+};
+
+static const struct mdev_parent_ops vfio_ap_matrix_ops = {
+ .owner = THIS_MODULE,
+ .supported_type_groups = vfio_ap_mdev_type_groups,
+ .create = vfio_ap_mdev_create,
+ .remove = vfio_ap_mdev_remove,
+};
+
+int vfio_ap_mdev_register(struct ap_matrix *ap_matrix)
+{
+ int ret;
+
+ ret = mdev_register_device(&ap_matrix->device, &vfio_ap_matrix_ops);
+ if (ret)
+ return ret;
+
+ ap_matrix->available_instances = AP_MATRIX_MAX_AVAILABLE_INSTANCES;
+
+ return 0;
+}
+
+void vfio_ap_mdev_unregister(struct ap_matrix *ap_matrix)
+{
+ ap_matrix->available_instances--;
+ mdev_unregister_device(&ap_matrix->device);
+}
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index 3505947..c264415 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -9,12 +9,30 @@
#define _VFIO_AP_PRIVATE_H_
#include <linux/types.h>
+#include <linux/mdev.h>
+#include "ap_bus.h"
#define VFIO_AP_MODULE_NAME "vfio_ap"
#define VFIO_AP_DRV_NAME "vfio_ap"
+#define VFIO_AP_MATRIX_MODULE_NAME "vfio_ap_matrix"
+/**
+ * There must be one mediated matrix device per guest. If every APQN is assigned
+ * to a guest, then the maximum number of guests with a unique APQN assigned
+ * would be 255 adapters x 255 domains = 72351 guests.
+ */
+#define AP_MATRIX_MAX_AVAILABLE_INSTANCES 72351
struct ap_matrix {
struct device device;
+ int available_instances;
};
+static inline struct ap_matrix *to_ap_matrix(struct device *dev)
+{
+ return container_of(dev, struct ap_matrix, device);
+}
+
+extern int vfio_ap_mdev_register(struct ap_matrix *ap_matrix);
+extern void vfio_ap_mdev_unregister(struct ap_matrix *ap_matrix);
+
#endif /* _VFIO_AP_PRIVATE_H_ */
--
1.7.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 07/15] KVM: s390: Interfaces to configure/deconfigure guest's AP matrix
2018-02-27 14:27 [PATCH v2 00/15] s390: vfio-ap: guest dedicated crypto adapters Tony Krowiak
` (5 preceding siblings ...)
2018-02-27 14:28 ` [PATCH v2 06/15] s390: vfio-ap: register matrix device with VFIO mdev framework Tony Krowiak
@ 2018-02-27 14:28 ` Tony Krowiak
2018-02-28 16:15 ` Pierre Morel
` (3 more replies)
2018-02-27 14:28 ` [PATCH v2 08/15] KVM: s390: interface to enable AP execution mode Tony Krowiak
` (8 subsequent siblings)
15 siblings, 4 replies; 56+ messages in thread
From: Tony Krowiak @ 2018-02-27 14:28 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen, Tony Krowiak
Provides interfaces to assign AP adapters, usage domains
and control domains to a KVM guest.
A KVM guest is started by executing the Start Interpretive Execution (SIE)
instruction. The SIE state description is a control block that contains the
state information for a KVM guest and is supplied as input to the SIE
instruction. The SIE state description has a satellite structure called the
Crypto Control Block (CRYCB). The CRYCB contains three bitmask fields
identifying the adapters, queues (domains) and control domains assigned to
the KVM guest:
* The AP Adapter Mask (APM) field identifies the AP adapters assigned to
the KVM guest
* The AP Queue Mask (AQM) field identifies the AP queues assigned to
the KVM guest. Each AP queue is connected to a usage domain within
an AP adapter.
* The AP Domain Mask (ADM) field identifies the control domains
assigned to the KVM guest.
Each adapter, queue (usage domain) and control domain are identified by
a number from 0 to 255. The bits in each mask, from most significant to
least significant bit, correspond to the numbers 0-255. When a bit is
set, the corresponding adapter, queue (usage domain) or control domain
is assigned to the KVM guest.
This patch will set the bits in the APM, AQM and ADM fields of the
CRYCB referenced by the KVM guest's SIE state description. The process
used is:
1. Verify that the bits to be set do not exceed the maximum bit
number for the given mask.
2. Verify that the APQNs that can be derived from the intersection
of the bits set in the APM and AQM fields of the KVM guest's CRYCB
are not assigned to any other KVM guest running on the same linux
host.
3. Set the APM, AQM and ADM in the CRYCB according to the matrix
configured for the mediated matrix device via its sysfs
adapter, domain and control domain attribute files respectively.
Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
arch/s390/include/asm/kvm-ap.h | 36 +++++
arch/s390/kvm/kvm-ap.c | 257 +++++++++++++++++++++++++++++++++
drivers/s390/crypto/vfio_ap_ops.c | 19 +++
drivers/s390/crypto/vfio_ap_private.h | 4 +
4 files changed, 316 insertions(+), 0 deletions(-)
diff --git a/arch/s390/include/asm/kvm-ap.h b/arch/s390/include/asm/kvm-ap.h
index ef749e7..46e7c5b 100644
--- a/arch/s390/include/asm/kvm-ap.h
+++ b/arch/s390/include/asm/kvm-ap.h
@@ -10,9 +10,45 @@
#define _ASM_KVM_AP
#include <linux/types.h>
#include <linux/kvm_host.h>
+#include <linux/types.h>
+#include <linux/kvm_host.h>
+#include <linux/bitops.h>
+
+#define KVM_AP_MASK_BYTES(n)(n / BITS_PER_BYTE)
+
+/**
+ * The AP matrix is comprised of three bit masks identifying the adapters,
+ * queues (domains) and control domains that belong to an AP matrix. The bits in
+ * each mask, from least significant to most significant bit, correspond to IDs
+ * 0 to the maximum ID allowed for a given mask. When a bit is set, the
+ * corresponding ID belongs to the matrix.
+ *
+ * @apm_max: max number of bits in @apm
+ * @apm identifies the AP adapters in the matrix
+ * @aqm_max: max number of bits in @aqm
+ * @aqm identifies the AP queues (domains) in the matrix
+ * @adm_max: max number of bits in @adm
+ * @adm identifies the AP control domains in the matrix
+ */
+struct kvm_ap_matrix {
+ int apm_max;
+ unsigned long *apm;
+ int aqm_max;
+ unsigned long *aqm;
+ int adm_max;
+ unsigned long *adm;
+};
void kvm_ap_set_crycb_format(struct kvm *kvm, __u32 *crycbd);
int kvm_ap_get_crycb_format(struct kvm *kvm);
+int kvm_ap_matrix_create(struct kvm_ap_matrix **ap_matrix);
+
+void kvm_ap_matrix_destroy(struct kvm_ap_matrix *ap_matrix);
+
+int kvm_ap_configure_matrix(struct kvm *kvm, struct kvm_ap_matrix *matrix);
+
+void kvm_ap_deconfigure_matrix(struct kvm *kvm);
+
#endif /* _ASM_KVM_AP */
diff --git a/arch/s390/kvm/kvm-ap.c b/arch/s390/kvm/kvm-ap.c
index bafe63b..bb29045 100644
--- a/arch/s390/kvm/kvm-ap.c
+++ b/arch/s390/kvm/kvm-ap.c
@@ -8,6 +8,7 @@
#include <asm/kvm-ap.h>
#include <asm/ap.h>
+#include <linux/bitops.h>
#include "kvm-s390.h"
@@ -16,6 +17,125 @@ int kvm_ap_get_crycb_format(struct kvm *kvm)
return kvm->arch.crypto.crycbd & CRYCB_FORMAT_MASK;
}
+static inline void kvm_ap_clear_crycb_masks(struct kvm *kvm)
+{
+ int crycb_fmt = kvm_ap_get_crycb_format(kvm);
+
+ if (crycb_fmt == CRYCB_FORMAT2)
+ memset(&kvm->arch.crypto.crycb->apcb1, 0,
+ sizeof(kvm->arch.crypto.crycb->apcb1));
+ else
+ memset(&kvm->arch.crypto.crycb->apcb0, 0,
+ sizeof(kvm->arch.crypto.crycb->apcb0));
+}
+
+static inline unsigned long *kvm_ap_get_crycb_apm(struct kvm *kvm)
+{
+ unsigned long *apm;
+ int crycb_fmt = kvm_ap_get_crycb_format(kvm);
+
+ if (crycb_fmt == CRYCB_FORMAT2)
+ apm = (unsigned long *)kvm->arch.crypto.crycb->apcb1.apm;
+ else
+ apm = (unsigned long *)kvm->arch.crypto.crycb->apcb0.apm;
+
+ return apm;
+}
+
+static inline unsigned long *kvm_ap_get_crycb_aqm(struct kvm *kvm)
+{
+ unsigned long *aqm;
+ int crycb_fmt = kvm_ap_get_crycb_format(kvm);
+
+ if (crycb_fmt == CRYCB_FORMAT2)
+ aqm = (unsigned long *)kvm->arch.crypto.crycb->apcb1.aqm;
+ else
+ aqm = (unsigned long *)kvm->arch.crypto.crycb->apcb0.aqm;
+
+ return aqm;
+}
+
+static inline unsigned long *kvm_ap_get_crycb_adm(struct kvm *kvm)
+{
+ unsigned long *adm;
+ int crycb_fmt = kvm_ap_get_crycb_format(kvm);
+
+ if (crycb_fmt == CRYCB_FORMAT2)
+ adm = (unsigned long *)kvm->arch.crypto.crycb->apcb1.adm;
+ else
+ adm = (unsigned long *)kvm->arch.crypto.crycb->apcb0.adm;
+
+ return adm;
+}
+
+static void kvm_ap_set_crycb_masks(struct kvm *kvm,
+ struct kvm_ap_matrix *matrix)
+{
+ unsigned long *apm = kvm_ap_get_crycb_apm(kvm);
+ unsigned long *aqm = kvm_ap_get_crycb_aqm(kvm);
+ unsigned long *adm = kvm_ap_get_crycb_adm(kvm);
+
+ kvm_ap_clear_crycb_masks(kvm);
+ memcpy(apm, matrix->apm, KVM_AP_MASK_BYTES(matrix->apm_max));
+ memcpy(aqm, matrix->aqm, KVM_AP_MASK_BYTES(matrix->aqm_max));
+
+ /*
+ * Merge the AQM and ADM since the ADM is a superset of the
+ * AQM by architectural convention.
+ */
+ bitmap_or(adm, adm, aqm, matrix->adm_max);
+}
+
+static void kvm_ap_log_sharing_err(struct kvm *kvm, unsigned long apid,
+ unsigned long apqi)
+{
+ pr_err("%s: AP queue %02lx.%04lx is registered to guest %s", __func__,
+ apid, apqi, kvm->arch.dbf->name);
+}
+
+/**
+ * kvm_ap_validate_queue_sharing
+ *
+ * Verifies that the APQNs derived from the intersection of the AP adapter IDs
+ * and AP queue indexes comprising the AP matrix are not configured for
+ * another guest. AP queue sharing is not allowed.
+ *
+ * @kvm: the KVM guest
+ * @matrix: the AP matrix
+ *
+ * Returns 0 if the APQNs are valid, otherwise; returns -EBUSY.
+ */
+static int kvm_ap_validate_queue_sharing(struct kvm *kvm,
+ struct kvm_ap_matrix *matrix)
+{
+ struct kvm *vm;
+ unsigned long *apm, *aqm;
+ unsigned long apid, apqi;
+
+
+ /* No other VM may share an AP Queue with the input VM */
+ list_for_each_entry(vm, &vm_list, vm_list) {
+ if (kvm == vm)
+ continue;
+
+ apm = kvm_ap_get_crycb_apm(vm);
+ if (!bitmap_and(apm, apm, matrix->apm, matrix->apm_max))
+ continue;
+
+ aqm = kvm_ap_get_crycb_aqm(vm);
+ if (!bitmap_and(aqm, aqm, matrix->aqm, matrix->aqm_max))
+ continue;
+
+ for_each_set_bit_inv(apid, apm, matrix->apm_max)
+ for_each_set_bit_inv(apqi, aqm, matrix->aqm_max)
+ kvm_ap_log_sharing_err(kvm, apid, apqi);
+
+ return -EBUSY;
+ }
+
+ return 0;
+}
+
static int kvm_ap_apxa_installed(void)
{
int ret;
@@ -50,3 +170,140 @@ void kvm_ap_set_crycb_format(struct kvm *kvm, __u32 *crycbd)
*crycbd |= CRYCB_FORMAT1;
}
}
+
+static int kvm_ap_matrix_apm_create(struct kvm_ap_matrix *ap_matrix, int apxa)
+{
+ if (apxa)
+ ap_matrix->apm_max = 256;
+ else
+ ap_matrix->apm_max = 64;
+
+ ap_matrix->apm = kzalloc(KVM_AP_MASK_BYTES(ap_matrix->apm_max),
+ GFP_KERNEL);
+ if (!ap_matrix->apm)
+ return -ENOMEM;
+
+ return 0;
+}
+
+static int kvm_ap_matrix_aqm_create(struct kvm_ap_matrix *ap_matrix, int apxa)
+{
+ if (apxa)
+ ap_matrix->aqm_max = 256;
+ else
+ ap_matrix->aqm_max = 16;
+
+ ap_matrix->aqm = kzalloc(KVM_AP_MASK_BYTES(ap_matrix->aqm_max),
+ GFP_KERNEL);
+ if (!ap_matrix->aqm)
+ return -ENOMEM;
+
+ return 0;
+}
+
+static int kvm_ap_matrix_adm_create(struct kvm_ap_matrix *ap_matrix, int apxa)
+{
+ if (apxa)
+ ap_matrix->adm_max = 256;
+ else
+ ap_matrix->adm_max = 16;
+
+ ap_matrix->adm = kzalloc(KVM_AP_MASK_BYTES(ap_matrix->adm_max),
+ GFP_KERNEL);
+ if (!ap_matrix->adm)
+ return -ENOMEM;
+
+ return 0;
+}
+
+static void kvm_ap_matrix_masks_destroy(struct kvm_ap_matrix *ap_matrix)
+{
+ kfree(ap_matrix->apm);
+ kfree(ap_matrix->aqm);
+ kfree(ap_matrix->adm);
+}
+
+int kvm_ap_matrix_create(struct kvm_ap_matrix **ap_matrix)
+{
+ int ret;
+ int apxa = kvm_ap_apxa_installed();
+ struct kvm_ap_matrix *matrix;
+
+ matrix = kzalloc(sizeof(*matrix), GFP_KERNEL);
+ if (!matrix)
+ return -ENOMEM;
+
+ ret = kvm_ap_matrix_apm_create(matrix, apxa);
+ if (ret)
+ goto mask_create_err;
+
+ ret = kvm_ap_matrix_aqm_create(matrix, apxa);
+ if (ret)
+ goto mask_create_err;
+
+ ret = kvm_ap_matrix_adm_create(matrix, apxa);
+ if (ret)
+ goto mask_create_err;
+
+ *ap_matrix = matrix;
+
+ return 0;
+
+mask_create_err:
+ kvm_ap_matrix_masks_destroy(matrix);
+ kfree(matrix);
+ return ret;
+}
+EXPORT_SYMBOL(kvm_ap_matrix_create);
+
+void kvm_ap_matrix_destroy(struct kvm_ap_matrix *ap_matrix)
+{
+ kvm_ap_matrix_masks_destroy(ap_matrix);
+ kfree(ap_matrix);
+}
+EXPORT_SYMBOL(kvm_ap_matrix_destroy);
+
+/**
+ * kvm_ap_configure_matrix
+ *
+ * Configure the AP matrix for a KVM guest.
+ *
+ * @kvm: the KVM guest
+ * @matrix: the matrix configuration information
+ *
+ * Returns 0 if the APQNs derived from the intersection of the set of adapter
+ * IDs (APM) and queue indexes (AQM) in @matrix are not configured for any
+ * other KVM guest running on the same linux host. Otherwise returns an error
+ * code.
+ */
+int kvm_ap_configure_matrix(struct kvm *kvm, struct kvm_ap_matrix *matrix)
+{
+ int ret = 0;
+
+ mutex_lock(&kvm->lock);
+
+ ret = kvm_ap_validate_queue_sharing(kvm, matrix);
+ if (ret)
+ return ret;
+
+ kvm_ap_set_crycb_masks(kvm, matrix);
+
+ mutex_unlock(&kvm->lock);
+
+ return 0;
+}
+EXPORT_SYMBOL(kvm_ap_configure_matrix);
+
+/**
+ * kvm_ap_deconfigure_matrix
+ *
+ * Deconfigure the AP matrix for a KVM guest. Clears all of the bits in the
+ * APM, AQM and ADM in the guest's CRYCB.
+ *
+ * @kvm: the KVM guest
+ */
+void kvm_ap_deconfigure_matrix(struct kvm *kvm)
+{
+ kvm_ap_clear_crycb_masks(kvm);
+}
+EXPORT_SYMBOL(kvm_ap_deconfigure_matrix);
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 4292a5e..4fda44e 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -10,6 +10,7 @@
#include <linux/device.h>
#include <linux/list.h>
#include <linux/ctype.h>
+#include <asm/kvm-ap.h>
#include "vfio_ap_private.h"
@@ -18,8 +19,23 @@
static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev)
{
+ int ret;
+ struct ap_matrix_mdev *matrix_mdev;
struct ap_matrix *ap_matrix = to_ap_matrix(mdev_parent_dev(mdev));
+ struct kvm_ap_matrix *matrix;
+
+ ret = kvm_ap_matrix_create(&matrix);
+ if (ret)
+ return ret;
+
+ matrix_mdev = kzalloc(sizeof(*matrix_mdev), GFP_KERNEL);
+ if (!matrix_mdev) {
+ kvm_ap_matrix_destroy(matrix);
+ return -ENOMEM;
+ }
+ matrix_mdev->matrix = matrix;
+ mdev_set_drvdata(mdev, matrix_mdev);
ap_matrix->available_instances--;
return 0;
@@ -28,7 +44,10 @@ static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev)
static int vfio_ap_mdev_remove(struct mdev_device *mdev)
{
struct ap_matrix *ap_matrix = to_ap_matrix(mdev_parent_dev(mdev));
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ kvm_ap_matrix_destroy(matrix_mdev->matrix);
+ kfree(matrix_mdev);
ap_matrix->available_instances++;
return 0;
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index c264415..522564e 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -27,6 +27,10 @@ struct ap_matrix {
int available_instances;
};
+struct ap_matrix_mdev {
+ struct kvm_ap_matrix *matrix;
+};
+
static inline struct ap_matrix *to_ap_matrix(struct device *dev)
{
return container_of(dev, struct ap_matrix, device);
--
1.7.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 08/15] KVM: s390: interface to enable AP execution mode
2018-02-27 14:27 [PATCH v2 00/15] s390: vfio-ap: guest dedicated crypto adapters Tony Krowiak
` (6 preceding siblings ...)
2018-02-27 14:28 ` [PATCH v2 07/15] KVM: s390: Interfaces to configure/deconfigure guest's AP matrix Tony Krowiak
@ 2018-02-27 14:28 ` Tony Krowiak
2018-02-28 9:42 ` David Hildenbrand
2018-02-28 9:44 ` David Hildenbrand
2018-02-27 14:28 ` [PATCH v2 09/15] s390: vfio-ap: sysfs interfaces to configure adapters Tony Krowiak
` (7 subsequent siblings)
15 siblings, 2 replies; 56+ messages in thread
From: Tony Krowiak @ 2018-02-27 14:28 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen, Tony Krowiak
Introduces a new interface to enable AP interpretive
execution (IE) mode for the KVM guest. When running
with IE mode enabled, AP instructions executed on the
KVM guest will be interpreted by the firmware and
passed directly through to an AP device installed on
the system. The CPU model feature for AP must
be enabled for the KVM guest in order to enable
interpretive execution mode.
This interface will be used in a subsequent patch
by the VFIO AP device driver.
Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
arch/s390/include/asm/kvm-ap.h | 2 ++
arch/s390/include/asm/kvm_host.h | 1 +
arch/s390/kvm/kvm-ap.c | 27 +++++++++++++++++++++++++++
arch/s390/kvm/kvm-s390.h | 1 +
4 files changed, 31 insertions(+), 0 deletions(-)
diff --git a/arch/s390/include/asm/kvm-ap.h b/arch/s390/include/asm/kvm-ap.h
index 46e7c5b..6bd6bfb 100644
--- a/arch/s390/include/asm/kvm-ap.h
+++ b/arch/s390/include/asm/kvm-ap.h
@@ -51,4 +51,6 @@ struct kvm_ap_matrix {
void kvm_ap_deconfigure_matrix(struct kvm *kvm);
+int kvm_ap_enable_ie_mode(struct kvm *kvm);
+
#endif /* _ASM_KVM_AP */
diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index a4c77d3..1eebdd6 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -186,6 +186,7 @@ struct kvm_s390_sie_block {
#define ECA_AIV 0x00200000
#define ECA_VX 0x00020000
#define ECA_PROTEXCI 0x00002000
+#define ECA_APIE 0x00000008
#define ECA_SII 0x00000001
__u32 eca; /* 0x004c */
#define ICPT_INST 0x04
diff --git a/arch/s390/kvm/kvm-ap.c b/arch/s390/kvm/kvm-ap.c
index bb29045..862e54b 100644
--- a/arch/s390/kvm/kvm-ap.c
+++ b/arch/s390/kvm/kvm-ap.c
@@ -307,3 +307,30 @@ void kvm_ap_deconfigure_matrix(struct kvm *kvm)
kvm_ap_clear_crycb_masks(kvm);
}
EXPORT_SYMBOL(kvm_ap_deconfigure_matrix);
+
+/**
+ * kvm_ap_enable_ie_mode
+ *
+ * Enable interpretrive execution of AP instructions for the guest. When
+ * enabled, AP instructions executed on the guest will be interpreted and
+ * passed through to an AP installed on the host system.
+ *
+ * Returns 0 if interpretrive execution is enabled. Returns -EOPNOTSUPP
+ * if AP facilities are not installed for the guest.
+ *
+ * @kvm: the guest's kvm structure
+ */
+int kvm_ap_enable_ie_mode(struct kvm *kvm)
+{
+ int i;
+ struct kvm_vcpu *vcpu;
+
+ if (!test_kvm_cpu_feat(kvm, KVM_S390_VM_CPU_FEAT_AP))
+ return -EOPNOTSUPP;
+
+ kvm_for_each_vcpu(i, vcpu, kvm)
+ vcpu->arch.sie_block->eca |= ECA_APIE;
+
+ return 0;
+}
+EXPORT_SYMBOL(kvm_ap_enable_ie_mode);
diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
index 1b5621f..3142541 100644
--- a/arch/s390/kvm/kvm-s390.h
+++ b/arch/s390/kvm/kvm-s390.h
@@ -18,6 +18,7 @@
#include <asm/facility.h>
#include <asm/processor.h>
#include <asm/sclp.h>
+#include <asm/ap.h>
/* Transactional Memory Execution related macros */
#define IS_TE_ENABLED(vcpu) ((vcpu->arch.sie_block->ecb & ECB_TE))
--
1.7.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 09/15] s390: vfio-ap: sysfs interfaces to configure adapters
2018-02-27 14:27 [PATCH v2 00/15] s390: vfio-ap: guest dedicated crypto adapters Tony Krowiak
` (7 preceding siblings ...)
2018-02-27 14:28 ` [PATCH v2 08/15] KVM: s390: interface to enable AP execution mode Tony Krowiak
@ 2018-02-27 14:28 ` Tony Krowiak
2018-02-27 14:28 ` [PATCH v2 10/15] s390: vfio-ap: sysfs interfaces to configure domains Tony Krowiak
` (6 subsequent siblings)
15 siblings, 0 replies; 56+ messages in thread
From: Tony Krowiak @ 2018-02-27 14:28 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen, Tony Krowiak
Provides the sysfs interfaces for assigning AP adapters to
and unassigning AP adapters from a mediated matrix device.
The IDs of the AP adapters assigned to the mediated matrix
device are stored in an AP mask (APM). The bits in the APM,
from most significant to least significant bit, correspond to
AP adapter numbers 0 to 255. When an adapter is assigned, the
bit corresponding adapter ID will be set in the APM. Likewise,
when an adapter is unassigned, the bit corresponding to the
adapter ID will be cleared from the APM.
The relevant sysfs structures are:
/sys/devices/vfio_ap
... [matrix]
...... [mdev_supported_types]
......... [vfio_ap-passthrough]
............ [devices]
...............[$uuid]
.................. assign_adapter
.................. unassign_adapter
To assign an adapter to the $uuid mediated matrix device's APM,
write the adapter ID (APID) to the assign_adapter file. To
unassign an adapter, write the APID to the unassign_adapter
file. The APID is specified using conventional semantics: If
it begins with 0x the number will be parsed as a hexadecimal
(case insensitive) number; otherwise, it will be parsed as a
decimal number.
For example, to assign adapter 173 (0xad) to the mediated matrix
device $uuid:
echo 173 > assign_adapter
or
echo 0xad > assign_adapter
To unassign adapter 173 (0xad):
echo 173 > unassign_adapter
or
echo 0xad > unassign_adapter
The assignment will be rejected:
* If the APID exceeds the maximum value for an AP adapter:
* If the AP Extended Addressing (APXA) facility is
installed, the max value is 255
* Else the max value is 64
* If no AP domains have yet been assigned and there are
no AP queues bound to the VFIO AP driver that have an APQN
with an APID matching that of the AP adapter being assigned.
* If any of the APQNs that can be derived from the intersection
of the APID being assigned and the AP queue index (APQI) of
each of the AP domains previously assigned can not be matched
with an APQN of an AP queue device reserved by the VFIO AP
driver.
Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
arch/s390/include/asm/kvm-ap.h | 1 +
drivers/s390/crypto/vfio_ap_ops.c | 296 +++++++++++++++++++++++++++++++++
drivers/s390/crypto/vfio_ap_private.h | 1 +
3 files changed, 298 insertions(+), 0 deletions(-)
diff --git a/arch/s390/include/asm/kvm-ap.h b/arch/s390/include/asm/kvm-ap.h
index 6bd6bfb..f660aaf 100644
--- a/arch/s390/include/asm/kvm-ap.h
+++ b/arch/s390/include/asm/kvm-ap.h
@@ -15,6 +15,7 @@
#include <linux/bitops.h>
#define KVM_AP_MASK_BYTES(n)(n / BITS_PER_BYTE)
+#define KVM_AP_MAX_APM_INDEX(matrix)(matrix->apm_max - 1)
/**
* The AP matrix is comprised of three bit masks identifying the adapters,
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 4fda44e..90512a6 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -97,9 +97,305 @@ static ssize_t device_api_show(struct kobject *kobj, struct device *dev,
NULL,
};
+struct vfio_apid_reserved {
+ unsigned long apid;
+ int reserved;
+};
+
+struct vfio_ap_qid_match {
+ qid_t qid;
+ struct device *dev;
+};
+
+/**
+ * vfio_ap_queue_match
+ *
+ * @dev: an AP queue device that has been reserved by the VFIO AP device
+ * driver
+ * @data: an AP queue identifier
+ *
+ * Returns 1 (true) if @data matches the AP queue identifier specified for @dev;
+ * otherwise, returns 0 (false);
+ */
+static int vfio_ap_queue_match(struct device *dev, void *data)
+{
+ struct vfio_ap_qid_match *qid_match = data;
+ struct ap_queue *ap_queue;
+
+ ap_queue = to_ap_queue(dev);
+ if (ap_queue->qid == qid_match->qid)
+ qid_match->dev = dev;
+
+ return 0;
+}
+
+/**
+ * vfio_ap_validate_queues_for_apid
+ *
+ * @ap_matrix: the matrix device
+ * @matrix_mdev: the mediated matrix device
+ * @apid: an AP adapter ID (APID)
+ *
+ * Verifies that each APQN that is derived from the intersection of @apid and
+ * each AP queue index (APQI) corresponding to an AP adapter assigned to the
+ * @matrix_mdev matches the APQN of an AP queue reserved by the VFIO AP device
+ * driver.
+ *
+ * Returns 0 if validation succeeds; otherwise, returns an error.
+ */
+static int vfio_ap_validate_queues_for_apid(struct ap_matrix *ap_matrix,
+ struct ap_matrix_mdev *matrix_mdev,
+ unsigned long apid)
+{
+ int ret;
+ struct vfio_ap_qid_match qid_match;
+ unsigned long apqi;
+ struct device_driver *drv = ap_matrix->device.driver;
+
+ /**
+ * Examine each APQN with the specified APID
+ */
+ for_each_set_bit_inv(apqi, matrix_mdev->matrix->aqm,
+ matrix_mdev->matrix->apm_max) {
+ qid_match.qid = AP_MKQID(apid, apqi);
+ qid_match.dev = NULL;
+
+ ret = driver_for_each_device(drv, NULL, &qid_match,
+ vfio_ap_queue_match);
+ if (ret)
+ return ret;
+
+ /*
+ * If the APQN identifies an AP queue that is reserved by the
+ * VFIO AP device driver, continue processing.
+ */
+ if (qid_match.dev)
+ continue;
+
+ pr_err("%s: AP queue %02lx.%04lx not reserved by %s driver",
+ VFIO_AP_MATRIX_MODULE_NAME, apid, apqi,
+ VFIO_AP_DRV_NAME);
+
+ return -ENXIO;
+ }
+
+ return 0;
+}
+
+struct vfio_ap_apid_reserved {
+ unsigned long apid;
+ bool reserved;
+};
+
+/**
+ * vfio_ap_queue_id_contains_apid
+ *
+ * @dev: an AP queue device
+ * @data: an AP adapter ID (APID)
+ *
+ * Returns 1 (true) if the APID (@data) is contained in the AP queue's (@data)
+ * identifier; otherwise, returns 0;
+ */
+static int vfio_ap_queue_id_contains_apid(struct device *dev, void *data)
+{
+ struct vfio_ap_apid_reserved *apid_res = data;
+ struct ap_queue *ap_queue = to_ap_queue(dev);
+
+ if (apid_res->apid == AP_QID_CARD(ap_queue->qid))
+ apid_res->reserved = true;
+
+ return 0;
+}
+
+/**
+ * vfio_ap_verify_apid_reserved
+ *
+ * @ap_matrix: the AP matrix configured for the mediated matrix device
+ * @apid: the AP adapter ID
+ *
+ * Verifies that at least one AP queue reserved by the VFIO AP device driver
+ * has an APQN containing @apid.
+ *
+ * Returns 0 if the APID is reserved; otherwise, returns -ENODEV.
+ */
+static int vfio_ap_verify_apid_reserved(struct ap_matrix *ap_matrix,
+ unsigned long apid)
+{
+ int ret;
+ struct vfio_ap_apid_reserved apid_res;
+
+ apid_res.apid = apid;
+
+ ret = driver_for_each_device(ap_matrix->device.driver, NULL,
+ &apid_res,
+ vfio_ap_queue_id_contains_apid);
+ if (ret)
+ return ret;
+
+ if (apid_res.reserved)
+ return 0;
+
+ pr_err("%s: no APQNs with adapter ID %02lx are reserved by %s driver",
+ VFIO_AP_MATRIX_MODULE_NAME, apid, VFIO_AP_DRV_NAME);
+
+ return -ENODEV;
+}
+
+/**
+ * vfio_ap_validate_apid
+ *
+ * @mdev: the mediated device
+ * @matrix_mdev: the mediated matrix device
+ * @apid: the APID to validate
+ *
+ * Validates the value of @apid:
+ * * If there are no AP domains assigned, then there must be at least
+ * one AP queue device reserved by the VFIO AP device driver with an
+ * APQN containing @apid.
+ *
+ * * Else each APQN that can be derived from the intersection of @apid and
+ * the IDs of the AP domains already assigned must identify an AP queue
+ * that has been reserved by the VFIO AP device driver.
+ *
+ * Returns 0 if the value of @apid is valid; otherwise, returns an error.
+ */
+static int vfio_ap_validate_apid(struct mdev_device *mdev,
+ struct ap_matrix_mdev *matrix_mdev,
+ unsigned long apid)
+{
+ int ret;
+ struct device *dev = mdev_parent_dev(mdev);
+ struct ap_matrix *ap_matrix = to_ap_matrix(dev);
+ unsigned long apqi;
+
+ apqi = find_first_bit_inv(matrix_mdev->matrix->aqm,
+ matrix_mdev->matrix->aqm_max);
+ if (apqi == matrix_mdev->matrix->aqm_max) {
+ ret = vfio_ap_verify_apid_reserved(ap_matrix, apid);
+ } else {
+ ret = vfio_ap_validate_queues_for_apid(ap_matrix, matrix_mdev,
+ apid);
+ }
+
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+/**
+ * assign_adapter_store
+ *
+ * @dev: the matrix device
+ * @attr: a mediated matrix device attribute
+ * @buf: a buffer containing the adapter ID (APID) to be assigned
+ * @count: the number of bytes in @buf
+ *
+ * Parses the APID from @buf and assigns it to the mediated matrix device. The
+ * APID must be a valid value:
+ * * The APID value must not exceed the maximum allowable AP adapter ID
+ *
+ * * If there are no AP domains assigned, then there must be at least
+ * one AP queue device reserved by the VFIO AP device driver with an
+ * APQN containing @apid.
+ *
+ * * Else each APQN that can be derived from the intersection of @apid and
+ * the IDs of the AP domains already assigned must identify an AP queue
+ * that has been reserved by the VFIO AP device driver.
+ *
+ * Returns the number of bytes processed if the APID is valid; otherwise returns
+ * an error.
+ */
+static ssize_t assign_adapter_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int ret;
+ unsigned long apid;
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ int maxid = KVM_AP_MAX_APM_INDEX(matrix_mdev->matrix);
+
+ ret = kstrtoul(buf, 0, &apid);
+ if (ret || (apid > maxid)) {
+ pr_err("%s: adapter id '%s' not a value from 0 to %02d(%#04x)",
+ VFIO_AP_MATRIX_MODULE_NAME, buf, maxid, maxid);
+
+ return ret ? ret : -EINVAL;
+ }
+
+ ret = vfio_ap_validate_apid(mdev, matrix_mdev, apid);
+ if (ret)
+ return ret;
+
+ /* Set the bit in the AP mask (APM) corresponding to the AP adapter
+ * number (APID). The bits in the mask, from most significant to least
+ * significant bit, correspond to APIDs 0-255.
+ */
+ set_bit_inv(apid, matrix_mdev->matrix->apm);
+
+ return count;
+}
+static DEVICE_ATTR_WO(assign_adapter);
+
+/**
+ * unassign_adapter_store
+ *
+ * @dev: the matrix device
+ * @attr: a mediated matrix device attribute
+ * @buf: a buffer containing the adapter ID (APID) to be assigned
+ * @count: the number of bytes in @buf
+ *
+ * Parses the APID from @buf and unassigns it from the mediated matrix device.
+ * The APID must be a valid value
+ *
+ * Returns the number of bytes processed if the APID is valid; otherwise returns
+ * an error.
+ */
+static ssize_t unassign_adapter_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int ret;
+ unsigned long apid;
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ int maxid = KVM_AP_MAX_APM_INDEX(matrix_mdev->matrix);
+
+ ret = kstrtoul(buf, 0, &apid);
+ if (ret || (apid > maxid)) {
+ pr_err("%s: adapter id '%s' must be a value from 0 to %02d(%#04x)",
+ VFIO_AP_MATRIX_MODULE_NAME, buf, maxid, maxid);
+
+ return ret ? ret : -EINVAL;
+ }
+
+ clear_bit_inv((unsigned long)apid,
+ (unsigned long *)matrix_mdev->matrix->apm);
+
+ return count;
+}
+DEVICE_ATTR_WO(unassign_adapter);
+
+static struct attribute *vfio_ap_mdev_attrs[] = {
+ &dev_attr_assign_adapter.attr,
+ &dev_attr_unassign_adapter.attr,
+ NULL
+};
+
+static struct attribute_group vfio_ap_mdev_attr_group = {
+ .attrs = vfio_ap_mdev_attrs
+};
+
+static const struct attribute_group *vfio_ap_mdev_attr_groups[] = {
+ &vfio_ap_mdev_attr_group,
+ NULL
+};
+
static const struct mdev_parent_ops vfio_ap_matrix_ops = {
.owner = THIS_MODULE,
.supported_type_groups = vfio_ap_mdev_type_groups,
+ .mdev_attr_groups = vfio_ap_mdev_attr_groups,
.create = vfio_ap_mdev_create,
.remove = vfio_ap_mdev_remove,
};
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index 522564e..a92d5ad 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -10,6 +10,7 @@
#include <linux/types.h>
#include <linux/mdev.h>
+#include <asm/kvm-ap.h>
#include "ap_bus.h"
#define VFIO_AP_MODULE_NAME "vfio_ap"
--
1.7.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 10/15] s390: vfio-ap: sysfs interfaces to configure domains
2018-02-27 14:27 [PATCH v2 00/15] s390: vfio-ap: guest dedicated crypto adapters Tony Krowiak
` (8 preceding siblings ...)
2018-02-27 14:28 ` [PATCH v2 09/15] s390: vfio-ap: sysfs interfaces to configure adapters Tony Krowiak
@ 2018-02-27 14:28 ` Tony Krowiak
2018-02-27 14:28 ` [PATCH v2 11/15] s390: vfio-ap: sysfs interfaces to configure control domains Tony Krowiak
` (5 subsequent siblings)
15 siblings, 0 replies; 56+ messages in thread
From: Tony Krowiak @ 2018-02-27 14:28 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen, Tony Krowiak
Provides the sysfs interfaces for assigning AP domains to
and unassigning AP domains from a mediated matrix device.
An AP domain ID corresponds to an AP queue index (APQI). For
each domain assigned to the mediated matrix device, its
corresponging APQI is stored in an AP queue mask (AQM).
The bits in the AQM, from most significant to least
significant bit, correspond to AP domain numbers 0 to 255.
When a domain is assigned, the bit corresponding to its
APQI will be set in the AQM. Likewise, when a domain is
unassigned, the bit corresponding to its APQI will be
cleared from the AQM.
The relevant sysfs structures are:
/sys/devices/vfio_ap
... [matrix]
...... [mdev_supported_types]
......... [vfio_ap-passthrough]
............ [devices]
...............[$uuid]
.................. assign_domain
.................. unassign_domain
To assign a domain to the $uuid mediated matrix device,
write the domain's ID to the assign_domain file. To
unassign a domain, write the domain's ID to the
unassign_domain file. The ID is specified using
conventional semantics: If it begins with 0x, the number
will be parsed as a hexadecimal (case insensitive) number;
otherwise, it will be parsed as a decimal number.
For example, to assign domain 173 (0xad) to the mediated matrix
device $uuid:
echo 173 > assign_domain
or
echo 0xad > assign_domain
To unassign domain 173 (0xad):
echo 173 > unassign_domain
or
echo 0xad > unassign_domain
The assignment will be rejected:
* If the domain ID exceeds the maximum value for an AP domain:
* If the AP Extended Addressing (APXA) facility is installed,
the max value is 255
* Else the max value is 15
* If no AP adapters have yet been assigned and there are
no AP queues reserved by the VFIO AP driver that have an APQN
with an APQI matching that of the AP domain number being
assigned.
* If any of the APQNs that can be derived from the intersection
of the APQI being assigned and the AP adapter ID (APID) of
each of the AP adapters previously assigned can not be matched
with an APQN of an AP queue device reserved by the VFIO AP
driver.
Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
arch/s390/include/asm/kvm-ap.h | 1 +
drivers/s390/crypto/vfio_ap_ops.c | 215 ++++++++++++++++++++++++++++++++++++-
2 files changed, 215 insertions(+), 1 deletions(-)
diff --git a/arch/s390/include/asm/kvm-ap.h b/arch/s390/include/asm/kvm-ap.h
index f660aaf..191071f 100644
--- a/arch/s390/include/asm/kvm-ap.h
+++ b/arch/s390/include/asm/kvm-ap.h
@@ -16,6 +16,7 @@
#define KVM_AP_MASK_BYTES(n)(n / BITS_PER_BYTE)
#define KVM_AP_MAX_APM_INDEX(matrix)(matrix->apm_max - 1)
+#define KVM_AP_MAX_AQM_INDEX(matrix)(matrix->aqm_max - 1)
/**
* The AP matrix is comprised of three bit masks identifying the adapters,
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 90512a6..c448835 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -377,10 +377,223 @@ static ssize_t unassign_adapter_store(struct device *dev,
}
DEVICE_ATTR_WO(unassign_adapter);
+/**
+ * vfio_ap_validate_queues_for_apqi
+ *
+ * @ap_matrix: the matrix device
+ * @matrix_mdev: the mediated matrix device
+ * @apqi: an AP queue index (APQI) - corresponds to a domain ID
+ *
+ * Verifies that each APQN that is derived from the intersection of @apqi and
+ * each AP adapter ID (APID) corresponding to an AP domain assigned to the
+ * @matrix_mdev matches the APQN of an AP queue reserved by the VFIO AP device
+ * driver.
+ *
+ * Returns 0 if validation succeeds; otherwise, returns an error.
+ */
+static int vfio_ap_validate_queues_for_apqi(struct ap_matrix *ap_matrix,
+ struct ap_matrix_mdev *matrix_mdev,
+ unsigned long apqi)
+{
+ int ret;
+ struct vfio_ap_qid_match qid_match;
+ unsigned long apid;
+ struct device_driver *drv = ap_matrix->device.driver;
+
+ /**
+ * Examine each APQN with the specified APQI
+ */
+ for_each_set_bit_inv(apid, matrix_mdev->matrix->apm,
+ matrix_mdev->matrix->apm_max) {
+ qid_match.qid = AP_MKQID(apid, apqi);
+ qid_match.dev = NULL;
+
+ ret = driver_for_each_device(drv, NULL, &qid_match,
+ vfio_ap_queue_match);
+ if (ret)
+ return ret;
+
+ /*
+ * If the APQN identifies an AP queue that is reserved by the
+ * VFIO AP device driver, continue processing.
+ */
+ if (qid_match.dev)
+ continue;
+
+ pr_err("%s: AP queue %02lx.%04lx not reserved by %s driver",
+ VFIO_AP_MATRIX_MODULE_NAME, apqi, apqi,
+ VFIO_AP_DRV_NAME);
+
+ return -ENXIO;
+ }
+
+ return 0;
+}
+
+struct vfio_ap_apqi_reserved {
+ unsigned long apqi;
+ bool reserved;
+};
+
+/**
+ * vfio_ap_queue_id_contains_apqi
+ *
+ * @dev: an AP queue device
+ * @data: an AP queue index (APQI)
+ *
+ * Returns 1 (true) if the APQI (@data) is contained in the AP queue's
+ * identifier; otherwise, returns 0;
+ */
+static int vfio_ap_queue_id_contains_apqi(struct device *dev, void *data)
+{
+ struct vfio_ap_apqi_reserved *apqi_res = data;
+ struct ap_queue *ap_queue = to_ap_queue(dev);
+
+ if (apqi_res->apqi == AP_QID_QUEUE(ap_queue->qid))
+ apqi_res->reserved = true;
+
+ return 0;
+}
+
+/**
+ * vfio_ap_verify_apqi_reserved
+ *
+ * @ap_matrix: the AP matrix configured for the mediated matrix device
+ * @apqi: the AP queue index (APQI) - corresponds to domain ID
+ *
+ * Verifies that at least one AP queue reserved by the VFIO AP device driver
+ * has an APQN containing @apqi.
+ *
+ * Returns 0 if the APQI is reserved; otherwise, returns -ENODEV.
+ */
+static int vfio_ap_verify_apqi_reserved(struct ap_matrix *ap_matrix,
+ unsigned long apqi)
+{
+ int ret;
+ struct vfio_ap_apqi_reserved apqi_res;
+
+ apqi_res.apqi = apqi;
+
+ ret = driver_for_each_device(ap_matrix->device.driver, NULL,
+ &apqi_res,
+ vfio_ap_queue_id_contains_apqi);
+ if (ret)
+ return ret;
+
+ if (apqi_res.reserved)
+ return 0;
+
+ pr_err("%s: no APQNs with domain ID %02lx are reserved by %s driver",
+ VFIO_AP_MATRIX_MODULE_NAME, apqi, VFIO_AP_DRV_NAME);
+
+ return -ENODEV;
+}
+
+/**
+ * vfio_ap_validate_apqi
+ *
+ * @matrix_mdev: the mediated matrix device
+ * @apqi: the APQI (domain ID) to validate
+ *
+ * Validates the value of @apqi:
+ * * If there are no AP adapters assigned, then there must be at least
+ * one AP queue device reserved by the VFIO AP device driver with an
+ * APQN containing @apqi.
+ *
+ * * Else each APQN that can be derived from the intersection of @apqi and
+ * the IDs of the AP adapters already assigned must identify an AP queue
+ * that has been reserved by the VFIO AP device driver.
+ *
+ * Returns 0 if the value of @apqi is valid; otherwise, returns an error.
+ */
+static int vfio_ap_validate_apqi(struct mdev_device *mdev,
+ struct ap_matrix_mdev *matrix_mdev,
+ unsigned long apqi)
+{
+ int ret;
+ struct device *dev = mdev_parent_dev(mdev);
+ struct ap_matrix *ap_matrix = to_ap_matrix(dev);
+ unsigned long apid;
+
+ apid = find_first_bit_inv(matrix_mdev->matrix->apm,
+ matrix_mdev->matrix->apm_max);
+ /* If there are no adapters assigned */
+ if (apid == matrix_mdev->matrix->apm_max) {
+ ret = vfio_ap_verify_apqi_reserved(ap_matrix, apqi);
+ } else {
+ ret = vfio_ap_validate_queues_for_apqi(ap_matrix, matrix_mdev,
+ apqi);
+ }
+
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static ssize_t assign_domain_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int ret;
+ unsigned long apqi;
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ int maxid = KVM_AP_MAX_AQM_INDEX(matrix_mdev->matrix);
+
+ ret = kstrtoul(buf, 0, &apqi);
+ if (ret || (apqi > maxid)) {
+ pr_err("%s: domain id '%s' not a value from 0 to %02d(%#04x)",
+ VFIO_AP_MATRIX_MODULE_NAME, buf, maxid, maxid);
+
+ return ret ? ret : -EINVAL;
+ }
+
+ ret = vfio_ap_validate_apqi(mdev, matrix_mdev, apqi);
+ if (ret)
+ return ret;
+
+ /* Set the bit in the AQM (bitmask) corresponding to the AP domain
+ * number (APQI). The bits in the mask, from most significant to least
+ * significant, correspond to numbers 0-255.
+ */
+ set_bit_inv(apqi, matrix_mdev->matrix->aqm);
+
+ return count;
+}
+DEVICE_ATTR_WO(assign_domain);
+
+static ssize_t unassign_domain_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int ret;
+ unsigned long apqi;
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ int maxid = KVM_AP_MAX_AQM_INDEX(matrix_mdev->matrix);
+
+ ret = kstrtoul(buf, 0, &apqi);
+ if (ret || (apqi > maxid)) {
+ pr_err("%s: domain id '%s' not a value from 0 to %02d(%#04x)",
+ VFIO_AP_MATRIX_MODULE_NAME, buf, maxid, maxid);
+
+ return ret ? ret : -EINVAL;
+ }
+
+ clear_bit_inv((unsigned long)apqi,
+ (unsigned long *)matrix_mdev->matrix->aqm);
+
+ return count;
+}
+DEVICE_ATTR_WO(unassign_domain);
+
static struct attribute *vfio_ap_mdev_attrs[] = {
&dev_attr_assign_adapter.attr,
&dev_attr_unassign_adapter.attr,
- NULL
+ &dev_attr_assign_domain.attr,
+ &dev_attr_unassign_domain.attr,
+ NULL,
};
static struct attribute_group vfio_ap_mdev_attr_group = {
--
1.7.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 11/15] s390: vfio-ap: sysfs interfaces to configure control domains
2018-02-27 14:27 [PATCH v2 00/15] s390: vfio-ap: guest dedicated crypto adapters Tony Krowiak
` (9 preceding siblings ...)
2018-02-27 14:28 ` [PATCH v2 10/15] s390: vfio-ap: sysfs interfaces to configure domains Tony Krowiak
@ 2018-02-27 14:28 ` Tony Krowiak
2018-02-27 14:28 ` [PATCH v2 12/15] s390: vfio-ap: sysfs interface to view matrix mdev matrix Tony Krowiak
` (4 subsequent siblings)
15 siblings, 0 replies; 56+ messages in thread
From: Tony Krowiak @ 2018-02-27 14:28 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen, Tony Krowiak
Provides the sysfs interfaces for assigning AP control domains
to and unassigning AP control domains from a mediated matrix device.
The IDs of the AP control domains assigned to the mediated matrix
device are stored in an AP domain mask (ADM). The bits in the ADM,
from most significant to least significant bit, correspond to
AP domain numbers 0 to 255. When a control domain is assigned,
the bit corresponding its domain ID will be set in the ADM.
Likewise, when a domain is unassigned, the bit corresponding
to its domain ID will be cleared in the ADM.
The relevant sysfs structures are:
/sys/devices/vfio_ap
... [matrix]
...... [mdev_supported_types]
......... [vfio_ap-passthrough]
............ [devices]
...............[$uuid]
.................. assign_control_domain
.................. unassign_control_domain
To assign a control domain to the $uuid mediated matrix device's
ADM, write its domain number to the assign_control_domain file.
To unassign a domain, write its domain number to the
unassign_control_domain file. The domain number is specified
using conventional semantics: If it begins with 0x the number
will be parsed as a hexadecimal (case insensitive) number;
otherwise, it will be parsed as a decimal number.
For example, to assign control domain 173 (0xad) to the mediated
matrix device $uuid:
echo 173 > assign_control_domain
or
echo 0xad > assign_control_domain
To unassign control domain 173 (0xad):
echo 173 > unassign_control_domain
or
echo 0xad > unassign_control_domain
The assignment will be rejected if the APQI exceeds the maximum
value for an AP domain:
* If the AP Extended Addressing (APXA) facility is installed,
the max value is 255
* Else the max value is 15
Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
arch/s390/include/asm/kvm-ap.h | 1 +
drivers/s390/crypto/vfio_ap_ops.c | 107 +++++++++++++++++++++++++++++++++++++
2 files changed, 108 insertions(+), 0 deletions(-)
diff --git a/arch/s390/include/asm/kvm-ap.h b/arch/s390/include/asm/kvm-ap.h
index 191071f..4171f4b 100644
--- a/arch/s390/include/asm/kvm-ap.h
+++ b/arch/s390/include/asm/kvm-ap.h
@@ -17,6 +17,7 @@
#define KVM_AP_MASK_BYTES(n)(n / BITS_PER_BYTE)
#define KVM_AP_MAX_APM_INDEX(matrix)(matrix->apm_max - 1)
#define KVM_AP_MAX_AQM_INDEX(matrix)(matrix->aqm_max - 1)
+#define KVM_AP_MAX_ADM_INDEX(matrix)(matrix->adm_max - 1)
/**
* The AP matrix is comprised of three bit masks identifying the adapters,
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index c448835..461d450 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -588,11 +588,118 @@ static ssize_t unassign_domain_store(struct device *dev,
}
DEVICE_ATTR_WO(unassign_domain);
+
+/**
+ * assign_control_domain_store
+ *
+ * @dev: the matrix device
+ * @attr: a mediated matrix device attribute
+ * @buf: a buffer containing the adapter ID (APID) to be assigned
+ * @count: the number of bytes in @buf
+ *
+ * Parses the domain ID from @buf and assigns it to the mediated matrix device.
+ *
+ * Returns the number of bytes processed if the domain ID is valid; otherwise
+ * returns an error.
+ */
+static ssize_t assign_control_domain_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int ret;
+ unsigned long id;
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ int maxid = KVM_AP_MAX_ADM_INDEX(matrix_mdev->matrix);
+
+ ret = kstrtoul(buf, 0, &id);
+ if (ret || (id > maxid)) {
+ pr_err("%s: control domain id '%s' not a value from 0 to %02d(%#04x)",
+ VFIO_AP_MATRIX_MODULE_NAME, buf, maxid, maxid);
+
+ return ret ? ret : -EINVAL;
+ }
+
+ /* Set the bit in the ADM (bitmask) corresponding to the AP control
+ * domain number (id). The bits in the mask, from most significant to
+ * least significant, correspond to IDs 0 up to the one less than the
+ * number of control domains that can be assigned.
+ */
+ set_bit_inv(id, matrix_mdev->matrix->adm);
+
+ return count;
+}
+DEVICE_ATTR_WO(assign_control_domain);
+
+/**
+ * unassign_control_domain_store
+ *
+ * @dev: the matrix device
+ * @attr: a mediated matrix device attribute
+ * @buf: a buffer containing the adapter ID (APID) to be assigned
+ * @count: the number of bytes in @buf
+ *
+ * Parses the domain ID from @buf and unassigns it from the mediated matrix
+ * device.
+ *
+ * Returns the number of bytes processed if the domain ID is valid; otherwise
+ * returns an error.
+ */
+static ssize_t unassign_control_domain_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int ret;
+ unsigned long apqi;
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ int maxid = KVM_AP_MAX_ADM_INDEX(matrix_mdev->matrix);
+
+ ret = kstrtoul(buf, 0, &apqi);
+ if (ret || (apqi > maxid)) {
+ pr_err("%s: control domain id '%s' not a value from 0 to %02d(%#04x)",
+ VFIO_AP_MATRIX_MODULE_NAME, buf, maxid, maxid);
+
+ return ret ? ret : -EINVAL;
+ }
+
+ clear_bit_inv((unsigned long)apqi,
+ (unsigned long *)matrix_mdev->matrix->adm);
+
+ return count;
+}
+DEVICE_ATTR_WO(unassign_control_domain);
+
+static ssize_t control_domains_show(struct device *dev,
+ struct device_attribute *dev_attr,
+ char *buf)
+{
+ unsigned long id;
+ int nchars = 0;
+ int n;
+ char *bufpos = buf;
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+ for_each_set_bit_inv(id, matrix_mdev->matrix->adm,
+ matrix_mdev->matrix->adm_max) {
+ n = sprintf(bufpos, "%04lx\n", id);
+ bufpos += n;
+ nchars += n;
+ }
+
+ return nchars;
+}
+DEVICE_ATTR_RO(control_domains);
+
static struct attribute *vfio_ap_mdev_attrs[] = {
&dev_attr_assign_adapter.attr,
&dev_attr_unassign_adapter.attr,
&dev_attr_assign_domain.attr,
&dev_attr_unassign_domain.attr,
+ &dev_attr_assign_control_domain.attr,
+ &dev_attr_unassign_control_domain.attr,
+ &dev_attr_control_domains.attr,
NULL,
};
--
1.7.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 12/15] s390: vfio-ap: sysfs interface to view matrix mdev matrix
2018-02-27 14:27 [PATCH v2 00/15] s390: vfio-ap: guest dedicated crypto adapters Tony Krowiak
` (10 preceding siblings ...)
2018-02-27 14:28 ` [PATCH v2 11/15] s390: vfio-ap: sysfs interfaces to configure control domains Tony Krowiak
@ 2018-02-27 14:28 ` Tony Krowiak
2018-02-27 14:28 ` [PATCH v2 13/15] KVM: s390: Configure the guest's CRYCB Tony Krowiak
` (3 subsequent siblings)
15 siblings, 0 replies; 56+ messages in thread
From: Tony Krowiak @ 2018-02-27 14:28 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen, Tony Krowiak
Provides a sysfs interface to view the AP matrix configured for the
mediated matrix device.
The relevant sysfs structures are:
/sys/devices/vfio_ap
... [matrix]
...... [mdev_supported_types]
......... [vfio_ap-passthrough]
............ [devices]
...............[$uuid]
.................. matrix
To view the matrix configured for the mediated matrix device,
print the matrix file:
cat matrix
Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
drivers/s390/crypto/vfio_ap_ops.c | 39 +++++++++++++++++++++++++++++++++++++
1 files changed, 39 insertions(+), 0 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 461d450..04f7a92 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -692,6 +692,44 @@ static ssize_t control_domains_show(struct device *dev,
}
DEVICE_ATTR_RO(control_domains);
+static ssize_t matrix_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ char *bufpos = buf;
+ unsigned long apid;
+ unsigned long apqi;
+ int nchars = 0;
+ int n;
+
+ n = sprintf(bufpos, "ADAPTER.DOMAIN\n");
+ bufpos += n;
+ nchars += n;
+
+ n = sprintf(bufpos, "--------------\n");
+ bufpos += n;
+ nchars += n;
+
+ for_each_set_bit_inv(apid, matrix_mdev->matrix->apm,
+ matrix_mdev->matrix->apm_max) {
+ n = sprintf(bufpos, "%02lx\n", apid);
+ bufpos += n;
+ nchars += n;
+
+ for_each_set_bit_inv(apqi, matrix_mdev->matrix->aqm,
+ matrix_mdev->matrix->aqm_max) {
+ n = sprintf(bufpos, "%02lx.%04lx\n", apid, apqi);
+ bufpos += n;
+ nchars += n;
+ }
+ }
+
+ return nchars;
+}
+DEVICE_ATTR_RO(matrix);
+
+
static struct attribute *vfio_ap_mdev_attrs[] = {
&dev_attr_assign_adapter.attr,
&dev_attr_unassign_adapter.attr,
@@ -700,6 +738,7 @@ static ssize_t control_domains_show(struct device *dev,
&dev_attr_assign_control_domain.attr,
&dev_attr_unassign_control_domain.attr,
&dev_attr_control_domains.attr,
+ &dev_attr_matrix.attr,
NULL,
};
--
1.7.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 13/15] KVM: s390: Configure the guest's CRYCB
2018-02-27 14:27 [PATCH v2 00/15] s390: vfio-ap: guest dedicated crypto adapters Tony Krowiak
` (11 preceding siblings ...)
2018-02-27 14:28 ` [PATCH v2 12/15] s390: vfio-ap: sysfs interface to view matrix mdev matrix Tony Krowiak
@ 2018-02-27 14:28 ` Tony Krowiak
2018-02-28 9:49 ` David Hildenbrand
2018-02-27 14:28 ` [PATCH v2 14/15] s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl Tony Krowiak
` (2 subsequent siblings)
15 siblings, 1 reply; 56+ messages in thread
From: Tony Krowiak @ 2018-02-27 14:28 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen, Tony Krowiak
Registers a group notifier during the open of the mediated
matrix device to get information on KVM presence through the
VFIO_GROUP_NOTIFY_SET_KVM event. When notified, the pointer
to the kvm structure is saved inside the mediated matrix
device. Once the VFIO AP device driver has access to KVM,
the AP matrix for the guest can be configured.
Guest access to AP adapters, usage domains and control domains
is controlled by three bit masks referenced from the
Crypto Control Block (CRYCB) referenced from the guest's SIE state
description:
* The AP Mask (APM) controls access to the AP adapters. Each bit
in the APM represents an adapter number - from most significant
to least significant bit - from 0 to 255. The bits in the APM
are set according to the adapter numbers assigned to the mediated
matrix device via its 'assign_adapter' sysfs attribute file.
* The AP Queue (AQM) controls access to the AP queues. Each bit
in the AQM represents an AP queue index - from most significant
to least significant bit - from 0 to 255. A queue index references
a specific domain and is synonymous with the domian number. The
bits in the AQM are set according to the domain numbers assigned
to the mediated matrix device via its 'assign_domain' sysfs
attribute file.
* The AP Domain Mask (ADM) controls access to the AP control domains.
Each bit in the ADM represents a control domain - from most
significant to least significant bit - from 0-255. The
bits in the ADM are set according to the domain numbers assigned
to the mediated matrix device via its 'assign_control_domain'
sysfs attribute file.
Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
drivers/s390/crypto/vfio_ap_ops.c | 46 +++++++++++++++++++++++++++++++++
drivers/s390/crypto/vfio_ap_private.h | 2 +
2 files changed, 48 insertions(+), 0 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 04f7a92..752d171 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -53,6 +53,50 @@ static int vfio_ap_mdev_remove(struct mdev_device *mdev)
return 0;
}
+static int vfio_ap_mdev_group_notifier(struct notifier_block *nb,
+ unsigned long action, void *data)
+{
+ struct ap_matrix_mdev *matrix_mdev;
+
+ if (action == VFIO_GROUP_NOTIFY_SET_KVM) {
+ matrix_mdev = container_of(nb, struct ap_matrix_mdev,
+ group_notifier);
+ matrix_mdev->kvm = data;
+ }
+
+ return NOTIFY_OK;
+}
+
+static int vfio_ap_mdev_open(struct mdev_device *mdev)
+{
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ unsigned long events;
+ int ret;
+
+ matrix_mdev->group_notifier.notifier_call = vfio_ap_mdev_group_notifier;
+ events = VFIO_GROUP_NOTIFY_SET_KVM;
+ ret = vfio_register_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
+ &events, &matrix_mdev->group_notifier);
+
+ ret = kvm_ap_configure_matrix(matrix_mdev->kvm,
+ matrix_mdev->matrix);
+ if (ret)
+ return ret;
+
+ ret = kvm_ap_enable_ie_mode(matrix_mdev->kvm);
+
+ return ret;
+}
+
+static void vfio_ap_mdev_release(struct mdev_device *mdev)
+{
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+ kvm_ap_deconfigure_matrix(matrix_mdev->kvm);
+ vfio_unregister_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
+ &matrix_mdev->group_notifier);
+}
+
static ssize_t name_show(struct kobject *kobj, struct device *dev, char *buf)
{
return sprintf(buf, "%s\n", VFIO_AP_MDEV_NAME_HWVIRT);
@@ -757,6 +801,8 @@ static ssize_t matrix_show(struct device *dev, struct device_attribute *attr,
.mdev_attr_groups = vfio_ap_mdev_attr_groups,
.create = vfio_ap_mdev_create,
.remove = vfio_ap_mdev_remove,
+ .open = vfio_ap_mdev_open,
+ .release = vfio_ap_mdev_release,
};
int vfio_ap_mdev_register(struct ap_matrix *ap_matrix)
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index a92d5ad..1bd3d42 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -30,6 +30,8 @@ struct ap_matrix {
struct ap_matrix_mdev {
struct kvm_ap_matrix *matrix;
+ struct notifier_block group_notifier;
+ struct kvm *kvm;
};
static inline struct ap_matrix *to_ap_matrix(struct device *dev)
--
1.7.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 14/15] s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl
2018-02-27 14:27 [PATCH v2 00/15] s390: vfio-ap: guest dedicated crypto adapters Tony Krowiak
` (12 preceding siblings ...)
2018-02-27 14:28 ` [PATCH v2 13/15] KVM: s390: Configure the guest's CRYCB Tony Krowiak
@ 2018-02-27 14:28 ` Tony Krowiak
2018-02-27 14:28 ` [PATCH v2 15/15] s390: doc: detailed specifications for AP virtualization Tony Krowiak
2018-02-27 14:58 ` [PATCH v2 00/15] s390: vfio-ap: guest dedicated crypto adapters Cornelia Huck
15 siblings, 0 replies; 56+ messages in thread
From: Tony Krowiak @ 2018-02-27 14:28 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen, Tony Krowiak
Introduces ioctl access to the VFIO AP Matrix device driver
by implementing the VFIO_DEVICE_GET_INFO ioctl. This ioctl
provides the VFIO AP Matrix device driver information to the
guest machine.
Reviewed-by: Pierre Morel <pmorel@linux.vnet.ibm.com>
Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
drivers/s390/crypto/vfio_ap_ops.c | 43 +++++++++++++++++++++++++++++++++++++
1 files changed, 43 insertions(+), 0 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 752d171..49c119d 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -97,6 +97,48 @@ static void vfio_ap_mdev_release(struct mdev_device *mdev)
&matrix_mdev->group_notifier);
}
+static int vfio_ap_mdev_get_device_info(unsigned long arg)
+{
+ unsigned long minsz;
+ struct vfio_device_info info;
+
+ minsz = offsetofend(struct vfio_device_info, num_irqs);
+
+ if (copy_from_user(&info, (void __user *)arg, minsz))
+ return -EFAULT;
+
+ if (info.argsz < minsz) {
+ pr_err("%s: Argument size %u less than min size %li",
+ VFIO_AP_MATRIX_MODULE_NAME, info.argsz, minsz);
+ return -EINVAL;
+ }
+
+ info.flags = VFIO_DEVICE_FLAGS_AP;
+ info.num_regions = 0;
+ info.num_irqs = 0;
+
+ return copy_to_user((void __user *)arg, &info, minsz);
+}
+
+static ssize_t vfio_ap_mdev_ioctl(struct mdev_device *mdev,
+ unsigned int cmd, unsigned long arg)
+{
+ int ret;
+
+ switch (cmd) {
+ case VFIO_DEVICE_GET_INFO:
+ ret = vfio_ap_mdev_get_device_info(arg);
+ break;
+ default:
+ pr_err("%s: ioctl command %d is not a supported command",
+ VFIO_AP_MATRIX_MODULE_NAME, cmd);
+ ret = -EOPNOTSUPP;
+ break;
+ }
+
+ return ret;
+}
+
static ssize_t name_show(struct kobject *kobj, struct device *dev, char *buf)
{
return sprintf(buf, "%s\n", VFIO_AP_MDEV_NAME_HWVIRT);
@@ -803,6 +845,7 @@ static ssize_t matrix_show(struct device *dev, struct device_attribute *attr,
.remove = vfio_ap_mdev_remove,
.open = vfio_ap_mdev_open,
.release = vfio_ap_mdev_release,
+ .ioctl = vfio_ap_mdev_ioctl,
};
int vfio_ap_mdev_register(struct ap_matrix *ap_matrix)
--
1.7.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 15/15] s390: doc: detailed specifications for AP virtualization
2018-02-27 14:27 [PATCH v2 00/15] s390: vfio-ap: guest dedicated crypto adapters Tony Krowiak
` (13 preceding siblings ...)
2018-02-27 14:28 ` [PATCH v2 14/15] s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl Tony Krowiak
@ 2018-02-27 14:28 ` Tony Krowiak
2018-02-27 15:57 ` Cornelia Huck
2018-02-27 14:58 ` [PATCH v2 00/15] s390: vfio-ap: guest dedicated crypto adapters Cornelia Huck
15 siblings, 1 reply; 56+ messages in thread
From: Tony Krowiak @ 2018-02-27 14:28 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen, Tony Krowiak
This patch provides documentation describing the AP architecture and
design concepts behind the virtualization of AP devices. It also
includes an example of how to configure AP devices for exclusive
use of KVM guests.
Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
Documentation/s390/vfio-ap.txt | 514 ++++++++++++++++++++++++++++++++++++++++
MAINTAINERS | 1 +
2 files changed, 515 insertions(+), 0 deletions(-)
create mode 100644 Documentation/s390/vfio-ap.txt
diff --git a/Documentation/s390/vfio-ap.txt b/Documentation/s390/vfio-ap.txt
new file mode 100644
index 0000000..c599f30
--- /dev/null
+++ b/Documentation/s390/vfio-ap.txt
@@ -0,0 +1,514 @@
+Introduction:
+============
+The Adjunct Processor (AP) facility is an IBM Z cryptographic facility comprised
+of three AP instructions and from 1 up to 256 PCIe cryptographic adapter cards.
+The AP devices provide cryptographic functions to all CPUs assigned to a
+linux system running in an IBM Z system LPAR.
+
+The AP adapter cards are exposed via the AP bus. The motivation for vfio-ap
+is to make AP cards available to KVM guests using the VFIO mediated device
+framework.
+
+AP Architectural Overview:
+=========================
+To facilitate the comprehension of the design, let's start with some
+definitions:
+
+* AP adapter
+
+ An AP adapter is an IBM Z adapter card that can perform cryptographic
+ functions. There can be from 0 to 256 adapters assigned to an LPAR. Adapters
+ assigned to the LPAR in which a linux host is running will be available to
+ the linux host. Each adapter is identified by a number from 0 to 255. When
+ installed, an AP adapter is accessed by AP instructions executed by any CPU.
+
+* AP domain
+
+ An adapter is partitioned into domains. Each domain can be thought of as
+ a set of hardware registers for processing AP instructions. An adapter can
+ hold up to 256 domains. Each domain is identified by a number from 0 to 255.
+ Domains can be further classified into two types:
+
+ * Usage domains are domains that can be accessed directly to process AP
+ commands
+
+ * Control domains are domains that are accessed indirectly by AP
+ commands sent to a usage domain to control or change the domain, for
+ example; to set a secure private key for the domain.
+
+* AP Queue
+
+ An AP queue is the means by which an AP command-request message is sent to an
+ AP usage domain inside a specific AP. An AP queue is identified by a tuple
+ comprised of an AP adapter ID (APID) and an AP queue index (APQI). The
+ APQI corresponds to a given usage domain number within the adapter. This tuple
+ forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP
+ instructions include a field containing the APQN to identify the AP queue to
+ which the AP command-request message is to be sent for processing.
+
+* AP Instructions:
+
+ There are three AP instructions:
+
+ * NQAP: to enqueue an AP command-request message to a queue
+ * DQAP: to dequeue an AP command-reply message from a queue
+ * PQAP: to administer the queues
+
+Start Interpretive Execution (SIE) Instruction:
+==============================================
+Let's now see how AP instructions are interpreted by the hardware.
+
+A KVM guest is started by executing the Start Interpretive Execution (SIE)
+instruction. The SIE state description is a control block that contains the
+state information for a KVM guest and is supplied as input to the SIE
+instruction. The SIE state description contains a field that references
+a Crypto Control Block (CRYCB). The CRYCB contains three fields to identify the
+adapters, usage domains and control domains assigned to the KVM guest:
+
+* The AP Mask (APM) field is a bit mask that identifies the AP adapters assigned
+ to the KVM guest. Each bit in the mask, from most significant to least
+ significant bit, corresponds to an APID from 0-255. If a bit is set, the
+ corresponding adapter is valid for use by the KVM guest.
+
+* The AP Queue Mask (AQM) field is a bit mask identifying the AP queues assigned
+ to the KVM guest. Each bit in the mask, from most significant to least
+ significant bit, corresponds to an AP queue index (APQI) from 0-255. If a bit
+ is set, the corresponding queue is valid for use by the KVM guest.
+
+* The AP Domain Mask field is a bit mask that identifies the AP control domains
+ assigned to the KVM guest. The ADM bit mask controls which domains can be
+ changed by an AP command-request message sent to a usage domain from the
+ guest. Each bit in the mask, from least significant to most significant bit,
+ corresponds to a domain from 0-255. If a bit is set, the corresponding domain
+ can be modified by an AP command-request message sent to a usage domain
+ configured for the KVM guest.
+
+If you recall from the description of an AP Queue, AP instructions include
+an APQN to identify the AP adapter and AP queue to which an AP command-request
+message is to be sent (NQAP and PQAP instructions), or from which a
+command-reply message is to be received (DQAP instruction). The validity of an
+APQN is defined by the matrix calculated from the APM and AQM; it is the
+intersection of all assigned adapter numbers (APM) with all assigned queue
+indexes (AQM). For example, if adapters 1 and 2 and usage domains 5 and 6 are
+assigned to a guest, the APQNs (1,5), (1,6), (2,5) and (2,6) will be valid for
+the guest.
+
+The APQNs provide secure key functionality - i.e., a private key is stored on
+the adapter card for each of its domains - so each APQN must be assigned to at
+most one guest or the linux host.
+
+ Example 1: Valid configuration:
+ ------------------------------
+ Guest1: adapters 1,2 domains 5,6
+ Guest2: adapter 1,2 domain 7
+
+ This is valid because both guests have a unique set of APQNs: Guest1 has
+ APQNs (1,5), (1,6), (2,5) and (2,6); Guest2 has APQNs (1,7) and (2,7).
+
+ Example 2: Invalid configuration:
+ --------------------------------is assigned by writing the adapter's number into the
+ Guest1: adapters 1,2 domains 5,6
+ Guest2: adapter 1 domains 6,7
+
+ This is an invalid configuration because both guests have access to
+ APQN (1,6).
+
+The Design:
+===========
+The design introduces three new objects:
+
+1. AP matrix device
+2. VFIO AP device driver (vfio_ap.ko)
+3. AP mediated matrix passthrough device
+
+The VFIO AP device driver
+-------------------------
+The VFIO AP (vfio_ap) device driver serves the following purposes:
+
+1. Provides the interfaces to reserve APQNs for exclusive use of KVM guests.
+
+2. Sets up the VFIO mediated device interfaces to manage the mediated matrix
+ device and create the sysfs interfaces for assigning adapters, usage domains,
+ and control domains comprising the matrix for a KVM guest.
+
+3. Configure the APM, AQM and ADM in the CRYCB referenced by a KVM guest's
+ SIE state description to grant the guest access to AP devices
+
+4. Initialize the CPU model feature indicating that a KVM guest may use
+ AP facilities installed on the linux host.
+
+5. Enable interpretive execution mode for the KVM guest.
+
+Reserve APQNs for exclusive use of KVM guests
+---------------------------------------------
+The following block diagram illustrates the mechanism by which APQNs are
+reserved:
+
+ +------------------+
+ remove | | unbind
+ +------------------->+ cex4queue driver +<-----------+
+ | | | |
+ | +------------------+ |
+ | |
+ | |
+ | |
++--------+---------+ register +------------------+ +-----+------+
+| +<---------+ | bind | |
+| ap_bus | | vfio_ap driver +<-----+ admin |
+| +--------->+ | | |
++------------------+ probe +---+--------+-----+ +------------+
+ | |
+ create | | store APQN
+ | |
+ v v
+ +---+--------+-----+
+ | |
+ | matrix device |
+ | |
+ +------------------+
+
+The process for reserving an AP queue for use by a KVM guest is:
+
+* The vfio-ap driver during its initialization will perform the following:
+ * Create the 'vfio_ap' root device - /sys/devices/vfio_ap
+ * Create the 'matrix' device in the 'vfio_ap' root
+ * Register the matrix device with the device core
+* Register with the ap_bus for AP queue devices of type CEX4, CEX5 and
+ CEX6 and to provide the vfio_ap driver's probe and remove callback interfaces.
+* The admin unbinds queue cc.qqqq from the cex4queue device driver. This results
+ in the ap_bus calling the the device driver's remove interface which
+ unbinds the cc.qqqq queue device from the driver.
+* The admin binds the cc.qqqq queue to the vfio_ap device driver. This results
+ in the ap_bus calling the device vfio_ap driver's probe interface to bind
+ queue cc.qqqq to the driver. The vfio_ap device driver will store the APQN for
+ the queue in the matrix device
+
+Set up the VFIO mediated device interfaces
+------------------------------------------
+The VFIO AP device driver utilizes the common interface of the VFIO mediated
+device core driver to:
+* Register an AP mediated bus driver to add a mediated matrix device to and
+ remove it from a VFIO group.
+* Create and destroy a mediated matrix device
+* Add a mediated matrix device to and remove it from the AP mediated bus driver
+* Add a mediated matrix device to and remove it from an IOMMU group
+
+The following high-level block diagram shows the main components and interfaces
+of the VFIO AP mediated matrix device driver:
+
+ +-------------+
+ | |
+ | +---------+ | mdev_register_driver() +--------------+
+ | | Mdev | +<-----------------------+ |
+ | | bus | | | vfio_mdev.ko |
+ | | driver | +----------------------->+ |<-> VFIO user
+ | +---------+ | probe()/remove() +--------------+ APIs
+ | |
+ | MDEV CORE |
+ | MODULE |
+ | mdev.ko |
+ | +---------+ | mdev_register_device() +--------------+
+ | |Physical | +<-----------------------+ |
+ | | device | | | vfio_ap.ko |<-> matrix
+ | |interface| +----------------------->+ | device
+ | +---------+ | callback +--------------+
+ +-------------+
+
+During initialization of the vfio_ap module, the matrix device is registered
+with an 'mdev_parent_ops' structure that provides the sysfs attribute
+structures, mdev functions and callback interfaces for managing the mediated
+matrix device.
+
+* sysfs attribute structures:
+ * supported_type_groups
+ The VFIO mediated device framework supports creation of user-defined
+ mediated device types. These mediated device types are specified
+ via the 'supported_type_groups' structure when a device is registered
+ with the mediated device framework. The registration process creates the
+ sysfs structures for each mediated device type specified in the
+ 'mdev_supported_types' sub-directory of the device being registered. Along
+ with the device type, the sysfs attributes of the mediated device type are
+ provided.
+
+ The VFIO AP device driver will register one mediated device type for
+ passthrough devices:
+ /sys/devices/vfio_ap/mdev_supported_types/vfio_ap-passthrough
+ Only the three read-only attributes required by the VFIO mdev framework will
+ be provided:
+ /sys/devices/vfio_ap/mdev_supported_types
+ ... name
+ ... device_api
+ ... available_instances
+ Where:
+ * name: specifies the name of the mediated device type
+ * device_api: the mediated device type's API
+ * available_instances: the number of mediated matrix passthrough devices
+ that can be created
+ * mdev_attr_groups
+ This attribute group identifies the user-defined sysfs attributes of the
+ mediated device. When a device is registered with the VFIO mediated device
+ framework, the sysfs attributes files identified in the 'mdev_attr_groups'
+ structure will be created in the mediated matrix device's directory. The
+ sysfs attributes for a mediated matrix device are:
+ * assign_adapter:
+ A write-only file for assigning an AP adapter to the mediated matrix
+ device. To assign an adapter, the APID of the adapter is written to the
+ file.
+ * assign_domain:
+ A write-only file for assigning an AP usage domain to the mediated matrix
+ device. To assign a domain, the APQI of the AP queue corresponding to a
+ usage domain is written to the file.
+ * assign_control_domain:
+ A write-only file for assigning an AP control domain to the mediated
+ matrix device. To assign a control domain, the ID of a domain to be
+ controlled is written to the file. By architectural convention, the set of
+ control domains will always include the set of usage domains, so it is
+ only necessary to assign control domains that are not also assigned as
+ usage domains.
+
+* functions:
+ * create:
+ allocates the ap_matrix_mdev structure used by the vfio_ap driver to:
+ * Keep track of the available instances
+ * Store the reference to the struct kvm for the KVM guest
+ * Provide the notifier callback that will get invoked to handle the
+ VFIO_GROUP_NOTIFY_SET_KVM event. When received, the vfio_ap driver will
+ store the reference in the mediated matrix device's ap_matrix_mdev
+ structure and enable the interpretive execution mode for the KVM guest.
+ * remove:
+ deallocates the mediated matrix device's ap_matrix_mdev structure.
+
+* callback interfaces
+ * open:
+ The vfio_ap driver uses this callback to register a
+ VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the mdev matrix
+ device. The notifier is invoked when QEMU connects the VFIO iommu group
+ for the mdev matrix device to the MDEV bus. Access to the KVM structure used
+ to set up the KVM guest is provided via this callback.
+ * release:
+ unregisters the VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the
+ mdev matrix device.
+
+Configure the APM, AQM and ADM in the CRYCB:
+-------------------------------------------
+Configuring the AP matrix for a KVM guest will be performed when the
+VFIO_GROUP_NOTIFY_SET_KVM notifier callback is invoked. The notifier is callback
+function is called when QEMU connects the VFIO iommu group for the mdev matrix
+device to the MDEV bus. The CRYCB is configured by:
+* Setting the bits in the APM corresponding to the APIDs assigned to the
+ mediated matrix device via its 'assign_adapter' interface.
+* Setting the bits in the AQM corresponding to the APQIs assigned to the
+ mediated matrix device via its 'assign_domain' interface.
+* Setting the bits in the ADM corresponding to the domain dIDs assigned to the
+ mediated matrix device via its 'assign_control_domains' interface.
+
+Initialize the CPU model feature for AP
+---------------------------------------
+This design exploits a feature of the SIE architecture called interpretive
+execution (IE). When IE is enabled for a KVM guest, the AP instructions
+executed in the guest will be interpreted by the firmware and the commands
+contained therein will be passed directly through to an AP device assigned to
+the linux host. In order to enable interpretive execution for a KVM guest, SIE
+must have access to the AP facilities installed on the linux host. A new CPU
+model feature is introduced by this design to indicate that the guest will
+directly access the host AP facilities. This feature will be enabled by the
+kernel only if the AP facilities are installed on the linux host. This feature
+is turned on for the guest via the qemu command line:
+
+ /usr/bin/qemu-system-s390x ... -cpu xxx,ap=on
+
+ Where xxx is the CPU model being used.
+
+If the CPU model feature is not enabled by the kernel, QEMU
+will fail and report that the feature is not supported.
+
+Example:
+=======
+Let's now provide an example to illustrate how KVM guests may be given
+access to AP facilities. For this example, we will show how to configure
+two guests such that executing the lszcrypt command on the guests would
+look like this:
+
+Guest1
+------
+CARD.DOMAIN TYPE MODE
+------------------------------
+05 CEX5C CCA-Coproc
+05.0004 CEX5C CCA-Coproc
+05.00ab CEX5C CCA-Coproc
+06 CEX5A Accelerator
+06.0004 CEX5A Accelerator
+06.00ab CEX5C CCA-Coproc
+
+Guest2
+------
+CARD.DOMAIN TYPE MODE
+------------------------------
+05 CEX5A Accelerator
+05.0047 CEX5A Accelerator
+05.00ff CEX5A Accelerator
+
+These are the steps:
+
+1. Install the vfio_ap module on the linux host. The dependency chain for the
+ vfio_ap module is:
+ * vfio
+ * mdev
+ * vfio_mdev
+ * vfio_ap
+
+2. Secure the AP queues to be used by the two guests so that the host can not
+ access them. This is done by unbinding each AP Queue device from its
+ respective AP driver. In our example, these queues are bound to the cex4queue
+ driver. The sysfs location of these devices is:
+
+ /sys/bus/ap
+ --- [drivers]
+ ------ [cex4queue]
+ --------- [05.0004]
+ --------- [05.0047]
+ --------- [05.00ab]
+ --------- [05.00ff]
+ --------- [06.0004]
+ --------- [06.00ab]
+ --------- unbind
+
+ To unbind AP queue 05.0004 from the cex4queue device driver:
+
+ echo 05.0004 > unbind
+
+ This must also be done for AP queues 05.00ab, 05.0047, 05.00ff, 06.0004,
+ and 06.00ab.
+
+3. Reserve the queues for use by the two KVM guests. This is accomplished by
+ binding them to the vfio_ap device driver. The sysfs location of the
+ device driver is:
+
+ /sys/bus/ap
+ ---[drivers]
+ ------ [vfio_ap]
+ ---------- bind
+
+ To bind queue 05.0004 to the vfio_ap driver:
+
+ echo 05.0004 > bind
+
+ This must also be done for AP queues 05.00ab, 05.0047, 05.00ff, 06.0004,
+ and 06.00ab.
+
+ Take note that the AP queues bound to the vfio_ap driver will be available
+ for guest usage until they are unbound from the driver, the vfio_ap module
+ is unloaded, or the host system is shut down.
+
+4. Create the mediated devices needed to configure the AP matrixes for the
+ two guests and to provide an interface to the vfio_ap driver for
+ use by the guests:
+
+ /sys/devices/
+ --- [vfio_ap]
+ ------ [matrix] (this is the matrix device)
+ --------- [mdev_supported_types]
+ ------------ [vfio_ap-passthrough] (passthrough mediated matrix device type)
+ --------------- create
+ --------------- [devices]
+
+ To create the mediated devices for the two guests:
+
+ uuidgen > create
+ uuidgen > create
+
+ This will create two mediated devices in the [devices] subdirectory named
+ with the UUID written to the create attribute file. We call them $uuid1
+ and $uuid2:
+
+ /sys/devices/
+ --- [vfio_ap]
+ ------ [matrix]
+ --------- [mdev_supported_types]
+ ------------ [vfio_ap-passthrough]
+ --------------- [devices]
+ ------------------ [$uuid1]
+ --------------------- assign_adapter
+ --------------------- assign_control_domain
+ --------------------- assign_domain
+ --------------------- matrix
+ --------------------- unassign_adapter
+ --------------------- unassign_control_domain
+ --------------------- unassign_domain
+
+ ------------------ [$uuid2]
+ --------------------- assign_adapter
+ --------------------- assign_cTo assign an adapter, the APID of the adapter is written to the
+ file. ontrol_domain
+ --------------------- assign_domain
+ --------------------- matrix
+ --------------------- unassign_adapter
+ --------------------- unassign_control_domain
+ --------------------- unassign_domain
+
+5. The administrator now needs to configure the matrixes for mediated
+ devices $uuid1 (for Guest1) and $uuid2 (for Guest2).
+
+ This is how the matrix is configured for Guest1:
+
+ echo 5 > assign_adapter
+ echo 6 > assign_adapter
+ echo 4 > assign_domain
+ echo 0xab > assign_domain
+
+ By architectural convention, all usage domains - i.e., domains assigned
+ via the assign_domain attribute file - will also be configured in the ADM
+ field of the KVM guest's CRYCB, so there is no need to assign control
+ domains here unless you want to assign control domains that are not
+ assigned as usage domains.
+
+ If a mistake is made configuring an adapter, domain or control domain,
+ you can use the unassign_xxx files to unassign the adapter, domain or
+ control domain.
+
+ To display the matrix configuration for Guest1:
+
+ cat matrix
+
+ This is how the matrix is configured for Guest2:
+
+ echo 5 > assign_adapter
+ echo 0x47 > assign_domain
+ echo 0xff > assign_domain
+
+6. Start Guest1:
+
+ /usr/bin/qemu-system-s390x ... -cpu xxx,ap=on \
+ -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid1 ...
+
+7. Start Guest2:
+
+ /usr/bin/qemu-system-s390x ... -cpu xxx,ap=on \
+ -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid2 ...
+
+When the guest is shut down, the mediated matrix device may be removed.
+
+Using our example again, to remove the mediated matrix device $uuid1:
+
+ /sys/devices/
+ --- [vfio_ap]
+ ------ [matrix]
+ --------- [mdev_supported_types]
+ ------------ [vfio_ap-passthrough]
+ --------------- [devices]
+ ------------------ [$uuid1]
+ --------------------- remove
+
+ echo 1 > remove
+
+ This will remove all of the mdev matrix device's sysfs structures. To
+ recreate and reconfigure the mdev matrix device, all of the steps starting
+ with step 4 will have to be performed again.
+
+ It is not necessary to remove an mdev matrix device, but one may want to
+ remove it if no guest will use it during the lifetime of the linux host. If
+ the mdev matrix device is removed, one may want to unbind the AP queues the
+ guest was using from the vfio_ap device driver and bind them back to the
+ default driver. Alternatively, the AP queues can be configured for another
+ mdev matrix (i.e., guest). In either case, one must take care to change the
+ secure key configured for the domain to which the queue is connected.
\ No newline at end of file
diff --git a/MAINTAINERS b/MAINTAINERS
index fb9e0d4..bfa94c8 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -11887,6 +11887,7 @@ F: arch/s390/kvm/kvm-ap.c
F: drivers/s390/crypto/vfio_ap_drv.c
F: drivers/s390/crypto/vfio_ap_private.h
F: drivers/s390/crypto/vfio_ap_ops.c
+F: Documentation/s390/vfio-ap.txt
S390 ZFCP DRIVER
M: Steffen Maier <maier@linux.vnet.ibm.com>
--
1.7.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* Re: [PATCH v2 00/15] s390: vfio-ap: guest dedicated crypto adapters
2018-02-27 14:27 [PATCH v2 00/15] s390: vfio-ap: guest dedicated crypto adapters Tony Krowiak
` (14 preceding siblings ...)
2018-02-27 14:28 ` [PATCH v2 15/15] s390: doc: detailed specifications for AP virtualization Tony Krowiak
@ 2018-02-27 14:58 ` Cornelia Huck
2018-02-27 15:57 ` Tony Krowiak
15 siblings, 1 reply; 56+ messages in thread
From: Cornelia Huck @ 2018-02-27 14:58 UTC (permalink / raw)
To: Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
fiuczy, buendgen
On Tue, 27 Feb 2018 09:27:58 -0500
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> On s390, we have cryptographic coprocessor cards, which are modeled on
> Linux as devices on the AP bus. Each card can be partitioned into domains
> which can be thought of as a set of hardware registers for processing
> crypto commands. Crypto commands are sent to a specific domain within a
> card is via a queue which is identified as a (card,domain) tuple. We model
> this something like the following (assuming we have access to cards 3 and
> 4 and domains 1 and 2):
>
> AP -> card3 -> queue (3,1)
> -> queue (3,2)
> -> card4 -> queue (4,1)
> -> queue (4,2)
>
> If we want to virtualize this, we can use a feature provided by the
> hardware. We basically attach a satellite control block to our main
> hardware virtualization control block and the hardware takes care of
> most of the rest.
>
> For this control block, we don't specify explicit tuples, but a list of
> cards and a list of domains. The guest will get access to the cross
> product.
>
> Because of this, we need to take care that the lists provided to
> different guests don't overlap; i.e., we need to enforce sane
> configurations. Otherwise, one guest may get access to things like
> secret keys for another guest.
>
> The idea of this patch set is to introduce a new device, the matrix
> device. This matrix device hangs off a different root and acts as the
> parent node for mdev devices.
>
> If you now want to give the tuples (4,1) and (4,2), you need to do the
> following:
>
> - Unbind the (4,1) and (4,2) tuples from their ap bus driver.
> - Bind the (4,1) and (4,2) tuples to the vfio_ap driver.
> - Create the mediated device.
> - Assign card 4 and domains 1 and 2 to the mediated device
>
> QEMU will now simply consume the mediated device and things should work.
>
> For a complete description of the architecture and concepts underlying the
> design, see the Documentation/s390/vfio-ap.txt file included with this
> patch set.
>
> v1 => v2 Change log:
> ===================
Uh, this is very long... It seems most of the changes are not related
to the basic approach, but concerned with details? The general
structure was fine last time IIRC.
> * Added documentation vfio-ap.txt
> * Renamed vfio_ap_matrix module and device driver to vfio_ap
> * Use device core device list instead of maintaining list of matrix
> devices in driver
> * Added VSIE support for AP
> * Create matrix device before registering VFIO AP device driver with the
> AP bus
> * Renamed the following files in drivers/s390/crypto:
> * vfio_ap_matrix.drv -> vfio_ap_drv
> * vfio_ap_matrix_private.h -> vfio_ap_private.h
> * vfio_ap_matrix_ops.c -> vfio_ap_ops.c
> * arch/s390/include/asm/kvm/ap-matrix-config.h
> * Renamed to kvm-ap.h
> * Changed the data type of the bit mask fields for the matrix structure
> to unsigned long and create them with DECLARE_BITMAP
> * Changed #define prefixes from AP_MATRIX to KVM_AP
> * Changed function and structure prefixes from ap_matrix to kvm_ap
> * Added function interface to check if AP Extended Addressing (APXA)
> facility is installedCRYCB_FORMAT_MASK
> * Added function interface to get the maximum ID for AP mask type
> * Added function interface to set the AP execution mode
> * arch/s390/kvm/ap-matrix-config.c
> * Renamed to kvm-ap.c
> * Changed function prefixes from ap_matrix to kvm_ap
> * Added function to check if AP Extended Addressing (APXA) facility is
> installed
> * Added function to get the maximum ID for AP mask type
> * Added function to set the AP execution mode
> * Added a boolean parameter to the functions that retrieve the APM, AQM
> and ADM bit mask fields from the CRYCB. If true, then the function
> will clear the bits in the mask before returning a reference to it
> * Added validation to verify that APM, AQM and ADM bits that are set do
> not exceed the maximum ID value allowed
> *
> * arch/s390/include/asm/kvm_host.h
> * Changed defined for ECA_AP to ECA_APIE - interpretive execution mode
> * Added a flag to struct kvm_s390_crypto to indicate whether the
> KVM_S390_VM_CPU_FEAT_AP CPU model feature for AP facilities is set
> * Added two CPU facilities features to set STFLE.12 and STFLE.15
> * arch/s390/kvm/kvm-s390.c
> * Added initialization for new KVM_S390_VM_CPU_FEAT_AP CPU model feature
> * Removed kvm_s390_apxa_installed() function
> * Changed call to kvm_s390_apxa_installed() which has been removed to a
> call to new kvm_ap_apxa_installed() function.
> * Added code to kvm_s390_vcpu_crypto_setup() to set the new CPU model
> feature flag in the kvm_s390_crypto structure
> * Added CRYCB_FORMAT_MASK to mask CRYCBD
> * arch/s390/tools/gen_facilities.c
> * Added STFLE.12 and STFLE.15 to struct facility _def
> * drivers/s390/crypto/vfio_ap_matrix_private.h
> * Changed name of file to vfio_ap.private.h
> * Changed #define prefixes from VFIO_AP_MATRIX to VFIO_AP
> * struct ap_matrix: removed list fields and locks
> * struct vfio_ap_queue: removed list field
> * Renamed functions ap_matrix_mdev_register and ap_matrix_mdev_unregister
> to vfio_ap_mdev_register and vfio_ap_mdev_unregister respectively
> * drivers/s390/crypto/vfio_ap_matrix_drv.c
> * Renamed file to drivers/s390/crypto/vfio_ap_drv.c
> * Changed all #define, structure and function prefixes to vfio_ap
> * probe function
> * Changed root device name for the matrix device to vfio_ap:
> i.e., /sys/devices/vfio_ap/matrix
> * No longer storing the AP queue device in a list, it is retrievable
> via the device core
> * Removed unnecessary check whether matrix device exists
> * Store the vfio_ap_queue structure in the private field of the
> ap_queue structure rather than using list interface
> * remove function
> * Retrieve vfio_ap_queue structure from the struct ap_queue private
> data rather than from a list
> * Removed unnecessary check
> * drivers/s390/crypto/vfio_ap_matrix_ops.c
> * Renamed file to vfio_ap_ops.c
> * Changed #define prefixes from AP_MATRIX to VFIO_AP
> * Changed function name prefixes from ap_matrix to vfio_ap
> * Removed ioctl to configure the CRYCB
> * create function
> * Removed ap_matrix_mdev_find_by_uuid() function - function is provided
> by mdev core
> * Removed available_instances verification, provided by mdev core
> * Removed check to see if mediated device exists, handled by mdev core
> * notifier function
> * Configuring matrix here instead of via ioctl
> * Set interpretive execution mode for all VCPUs
> * Removed R/O attributes to display adapters and domains
> * Added an R/O attribute to display the matrix
> * assign_control_domain mdev attribute:
> * Removed check to see if the domain is installed on the linux host
> * Added check to verify the control domain ID does not exceed the max
> value
> * assign_adapter mdev attribute:
> * Added check to verify the adapter ID does not exceed the max
> value
> * If any APQNs configured for the mediated matrix device that
> have an APID matching the adapter ID being assigned are not
> bound to the vfio_ap device driver then it is assumed that the APQN
> is bound to another driver and assignment will fail
> * assign_domain mdev attribute:
> * Added check to verify the domain ID does not exceed the max
> value
> * If any APQNs configured for the mediated matrix device that
> have an APQI matching the domain ID being assigned are not
> bound to the vfio_ap device driver then it is assumed that the APQN
> is bound to another driver and assignment will fail
> * tools/arch/s390/include/uapi/asm/kvm.h
> * removed KVM_S390_VM_CPU_FEAT_AP feature definition
>
> Tony Krowiak (15):
> KVM: s390: refactor crypto initialization
> s390: vsie: implement AP support for second level guest
> s390: zcrypt: externalize AP instructions available function
> KVM: s390: CPU model support for AP virtualization
> s390: vfio-ap: base implementation of VFIO AP device driver
> s390: vfio-ap: register matrix device with VFIO mdev framework
> KVM: s390: Interfaces to configure/deconfigure guest's AP matrix
> KVM: s390: interface to enable AP execution mode
> s390: vfio-ap: sysfs interfaces to configure adapters
> s390: vfio-ap: sysfs interfaces to configure domains
> s390: vfio-ap: sysfs interfaces to configure control domains
> s390: vfio-ap: sysfs interface to view matrix mdev matrix
> KVM: s390: Configure the guest's CRYCB
> s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl
> s390: doc: detailed specifications for AP virtualization
>
> Documentation/s390/vfio-ap.txt | 514 ++++++++++++++++++
> MAINTAINERS | 14 +
> arch/s390/Kconfig | 8 +
> arch/s390/configs/default_defconfig | 3 +
> arch/s390/configs/gcov_defconfig | 3 +
> arch/s390/configs/performance_defconfig | 3 +
> arch/s390/defconfig | 3 +
> arch/s390/include/asm/ap.h | 7 +
> arch/s390/include/asm/kvm-ap.h | 59 +++
> arch/s390/include/asm/kvm_host.h | 2 +
> arch/s390/include/uapi/asm/kvm.h | 1 +
> arch/s390/kvm/Makefile | 2 +-
> arch/s390/kvm/kvm-ap.c | 336 ++++++++++++
> arch/s390/kvm/kvm-s390.c | 66 +--
> arch/s390/kvm/kvm-s390.h | 1 +
> arch/s390/kvm/vsie.c | 71 +++-
> arch/s390/tools/gen_facilities.c | 2 +
> drivers/s390/crypto/Makefile | 4 +
> drivers/s390/crypto/ap_bus.c | 6 +
> drivers/s390/crypto/vfio_ap_drv.c | 143 +++++
> drivers/s390/crypto/vfio_ap_ops.c | 868 +++++++++++++++++++++++++++++++
> drivers/s390/crypto/vfio_ap_private.h | 45 ++
> include/uapi/linux/vfio.h | 2 +
> 23 files changed, 2096 insertions(+), 67 deletions(-)
> create mode 100644 Documentation/s390/vfio-ap.txt
> create mode 100644 arch/s390/include/asm/kvm-ap.h
> create mode 100644 arch/s390/kvm/kvm-ap.c
> create mode 100644 drivers/s390/crypto/vfio_ap_drv.c
> create mode 100644 drivers/s390/crypto/vfio_ap_ops.c
> create mode 100644 drivers/s390/crypto/vfio_ap_private.h
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 15/15] s390: doc: detailed specifications for AP virtualization
2018-02-27 14:28 ` [PATCH v2 15/15] s390: doc: detailed specifications for AP virtualization Tony Krowiak
@ 2018-02-27 15:57 ` Cornelia Huck
2018-02-27 18:12 ` Tony Krowiak
0 siblings, 1 reply; 56+ messages in thread
From: Cornelia Huck @ 2018-02-27 15:57 UTC (permalink / raw)
To: Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
fiuczy, buendgen
On Tue, 27 Feb 2018 09:28:13 -0500
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> This patch provides documentation describing the AP architecture and
> design concepts behind the virtualization of AP devices. It also
> includes an example of how to configure AP devices for exclusive
> use of KVM guests.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
> Documentation/s390/vfio-ap.txt | 514 ++++++++++++++++++++++++++++++++++++++++
> MAINTAINERS | 1 +
> 2 files changed, 515 insertions(+), 0 deletions(-)
> create mode 100644 Documentation/s390/vfio-ap.txt
>
> diff --git a/Documentation/s390/vfio-ap.txt b/Documentation/s390/vfio-ap.txt
> new file mode 100644
> index 0000000..c599f30
> --- /dev/null
> +++ b/Documentation/s390/vfio-ap.txt
> @@ -0,0 +1,514 @@
> +Introduction:
> +============
> +The Adjunct Processor (AP) facility is an IBM Z cryptographic facility comprised
> +of three AP instructions and from 1 up to 256 PCIe cryptographic adapter cards.
> +The AP devices provide cryptographic functions to all CPUs assigned to a
> +linux system running in an IBM Z system LPAR.
> +
> +The AP adapter cards are exposed via the AP bus. The motivation for vfio-ap
> +is to make AP cards available to KVM guests using the VFIO mediated device
> +framework.
Maybe drop a sentence in here that this makes heavy usage of the s390
virtualization facilities, which do the heavy lifting?
> +
> +AP Architectural Overview:
> +=========================
> +To facilitate the comprehension of the design, let's start with some
> +definitions:
> +
> +* AP adapter
> +
> + An AP adapter is an IBM Z adapter card that can perform cryptographic
> + functions. There can be from 0 to 256 adapters assigned to an LPAR. Adapters
> + assigned to the LPAR in which a linux host is running will be available to
> + the linux host. Each adapter is identified by a number from 0 to 255. When
> + installed, an AP adapter is accessed by AP instructions executed by any CPU.
> +
> +* AP domain
> +
> + An adapter is partitioned into domains. Each domain can be thought of as
> + a set of hardware registers for processing AP instructions. An adapter can
> + hold up to 256 domains. Each domain is identified by a number from 0 to 255.
> + Domains can be further classified into two types:
> +
> + * Usage domains are domains that can be accessed directly to process AP
> + commands
> +
> + * Control domains are domains that are accessed indirectly by AP
> + commands sent to a usage domain to control or change the domain, for
> + example; to set a secure private key for the domain.
> +
> +* AP Queue
> +
> + An AP queue is the means by which an AP command-request message is sent to an
> + AP usage domain inside a specific AP. An AP queue is identified by a tuple
> + comprised of an AP adapter ID (APID) and an AP queue index (APQI). The
> + APQI corresponds to a given usage domain number within the adapter. This tuple
> + forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP
> + instructions include a field containing the APQN to identify the AP queue to
> + which the AP command-request message is to be sent for processing.
> +
> +* AP Instructions:
> +
> + There are three AP instructions:
> +
> + * NQAP: to enqueue an AP command-request message to a queue
> + * DQAP: to dequeue an AP command-reply message from a queue
> + * PQAP: to administer the queues
Do you also want to explain how these entities show up on the ap bus in
Linux? It might make the explanations further down easier to
understand. (Is there any document for the ap bus you could point to?)
> +
> +Start Interpretive Execution (SIE) Instruction:
> +==============================================
Call this "AP and SIE" or so? You're not trying to explain the whole
SIE architecture :)
> +Let's now see how AP instructions are interpreted by the hardware.
(...)
> +Reserve APQNs for exclusive use of KVM guests
> +---------------------------------------------
> +The following block diagram illustrates the mechanism by which APQNs are
> +reserved:
> +
> + +------------------+
> + remove | | unbind
> + +------------------->+ cex4queue driver +<-----------+
> + | | | |
> + | +------------------+ |
> + | |
> + | |
> + | |
> ++--------+---------+ register +------------------+ +-----+------+
> +| +<---------+ | bind | |
> +| ap_bus | | vfio_ap driver +<-----+ admin |
> +| +--------->+ | | |
> ++------------------+ probe +---+--------+-----+ +------------+
> + | |
> + create | | store APQN
> + | |
> + v v
> + +---+--------+-----+
> + | |
> + | matrix device |
> + | |
> + +------------------+
> +
Thank you for including diagrams, these are really helpful.
(...)
> +Initialize the CPU model feature for AP
> +---------------------------------------
> +This design exploits a feature of the SIE architecture called interpretive
> +execution (IE). When IE is enabled for a KVM guest, the AP instructions
> +executed in the guest will be interpreted by the firmware and the commands
> +contained therein will be passed directly through to an AP device assigned to
> +the linux host. In order to enable interpretive execution for a KVM guest, SIE
> +must have access to the AP facilities installed on the linux host. A new CPU
> +model feature is introduced by this design to indicate that the guest will
> +directly access the host AP facilities. This feature will be enabled by the
> +kernel only if the AP facilities are installed on the linux host. This feature
> +is turned on for the guest via the qemu command line:
> +
> + /usr/bin/qemu-system-s390x ... -cpu xxx,ap=on
> +
> + Where xxx is the CPU model being used.
> +
> +If the CPU model feature is not enabled by the kernel, QEMU
> +will fail and report that the feature is not supported.
The cpu model interface is supposed to be user space agnostic, although
it is only used by QEMU in practice. Mark this as an example more
clearly?
(...)
I have not looked at this in detail (will probably come back to this
later), but this looks like a useful document.
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 00/15] s390: vfio-ap: guest dedicated crypto adapters
2018-02-27 14:58 ` [PATCH v2 00/15] s390: vfio-ap: guest dedicated crypto adapters Cornelia Huck
@ 2018-02-27 15:57 ` Tony Krowiak
0 siblings, 0 replies; 56+ messages in thread
From: Tony Krowiak @ 2018-02-27 15:57 UTC (permalink / raw)
To: Cornelia Huck
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
fiuczy, buendgen
On 02/27/2018 09:58 AM, Cornelia Huck wrote:
> On Tue, 27 Feb 2018 09:27:58 -0500
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> On s390, we have cryptographic coprocessor cards, which are modeled on
>> Linux as devices on the AP bus. Each card can be partitioned into domains
>> which can be thought of as a set of hardware registers for processing
>> crypto commands. Crypto commands are sent to a specific domain within a
>> card is via a queue which is identified as a (card,domain) tuple. We model
>> this something like the following (assuming we have access to cards 3 and
>> 4 and domains 1 and 2):
>>
>> AP -> card3 -> queue (3,1)
>> -> queue (3,2)
>> -> card4 -> queue (4,1)
>> -> queue (4,2)
>>
>> If we want to virtualize this, we can use a feature provided by the
>> hardware. We basically attach a satellite control block to our main
>> hardware virtualization control block and the hardware takes care of
>> most of the rest.
>>
>> For this control block, we don't specify explicit tuples, but a list of
>> cards and a list of domains. The guest will get access to the cross
>> product.
>>
>> Because of this, we need to take care that the lists provided to
>> different guests don't overlap; i.e., we need to enforce sane
>> configurations. Otherwise, one guest may get access to things like
>> secret keys for another guest.
>>
>> The idea of this patch set is to introduce a new device, the matrix
>> device. This matrix device hangs off a different root and acts as the
>> parent node for mdev devices.
>>
>> If you now want to give the tuples (4,1) and (4,2), you need to do the
>> following:
>>
>> - Unbind the (4,1) and (4,2) tuples from their ap bus driver.
>> - Bind the (4,1) and (4,2) tuples to the vfio_ap driver.
>> - Create the mediated device.
>> - Assign card 4 and domains 1 and 2 to the mediated device
>>
>> QEMU will now simply consume the mediated device and things should work.
>>
>> For a complete description of the architecture and concepts underlying the
>> design, see the Documentation/s390/vfio-ap.txt file included with this
>> patch set.
>>
>> v1 => v2 Change log:
>> ===================
> Uh, this is very long... It seems most of the changes are not related
> to the basic approach, but concerned with details? The general
> structure was fine last time IIRC.
I'm apologize, I am not well-versed in how much detail is appropriate. This
also went through several internal reviews before re-posting to the kernel
lists.
>
>> * Added documentation vfio-ap.txt
>> * Renamed vfio_ap_matrix module and device driver to vfio_ap
>> * Use device core device list instead of maintaining list of matrix
>> devices in driver
>> * Added VSIE support for AP
>> * Create matrix device before registering VFIO AP device driver with the
>> AP bus
>> * Renamed the following files in drivers/s390/crypto:
>> * vfio_ap_matrix.drv -> vfio_ap_drv
>> * vfio_ap_matrix_private.h -> vfio_ap_private.h
>> * vfio_ap_matrix_ops.c -> vfio_ap_ops.c
>> * arch/s390/include/asm/kvm/ap-matrix-config.h
>> * Renamed to kvm-ap.h
>> * Changed the data type of the bit mask fields for the matrix structure
>> to unsigned long and create them with DECLARE_BITMAP
>> * Changed #define prefixes from AP_MATRIX to KVM_AP
>> * Changed function and structure prefixes from ap_matrix to kvm_ap
>> * Added function interface to check if AP Extended Addressing (APXA)
>> facility is installedCRYCB_FORMAT_MASK
>> * Added function interface to get the maximum ID for AP mask type
>> * Added function interface to set the AP execution mode
>> * arch/s390/kvm/ap-matrix-config.c
>> * Renamed to kvm-ap.c
>> * Changed function prefixes from ap_matrix to kvm_ap
>> * Added function to check if AP Extended Addressing (APXA) facility is
>> installed
>> * Added function to get the maximum ID for AP mask type
>> * Added function to set the AP execution mode
>> * Added a boolean parameter to the functions that retrieve the APM, AQM
>> and ADM bit mask fields from the CRYCB. If true, then the function
>> will clear the bits in the mask before returning a reference to it
>> * Added validation to verify that APM, AQM and ADM bits that are set do
>> not exceed the maximum ID value allowed
>> *
>> * arch/s390/include/asm/kvm_host.h
>> * Changed defined for ECA_AP to ECA_APIE - interpretive execution mode
>> * Added a flag to struct kvm_s390_crypto to indicate whether the
>> KVM_S390_VM_CPU_FEAT_AP CPU model feature for AP facilities is set
>> * Added two CPU facilities features to set STFLE.12 and STFLE.15
>> * arch/s390/kvm/kvm-s390.c
>> * Added initialization for new KVM_S390_VM_CPU_FEAT_AP CPU model feature
>> * Removed kvm_s390_apxa_installed() function
>> * Changed call to kvm_s390_apxa_installed() which has been removed to a
>> call to new kvm_ap_apxa_installed() function.
>> * Added code to kvm_s390_vcpu_crypto_setup() to set the new CPU model
>> feature flag in the kvm_s390_crypto structure
>> * Added CRYCB_FORMAT_MASK to mask CRYCBD
>> * arch/s390/tools/gen_facilities.c
>> * Added STFLE.12 and STFLE.15 to struct facility _def
>> * drivers/s390/crypto/vfio_ap_matrix_private.h
>> * Changed name of file to vfio_ap.private.h
>> * Changed #define prefixes from VFIO_AP_MATRIX to VFIO_AP
>> * struct ap_matrix: removed list fields and locks
>> * struct vfio_ap_queue: removed list field
>> * Renamed functions ap_matrix_mdev_register and ap_matrix_mdev_unregister
>> to vfio_ap_mdev_register and vfio_ap_mdev_unregister respectively
>> * drivers/s390/crypto/vfio_ap_matrix_drv.c
>> * Renamed file to drivers/s390/crypto/vfio_ap_drv.c
>> * Changed all #define, structure and function prefixes to vfio_ap
>> * probe function
>> * Changed root device name for the matrix device to vfio_ap:
>> i.e., /sys/devices/vfio_ap/matrix
>> * No longer storing the AP queue device in a list, it is retrievable
>> via the device core
>> * Removed unnecessary check whether matrix device exists
>> * Store the vfio_ap_queue structure in the private field of the
>> ap_queue structure rather than using list interface
>> * remove function
>> * Retrieve vfio_ap_queue structure from the struct ap_queue private
>> data rather than from a list
>> * Removed unnecessary check
>> * drivers/s390/crypto/vfio_ap_matrix_ops.c
>> * Renamed file to vfio_ap_ops.c
>> * Changed #define prefixes from AP_MATRIX to VFIO_AP
>> * Changed function name prefixes from ap_matrix to vfio_ap
>> * Removed ioctl to configure the CRYCB
>> * create function
>> * Removed ap_matrix_mdev_find_by_uuid() function - function is provided
>> by mdev core
>> * Removed available_instances verification, provided by mdev core
>> * Removed check to see if mediated device exists, handled by mdev core
>> * notifier function
>> * Configuring matrix here instead of via ioctl
>> * Set interpretive execution mode for all VCPUs
>> * Removed R/O attributes to display adapters and domains
>> * Added an R/O attribute to display the matrix
>> * assign_control_domain mdev attribute:
>> * Removed check to see if the domain is installed on the linux host
>> * Added check to verify the control domain ID does not exceed the max
>> value
>> * assign_adapter mdev attribute:
>> * Added check to verify the adapter ID does not exceed the max
>> value
>> * If any APQNs configured for the mediated matrix device that
>> have an APID matching the adapter ID being assigned are not
>> bound to the vfio_ap device driver then it is assumed that the APQN
>> is bound to another driver and assignment will fail
>> * assign_domain mdev attribute:
>> * Added check to verify the domain ID does not exceed the max
>> value
>> * If any APQNs configured for the mediated matrix device that
>> have an APQI matching the domain ID being assigned are not
>> bound to the vfio_ap device driver then it is assumed that the APQN
>> is bound to another driver and assignment will fail
>> * tools/arch/s390/include/uapi/asm/kvm.h
>> * removed KVM_S390_VM_CPU_FEAT_AP feature definition
>>
>> Tony Krowiak (15):
>> KVM: s390: refactor crypto initialization
>> s390: vsie: implement AP support for second level guest
>> s390: zcrypt: externalize AP instructions available function
>> KVM: s390: CPU model support for AP virtualization
>> s390: vfio-ap: base implementation of VFIO AP device driver
>> s390: vfio-ap: register matrix device with VFIO mdev framework
>> KVM: s390: Interfaces to configure/deconfigure guest's AP matrix
>> KVM: s390: interface to enable AP execution mode
>> s390: vfio-ap: sysfs interfaces to configure adapters
>> s390: vfio-ap: sysfs interfaces to configure domains
>> s390: vfio-ap: sysfs interfaces to configure control domains
>> s390: vfio-ap: sysfs interface to view matrix mdev matrix
>> KVM: s390: Configure the guest's CRYCB
>> s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl
>> s390: doc: detailed specifications for AP virtualization
>>
>> Documentation/s390/vfio-ap.txt | 514 ++++++++++++++++++
>> MAINTAINERS | 14 +
>> arch/s390/Kconfig | 8 +
>> arch/s390/configs/default_defconfig | 3 +
>> arch/s390/configs/gcov_defconfig | 3 +
>> arch/s390/configs/performance_defconfig | 3 +
>> arch/s390/defconfig | 3 +
>> arch/s390/include/asm/ap.h | 7 +
>> arch/s390/include/asm/kvm-ap.h | 59 +++
>> arch/s390/include/asm/kvm_host.h | 2 +
>> arch/s390/include/uapi/asm/kvm.h | 1 +
>> arch/s390/kvm/Makefile | 2 +-
>> arch/s390/kvm/kvm-ap.c | 336 ++++++++++++
>> arch/s390/kvm/kvm-s390.c | 66 +--
>> arch/s390/kvm/kvm-s390.h | 1 +
>> arch/s390/kvm/vsie.c | 71 +++-
>> arch/s390/tools/gen_facilities.c | 2 +
>> drivers/s390/crypto/Makefile | 4 +
>> drivers/s390/crypto/ap_bus.c | 6 +
>> drivers/s390/crypto/vfio_ap_drv.c | 143 +++++
>> drivers/s390/crypto/vfio_ap_ops.c | 868 +++++++++++++++++++++++++++++++
>> drivers/s390/crypto/vfio_ap_private.h | 45 ++
>> include/uapi/linux/vfio.h | 2 +
>> 23 files changed, 2096 insertions(+), 67 deletions(-)
>> create mode 100644 Documentation/s390/vfio-ap.txt
>> create mode 100644 arch/s390/include/asm/kvm-ap.h
>> create mode 100644 arch/s390/kvm/kvm-ap.c
>> create mode 100644 drivers/s390/crypto/vfio_ap_drv.c
>> create mode 100644 drivers/s390/crypto/vfio_ap_ops.c
>> create mode 100644 drivers/s390/crypto/vfio_ap_private.h
>>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 15/15] s390: doc: detailed specifications for AP virtualization
2018-02-27 15:57 ` Cornelia Huck
@ 2018-02-27 18:12 ` Tony Krowiak
0 siblings, 0 replies; 56+ messages in thread
From: Tony Krowiak @ 2018-02-27 18:12 UTC (permalink / raw)
To: Cornelia Huck
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
fiuczy, buendgen
On 02/27/2018 10:57 AM, Cornelia Huck wrote:
> On Tue, 27 Feb 2018 09:28:13 -0500
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> This patch provides documentation describing the AP architecture and
>> design concepts behind the virtualization of AP devices. It also
>> includes an example of how to configure AP devices for exclusive
>> use of KVM guests.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>> Documentation/s390/vfio-ap.txt | 514 ++++++++++++++++++++++++++++++++++++++++
>> MAINTAINERS | 1 +
>> 2 files changed, 515 insertions(+), 0 deletions(-)
>> create mode 100644 Documentation/s390/vfio-ap.txt
>>
>> diff --git a/Documentation/s390/vfio-ap.txt b/Documentation/s390/vfio-ap.txt
>> new file mode 100644
>> index 0000000..c599f30
>> --- /dev/null
>> +++ b/Documentation/s390/vfio-ap.txt
>> @@ -0,0 +1,514 @@
>> +Introduction:
>> +============
>> +The Adjunct Processor (AP) facility is an IBM Z cryptographic facility comprised
>> +of three AP instructions and from 1 up to 256 PCIe cryptographic adapter cards.
>> +The AP devices provide cryptographic functions to all CPUs assigned to a
>> +linux system running in an IBM Z system LPAR.
>> +
>> +The AP adapter cards are exposed via the AP bus. The motivation for vfio-ap
>> +is to make AP cards available to KVM guests using the VFIO mediated device
>> +framework.
> Maybe drop a sentence in here that this makes heavy usage of the s390
> virtualization facilities, which do the heavy lifting?
Will do.
>
>> +
>> +AP Architectural Overview:
>> +=========================
>> +To facilitate the comprehension of the design, let's start with some
>> +definitions:
>> +
>> +* AP adapter
>> +
>> + An AP adapter is an IBM Z adapter card that can perform cryptographic
>> + functions. There can be from 0 to 256 adapters assigned to an LPAR. Adapters
>> + assigned to the LPAR in which a linux host is running will be available to
>> + the linux host. Each adapter is identified by a number from 0 to 255. When
>> + installed, an AP adapter is accessed by AP instructions executed by any CPU.
>> +
>> +* AP domain
>> +
>> + An adapter is partitioned into domains. Each domain can be thought of as
>> + a set of hardware registers for processing AP instructions. An adapter can
>> + hold up to 256 domains. Each domain is identified by a number from 0 to 255.
>> + Domains can be further classified into two types:
>> +
>> + * Usage domains are domains that can be accessed directly to process AP
>> + commands
>> +
>> + * Control domains are domains that are accessed indirectly by AP
>> + commands sent to a usage domain to control or change the domain, for
>> + example; to set a secure private key for the domain.
>> +
>> +* AP Queue
>> +
>> + An AP queue is the means by which an AP command-request message is sent to an
>> + AP usage domain inside a specific AP. An AP queue is identified by a tuple
>> + comprised of an AP adapter ID (APID) and an AP queue index (APQI). The
>> + APQI corresponds to a given usage domain number within the adapter. This tuple
>> + forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP
>> + instructions include a field containing the APQN to identify the AP queue to
>> + which the AP command-request message is to be sent for processing.
>> +
>> +* AP Instructions:
>> +
>> + There are three AP instructions:
>> +
>> + * NQAP: to enqueue an AP command-request message to a queue
>> + * DQAP: to dequeue an AP command-reply message from a queue
>> + * PQAP: to administer the queues
> Do you also want to explain how these entities show up on the ap bus in
> Linux? It might make the explanations further down easier to
> understand. (Is there any document for the ap bus you could point to?)
I am not aware of any documentation for the AP bus, but I can provide a
few sentences to explain how AP devices are managed by the AP bus.
>
>> +
>> +Start Interpretive Execution (SIE) Instruction:
>> +==============================================
> Call this "AP and SIE" or so? You're not trying to explain the whole
> SIE architecture :)
Okay
>
>
>> +Let's now see how AP instructions are interpreted by the hardware.
> (...)
>
>> +Reserve APQNs for exclusive use of KVM guests
>> +---------------------------------------------
>> +The following block diagram illustrates the mechanism by which APQNs are
>> +reserved:
>> +
>> + +------------------+
>> + remove | | unbind
>> + +------------------->+ cex4queue driver +<-----------+
>> + | | | |
>> + | +------------------+ |
>> + | |
>> + | |
>> + | |
>> ++--------+---------+ register +------------------+ +-----+------+
>> +| +<---------+ | bind | |
>> +| ap_bus | | vfio_ap driver +<-----+ admin |
>> +| +--------->+ | | |
>> ++------------------+ probe +---+--------+-----+ +------------+
>> + | |
>> + create | | store APQN
>> + | |
>> + v v
>> + +---+--------+-----+
>> + | |
>> + | matrix device |
>> + | |
>> + +------------------+
>> +
> Thank you for including diagrams, these are really helpful.
You are welcomed
>
> (...)
>
>> +Initialize the CPU model feature for AP
>> +---------------------------------------
>> +This design exploits a feature of the SIE architecture called interpretive
>> +execution (IE). When IE is enabled for a KVM guest, the AP instructions
>> +executed in the guest will be interpreted by the firmware and the commands
>> +contained therein will be passed directly through to an AP device assigned to
>> +the linux host. In order to enable interpretive execution for a KVM guest, SIE
>> +must have access to the AP facilities installed on the linux host. A new CPU
>> +model feature is introduced by this design to indicate that the guest will
>> +directly access the host AP facilities. This feature will be enabled by the
>> +kernel only if the AP facilities are installed on the linux host. This feature
>> +must be turned turned on for the guest in order to ac:
>> +
>> + /usr/bin/qemu-system-s390x ... -cpu xxx,ap=on
>> +
>> + Where xxx is the CPU model being used.
>> +
>> +If the CPU model feature is not enabled by the kernel, QEMU
>> +will fail and report that the feature is not supported.
> The cpu model interface is supposed to be user space agnostic, although
> it is only used by QEMU in practice. Mark this as an example more
> clearly?
Will do
>
> (...)
>
> I have not looked at this in detail (will probably come back to this
> later), but this looks like a useful document.
Excellent, I'm glad to hear that. It took some time to put together.
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 02/15] s390: vsie: implement AP support for second level guest
2018-02-27 14:28 ` [PATCH v2 02/15] s390: vsie: implement AP support for second level guest Tony Krowiak
@ 2018-02-28 9:39 ` David Hildenbrand
2018-02-28 10:03 ` Pierre Morel
0 siblings, 1 reply; 56+ messages in thread
From: David Hildenbrand @ 2018-02-28 9:39 UTC (permalink / raw)
To: Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen
On 27.02.2018 15:28, Tony Krowiak wrote:
> Set effective masks and CRYCB format in the shadow copy of the
> guest level 2 CRYCB.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
> arch/s390/include/asm/kvm-ap.h | 2 +
> arch/s390/kvm/kvm-ap.c | 5 +++
> arch/s390/kvm/vsie.c | 71 +++++++++++++++++++++++++++++++++-------
> 3 files changed, 66 insertions(+), 12 deletions(-)
>
> diff --git a/arch/s390/include/asm/kvm-ap.h b/arch/s390/include/asm/kvm-ap.h
> index 4e43117..ef749e7 100644
> --- a/arch/s390/include/asm/kvm-ap.h
> +++ b/arch/s390/include/asm/kvm-ap.h
> @@ -13,4 +13,6 @@
>
> void kvm_ap_set_crycb_format(struct kvm *kvm, __u32 *crycbd);
>
> +int kvm_ap_get_crycb_format(struct kvm *kvm);
> +
> #endif /* _ASM_KVM_AP */
> diff --git a/arch/s390/kvm/kvm-ap.c b/arch/s390/kvm/kvm-ap.c
> index 5305f4c..bafe63b 100644
> --- a/arch/s390/kvm/kvm-ap.c
> +++ b/arch/s390/kvm/kvm-ap.c
> @@ -11,6 +11,11 @@
>
> #include "kvm-s390.h"
>
> +int kvm_ap_get_crycb_format(struct kvm *kvm)
> +{
> + return kvm->arch.crypto.crycbd & CRYCB_FORMAT_MASK;
> +}
Why exactly do we need this function? If there are no other users, just
do it directly in the code below.
> +
> static int kvm_ap_apxa_installed(void)
> {
> int ret;
> diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c
> index 8961e39..93076ba 100644
> --- a/arch/s390/kvm/vsie.c
> +++ b/arch/s390/kvm/vsie.c
> @@ -18,6 +18,7 @@
> #include <asm/sclp.h>
> #include <asm/nmi.h>
> #include <asm/dis.h>
> +#include <asm/kvm-ap.h>
> #include "kvm-s390.h"
> #include "gaccess.h"
>
> @@ -137,12 +138,56 @@ static int prepare_cpuflags(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
> }
>
> /*
> + * Set up the effective masks for the shadow copy of the crycb. The effective
> + * masks for guest 3 are set by performing a logical bitwise AND of the guest 3
> + * masks with the guest 2 masks.
> + */
> +static void set_crycb_emasks(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
> +{
> + int fmt = kvm_ap_get_crycb_format(vcpu->kvm);
> + unsigned long *mask1, *mask2;
> +
> + switch (fmt) {
> + case CRYCB_FORMAT1:
> + case CRYCB_FORMAT2:
Assume L2 gave us FORMAT0 and we are using FORMAT2. You would copy wrong
bits. See below, maybe we need conversion.
> + mask1 = (unsigned long *)vsie_page->crycb.apcb1.apm;
> + mask2 = (unsigned long *)
> + vcpu->kvm->arch.crypto.crycb->apcb1.apm;
> + bitmap_and(mask1, mask1, mask2, APCB1_MASK_SIZE);
> +
> + mask1 = (unsigned long *)vsie_page->crycb.apcb1.aqm;
> + mask2 = (unsigned long *)
> + vcpu->kvm->arch.crypto.crycb->apcb1.aqm;
> + bitmap_and(mask1, mask1, mask2, APCB1_MASK_SIZE);
> +
> + mask1 = (unsigned long *)vsie_page->crycb.apcb1.adm;
> + mask2 = (unsigned long *)
> + vcpu->kvm->arch.crypto.crycb->apcb1.adm;
> + bitmap_and(mask1, mask1, mask2, APCB1_MASK_SIZE);
> + break;
> + default:
> + mask1 = (unsigned long *)vsie_page->crycb.apcb0.apm;
> + mask2 = (unsigned long *)
> + vcpu->kvm->arch.crypto.crycb->apcb1.apm;
> + bitmap_and(mask1, mask1, mask2, APCB0_MASK_SIZE);
> +
> + mask1 = (unsigned long *)vsie_page->crycb.apcb0.aqm;
> + mask2 = (unsigned long *)
> + vcpu->kvm->arch.crypto.crycb->apcb1.aqm;
> + bitmap_and(mask1, mask1, mask2, APCB0_MASK_SIZE);
> +
> + mask1 = (unsigned long *)vsie_page->crycb.apcb0.adm;
> + mask2 = (unsigned long *)
> + vcpu->kvm->arch.crypto.crycb->apcb1.adm;
> + bitmap_and(mask1, mask1, mask2, APCB0_MASK_SIZE);
> + break;
> + }
> +}
> +
> +/*
> * Create a shadow copy of the crycb block and setup key wrapping, if
> * requested for guest 3 and enabled for guest 2.
> *
> - * We only accept format-1 (no AP in g2), but convert it into format-2
> - * There is nothing to do for format-0.
> - *
> * Returns: - 0 if shadowed or nothing to do
> * - > 0 if control has to be given to guest 2
> */
> @@ -155,9 +200,17 @@ static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
> unsigned long *b1, *b2;
> u8 ecb3_flags;
>
> - scb_s->crycbd = 0;
> - if (!(crycbd_o & vcpu->arch.sie_block->crycbd & CRYCB_FORMAT1))
> - return 0;
> + scb_s->crycbd = ((__u32)(__u64) &vsie_page->crycb);
> + kvm_ap_set_crycb_format(vcpu->kvm, &scb_s->crycbd);
> +
> + /* copy the crycb */
> + if (read_guest_real(vcpu, crycb_addr, &vsie_page->crycb,
> + sizeof(vsie_page->crycb)))
> + return set_validity_icpt(scb_s, 0x0035U);
This looks wrong. You are always copying from L2, although L2 might not
even set up a crycb for L3. You removed the important crycbd_o check.
Don't we need the following?
a) Determine the _allowed_ crycb format for L2 -> L3. (How can we tell
which format L2 is allowed to use for L3)
b) Determine the target crycb format. This is basically
vm_ap_set_crycb_format afaics.
c) Convert the given crycb to the target crycb format.
So importantly, can you clarify (as I don't have access to documentation):
- When is L2 allowed to use format-0? So when will the format not be
ignored but is actually used?
- When is L2 allowed to use fortma-2? So when will the format not be
ignored but is actually used?
- When does the SIE start interpreting AP instructions? So how can we
hinder it from doing so? Empty masks? crycb format?
> +
> + /* set up the effective masks */
> + set_crycb_emasks(vcpu, vsie_page);
> +
> /* format-1 is supported with message-security-assist extension 3 */
> if (!test_kvm_facility(vcpu->kvm, 76))
> return 0;
You now already copied the wrapping keys. So they could be !0. Please
add a comment why we don't care.
/* wrapping keys are ignored without ECB3_AES / ECB3_DEA */
> @@ -172,13 +225,7 @@ static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
> else if (!crycb_addr)
> return set_validity_icpt(scb_s, 0x0039U);
>
> - /* copy only the wrapping keys */
> - if (read_guest_real(vcpu, crycb_addr + 72, &vsie_page->crycb, 56))
> - return set_validity_icpt(scb_s, 0x0035U);
> -
> scb_s->ecb3 |= ecb3_flags;
> - scb_s->crycbd = ((__u32)(__u64) &vsie_page->crycb) | CRYCB_FORMAT1 |
> - CRYCB_FORMAT2;
>
> /* xor both blocks in one run */
> b1 = (unsigned long *) vsie_page->crycb.dea_wrapping_key_mask;
>
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 08/15] KVM: s390: interface to enable AP execution mode
2018-02-27 14:28 ` [PATCH v2 08/15] KVM: s390: interface to enable AP execution mode Tony Krowiak
@ 2018-02-28 9:42 ` David Hildenbrand
2018-02-28 20:37 ` Tony Krowiak
2018-02-28 9:44 ` David Hildenbrand
1 sibling, 1 reply; 56+ messages in thread
From: David Hildenbrand @ 2018-02-28 9:42 UTC (permalink / raw)
To: Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen
On 27.02.2018 15:28, Tony Krowiak wrote:
> Introduces a new interface to enable AP interpretive
> execution (IE) mode for the KVM guest. When running
> with IE mode enabled, AP instructions executed on the
> KVM guest will be interpreted by the firmware and
> passed directly through to an AP device installed on
> the system. The CPU model feature for AP must
> be enabled for the KVM guest in order to enable
> interpretive execution mode.
>
> This interface will be used in a subsequent patch
> by the VFIO AP device driver.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
> arch/s390/include/asm/kvm-ap.h | 2 ++
> arch/s390/include/asm/kvm_host.h | 1 +
> arch/s390/kvm/kvm-ap.c | 27 +++++++++++++++++++++++++++
> arch/s390/kvm/kvm-s390.h | 1 +
> 4 files changed, 31 insertions(+), 0 deletions(-)
>
> diff --git a/arch/s390/include/asm/kvm-ap.h b/arch/s390/include/asm/kvm-ap.h
> index 46e7c5b..6bd6bfb 100644
> --- a/arch/s390/include/asm/kvm-ap.h
> +++ b/arch/s390/include/asm/kvm-ap.h
> @@ -51,4 +51,6 @@ struct kvm_ap_matrix {
>
> void kvm_ap_deconfigure_matrix(struct kvm *kvm);
>
> +int kvm_ap_enable_ie_mode(struct kvm *kvm);
> +
> #endif /* _ASM_KVM_AP */
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index a4c77d3..1eebdd6 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -186,6 +186,7 @@ struct kvm_s390_sie_block {
> #define ECA_AIV 0x00200000
> #define ECA_VX 0x00020000
> #define ECA_PROTEXCI 0x00002000
> +#define ECA_APIE 0x00000008
> #define ECA_SII 0x00000001
> __u32 eca; /* 0x004c */
> #define ICPT_INST 0x04
> diff --git a/arch/s390/kvm/kvm-ap.c b/arch/s390/kvm/kvm-ap.c
> index bb29045..862e54b 100644
> --- a/arch/s390/kvm/kvm-ap.c
> +++ b/arch/s390/kvm/kvm-ap.c
> @@ -307,3 +307,30 @@ void kvm_ap_deconfigure_matrix(struct kvm *kvm)
> kvm_ap_clear_crycb_masks(kvm);
> }
> EXPORT_SYMBOL(kvm_ap_deconfigure_matrix);
> +
> +/**
> + * kvm_ap_enable_ie_mode
> + *
> + * Enable interpretrive execution of AP instructions for the guest. When
> + * enabled, AP instructions executed on the guest will be interpreted and
> + * passed through to an AP installed on the host system.
> + *
> + * Returns 0 if interpretrive execution is enabled. Returns -EOPNOTSUPP
> + * if AP facilities are not installed for the guest.
> + *
> + * @kvm: the guest's kvm structure
> + */
> +int kvm_ap_enable_ie_mode(struct kvm *kvm)
> +{
> + int i;
> + struct kvm_vcpu *vcpu;
> +
> + if (!test_kvm_cpu_feat(kvm, KVM_S390_VM_CPU_FEAT_AP))
> + return -EOPNOTSUPP;
> +
> + kvm_for_each_vcpu(i, vcpu, kvm)
> + vcpu->arch.sie_block->eca |= ECA_APIE;
> +
> + return 0;
> +}
> +EXPORT_SYMBOL(kvm_ap_enable_ie_mode);
> diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
> index 1b5621f..3142541 100644
> --- a/arch/s390/kvm/kvm-s390.h
> +++ b/arch/s390/kvm/kvm-s390.h
> @@ -18,6 +18,7 @@
> #include <asm/facility.h>
> #include <asm/processor.h>
> #include <asm/sclp.h>
> +#include <asm/ap.h>
>
> /* Transactional Memory Execution related macros */
> #define IS_TE_ENABLED(vcpu) ((vcpu->arch.sie_block->ecb & ECB_TE))
>
What about VSIE?
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 08/15] KVM: s390: interface to enable AP execution mode
2018-02-27 14:28 ` [PATCH v2 08/15] KVM: s390: interface to enable AP execution mode Tony Krowiak
2018-02-28 9:42 ` David Hildenbrand
@ 2018-02-28 9:44 ` David Hildenbrand
2018-02-28 20:39 ` Tony Krowiak
1 sibling, 1 reply; 56+ messages in thread
From: David Hildenbrand @ 2018-02-28 9:44 UTC (permalink / raw)
To: Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen
On 27.02.2018 15:28, Tony Krowiak wrote:
> Introduces a new interface to enable AP interpretive
> execution (IE) mode for the KVM guest. When running
> with IE mode enabled, AP instructions executed on the
> KVM guest will be interpreted by the firmware and
> passed directly through to an AP device installed on
> the system. The CPU model feature for AP must
> be enabled for the KVM guest in order to enable
> interpretive execution mode.
>
> This interface will be used in a subsequent patch
> by the VFIO AP device driver.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
> arch/s390/include/asm/kvm-ap.h | 2 ++
> arch/s390/include/asm/kvm_host.h | 1 +
> arch/s390/kvm/kvm-ap.c | 27 +++++++++++++++++++++++++++
> arch/s390/kvm/kvm-s390.h | 1 +
> 4 files changed, 31 insertions(+), 0 deletions(-)
>
> diff --git a/arch/s390/include/asm/kvm-ap.h b/arch/s390/include/asm/kvm-ap.h
> index 46e7c5b..6bd6bfb 100644
> --- a/arch/s390/include/asm/kvm-ap.h
> +++ b/arch/s390/include/asm/kvm-ap.h
> @@ -51,4 +51,6 @@ struct kvm_ap_matrix {
>
> void kvm_ap_deconfigure_matrix(struct kvm *kvm);
>
> +int kvm_ap_enable_ie_mode(struct kvm *kvm);
> +
> #endif /* _ASM_KVM_AP */
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index a4c77d3..1eebdd6 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -186,6 +186,7 @@ struct kvm_s390_sie_block {
> #define ECA_AIV 0x00200000
> #define ECA_VX 0x00020000
> #define ECA_PROTEXCI 0x00002000
> +#define ECA_APIE 0x00000008
> #define ECA_SII 0x00000001
> __u32 eca; /* 0x004c */
> #define ICPT_INST 0x04
> diff --git a/arch/s390/kvm/kvm-ap.c b/arch/s390/kvm/kvm-ap.c
> index bb29045..862e54b 100644
> --- a/arch/s390/kvm/kvm-ap.c
> +++ b/arch/s390/kvm/kvm-ap.c
> @@ -307,3 +307,30 @@ void kvm_ap_deconfigure_matrix(struct kvm *kvm)
> kvm_ap_clear_crycb_masks(kvm);
> }
> EXPORT_SYMBOL(kvm_ap_deconfigure_matrix);
> +
> +/**
> + * kvm_ap_enable_ie_mode
> + *
> + * Enable interpretrive execution of AP instructions for the guest. When
> + * enabled, AP instructions executed on the guest will be interpreted and
> + * passed through to an AP installed on the host system.
> + *
> + * Returns 0 if interpretrive execution is enabled. Returns -EOPNOTSUPP
> + * if AP facilities are not installed for the guest.
> + *
> + * @kvm: the guest's kvm structure
> + */
> +int kvm_ap_enable_ie_mode(struct kvm *kvm)
> +{
> + int i;
> + struct kvm_vcpu *vcpu;
> +
> + if (!test_kvm_cpu_feat(kvm, KVM_S390_VM_CPU_FEAT_AP))
> + return -EOPNOTSUPP;
> +
> + kvm_for_each_vcpu(i, vcpu, kvm)
> + vcpu->arch.sie_block->eca |= ECA_APIE;
> +
> + return 0;
> +}
> +EXPORT_SYMBOL(kvm_ap_enable_ie_mode);
> diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
> index 1b5621f..3142541 100644
> --- a/arch/s390/kvm/kvm-s390.h
> +++ b/arch/s390/kvm/kvm-s390.h
> @@ -18,6 +18,7 @@
> #include <asm/facility.h>
> #include <asm/processor.h>
> #include <asm/sclp.h>
> +#include <asm/ap.h>
>
> /* Transactional Memory Execution related macros */
> #define IS_TE_ENABLED(vcpu) ((vcpu->arch.sie_block->ecb & ECB_TE))
>
And also, what about hot-plugged CPUs?
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 04/15] KVM: s390: CPU model support for AP virtualization
2018-02-27 14:28 ` [PATCH v2 04/15] KVM: s390: CPU model support for AP virtualization Tony Krowiak
@ 2018-02-28 9:48 ` David Hildenbrand
2018-02-28 11:40 ` Christian Borntraeger
0 siblings, 1 reply; 56+ messages in thread
From: David Hildenbrand @ 2018-02-28 9:48 UTC (permalink / raw)
To: Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen
On 27.02.2018 15:28, Tony Krowiak wrote:
> Introduces a new CPU model feature and two CPU model
> facilities to support AP virtualization for KVM guests.
>
> CPU model feature:
>
> The KVM_S390_VM_CPU_FEAT_AP feature indicates that the
> AP facilities are installed on the KVM guest. This
> feature will be enabled by the kernel only if the AP
> facilities are installed on the linux host. This feature
> must be specifically turned on for the KVM guest from
> userspace to allow guest access to AP devices installed
> on the linux host.
>
> CPU model facilities:
>
> 1. AP Query Configuration Information (QCI) facility is installed.
>
> This is indicated by setting facilities bit 12 for
> the guest. The kernel will not enable this facility
> for the guest if it is not set on the host. This facility
> must not be set by userspace if the KVM_S390_VM_CPU_FEAT_AP
> feature is not installed.
>
> 2. AP Facilities Test facility (APFT) is installed.
>
> This is indicated by setting facilities bit 15 for
> the guest. The kernel will not enable this facility for
> the guest if it is not set on the host. This facility
> must not be set by userspace if the KVM_S390_VM_CPU_FEAT_AP
> feature is not installed.
>
> Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com>
> Reviewed-by: Halil Pasic <pasic@linux.vnet.ibm.com>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
> arch/s390/include/uapi/asm/kvm.h | 1 +
> arch/s390/kvm/kvm-s390.c | 4 ++++
> arch/s390/tools/gen_facilities.c | 2 ++
> 3 files changed, 7 insertions(+), 0 deletions(-)
>
> diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h
> index 4cdaa55..a580dec 100644
> --- a/arch/s390/include/uapi/asm/kvm.h
> +++ b/arch/s390/include/uapi/asm/kvm.h
> @@ -130,6 +130,7 @@ struct kvm_s390_vm_cpu_machine {
> #define KVM_S390_VM_CPU_FEAT_PFMFI 11
> #define KVM_S390_VM_CPU_FEAT_SIGPIF 12
> #define KVM_S390_VM_CPU_FEAT_KSS 13
> +#define KVM_S390_VM_CPU_FEAT_AP 14
> struct kvm_s390_vm_cpu_feat {
> __u64 feat[16];
> };
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index de1e299..c68ca86 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -347,6 +347,10 @@ static void kvm_s390_cpu_feat_init(void)
>
> if (MACHINE_HAS_ESOP)
> allow_cpu_feat(KVM_S390_VM_CPU_FEAT_ESOP);
> +
> + if (ap_instructions_installed()) /* AP instructions installed on host */
> + allow_cpu_feat(KVM_S390_VM_CPU_FEAT_AP);
Don't we have a SIE specific AP feature? So is it true, that once we
have AP instructions, we are allowed to use them for SIE? Isn't there a
"AP interpretation facility" or anything like that? (that unlocks ECA_APIE)
> +
> /*
> * We need SIE support, ESOP (PROT_READ protection for gmap_shadow),
> * 64bit SCAO (SCA passthrough) and IDTE (for gmap_shadow unshadowing).
> diff --git a/arch/s390/tools/gen_facilities.c b/arch/s390/tools/gen_facilities.c
> index 90a8c9e..a52290b 100644
> --- a/arch/s390/tools/gen_facilities.c
> +++ b/arch/s390/tools/gen_facilities.c
> @@ -106,6 +106,8 @@ struct facility_def {
>
> .name = "FACILITIES_KVM_CPUMODEL",
> .bits = (int[]){
> + 12, /* AP Query Configuration Information */
> + 15, /* AP Facilities Test */
> -1 /* END */
> }
> },
>
So only if this feature is enabled, we later on allow to
kvm_ap_enable_ie_mode, right?
So basically, without this feature:
1. We will never set the execution control ECA_APIE.
2. The masks will always be 0.
Which also results in VSIE never having masks set.
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 13/15] KVM: s390: Configure the guest's CRYCB
2018-02-27 14:28 ` [PATCH v2 13/15] KVM: s390: Configure the guest's CRYCB Tony Krowiak
@ 2018-02-28 9:49 ` David Hildenbrand
2018-02-28 20:45 ` Tony Krowiak
0 siblings, 1 reply; 56+ messages in thread
From: David Hildenbrand @ 2018-02-28 9:49 UTC (permalink / raw)
To: Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen
> +static int vfio_ap_mdev_open(struct mdev_device *mdev)
> +{
> + struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
> + unsigned long events;
> + int ret;
> +
> + matrix_mdev->group_notifier.notifier_call = vfio_ap_mdev_group_notifier;
> + events = VFIO_GROUP_NOTIFY_SET_KVM;
> + ret = vfio_register_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
> + &events, &matrix_mdev->group_notifier);
> +
> + ret = kvm_ap_configure_matrix(matrix_mdev->kvm,
> + matrix_mdev->matrix);
> + if (ret)
> + return ret;
> +
> + ret = kvm_ap_enable_ie_mode(matrix_mdev->kvm);
Can't this happen while the guest is already running? Or what hinders us
from doing that?
> +
> + return ret;
> +}
> +
> +static void vfio_ap_mdev_release(struct mdev_device *mdev)
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 02/15] s390: vsie: implement AP support for second level guest
2018-02-28 9:39 ` David Hildenbrand
@ 2018-02-28 10:03 ` Pierre Morel
0 siblings, 0 replies; 56+ messages in thread
From: Pierre Morel @ 2018-02-28 10:03 UTC (permalink / raw)
To: David Hildenbrand, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, alifm, mjrosato,
jjherne, thuth, pasic, fiuczy, buendgen
On 28/02/2018 10:39, David Hildenbrand wrote:
> On 27.02.2018 15:28, Tony Krowiak wrote:
>> Set effective masks and CRYCB format in the shadow copy of the
>> guest level 2 CRYCB.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>> arch/s390/include/asm/kvm-ap.h | 2 +
>> arch/s390/kvm/kvm-ap.c | 5 +++
>> arch/s390/kvm/vsie.c | 71 +++++++++++++++++++++++++++++++++-------
>> 3 files changed, 66 insertions(+), 12 deletions(-)
>>
>> diff --git a/arch/s390/include/asm/kvm-ap.h b/arch/s390/include/asm/kvm-ap.h
>> index 4e43117..ef749e7 100644
>> --- a/arch/s390/include/asm/kvm-ap.h
>> +++ b/arch/s390/include/asm/kvm-ap.h
>> @@ -13,4 +13,6 @@
>>
>> void kvm_ap_set_crycb_format(struct kvm *kvm, __u32 *crycbd);
>>
>> +int kvm_ap_get_crycb_format(struct kvm *kvm);
>> +
>> #endif /* _ASM_KVM_AP */
>> diff --git a/arch/s390/kvm/kvm-ap.c b/arch/s390/kvm/kvm-ap.c
>> index 5305f4c..bafe63b 100644
>> --- a/arch/s390/kvm/kvm-ap.c
>> +++ b/arch/s390/kvm/kvm-ap.c
>> @@ -11,6 +11,11 @@
>>
>> #include "kvm-s390.h"
>>
>> +int kvm_ap_get_crycb_format(struct kvm *kvm)
>> +{
>> + return kvm->arch.crypto.crycbd & CRYCB_FORMAT_MASK;
>> +}
> Why exactly do we need this function? If there are no other users, just
> do it directly in the code below.
>
>> +
>> static int kvm_ap_apxa_installed(void)
>> {
>> int ret;
>> diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c
>> index 8961e39..93076ba 100644
>> --- a/arch/s390/kvm/vsie.c
>> +++ b/arch/s390/kvm/vsie.c
>> @@ -18,6 +18,7 @@
>> #include <asm/sclp.h>
>> #include <asm/nmi.h>
>> #include <asm/dis.h>
>> +#include <asm/kvm-ap.h>
>> #include "kvm-s390.h"
>> #include "gaccess.h"
>>
>> @@ -137,12 +138,56 @@ static int prepare_cpuflags(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
>> }
>>
>> /*
>> + * Set up the effective masks for the shadow copy of the crycb. The effective
>> + * masks for guest 3 are set by performing a logical bitwise AND of the guest 3
>> + * masks with the guest 2 masks.
>> + */
>> +static void set_crycb_emasks(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
>> +{
>> + int fmt = kvm_ap_get_crycb_format(vcpu->kvm);
>> + unsigned long *mask1, *mask2;
>> +
>> + switch (fmt) {
>> + case CRYCB_FORMAT1:
>> + case CRYCB_FORMAT2:
> Assume L2 gave us FORMAT0 and we are using FORMAT2. You would copy wrong
> bits. See below, maybe we need conversion.
+1
and in the case we have FORTMAT1 in L2 and/or L3, you also copy them
from and/or to the wrong place.
>
>> + mask1 = (unsigned long *)vsie_page->crycb.apcb1.apm;
>> + mask2 = (unsigned long *)
>> + vcpu->kvm->arch.crypto.crycb->apcb1.apm;
>> + bitmap_and(mask1, mask1, mask2, APCB1_MASK_SIZE);
>> +
>> + mask1 = (unsigned long *)vsie_page->crycb.apcb1.aqm;
>> + mask2 = (unsigned long *)
>> + vcpu->kvm->arch.crypto.crycb->apcb1.aqm;
>> + bitmap_and(mask1, mask1, mask2, APCB1_MASK_SIZE);
>> +
>> + mask1 = (unsigned long *)vsie_page->crycb.apcb1.adm;
>> + mask2 = (unsigned long *)
>> + vcpu->kvm->arch.crypto.crycb->apcb1.adm;
>> + bitmap_and(mask1, mask1, mask2, APCB1_MASK_SIZE);
>> + break;
>> + default:
>> + mask1 = (unsigned long *)vsie_page->crycb.apcb0.apm;
>> + mask2 = (unsigned long *)
>> + vcpu->kvm->arch.crypto.crycb->apcb1.apm;
>> + bitmap_and(mask1, mask1, mask2, APCB0_MASK_SIZE);
>> +
>> + mask1 = (unsigned long *)vsie_page->crycb.apcb0.aqm;
>> + mask2 = (unsigned long *)
>> + vcpu->kvm->arch.crypto.crycb->apcb1.aqm;
>> + bitmap_and(mask1, mask1, mask2, APCB0_MASK_SIZE);
>> +
>> + mask1 = (unsigned long *)vsie_page->crycb.apcb0.adm;
>> + mask2 = (unsigned long *)
>> + vcpu->kvm->arch.crypto.crycb->apcb1.adm;
>> + bitmap_and(mask1, mask1, mask2, APCB0_MASK_SIZE);
>> + break;
>> + }
>> +}
>> +
>> +/*
>> * Create a shadow copy of the crycb block and setup key wrapping, if
>> * requested for guest 3 and enabled for guest 2.
>> *
>> - * We only accept format-1 (no AP in g2), but convert it into format-2
>> - * There is nothing to do for format-0.
>> - *
>> * Returns: - 0 if shadowed or nothing to do
>> * - > 0 if control has to be given to guest 2
>> */
>> @@ -155,9 +200,17 @@ static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
>> unsigned long *b1, *b2;
>> u8 ecb3_flags;
>>
>> - scb_s->crycbd = 0;
>> - if (!(crycbd_o & vcpu->arch.sie_block->crycbd & CRYCB_FORMAT1))
>> - return 0;
>> + scb_s->crycbd = ((__u32)(__u64) &vsie_page->crycb);
>> + kvm_ap_set_crycb_format(vcpu->kvm, &scb_s->crycbd);
>> +
>> + /* copy the crycb */
>> + if (read_guest_real(vcpu, crycb_addr, &vsie_page->crycb,
>> + sizeof(vsie_page->crycb)))
>> + return set_validity_icpt(scb_s, 0x0035U);
> This looks wrong. You are always copying from L2, although L2 might not
> even set up a crycb for L3. You removed the important crycbd_o check.
>
> Don't we need the following?
>
> a) Determine the _allowed_ crycb format for L2 -> L3. (How can we tell
> which format L2 is allowed to use for L3)
> b) Determine the target crycb format. This is basically
> vm_ap_set_crycb_format afaics.
> c) Convert the given crycb to the target crycb format.
>
> So importantly, can you clarify (as I don't have access to documentation):
>
> - When is L2 allowed to use format-0? So when will the format not be
> ignored but is actually used?
>
> - When is L2 allowed to use fortma-2? So when will the format not be
> ignored but is actually used?
>
> - When does the SIE start interpreting AP instructions? So how can we
> hinder it from doing so? Empty masks? crycb format?
>
>> +
>> + /* set up the effective masks */
>> + set_crycb_emasks(vcpu, vsie_page);
>> +
>> /* format-1 is supported with message-security-assist extension 3 */
>> if (!test_kvm_facility(vcpu->kvm, 76))
>> return 0;
> You now already copied the wrapping keys. So they could be !0. Please
> add a comment why we don't care.
>
> /* wrapping keys are ignored without ECB3_AES / ECB3_DEA */
>
>> @@ -172,13 +225,7 @@ static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
>> else if (!crycb_addr)
>> return set_validity_icpt(scb_s, 0x0039U);
>>
>> - /* copy only the wrapping keys */
>> - if (read_guest_real(vcpu, crycb_addr + 72, &vsie_page->crycb, 56))
>> - return set_validity_icpt(scb_s, 0x0035U);
>> -
>> scb_s->ecb3 |= ecb3_flags;
>> - scb_s->crycbd = ((__u32)(__u64) &vsie_page->crycb) | CRYCB_FORMAT1 |
>> - CRYCB_FORMAT2;
>>
>> /* xor both blocks in one run */
>> b1 = (unsigned long *) vsie_page->crycb.dea_wrapping_key_mask;
>>
>
--
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 03/15] s390: zcrypt: externalize AP instructions available function
2018-02-27 14:28 ` [PATCH v2 03/15] s390: zcrypt: externalize AP instructions available function Tony Krowiak
@ 2018-02-28 11:40 ` Harald Freudenberger
2018-02-28 17:41 ` Cornelia Huck
1 sibling, 0 replies; 56+ messages in thread
From: Harald Freudenberger @ 2018-02-28 11:40 UTC (permalink / raw)
To: Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen
On 02/27/2018 03:28 PM, Tony Krowiak wrote:
> If the AP instructions are not available on the linux host, then
> AP devices can not be interpreted by the SIE. The AP bus has a
> function it uses to determine if the AP instructions are
> available. This patch provides a new function that wraps the
> AP bus's function to externalize it for use by KVM.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> Reviewed-by: Pierre Morel <pmorel@linux.vnet.ibm.com>
> ---
> arch/s390/include/asm/ap.h | 7 +++++++
> drivers/s390/crypto/ap_bus.c | 6 ++++++
> 2 files changed, 13 insertions(+), 0 deletions(-)
>
> diff --git a/arch/s390/include/asm/ap.h b/arch/s390/include/asm/ap.h
> index cfce683..1df6b59 100644
> --- a/arch/s390/include/asm/ap.h
> +++ b/arch/s390/include/asm/ap.h
> @@ -120,4 +120,11 @@ struct ap_queue_status ap_queue_irq_ctrl(ap_qid_t qid,
> struct ap_qirq_ctrl qirqctrl,
> void *ind);
>
> +/**
> + * ap_instructions_installed() - Tests whether AP instructions are installed
> + *
> + * Returns 1 if the AP instructions are installed, otherwise; returns 0
> + */
> +int ap_instructions_installed(void);
> +
> #endif /* _ASM_S390_AP_H_ */
> diff --git a/drivers/s390/crypto/ap_bus.c b/drivers/s390/crypto/ap_bus.c
> index 48d55dc..089b1cf 100644
> --- a/drivers/s390/crypto/ap_bus.c
> +++ b/drivers/s390/crypto/ap_bus.c
> @@ -211,6 +211,12 @@ int ap_query_configuration(struct ap_config_info *info)
> }
> EXPORT_SYMBOL(ap_query_configuration);
>
> +int ap_instructions_installed(void)
> +{
> + return (ap_instructions_available() == 0);
> +}
> +EXPORT_SYMBOL(ap_instructions_installed);
> +
> /**
> * ap_init_configuration(): Allocate and query configuration array.
> */
Reviewed-by: Harald Freudenberger <freude@linux.vnet.ibm.com>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 04/15] KVM: s390: CPU model support for AP virtualization
2018-02-28 9:48 ` David Hildenbrand
@ 2018-02-28 11:40 ` Christian Borntraeger
2018-02-28 11:58 ` David Hildenbrand
0 siblings, 1 reply; 56+ messages in thread
From: Christian Borntraeger @ 2018-02-28 11:40 UTC (permalink / raw)
To: David Hildenbrand, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, cohuck, kwankhede, bjsdjshi,
pbonzini, alex.williamson, pmorel, alifm, mjrosato, jjherne,
thuth, pasic, fiuczy, buendgen
On 02/28/2018 10:48 AM, David Hildenbrand wrote:
> On 27.02.2018 15:28, Tony Krowiak wrote:
>> Introduces a new CPU model feature and two CPU model
>> facilities to support AP virtualization for KVM guests.
>>
>> CPU model feature:
>>
>> The KVM_S390_VM_CPU_FEAT_AP feature indicates that the
>> AP facilities are installed on the KVM guest. This
>> feature will be enabled by the kernel only if the AP
>> facilities are installed on the linux host. This feature
>> must be specifically turned on for the KVM guest from
>> userspace to allow guest access to AP devices installed
>> on the linux host.
>>
>> CPU model facilities:
>>
>> 1. AP Query Configuration Information (QCI) facility is installed.
>>
>> This is indicated by setting facilities bit 12 for
>> the guest. The kernel will not enable this facility
>> for the guest if it is not set on the host. This facility
>> must not be set by userspace if the KVM_S390_VM_CPU_FEAT_AP
>> feature is not installed.
>>
>> 2. AP Facilities Test facility (APFT) is installed.
>>
>> This is indicated by setting facilities bit 15 for
>> the guest. The kernel will not enable this facility for
>> the guest if it is not set on the host. This facility
>> must not be set by userspace if the KVM_S390_VM_CPU_FEAT_AP
>> feature is not installed.
>>
>> Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> Reviewed-by: Halil Pasic <pasic@linux.vnet.ibm.com>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>> arch/s390/include/uapi/asm/kvm.h | 1 +
>> arch/s390/kvm/kvm-s390.c | 4 ++++
>> arch/s390/tools/gen_facilities.c | 2 ++
>> 3 files changed, 7 insertions(+), 0 deletions(-)
>>
>> diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h
>> index 4cdaa55..a580dec 100644
>> --- a/arch/s390/include/uapi/asm/kvm.h
>> +++ b/arch/s390/include/uapi/asm/kvm.h
>> @@ -130,6 +130,7 @@ struct kvm_s390_vm_cpu_machine {
>> #define KVM_S390_VM_CPU_FEAT_PFMFI 11
>> #define KVM_S390_VM_CPU_FEAT_SIGPIF 12
>> #define KVM_S390_VM_CPU_FEAT_KSS 13
>> +#define KVM_S390_VM_CPU_FEAT_AP 14
>> struct kvm_s390_vm_cpu_feat {
>> __u64 feat[16];
>> };
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index de1e299..c68ca86 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -347,6 +347,10 @@ static void kvm_s390_cpu_feat_init(void)
>>
>> if (MACHINE_HAS_ESOP)
>> allow_cpu_feat(KVM_S390_VM_CPU_FEAT_ESOP);
>> +
>> + if (ap_instructions_installed()) /* AP instructions installed on host */
>> + allow_cpu_feat(KVM_S390_VM_CPU_FEAT_AP);
>
> Don't we have a SIE specific AP feature? So is it true, that once we
> have AP instructions, we are allowed to use them for SIE? Isn't there a
> "AP interpretation facility" or anything like that? (that unlocks ECA_APIE)
This seems to be coupled to the AP facility and there is no facility or scp bit.
AP is too old to have that (predates STFLE)
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 04/15] KVM: s390: CPU model support for AP virtualization
2018-02-28 11:40 ` Christian Borntraeger
@ 2018-02-28 11:58 ` David Hildenbrand
2018-02-28 15:59 ` Tony Krowiak
0 siblings, 1 reply; 56+ messages in thread
From: David Hildenbrand @ 2018-02-28 11:58 UTC (permalink / raw)
To: Christian Borntraeger, Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, cohuck, kwankhede, bjsdjshi,
pbonzini, alex.williamson, pmorel, alifm, mjrosato, jjherne,
thuth, pasic, fiuczy, buendgen
On 28.02.2018 12:40, Christian Borntraeger wrote:
>
>
> On 02/28/2018 10:48 AM, David Hildenbrand wrote:
>> On 27.02.2018 15:28, Tony Krowiak wrote:
>>> Introduces a new CPU model feature and two CPU model
>>> facilities to support AP virtualization for KVM guests.
>>>
>>> CPU model feature:
>>>
>>> The KVM_S390_VM_CPU_FEAT_AP feature indicates that the
>>> AP facilities are installed on the KVM guest. This
>>> feature will be enabled by the kernel only if the AP
>>> facilities are installed on the linux host. This feature
>>> must be specifically turned on for the KVM guest from
>>> userspace to allow guest access to AP devices installed
>>> on the linux host.
>>>
>>> CPU model facilities:
>>>
>>> 1. AP Query Configuration Information (QCI) facility is installed.
>>>
>>> This is indicated by setting facilities bit 12 for
>>> the guest. The kernel will not enable this facility
>>> for the guest if it is not set on the host. This facility
>>> must not be set by userspace if the KVM_S390_VM_CPU_FEAT_AP
>>> feature is not installed.
>>>
>>> 2. AP Facilities Test facility (APFT) is installed.
>>>
>>> This is indicated by setting facilities bit 15 for
>>> the guest. The kernel will not enable this facility for
>>> the guest if it is not set on the host. This facility
>>> must not be set by userspace if the KVM_S390_VM_CPU_FEAT_AP
>>> feature is not installed.
>>>
>>> Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com>
>>> Reviewed-by: Halil Pasic <pasic@linux.vnet.ibm.com>
>>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>>> ---
>>> arch/s390/include/uapi/asm/kvm.h | 1 +
>>> arch/s390/kvm/kvm-s390.c | 4 ++++
>>> arch/s390/tools/gen_facilities.c | 2 ++
>>> 3 files changed, 7 insertions(+), 0 deletions(-)
>>>
>>> diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h
>>> index 4cdaa55..a580dec 100644
>>> --- a/arch/s390/include/uapi/asm/kvm.h
>>> +++ b/arch/s390/include/uapi/asm/kvm.h
>>> @@ -130,6 +130,7 @@ struct kvm_s390_vm_cpu_machine {
>>> #define KVM_S390_VM_CPU_FEAT_PFMFI 11
>>> #define KVM_S390_VM_CPU_FEAT_SIGPIF 12
>>> #define KVM_S390_VM_CPU_FEAT_KSS 13
>>> +#define KVM_S390_VM_CPU_FEAT_AP 14
>>> struct kvm_s390_vm_cpu_feat {
>>> __u64 feat[16];
>>> };
>>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>>> index de1e299..c68ca86 100644
>>> --- a/arch/s390/kvm/kvm-s390.c
>>> +++ b/arch/s390/kvm/kvm-s390.c
>>> @@ -347,6 +347,10 @@ static void kvm_s390_cpu_feat_init(void)
>>>
>>> if (MACHINE_HAS_ESOP)
>>> allow_cpu_feat(KVM_S390_VM_CPU_FEAT_ESOP);
>>> +
>>> + if (ap_instructions_installed()) /* AP instructions installed on host */
>>> + allow_cpu_feat(KVM_S390_VM_CPU_FEAT_AP);
>>
>> Don't we have a SIE specific AP feature? So is it true, that once we
>> have AP instructions, we are allowed to use them for SIE? Isn't there a
>> "AP interpretation facility" or anything like that? (that unlocks ECA_APIE)
>
> This seems to be coupled to the AP facility and there is no facility or scp bit.
> AP is too old to have that (predates STFLE)
>
Interesting, so we (I :) ) didn't implement vSIE support back then
because we never indicated the AP facility (sensed by executing the
instruction) to the guest.
So ECA_APIE can be used when we sense the AP facility. Thanks!
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 05/15] s390: vfio-ap: base implementation of VFIO AP device driver
2018-02-27 14:28 ` [PATCH v2 05/15] s390: vfio-ap: base implementation of VFIO AP device driver Tony Krowiak
@ 2018-02-28 15:33 ` Pierre Morel
2018-02-28 16:43 ` Tony Krowiak
0 siblings, 1 reply; 56+ messages in thread
From: Pierre Morel @ 2018-02-28 15:33 UTC (permalink / raw)
To: Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, alifm, mjrosato,
jjherne, thuth, pasic, fiuczy, buendgen
On 27/02/2018 15:28, Tony Krowiak wrote:
> Introduces a new AP device driver. This device driver
> is built on the VFIO mediated device framework. The framework
> provides sysfs interfaces that facilitate passthrough
> access by guests to devices installed on the linux host.
>
> The VFIO AP device driver will serve two purposes:
>
> 1. Provide the interfaces to reserve AP devices for exclusive
> use by KVM guests. This is accomplished by unbinding the
> devices to be reserved for guest usage from the default AP
> device driver and binding them to the VFIO AP device driver.
>
> 2. Implements the functions, callbacks and sysfs attribute
> interfaces required to create one or more VFIO mediated
> devices each of which will be used to configure the AP
> matrix for a guest and serve as a file descriptor
> for facilitating communication between QEMU and the
> VFIO AP device driver.
>
> When the VFIO AP device driver is initialized:
>
> * It registers with the AP bus for control of type 10 (CEX4
> and newer) AP queue devices. The probe and remove callbacks
> will be provided to support the binding/unbinding of
> AP queue devices to/from the VFIO AP device driver.
>
> * Creates a /sys/devices/vfio-ap/matrix device to hold
> the APQNs of the AP devices bound to the VFIO
> AP device driver and serves as the parent of the
> mediated devices created for each guest.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
> MAINTAINERS | 2 +
> arch/s390/Kconfig | 8 ++
> arch/s390/configs/default_defconfig | 3 +
> arch/s390/configs/gcov_defconfig | 3 +
> arch/s390/configs/performance_defconfig | 3 +
> arch/s390/defconfig | 3 +
> drivers/s390/crypto/Makefile | 4 +
> drivers/s390/crypto/vfio_ap_drv.c | 134 +++++++++++++++++++++++++++++++
> drivers/s390/crypto/vfio_ap_private.h | 20 +++++
> include/uapi/linux/vfio.h | 2 +
> 10 files changed, 182 insertions(+), 0 deletions(-)
> create mode 100644 drivers/s390/crypto/vfio_ap_drv.c
> create mode 100644 drivers/s390/crypto/vfio_ap_private.h
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 4acf7c2..a2f232d 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -11884,6 +11884,8 @@ W: http://www.ibm.com/developerworks/linux/linux390/
> S: Supported
> F: arch/s390/include/asm/kvm/kvm-ap.h
> F: arch/s390/kvm/kvm-ap.c
> +F: drivers/s390/crypto/vfio_ap_drv.c
> +F: drivers/s390/crypto/vfio_ap_private.h
>
> S390 ZFCP DRIVER
> M: Steffen Maier <maier@linux.vnet.ibm.com>
> diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
> index cbe1d97..9f23caf 100644
> --- a/arch/s390/Kconfig
> +++ b/arch/s390/Kconfig
> @@ -771,6 +771,14 @@ config VFIO_CCW
> To compile this driver as a module, choose M here: the
> module will be called vfio_ccw.
>
> +config VFIO_AP
> + def_tristate m
> + prompt "Support for virtual Adjunct Processor device interface"
The VFIO AP devices are not virtual.
What about
"VFIO support for AP devices"
> + depends on ZCRYPT && VFIO_MDEV_DEVICE
> + help
> + driver grants access to Adjunct Processor (AP) devices
> + via the VFIO mediated device interface.
> +
> endmenu
>
> menu "Dump support"
> diff --git a/arch/s390/configs/default_defconfig b/arch/s390/configs/default_defconfig
> index 5af8458..40fa3f6 100644
> --- a/arch/s390/configs/default_defconfig
> +++ b/arch/s390/configs/default_defconfig
Not sure that this file belongs to this patch
> @@ -719,3 +719,6 @@ CONFIG_APPLDATA_BASE=y
> CONFIG_KVM=m
> CONFIG_KVM_S390_UCONTROL=y
> CONFIG_VHOST_NET=m
> +VFIO_MDEV=m
> +VFIO_MDEV_DEVICE=m
> +CONFIG_VFIO_AP=m
What is your goal when modifying this three files?
Could you add a comment in the commit message?
> diff --git a/arch/s390/configs/gcov_defconfig b/arch/s390/configs/gcov_defconfig
> index d52eafe..377d40f 100644
> --- a/arch/s390/configs/gcov_defconfig
> +++ b/arch/s390/configs/gcov_defconfig
> @@ -659,3 +659,6 @@ CONFIG_APPLDATA_BASE=y
> CONFIG_KVM=m
> CONFIG_KVM_S390_UCONTROL=y
> CONFIG_VHOST_NET=m
> +VFIO_MDEV=m
> +VFIO_MDEV_DEVICE=m
> +CONFIG_VFIO_AP=m
> diff --git a/arch/s390/configs/performance_defconfig b/arch/s390/configs/performance_defconfig
> index 20ed149..f0c52df 100644
> --- a/arch/s390/configs/performance_defconfig
> +++ b/arch/s390/configs/performance_defconfig
> @@ -657,3 +657,6 @@ CONFIG_APPLDATA_BASE=y
> CONFIG_KVM=m
> CONFIG_KVM_S390_UCONTROL=y
> CONFIG_VHOST_NET=m
> +VFIO_MDEV=m
> +VFIO_MDEV_DEVICE=m
> +CONFIG_VFIO_AP=m
> diff --git a/arch/s390/defconfig b/arch/s390/defconfig
> index 46a3178..0996eb7 100644
> --- a/arch/s390/defconfig
> +++ b/arch/s390/defconfig
> @@ -239,3 +239,6 @@ CONFIG_CRC7=m
> # CONFIG_XZ_DEC_ARMTHUMB is not set
> # CONFIG_XZ_DEC_SPARC is not set
> CONFIG_CMM=m
> +VFIO_MDEV=m
> +VFIO_MDEV_DEVICE=m
> +CONFIG_VFIO_AP=m
> diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile
> index b59af54..48e466e 100644
> --- a/drivers/s390/crypto/Makefile
> +++ b/drivers/s390/crypto/Makefile
> @@ -15,3 +15,7 @@ obj-$(CONFIG_ZCRYPT) += zcrypt_pcixcc.o zcrypt_cex2a.o zcrypt_cex4.o
> # pkey kernel module
> pkey-objs := pkey_api.o
> obj-$(CONFIG_PKEY) += pkey.o
> +
> +# adjunct processor matrix
> +vfio_ap-objs := vfio_ap_drv.o
> +obj-$(CONFIG_VFIO_AP) += vfio_ap.o
> diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c
> new file mode 100644
> index 0000000..8bb72af
> --- /dev/null
> +++ b/drivers/s390/crypto/vfio_ap_drv.c
> @@ -0,0 +1,134 @@
> +/*
> + * VFIO based AP device driver
> + *
> + * Copyright IBM Corp. 2017
> + *
> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> + */
> +
> +#include <linux/module.h>
> +#include <linux/mod_devicetable.h>
> +#include <linux/slab.h>
> +
> +#include "ap_bus.h"
Since you do it in a later patch, may be put this include of ap_bus.h in
ap_private.h now.
> +#include "vfio_ap_private.h"
> +
> +#define VFIO_AP_ROOT_NAME "vfio_ap"
> +#define VFIO_AP_DEV_TYPE_NAME "ap_matrix"
> +#define VFIO_AP_DEV_NAME "matrix"
> +
> +MODULE_AUTHOR("IBM Corporation");
> +MODULE_DESCRIPTION("VFIO AP device driver, Copyright IBM Corp. 2017");
> +MODULE_LICENSE("GPL v2");
> +
> +static struct device *vfio_ap_root_device;
> +
> +static struct ap_driver vfio_ap_drv;
> +
> +static struct ap_matrix *ap_matrix;
> +
> +static struct device_type vfio_ap_dev_type = {
> + .name = VFIO_AP_DEV_TYPE_NAME,
> +};
> +
> +/* Only type 10 adapters (CEX4 and later) are supported
> + * by the AP matrix device driver
> + */
> +static struct ap_device_id ap_queue_ids[] = {
> + { .dev_type = AP_DEVICE_TYPE_CEX4,
> + .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
> + { .dev_type = AP_DEVICE_TYPE_CEX5,
> + .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
> + { .dev_type = AP_DEVICE_TYPE_CEX6,
> + .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
> + { /* end of sibling */ },
> +};
> +
> +MODULE_DEVICE_TABLE(vfio_ap, ap_queue_ids);
> +
> +static int vfio_ap_queue_dev_probe(struct ap_device *apdev)
> +{
> + return 0;
> +}
> +
> +static void vfio_ap_matrix_dev_release(struct device *dev)
> +{
> + struct ap_matrix *ap_matrix = dev_get_drvdata(dev);
> +
> + kfree(ap_matrix);
> +}
> +
> +static int vfio_ap_matrix_dev_create(void)
> +{
> + int ret;
> +
> + vfio_ap_root_device = root_device_register(VFIO_AP_ROOT_NAME);
> +
> + ret = PTR_ERR_OR_ZERO(vfio_ap_root_device);
IS_ERR() is enough, root_device_register() never return NULL.
> + if (ret)
> + goto done;
> +
> + ap_matrix = kzalloc(sizeof(*ap_matrix), GFP_KERNEL);
> + if (!ap_matrix) {
> + ret = -ENOMEM;
> + goto matrix_alloc_err;
> + }
> +
> + ap_matrix->device.type = &vfio_ap_dev_type;
> + dev_set_name(&ap_matrix->device, "%s", VFIO_AP_DEV_NAME);
> + ap_matrix->device.parent = vfio_ap_root_device;
> + ap_matrix->device.release = vfio_ap_matrix_dev_release;
> + ap_matrix->device.driver = &vfio_ap_drv.driver;
> +
> + ret = device_register(&ap_matrix->device);
> + if (ret)
> + goto matrix_reg_err;
> +
> + goto done;
> +
> +matrix_reg_err:
> + put_device(&ap_matrix->device);
> + kfree(ap_matrix);
> +
> +matrix_alloc_err:
> + root_device_unregister(vfio_ap_root_device);
> +
> +done:
> + return ret;
> +}
> +
> +static void vfio_ap_matrix_dev_destroy(struct ap_matrix *ap_matrix)
> +{
> + device_unregister(&ap_matrix->device);
> + root_device_unregister(vfio_ap_root_device);
> +}
> +
> +int __init vfio_ap_init(void)
> +{
> + int ret;
> +
> + ret = vfio_ap_matrix_dev_create();
> + if (ret)
> + return ret;
> +
> + memset(&vfio_ap_drv, 0, sizeof(vfio_ap_drv));
> + vfio_ap_drv.probe = vfio_ap_queue_dev_probe;
> + vfio_ap_drv.ids = ap_queue_ids;
> +
> + ret = ap_driver_register(&vfio_ap_drv, THIS_MODULE, VFIO_AP_DRV_NAME);
> + if (ret) {
> + vfio_ap_matrix_dev_destroy(ap_matrix);
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> +void __exit vfio_ap_exit(void)
> +{
> + ap_driver_unregister(&vfio_ap_drv);
> + vfio_ap_matrix_dev_destroy(ap_matrix);
> +}
> +
> +module_init(vfio_ap_init);
> +module_exit(vfio_ap_exit);
> diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
> new file mode 100644
> index 0000000..3505947
> --- /dev/null
> +++ b/drivers/s390/crypto/vfio_ap_private.h
> @@ -0,0 +1,20 @@
> +/*
> + * Private data and functions for adjunct processor VFIO matrix driver.
> + *
> + * Copyright IBM Corp. 2017
> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> + */
> +
> +#ifndef _VFIO_AP_PRIVATE_H_
> +#define _VFIO_AP_PRIVATE_H_
> +
> +#include <linux/types.h>
> +
> +#define VFIO_AP_MODULE_NAME "vfio_ap"
> +#define VFIO_AP_DRV_NAME "vfio_ap"
> +
> +struct ap_matrix {
> + struct device device;
> +};
> +
> +#endif /* _VFIO_AP_PRIVATE_H_ */
> diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
> index e3301db..cf2a5e9 100644
> --- a/include/uapi/linux/vfio.h
> +++ b/include/uapi/linux/vfio.h
> @@ -200,6 +200,7 @@ struct vfio_device_info {
> #define VFIO_DEVICE_FLAGS_PLATFORM (1 << 2) /* vfio-platform device */
> #define VFIO_DEVICE_FLAGS_AMBA (1 << 3) /* vfio-amba device */
> #define VFIO_DEVICE_FLAGS_CCW (1 << 4) /* vfio-ccw device */
> +#define VFIO_DEVICE_FLAGS_AP (1 << 5) /* vfio-ap device */
> __u32 num_regions; /* Max region index + 1 */
> __u32 num_irqs; /* Max IRQ index + 1 */
> };
> @@ -215,6 +216,7 @@ struct vfio_device_info {
> #define VFIO_DEVICE_API_PLATFORM_STRING "vfio-platform"
> #define VFIO_DEVICE_API_AMBA_STRING "vfio-amba"
> #define VFIO_DEVICE_API_CCW_STRING "vfio-ccw"
> +#define VFIO_DEVICE_API_AP_STRING "vfio-ap"
>
> /**
> * VFIO_DEVICE_GET_REGION_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 8,
--
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 04/15] KVM: s390: CPU model support for AP virtualization
2018-02-28 11:58 ` David Hildenbrand
@ 2018-02-28 15:59 ` Tony Krowiak
0 siblings, 0 replies; 56+ messages in thread
From: Tony Krowiak @ 2018-02-28 15:59 UTC (permalink / raw)
To: David Hildenbrand, Christian Borntraeger, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, cohuck, kwankhede, bjsdjshi,
pbonzini, alex.williamson, pmorel, alifm, mjrosato, jjherne,
thuth, pasic, fiuczy, buendgen
On 02/28/2018 06:58 AM, David Hildenbrand wrote:
> On 28.02.2018 12:40, Christian Borntraeger wrote:
>>
>> On 02/28/2018 10:48 AM, David Hildenbrand wrote:
>>> On 27.02.2018 15:28, Tony Krowiak wrote:
>>>> Introduces a new CPU model feature and two CPU model
>>>> facilities to support AP virtualization for KVM guests.
>>>>
>>>> CPU model feature:
>>>>
>>>> The KVM_S390_VM_CPU_FEAT_AP feature indicates that the
>>>> AP facilities are installed on the KVM guest. This
>>>> feature will be enabled by the kernel only if the AP
>>>> facilities are installed on the linux host. This feature
>>>> must be specifically turned on for the KVM guest from
>>>> userspace to allow guest access to AP devices installed
>>>> on the linux host.
>>>>
>>>> CPU model facilities:
>>>>
>>>> 1. AP Query Configuration Information (QCI) facility is installed.
>>>>
>>>> This is indicated by setting facilities bit 12 for
>>>> the guest. The kernel will not enable this facility
>>>> for the guest if it is not set on the host. This facility
>>>> must not be set by userspace if the KVM_S390_VM_CPU_FEAT_AP
>>>> feature is not installed.
>>>>
>>>> 2. AP Facilities Test facility (APFT) is installed.
>>>>
>>>> This is indicated by setting facilities bit 15 for
>>>> the guest. The kernel will not enable this facility for
>>>> the guest if it is not set on the host. This facility
>>>> must not be set by userspace if the KVM_S390_VM_CPU_FEAT_AP
>>>> feature is not installed.
>>>>
>>>> Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com>
>>>> Reviewed-by: Halil Pasic <pasic@linux.vnet.ibm.com>
>>>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>>>> ---
>>>> arch/s390/include/uapi/asm/kvm.h | 1 +
>>>> arch/s390/kvm/kvm-s390.c | 4 ++++
>>>> arch/s390/tools/gen_facilities.c | 2 ++
>>>> 3 files changed, 7 insertions(+), 0 deletions(-)
>>>>
>>>> diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h
>>>> index 4cdaa55..a580dec 100644
>>>> --- a/arch/s390/include/uapi/asm/kvm.h
>>>> +++ b/arch/s390/include/uapi/asm/kvm.h
>>>> @@ -130,6 +130,7 @@ struct kvm_s390_vm_cpu_machine {
>>>> #define KVM_S390_VM_CPU_FEAT_PFMFI 11
>>>> #define KVM_S390_VM_CPU_FEAT_SIGPIF 12
>>>> #define KVM_S390_VM_CPU_FEAT_KSS 13
>>>> +#define KVM_S390_VM_CPU_FEAT_AP 14
>>>> struct kvm_s390_vm_cpu_feat {
>>>> __u64 feat[16];
>>>> };
>>>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>>>> index de1e299..c68ca86 100644
>>>> --- a/arch/s390/kvm/kvm-s390.c
>>>> +++ b/arch/s390/kvm/kvm-s390.c
>>>> @@ -347,6 +347,10 @@ static void kvm_s390_cpu_feat_init(void)
>>>>
>>>> if (MACHINE_HAS_ESOP)
>>>> allow_cpu_feat(KVM_S390_VM_CPU_FEAT_ESOP);
>>>> +
>>>> + if (ap_instructions_installed()) /* AP instructions installed on host */
>>>> + allow_cpu_feat(KVM_S390_VM_CPU_FEAT_AP);
>>> Don't we have a SIE specific AP feature? So is it true, that once we
>>> have AP instructions, we are allowed to use them for SIE? Isn't there a
>>> "AP interpretation facility" or anything like that? (that unlocks ECA_APIE)
>> This seems to be coupled to the AP facility and there is no facility or scp bit.
>> AP is too old to have that (predates STFLE)
>>
> Interesting, so we (I :) ) didn't implement vSIE support back then
> because we never indicated the AP facility (sensed by executing the
> instruction) to the guest.
>
> So ECA_APIE can be used when we sense the AP facility. Thanks!
All true.
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 07/15] KVM: s390: Interfaces to configure/deconfigure guest's AP matrix
2018-02-27 14:28 ` [PATCH v2 07/15] KVM: s390: Interfaces to configure/deconfigure guest's AP matrix Tony Krowiak
@ 2018-02-28 16:15 ` Pierre Morel
2018-02-28 19:11 ` Tony Krowiak
2018-02-28 18:50 ` Tony Krowiak
` (2 subsequent siblings)
3 siblings, 1 reply; 56+ messages in thread
From: Pierre Morel @ 2018-02-28 16:15 UTC (permalink / raw)
To: Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, alifm, mjrosato,
jjherne, thuth, pasic, fiuczy, buendgen
On 27/02/2018 15:28, Tony Krowiak wrote:
> Provides interfaces to assign AP adapters, usage domains
> and control domains to a KVM guest.
>
> A KVM guest is started by executing the Start Interpretive Execution (SIE)
> instruction. The SIE state description is a control block that contains the
> state information for a KVM guest and is supplied as input to the SIE
> instruction. The SIE state description has a satellite structure called the
> Crypto Control Block (CRYCB). The CRYCB contains three bitmask fields
> identifying the adapters, queues (domains) and control domains assigned to
> the KVM guest:
>
> * The AP Adapter Mask (APM) field identifies the AP adapters assigned to
> the KVM guest
>
> * The AP Queue Mask (AQM) field identifies the AP queues assigned to
> the KVM guest. Each AP queue is connected to a usage domain within
> an AP adapter.
>
> * The AP Domain Mask (ADM) field identifies the control domains
> assigned to the KVM guest.
>
> Each adapter, queue (usage domain) and control domain are identified by
> a number from 0 to 255. The bits in each mask, from most significant to
> least significant bit, correspond to the numbers 0-255. When a bit is
> set, the corresponding adapter, queue (usage domain) or control domain
> is assigned to the KVM guest.
...snip...
> static int kvm_ap_apxa_installed(void)
> {
> int ret;
> @@ -50,3 +170,140 @@ void kvm_ap_set_crycb_format(struct kvm *kvm, __u32 *crycbd)
> *crycbd |= CRYCB_FORMAT1;
> }
> }
> +
> +static int kvm_ap_matrix_apm_create(struct kvm_ap_matrix *ap_matrix, int apxa)
> +{
> + if (apxa)
> + ap_matrix->apm_max = 256;
AFAIK the number of possible bits in the masks for a system is not a
generic value but is
returned by the QCI instruction.
Is there a reason to use a fix value?
> + else
> + ap_matrix->apm_max = 64;
> +
> + ap_matrix->apm = kzalloc(KVM_AP_MASK_BYTES(ap_matrix->apm_max),
> + GFP_KERNEL);
> + if (!ap_matrix->apm)
> + return -ENOMEM;
> +
> + return 0;
> +}
> +
> +static int kvm_ap_matrix_aqm_create(struct kvm_ap_matrix *ap_matrix, int apxa)
> +{
> + if (apxa)
> + ap_matrix->aqm_max = 256;
same here
> + else
> + ap_matrix->aqm_max = 16;
> +
> + ap_matrix->aqm = kzalloc(KVM_AP_MASK_BYTES(ap_matrix->aqm_max),
> + GFP_KERNEL);
> + if (!ap_matrix->aqm)
> + return -ENOMEM;
> +
> + return 0;
> +}
> +
> +static int kvm_ap_matrix_adm_create(struct kvm_ap_matrix *ap_matrix, int apxa)
> +{
> + if (apxa)
> + ap_matrix->adm_max = 256;
and here
Pierre
--
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 05/15] s390: vfio-ap: base implementation of VFIO AP device driver
2018-02-28 15:33 ` Pierre Morel
@ 2018-02-28 16:43 ` Tony Krowiak
2018-02-28 18:10 ` Cornelia Huck
0 siblings, 1 reply; 56+ messages in thread
From: Tony Krowiak @ 2018-02-28 16:43 UTC (permalink / raw)
To: Pierre Morel, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, alifm, mjrosato,
jjherne, thuth, pasic, fiuczy, buendgen
On 02/28/2018 10:33 AM, Pierre Morel wrote:
> On 27/02/2018 15:28, Tony Krowiak wrote:
>> Introduces a new AP device driver. This device driver
>> is built on the VFIO mediated device framework. The framework
>> provides sysfs interfaces that facilitate passthrough
>> access by guests to devices installed on the linux host.
>>
>> The VFIO AP device driver will serve two purposes:
>>
>> 1. Provide the interfaces to reserve AP devices for exclusive
>> use by KVM guests. This is accomplished by unbinding the
>> devices to be reserved for guest usage from the default AP
>> device driver and binding them to the VFIO AP device driver.
>>
>> 2. Implements the functions, callbacks and sysfs attribute
>> interfaces required to create one or more VFIO mediated
>> devices each of which will be used to configure the AP
>> matrix for a guest and serve as a file descriptor
>> for facilitating communication between QEMU and the
>> VFIO AP device driver.
>>
>> When the VFIO AP device driver is initialized:
>>
>> * It registers with the AP bus for control of type 10 (CEX4
>> and newer) AP queue devices. The probe and remove callbacks
>> will be provided to support the binding/unbinding of
>> AP queue devices to/from the VFIO AP device driver.
>>
>> * Creates a /sys/devices/vfio-ap/matrix device to hold
>> the APQNs of the AP devices bound to the VFIO
>> AP device driver and serves as the parent of the
>> mediated devices created for each guest.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>> MAINTAINERS | 2 +
>> arch/s390/Kconfig | 8 ++
>> arch/s390/configs/default_defconfig | 3 +
>> arch/s390/configs/gcov_defconfig | 3 +
>> arch/s390/configs/performance_defconfig | 3 +
>> arch/s390/defconfig | 3 +
>> drivers/s390/crypto/Makefile | 4 +
>> drivers/s390/crypto/vfio_ap_drv.c | 134
>> +++++++++++++++++++++++++++++++
>> drivers/s390/crypto/vfio_ap_private.h | 20 +++++
>> include/uapi/linux/vfio.h | 2 +
>> 10 files changed, 182 insertions(+), 0 deletions(-)
>> create mode 100644 drivers/s390/crypto/vfio_ap_drv.c
>> create mode 100644 drivers/s390/crypto/vfio_ap_private.h
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index 4acf7c2..a2f232d 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -11884,6 +11884,8 @@ W:
>> http://www.ibm.com/developerworks/linux/linux390/
>> S: Supported
>> F: arch/s390/include/asm/kvm/kvm-ap.h
>> F: arch/s390/kvm/kvm-ap.c
>> +F: drivers/s390/crypto/vfio_ap_drv.c
>> +F: drivers/s390/crypto/vfio_ap_private.h
>>
>> S390 ZFCP DRIVER
>> M: Steffen Maier <maier@linux.vnet.ibm.com>
>> diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
>> index cbe1d97..9f23caf 100644
>> --- a/arch/s390/Kconfig
>> +++ b/arch/s390/Kconfig
>> @@ -771,6 +771,14 @@ config VFIO_CCW
>> To compile this driver as a module, choose M here: the
>> module will be called vfio_ccw.
>>
>> +config VFIO_AP
>> + def_tristate m
>> + prompt "Support for virtual Adjunct Processor device interface"
>
> The VFIO AP devices are not virtual.
> What about
> "VFIO support for AP devices"
Sounds good.
>
>> + depends on ZCRYPT && VFIO_MDEV_DEVICE
>> + help
>> + driver grants access to Adjunct Processor (AP) devices
>> + via the VFIO mediated device interface.
>> +
>> endmenu
>>
>> menu "Dump support"
>> diff --git a/arch/s390/configs/default_defconfig
>> b/arch/s390/configs/default_defconfig
>> index 5af8458..40fa3f6 100644
>> --- a/arch/s390/configs/default_defconfig
>> +++ b/arch/s390/configs/default_defconfig
>
> Not sure that this file belongs to this patch
Neither am I, but at the time I inserted this here - well before August
of last year - I was using vfio-ccw as a model.
If someone can verify this does not belong here, I'd be more than happy
to remove it.
>
>
>> @@ -719,3 +719,6 @@ CONFIG_APPLDATA_BASE=y
>> CONFIG_KVM=m
>> CONFIG_KVM_S390_UCONTROL=y
>> CONFIG_VHOST_NET=m
>> +VFIO_MDEV=m
>> +VFIO_MDEV_DEVICE=m
>> +CONFIG_VFIO_AP=m
>
> What is your goal when modifying this three files?
> Could you add a comment in the commit message?
As stated above, this was originally based on the vfio-ccw model and has
been in the
patch series since its inception. I'd be happy to remove it if it is not
necessary.
>
>
>
>> diff --git a/arch/s390/configs/gcov_defconfig
>> b/arch/s390/configs/gcov_defconfig
>> index d52eafe..377d40f 100644
>> --- a/arch/s390/configs/gcov_defconfig
>> +++ b/arch/s390/configs/gcov_defconfig
>> @@ -659,3 +659,6 @@ CONFIG_APPLDATA_BASE=y
>> CONFIG_KVM=m
>> CONFIG_KVM_S390_UCONTROL=y
>> CONFIG_VHOST_NET=m
>> +VFIO_MDEV=m
>> +VFIO_MDEV_DEVICE=m
>> +CONFIG_VFIO_AP=m
>> diff --git a/arch/s390/configs/performance_defconfig
>> b/arch/s390/configs/performance_defconfig
>> index 20ed149..f0c52df 100644
>> --- a/arch/s390/configs/performance_defconfig
>> +++ b/arch/s390/configs/performance_defconfig
>> @@ -657,3 +657,6 @@ CONFIG_APPLDATA_BASE=y
>> CONFIG_KVM=m
>> CONFIG_KVM_S390_UCONTROL=y
>> CONFIG_VHOST_NET=m
>> +VFIO_MDEV=m
>> +VFIO_MDEV_DEVICE=m
>> +CONFIG_VFIO_AP=m
>> diff --git a/arch/s390/defconfig b/arch/s390/defconfig
>> index 46a3178..0996eb7 100644
>> --- a/arch/s390/defconfig
>> +++ b/arch/s390/defconfig
>> @@ -239,3 +239,6 @@ CONFIG_CRC7=m
>> # CONFIG_XZ_DEC_ARMTHUMB is not set
>> # CONFIG_XZ_DEC_SPARC is not set
>> CONFIG_CMM=m
>> +VFIO_MDEV=m
>> +VFIO_MDEV_DEVICE=m
>> +CONFIG_VFIO_AP=m
>> diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile
>> index b59af54..48e466e 100644
>> --- a/drivers/s390/crypto/Makefile
>> +++ b/drivers/s390/crypto/Makefile
>> @@ -15,3 +15,7 @@ obj-$(CONFIG_ZCRYPT) += zcrypt_pcixcc.o
>> zcrypt_cex2a.o zcrypt_cex4.o
>> # pkey kernel module
>> pkey-objs := pkey_api.o
>> obj-$(CONFIG_PKEY) += pkey.o
>> +
>> +# adjunct processor matrix
>> +vfio_ap-objs := vfio_ap_drv.o
>> +obj-$(CONFIG_VFIO_AP) += vfio_ap.o
>> diff --git a/drivers/s390/crypto/vfio_ap_drv.c
>> b/drivers/s390/crypto/vfio_ap_drv.c
>> new file mode 100644
>> index 0000000..8bb72af
>> --- /dev/null
>> +++ b/drivers/s390/crypto/vfio_ap_drv.c
>> @@ -0,0 +1,134 @@
>> +/*
>> + * VFIO based AP device driver
>> + *
>> + * Copyright IBM Corp. 2017
>> + *
>> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> + */
>> +
>> +#include <linux/module.h>
>> +#include <linux/mod_devicetable.h>
>> +#include <linux/slab.h>
>> +
>> +#include "ap_bus.h"
>
> Since you do it in a later patch, may be put this include of ap_bus.h
> in ap_private.h now.
Okay
>
>
>> +#include "vfio_ap_private.h"
>> +
>> +#define VFIO_AP_ROOT_NAME "vfio_ap"
>> +#define VFIO_AP_DEV_TYPE_NAME "ap_matrix"
>> +#define VFIO_AP_DEV_NAME "matrix"
>> +
>> +MODULE_AUTHOR("IBM Corporation");
>> +MODULE_DESCRIPTION("VFIO AP device driver, Copyright IBM Corp. 2017");
>> +MODULE_LICENSE("GPL v2");
>> +
>> +static struct device *vfio_ap_root_device;
>> +
>> +static struct ap_driver vfio_ap_drv;
>> +
>> +static struct ap_matrix *ap_matrix;
>> +
>> +static struct device_type vfio_ap_dev_type = {
>> + .name = VFIO_AP_DEV_TYPE_NAME,
>> +};
>> +
>> +/* Only type 10 adapters (CEX4 and later) are supported
>> + * by the AP matrix device driver
>> + */
>> +static struct ap_device_id ap_queue_ids[] = {
>> + { .dev_type = AP_DEVICE_TYPE_CEX4,
>> + .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
>> + { .dev_type = AP_DEVICE_TYPE_CEX5,
>> + .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
>> + { .dev_type = AP_DEVICE_TYPE_CEX6,
>> + .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
>> + { /* end of sibling */ },
>> +};
>> +
>> +MODULE_DEVICE_TABLE(vfio_ap, ap_queue_ids);
>> +
>> +static int vfio_ap_queue_dev_probe(struct ap_device *apdev)
>> +{
>> + return 0;
>> +}
>> +
>> +static void vfio_ap_matrix_dev_release(struct device *dev)
>> +{
>> + struct ap_matrix *ap_matrix = dev_get_drvdata(dev);
>> +
>> + kfree(ap_matrix);
>> +}
>> +
>> +static int vfio_ap_matrix_dev_create(void)
>> +{
>> + int ret;
>> +
>> + vfio_ap_root_device = root_device_register(VFIO_AP_ROOT_NAME);
>> +
>> + ret = PTR_ERR_OR_ZERO(vfio_ap_root_device);
>
> IS_ERR() is enough, root_device_register() never return NULL.
I searched the kernel code to look at other places the
root_device_register()
function is called to see how the return value is handled. I've seen all
of the
following used:
if (IS_ERR())
ret = PTR_ERR()
PTR_ERR()
PTR_ERR_OR_ZERO()
I'm not sure why this is a concern, but I'll use the first option above
since PTR_ERR_OR_ZERO() also embeds the first option.
>> + if (ret)
>> + goto done;
>> +
>> + ap_matrix = kzalloc(sizeof(*ap_matrix), GFP_KERNEL);
>> + if (!ap_matrix) {
>> + ret = -ENOMEM;
>> + goto matrix_alloc_err;
>> + }
>> +
>> + ap_matrix->device.type = &vfio_ap_dev_type;
>> + dev_set_name(&ap_matrix->device, "%s", VFIO_AP_DEV_NAME);
>> + ap_matrix->device.parent = vfio_ap_root_device;
>> + ap_matrix->device.release = vfio_ap_matrix_dev_release;
>> + ap_matrix->device.driver = &vfio_ap_drv.driver;
>> +
>> + ret = device_register(&ap_matrix->device);
>> + if (ret)
>> + goto matrix_reg_err;
>> +
>> + goto done;
>> +
>> +matrix_reg_err:
>> + put_device(&ap_matrix->device);
>> + kfree(ap_matrix);
>> +
>> +matrix_alloc_err:
>> + root_device_unregister(vfio_ap_root_device);
>> +
>> +done:
>> + return ret;
>> +}
>> +
>> +static void vfio_ap_matrix_dev_destroy(struct ap_matrix *ap_matrix)
>> +{
>> + device_unregister(&ap_matrix->device);
>> + root_device_unregister(vfio_ap_root_device);
>> +}
>> +
>> +int __init vfio_ap_init(void)
>> +{
>> + int ret;
>> +
>> + ret = vfio_ap_matrix_dev_create();
>> + if (ret)
>> + return ret;
>> +
>> + memset(&vfio_ap_drv, 0, sizeof(vfio_ap_drv));
>> + vfio_ap_drv.probe = vfio_ap_queue_dev_probe;
>> + vfio_ap_drv.ids = ap_queue_ids;
>> +
>> + ret = ap_driver_register(&vfio_ap_drv, THIS_MODULE,
>> VFIO_AP_DRV_NAME);
>> + if (ret) {
>> + vfio_ap_matrix_dev_destroy(ap_matrix);
>> + return ret;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +void __exit vfio_ap_exit(void)
>> +{
>> + ap_driver_unregister(&vfio_ap_drv);
>> + vfio_ap_matrix_dev_destroy(ap_matrix);
>> +}
>> +
>> +module_init(vfio_ap_init);
>> +module_exit(vfio_ap_exit);
>> diff --git a/drivers/s390/crypto/vfio_ap_private.h
>> b/drivers/s390/crypto/vfio_ap_private.h
>> new file mode 100644
>> index 0000000..3505947
>> --- /dev/null
>> +++ b/drivers/s390/crypto/vfio_ap_private.h
>> @@ -0,0 +1,20 @@
>> +/*
>> + * Private data and functions for adjunct processor VFIO matrix driver.
>> + *
>> + * Copyright IBM Corp. 2017
>> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> + */
>> +
>> +#ifndef _VFIO_AP_PRIVATE_H_
>> +#define _VFIO_AP_PRIVATE_H_
>> +
>> +#include <linux/types.h>
>> +
>> +#define VFIO_AP_MODULE_NAME "vfio_ap"
>> +#define VFIO_AP_DRV_NAME "vfio_ap"
>> +
>> +struct ap_matrix {
>> + struct device device;
>> +};
>> +
>> +#endif /* _VFIO_AP_PRIVATE_H_ */
>> diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
>> index e3301db..cf2a5e9 100644
>> --- a/include/uapi/linux/vfio.h
>> +++ b/include/uapi/linux/vfio.h
>> @@ -200,6 +200,7 @@ struct vfio_device_info {
>> #define VFIO_DEVICE_FLAGS_PLATFORM (1 << 2) /* vfio-platform
>> device */
>> #define VFIO_DEVICE_FLAGS_AMBA (1 << 3) /* vfio-amba device */
>> #define VFIO_DEVICE_FLAGS_CCW (1 << 4) /* vfio-ccw device */
>> +#define VFIO_DEVICE_FLAGS_AP (1 << 5) /* vfio-ap device */
>> __u32 num_regions; /* Max region index + 1 */
>> __u32 num_irqs; /* Max IRQ index + 1 */
>> };
>> @@ -215,6 +216,7 @@ struct vfio_device_info {
>> #define VFIO_DEVICE_API_PLATFORM_STRING "vfio-platform"
>> #define VFIO_DEVICE_API_AMBA_STRING "vfio-amba"
>> #define VFIO_DEVICE_API_CCW_STRING "vfio-ccw"
>> +#define VFIO_DEVICE_API_AP_STRING "vfio-ap"
>>
>> /**
>> * VFIO_DEVICE_GET_REGION_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 8,
>
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 01/15] KVM: s390: refactor crypto initialization
2018-02-27 14:27 ` [PATCH v2 01/15] KVM: s390: refactor crypto initialization Tony Krowiak
@ 2018-02-28 17:37 ` Cornelia Huck
2018-02-28 21:23 ` Tony Krowiak
0 siblings, 1 reply; 56+ messages in thread
From: Cornelia Huck @ 2018-02-28 17:37 UTC (permalink / raw)
To: Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
fiuczy, buendgen
On Tue, 27 Feb 2018 09:27:59 -0500
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> The crypto control block designation (CRYCBD) is a 32-bit
> field in the KVM guest's SIE state description. The
> contents of bits 1-28 of this field, with three zero bits
> appended on the right, designate the host real 31-bit
> address of a crypto control block (CRYCB). Bits 30-31
> specify the format of the CRYCB. In the current
> implementation, the address of the CRYCB is stored in
> the CRYCBD only if the Message-Security-Assist extension
> 3 (MSA3) facility is installed. Virtualization of AP
> facilities, however, requires that a CRYCB of the
> appropriate format be made available to SIE regardless
> of whether MSA3 is installed or not.
>
> This patch introduces a new compilation unit to provide
> all interfaces related to configuration of AP facilities.
> Let's start by moving the function for setting the CRYCB
> format from arch/s390/kvm/kvm-s390 to this new AP
> configuration interface.
Hm, I would tweak this patch description a bit. First, you talk about
what the crycbd is; then, what needs to be done for vfio-ap support;
then you simply state that you move some interfaces to a new file. I'd
like to see a connection between those parts :)
[It sounds a bit like you'd just introduce a new file and move some
functions, while you do have more changes in there.]
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
> MAINTAINERS | 10 ++++++
> arch/s390/include/asm/kvm-ap.h | 16 ++++++++++
> arch/s390/include/asm/kvm_host.h | 1 +
> arch/s390/kvm/Makefile | 2 +-
> arch/s390/kvm/kvm-ap.c | 47 ++++++++++++++++++++++++++++
> arch/s390/kvm/kvm-s390.c | 62 +++++---------------------------------
> 6 files changed, 83 insertions(+), 55 deletions(-)
> create mode 100644 arch/s390/include/asm/kvm-ap.h
> create mode 100644 arch/s390/kvm/kvm-ap.c
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 0ec5881..4acf7c2 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -11875,6 +11875,16 @@ W: http://www.ibm.com/developerworks/linux/linux390/
> S: Supported
> F: drivers/s390/crypto/
>
> +S390 VFIO AP DRIVER
> +M: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> +M: Christian BornTraeger <borntraeger@de.ibm.com>
Typo.
> +M: Martin Schwidefsky <schwidefsky@de.ibm.com>
> +L: linux-s390@vger.kernel.org
> +W: http://www.ibm.com/developerworks/linux/linux390/
> +S: Supported
> +F: arch/s390/include/asm/kvm/kvm-ap.h
> +F: arch/s390/kvm/kvm-ap.c
> +
> S390 ZFCP DRIVER
> M: Steffen Maier <maier@linux.vnet.ibm.com>
> M: Benjamin Block <bblock@linux.vnet.ibm.com>
(...)
> diff --git a/arch/s390/kvm/kvm-ap.c b/arch/s390/kvm/kvm-ap.c
> new file mode 100644
> index 0000000..5305f4c
> --- /dev/null
> +++ b/arch/s390/kvm/kvm-ap.c
> @@ -0,0 +1,47 @@
> +/*
> + * Adjunct Processor (AP) configuration management for KVM guests
> + *
> + * Copyright IBM Corp. 2017
> + *
> + * Author(s): Tony Krowiak <akrowia@linux.vnet.ibm.com>
> + */
> +
> +#include <asm/kvm-ap.h>
> +#include <asm/ap.h>
> +
> +#include "kvm-s390.h"
> +
> +static int kvm_ap_apxa_installed(void)
> +{
> + int ret;
> + struct ap_config_info config;
> +
> + ret = ap_query_configuration(&config);
Doesn't that introduce a dependency on CONFIG_ZCRYPT?
> + if (ret)
> + return 0;
> +
> + return (config.apxa == 1);
> +}
> +
> +/**
> + * kvm_ap_set_crycb_format
> + *
> + * Set the CRYCB format in the CRYCBD for the KVM guest.
Spell out "crypto control block" somewhere?
> + *
> + * @kvm: the KVM guest
> + * @crycbd: the CRYCB descriptor
> + */
> +void kvm_ap_set_crycb_format(struct kvm *kvm, __u32 *crycbd)
> +{
> + *crycbd = (__u32)(unsigned long) kvm->arch.crypto.crycb;
> +
> + *crycbd &= ~(CRYCB_FORMAT_MASK);
> +
> + /* If the MSAX3 is installed */
/* check whether MSAX3 is installed */ ?
> + if (test_kvm_facility(kvm, 76)) {
> + if (kvm_ap_apxa_installed())
> + *crycbd |= CRYCB_FORMAT2;
> + else
> + *crycbd |= CRYCB_FORMAT1;
> + }
> +}
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 5f5a4cb..de1e299 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -1913,12 +1866,13 @@ static u64 kvm_s390_get_initial_cpuid(void)
>
> static void kvm_s390_crypto_init(struct kvm *kvm)
> {
> + kvm->arch.crypto.crycb = &kvm->arch.sie_page2->crycb;
> + kvm->arch.crypto.crycbd = (__u32)(unsigned long) kvm->arch.crypto.crycb;
> + kvm_ap_set_crycb_format(kvm, &kvm->arch.crypto.crycbd);
Doesn't kvm_ap_set_crycb_format() already initialize its second
parameter?
Would it make sense to do
kvm->arch.crypto.crycbd = kvm_ap_build_crycbd(kvm);
or so instead?
> +
> if (!test_kvm_facility(kvm, 76))
> return;
>
> - kvm->arch.crypto.crycb = &kvm->arch.sie_page2->crycb;
> - kvm_s390_set_crycb_format(kvm);
> -
> /* Enable AES/DEA protected key functions by default */
> kvm->arch.crypto.aes_kw = 1;
> kvm->arch.crypto.dea_kw = 1;
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 03/15] s390: zcrypt: externalize AP instructions available function
2018-02-27 14:28 ` [PATCH v2 03/15] s390: zcrypt: externalize AP instructions available function Tony Krowiak
2018-02-28 11:40 ` Harald Freudenberger
@ 2018-02-28 17:41 ` Cornelia Huck
2018-03-01 8:38 ` Harald Freudenberger
1 sibling, 1 reply; 56+ messages in thread
From: Cornelia Huck @ 2018-02-28 17:41 UTC (permalink / raw)
To: Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
fiuczy, buendgen
On Tue, 27 Feb 2018 09:28:01 -0500
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> If the AP instructions are not available on the linux host, then
> AP devices can not be interpreted by the SIE. The AP bus has a
> function it uses to determine if the AP instructions are
> available. This patch provides a new function that wraps the
> AP bus's function to externalize it for use by KVM.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> Reviewed-by: Pierre Morel <pmorel@linux.vnet.ibm.com>
> ---
> arch/s390/include/asm/ap.h | 7 +++++++
> drivers/s390/crypto/ap_bus.c | 6 ++++++
> 2 files changed, 13 insertions(+), 0 deletions(-)
While I don't see anything wrong with this patch as it stands, you need
to be careful to call this only from vfio-ap code (which depends on
CONFIG_ZCRYPT).
Speaking of which: the ap_bus part cannot be a module, but only
built-in, right? So we don't get additional module complications?
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 05/15] s390: vfio-ap: base implementation of VFIO AP device driver
2018-02-28 16:43 ` Tony Krowiak
@ 2018-02-28 18:10 ` Cornelia Huck
2018-02-28 20:34 ` Tony Krowiak
0 siblings, 1 reply; 56+ messages in thread
From: Cornelia Huck @ 2018-02-28 18:10 UTC (permalink / raw)
To: Tony Krowiak
Cc: Pierre Morel, linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, alifm, mjrosato, jjherne, thuth, pasic, fiuczy,
buendgen
On Wed, 28 Feb 2018 11:43:37 -0500
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> On 02/28/2018 10:33 AM, Pierre Morel wrote:
> > On 27/02/2018 15:28, Tony Krowiak wrote:
(...)
> >> diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
> >> index cbe1d97..9f23caf 100644
> >> --- a/arch/s390/Kconfig
> >> +++ b/arch/s390/Kconfig
> >> @@ -771,6 +771,14 @@ config VFIO_CCW
> >> To compile this driver as a module, choose M here: the
> >> module will be called vfio_ccw.
> >>
> >> +config VFIO_AP
> >> + def_tristate m
Any reason you default to m instead of n here?
> >> + prompt "Support for virtual Adjunct Processor device interface"
> >
> > The VFIO AP devices are not virtual.
> > What about
> > "VFIO support for AP devices"
> Sounds good.
+1
> >
> >> + depends on ZCRYPT && VFIO_MDEV_DEVICE
> >> + help
> >> + driver grants access to Adjunct Processor (AP) devices
s/driver/This driver/
> >> + via the VFIO mediated device interface.
You also might want to add
"To compile this driver as a module, choose M here: the
module will be called..."
> >> +
> >> endmenu
It's a tad confusing to find this in the I/O submenu, but I don't
really have a better idea.
> >>
> >> menu "Dump support"
> >> diff --git a/arch/s390/configs/default_defconfig
> >> b/arch/s390/configs/default_defconfig
> >> index 5af8458..40fa3f6 100644
> >> --- a/arch/s390/configs/default_defconfig
> >> +++ b/arch/s390/configs/default_defconfig
> >
> > Not sure that this file belongs to this patch
> Neither am I, but at the time I inserted this here - well before August
> of last year - I was using vfio-ccw as a model.
> If someone can verify this does not belong here, I'd be more than happy
> to remove it.
I don't see any entry for VFIO_CCW in there?
> >
> >
> >> @@ -719,3 +719,6 @@ CONFIG_APPLDATA_BASE=y
> >> CONFIG_KVM=m
> >> CONFIG_KVM_S390_UCONTROL=y
> >> CONFIG_VHOST_NET=m
> >> +VFIO_MDEV=m
> >> +VFIO_MDEV_DEVICE=m
> >> +CONFIG_VFIO_AP=m
> >
> > What is your goal when modifying this three files?
> > Could you add a comment in the commit message?
> As stated above, this was originally based on the vfio-ccw model and has
> been in the
> patch series since its inception. I'd be happy to remove it if it is not
> necessary.
I'd vote for removing it.
(...)
> >> +static int vfio_ap_matrix_dev_create(void)
> >> +{
> >> + int ret;
> >> +
> >> + vfio_ap_root_device = root_device_register(VFIO_AP_ROOT_NAME);
> >> +
> >> + ret = PTR_ERR_OR_ZERO(vfio_ap_root_device);
> >
> > IS_ERR() is enough, root_device_register() never return NULL.
> I searched the kernel code to look at other places the
> root_device_register()
> function is called to see how the return value is handled. I've seen all
> of the
> following used:
> if (IS_ERR())
> ret = PTR_ERR()
> PTR_ERR()
> PTR_ERR_OR_ZERO()
>
> I'm not sure why this is a concern, but I'll use the first option above
> since PTR_ERR_OR_ZERO() also embeds the first option.
PTR_ERR_OR_ZERO() seems like the best choice for the way the return
code is processed here. (It's just unfortunate that its name conjures
up connotations of NULL-pointer handling.)
> >> + if (ret)
> >> + goto done;
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 07/15] KVM: s390: Interfaces to configure/deconfigure guest's AP matrix
2018-02-27 14:28 ` [PATCH v2 07/15] KVM: s390: Interfaces to configure/deconfigure guest's AP matrix Tony Krowiak
2018-02-28 16:15 ` Pierre Morel
@ 2018-02-28 18:50 ` Tony Krowiak
2018-02-28 19:38 ` Tony Krowiak
2018-03-08 23:03 ` Tony Krowiak
3 siblings, 0 replies; 56+ messages in thread
From: Tony Krowiak @ 2018-02-28 18:50 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen
On 02/27/2018 09:28 AM, Tony Krowiak wrote:
> Provides interfaces to assign AP adapters, usage domains
> and control domains to a KVM guest.
>
> A KVM guest is started by executing the Start Interpretive Execution (SIE)
> instruction. The SIE state description is a control block that contains the
> state information for a KVM guest and is supplied as input to the SIE
> instruction. The SIE state description has a satellite structure called the
> Crypto Control Block (CRYCB). The CRYCB contains three bitmask fields
> identifying the adapters, queues (domains) and control domains assigned to
> the KVM guest:
>
> * The AP Adapter Mask (APM) field identifies the AP adapters assigned to
> the KVM guest
>
> * The AP Queue Mask (AQM) field identifies the AP queues assigned to
> the KVM guest. Each AP queue is connected to a usage domain within
> an AP adapter.
>
> * The AP Domain Mask (ADM) field identifies the control domains
> assigned to the KVM guest.
>
> Each adapter, queue (usage domain) and control domain are identified by
> a number from 0 to 255. The bits in each mask, from most significant to
> least significant bit, correspond to the numbers 0-255. When a bit is
> set, the corresponding adapter, queue (usage domain) or control domain
> is assigned to the KVM guest.
>
> This patch will set the bits in the APM, AQM and ADM fields of the
> CRYCB referenced by the KVM guest's SIE state description. The process
> used is:
>
> 1. Verify that the bits to be set do not exceed the maximum bit
> number for the given mask.
>
> 2. Verify that the APQNs that can be derived from the intersection
> of the bits set in the APM and AQM fields of the KVM guest's CRYCB
> are not assigned to any other KVM guest running on the same linux
> host.
>
> 3. Set the APM, AQM and ADM in the CRYCB according to the matrix
> configured for the mediated matrix device via its sysfs
> adapter, domain and control domain attribute files respectively.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
> arch/s390/include/asm/kvm-ap.h | 36 +++++
> arch/s390/kvm/kvm-ap.c | 257 +++++++++++++++++++++++++++++++++
> drivers/s390/crypto/vfio_ap_ops.c | 19 +++
> drivers/s390/crypto/vfio_ap_private.h | 4 +
> 4 files changed, 316 insertions(+), 0 deletions(-)
>
> diff --git a/arch/s390/include/asm/kvm-ap.h b/arch/s390/include/asm/kvm-ap.h
> index ef749e7..46e7c5b 100644
> --- a/arch/s390/include/asm/kvm-ap.h
> +++ b/arch/s390/include/asm/kvm-ap.h
> @@ -10,9 +10,45 @@
> #define _ASM_KVM_AP
> #include <linux/types.h>
> #include <linux/kvm_host.h>
> +#include <linux/types.h>
> +#include <linux/kvm_host.h>
> +#include <linux/bitops.h>
> +
> +#define KVM_AP_MASK_BYTES(n)(n / BITS_PER_BYTE)
I don't know how I missed it, but there is a BITS_TO_BYTES macro in
linux/bitops.h.
It makes no sense to reinvent the wheel. Also, the above will only produce
a valid value if (n % BITS_PER_BYTE) == 0. That would most likely hold
true for
the cases in which the macro is used, but the BITS_TO_BYTES macro
compensates for
the case where n is not evenly divisible by 8.
> +
> +/**
> + * The AP matrix is comprised of three bit masks identifying the adapters,
> + * queues (domains) and control domains that belong to an AP matrix. The bits in
> + * each mask, from least significant to most significant bit, correspond to IDs
> + * 0 to the maximum ID allowed for a given mask. When a bit is set, the
> + * corresponding ID belongs to the matrix.
> + *
> + * @apm_max: max number of bits in @apm
> + * @apm identifies the AP adapters in the matrix
> + * @aqm_max: max number of bits in @aqm
> + * @aqm identifies the AP queues (domains) in the matrix
> + * @adm_max: max number of bits in @adm
> + * @adm identifies the AP control domains in the matrix
> + */
> +struct kvm_ap_matrix {
> + int apm_max;
> + unsigned long *apm;
> + int aqm_max;
> + unsigned long *aqm;
> + int adm_max;
> + unsigned long *adm;
> +};
>
> void kvm_ap_set_crycb_format(struct kvm *kvm, __u32 *crycbd);
>
> int kvm_ap_get_crycb_format(struct kvm *kvm);
>
> +int kvm_ap_matrix_create(struct kvm_ap_matrix **ap_matrix);
> +
> +void kvm_ap_matrix_destroy(struct kvm_ap_matrix *ap_matrix);
> +
> +int kvm_ap_configure_matrix(struct kvm *kvm, struct kvm_ap_matrix *matrix);
> +
> +void kvm_ap_deconfigure_matrix(struct kvm *kvm);
> +
> #endif /* _ASM_KVM_AP */
> diff --git a/arch/s390/kvm/kvm-ap.c b/arch/s390/kvm/kvm-ap.c
> index bafe63b..bb29045 100644
> --- a/arch/s390/kvm/kvm-ap.c
> +++ b/arch/s390/kvm/kvm-ap.c
> @@ -8,6 +8,7 @@
>
> #include <asm/kvm-ap.h>
> #include <asm/ap.h>
> +#include <linux/bitops.h>
>
> #include "kvm-s390.h"
>
> @@ -16,6 +17,125 @@ int kvm_ap_get_crycb_format(struct kvm *kvm)
> return kvm->arch.crypto.crycbd & CRYCB_FORMAT_MASK;
> }
>
> +static inline void kvm_ap_clear_crycb_masks(struct kvm *kvm)
> +{
> + int crycb_fmt = kvm_ap_get_crycb_format(kvm);
> +
> + if (crycb_fmt == CRYCB_FORMAT2)
> + memset(&kvm->arch.crypto.crycb->apcb1, 0,
> + sizeof(kvm->arch.crypto.crycb->apcb1));
> + else
> + memset(&kvm->arch.crypto.crycb->apcb0, 0,
> + sizeof(kvm->arch.crypto.crycb->apcb0));
> +}
> +
> +static inline unsigned long *kvm_ap_get_crycb_apm(struct kvm *kvm)
> +{
> + unsigned long *apm;
> + int crycb_fmt = kvm_ap_get_crycb_format(kvm);
> +
> + if (crycb_fmt == CRYCB_FORMAT2)
> + apm = (unsigned long *)kvm->arch.crypto.crycb->apcb1.apm;
> + else
> + apm = (unsigned long *)kvm->arch.crypto.crycb->apcb0.apm;
> +
> + return apm;
> +}
> +
> +static inline unsigned long *kvm_ap_get_crycb_aqm(struct kvm *kvm)
> +{
> + unsigned long *aqm;
> + int crycb_fmt = kvm_ap_get_crycb_format(kvm);
> +
> + if (crycb_fmt == CRYCB_FORMAT2)
> + aqm = (unsigned long *)kvm->arch.crypto.crycb->apcb1.aqm;
> + else
> + aqm = (unsigned long *)kvm->arch.crypto.crycb->apcb0.aqm;
> +
> + return aqm;
> +}
> +
> +static inline unsigned long *kvm_ap_get_crycb_adm(struct kvm *kvm)
> +{
> + unsigned long *adm;
> + int crycb_fmt = kvm_ap_get_crycb_format(kvm);
> +
> + if (crycb_fmt == CRYCB_FORMAT2)
> + adm = (unsigned long *)kvm->arch.crypto.crycb->apcb1.adm;
> + else
> + adm = (unsigned long *)kvm->arch.crypto.crycb->apcb0.adm;
> +
> + return adm;
> +}
> +
> +static void kvm_ap_set_crycb_masks(struct kvm *kvm,
> + struct kvm_ap_matrix *matrix)
> +{
> + unsigned long *apm = kvm_ap_get_crycb_apm(kvm);
> + unsigned long *aqm = kvm_ap_get_crycb_aqm(kvm);
> + unsigned long *adm = kvm_ap_get_crycb_adm(kvm);
> +
> + kvm_ap_clear_crycb_masks(kvm);
> + memcpy(apm, matrix->apm, KVM_AP_MASK_BYTES(matrix->apm_max));
> + memcpy(aqm, matrix->aqm, KVM_AP_MASK_BYTES(matrix->aqm_max));
Replace KVM_AP_MASK_BYTES macro with BITS_TO_BYTES macro
> +
> + /*
> + * Merge the AQM and ADM since the ADM is a superset of the
> + * AQM by architectural convention.
> + */
> + bitmap_or(adm, adm, aqm, matrix->adm_max);
> +}
> +
> +static void kvm_ap_log_sharing_err(struct kvm *kvm, unsigned long apid,
> + unsigned long apqi)
> +{
> + pr_err("%s: AP queue %02lx.%04lx is registered to guest %s", __func__,
> + apid, apqi, kvm->arch.dbf->name);
> +}
> +
> +/**
> + * kvm_ap_validate_queue_sharing
> + *
> + * Verifies that the APQNs derived from the intersection of the AP adapter IDs
> + * and AP queue indexes comprising the AP matrix are not configured for
> + * another guest. AP queue sharing is not allowed.
> + *
> + * @kvm: the KVM guest
> + * @matrix: the AP matrix
> + *
> + * Returns 0 if the APQNs are valid, otherwise; returns -EBUSY.
> + */
> +static int kvm_ap_validate_queue_sharing(struct kvm *kvm,
> + struct kvm_ap_matrix *matrix)
> +{
> + struct kvm *vm;
> + unsigned long *apm, *aqm;
> + unsigned long apid, apqi;
> +
> +
> + /* No other VM may share an AP Queue with the input VM */
> + list_for_each_entry(vm, &vm_list, vm_list) {
> + if (kvm == vm)
> + continue;
> +
> + apm = kvm_ap_get_crycb_apm(vm);
> + if (!bitmap_and(apm, apm, matrix->apm, matrix->apm_max))
> + continue;
> +
> + aqm = kvm_ap_get_crycb_aqm(vm);
> + if (!bitmap_and(aqm, aqm, matrix->aqm, matrix->aqm_max))
> + continue;
> +
> + for_each_set_bit_inv(apid, apm, matrix->apm_max)
> + for_each_set_bit_inv(apqi, aqm, matrix->aqm_max)
> + kvm_ap_log_sharing_err(kvm, apid, apqi);
> +
> + return -EBUSY;
> + }
> +
> + return 0;
> +}
> +
> static int kvm_ap_apxa_installed(void)
> {
> int ret;
> @@ -50,3 +170,140 @@ void kvm_ap_set_crycb_format(struct kvm *kvm, __u32 *crycbd)
> *crycbd |= CRYCB_FORMAT1;
> }
> }
> +
> +static int kvm_ap_matrix_apm_create(struct kvm_ap_matrix *ap_matrix, int apxa)
> +{
> + if (apxa)
> + ap_matrix->apm_max = 256;
> + else
> + ap_matrix->apm_max = 64;
> +
> + ap_matrix->apm = kzalloc(KVM_AP_MASK_BYTES(ap_matrix->apm_max),
> + GFP_KERNEL);
Replace KVM_AP_MASK_BYTES macro with BITS_TO_BYTES macro
> + if (!ap_matrix->apm)
> + return -ENOMEM;
> +
> + return 0;
> +}
> +
> +static int kvm_ap_matrix_aqm_create(struct kvm_ap_matrix *ap_matrix, int apxa)
> +{
> + if (apxa)
> + ap_matrix->aqm_max = 256;
> + else
> + ap_matrix->aqm_max = 16;
> +
> + ap_matrix->aqm = kzalloc(KVM_AP_MASK_BYTES(ap_matrix->aqm_max),
> + GFP_KERNEL);
Replace KVM_AP_MASK_BYTES macro with BITS_TO_BYTES macro
> + if (!ap_matrix->aqm)
> + return -ENOMEM;
> +
> + return 0;
> +}
> +
> +static int kvm_ap_matrix_adm_create(struct kvm_ap_matrix *ap_matrix, int apxa)
> +{
> + if (apxa)
> + ap_matrix->adm_max = 256;
> + else
> + ap_matrix->adm_max = 16;
> +
> + ap_matrix->adm = kzalloc(KVM_AP_MASK_BYTES(ap_matrix->adm_max),
> + GFP_KERNEL);
Replace KVM_AP_MASK_BYTES macro with BITS_TO_BYTES macro
> + if (!ap_matrix->adm)
> + return -ENOMEM;
> +
> + return 0;
> +}
> +
> +static void kvm_ap_matrix_masks_destroy(struct kvm_ap_matrix *ap_matrix)
> +{
> + kfree(ap_matrix->apm);
> + kfree(ap_matrix->aqm);
> + kfree(ap_matrix->adm);
> +}
> +
> +int kvm_ap_matrix_create(struct kvm_ap_matrix **ap_matrix)
> +{
> + int ret;
> + int apxa = kvm_ap_apxa_installed();
> + struct kvm_ap_matrix *matrix;
> +
> + matrix = kzalloc(sizeof(*matrix), GFP_KERNEL);
> + if (!matrix)
> + return -ENOMEM;
> +
> + ret = kvm_ap_matrix_apm_create(matrix, apxa);
> + if (ret)
> + goto mask_create_err;
> +
> + ret = kvm_ap_matrix_aqm_create(matrix, apxa);
> + if (ret)
> + goto mask_create_err;
> +
> + ret = kvm_ap_matrix_adm_create(matrix, apxa);
> + if (ret)
> + goto mask_create_err;
> +
> + *ap_matrix = matrix;
> +
> + return 0;
> +
> +mask_create_err:
> + kvm_ap_matrix_masks_destroy(matrix);
> + kfree(matrix);
> + return ret;
> +}
> +EXPORT_SYMBOL(kvm_ap_matrix_create);
> +
> +void kvm_ap_matrix_destroy(struct kvm_ap_matrix *ap_matrix)
> +{
> + kvm_ap_matrix_masks_destroy(ap_matrix);
> + kfree(ap_matrix);
> +}
> +EXPORT_SYMBOL(kvm_ap_matrix_destroy);
> +
> +/**
> + * kvm_ap_configure_matrix
> + *
> + * Configure the AP matrix for a KVM guest.
> + *
> + * @kvm: the KVM guest
> + * @matrix: the matrix configuration information
> + *
> + * Returns 0 if the APQNs derived from the intersection of the set of adapter
> + * IDs (APM) and queue indexes (AQM) in @matrix are not configured for any
> + * other KVM guest running on the same linux host. Otherwise returns an error
> + * code.
> + */
> +int kvm_ap_configure_matrix(struct kvm *kvm, struct kvm_ap_matrix *matrix)
> +{
> + int ret = 0;
> +
> + mutex_lock(&kvm->lock);
> +
> + ret = kvm_ap_validate_queue_sharing(kvm, matrix);
> + if (ret)
> + return ret;
> +
> + kvm_ap_set_crycb_masks(kvm, matrix);
> +
> + mutex_unlock(&kvm->lock);
> +
> + return 0;
> +}
> +EXPORT_SYMBOL(kvm_ap_configure_matrix);
> +
> +/**
> + * kvm_ap_deconfigure_matrix
> + *
> + * Deconfigure the AP matrix for a KVM guest. Clears all of the bits in the
> + * APM, AQM and ADM in the guest's CRYCB.
> + *
> + * @kvm: the KVM guest
> + */
> +void kvm_ap_deconfigure_matrix(struct kvm *kvm)
> +{
> + kvm_ap_clear_crycb_masks(kvm);
> +}
> +EXPORT_SYMBOL(kvm_ap_deconfigure_matrix);
> diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
> index 4292a5e..4fda44e 100644
> --- a/drivers/s390/crypto/vfio_ap_ops.c
> +++ b/drivers/s390/crypto/vfio_ap_ops.c
> @@ -10,6 +10,7 @@
> #include <linux/device.h>
> #include <linux/list.h>
> #include <linux/ctype.h>
> +#include <asm/kvm-ap.h>
>
> #include "vfio_ap_private.h"
>
> @@ -18,8 +19,23 @@
>
> static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev)
> {
> + int ret;
> + struct ap_matrix_mdev *matrix_mdev;
> struct ap_matrix *ap_matrix = to_ap_matrix(mdev_parent_dev(mdev));
> + struct kvm_ap_matrix *matrix;
> +
> + ret = kvm_ap_matrix_create(&matrix);
> + if (ret)
> + return ret;
> +
> + matrix_mdev = kzalloc(sizeof(*matrix_mdev), GFP_KERNEL);
> + if (!matrix_mdev) {
> + kvm_ap_matrix_destroy(matrix);
> + return -ENOMEM;
> + }
>
> + matrix_mdev->matrix = matrix;
> + mdev_set_drvdata(mdev, matrix_mdev);
> ap_matrix->available_instances--;
>
> return 0;
> @@ -28,7 +44,10 @@ static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev)
> static int vfio_ap_mdev_remove(struct mdev_device *mdev)
> {
> struct ap_matrix *ap_matrix = to_ap_matrix(mdev_parent_dev(mdev));
> + struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
>
> + kvm_ap_matrix_destroy(matrix_mdev->matrix);
> + kfree(matrix_mdev);
> ap_matrix->available_instances++;
>
> return 0;
> diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
> index c264415..522564e 100644
> --- a/drivers/s390/crypto/vfio_ap_private.h
> +++ b/drivers/s390/crypto/vfio_ap_private.h
> @@ -27,6 +27,10 @@ struct ap_matrix {
> int available_instances;
> };
>
> +struct ap_matrix_mdev {
> + struct kvm_ap_matrix *matrix;
> +};
> +
> static inline struct ap_matrix *to_ap_matrix(struct device *dev)
> {
> return container_of(dev, struct ap_matrix, device);
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 07/15] KVM: s390: Interfaces to configure/deconfigure guest's AP matrix
2018-02-28 16:15 ` Pierre Morel
@ 2018-02-28 19:11 ` Tony Krowiak
0 siblings, 0 replies; 56+ messages in thread
From: Tony Krowiak @ 2018-02-28 19:11 UTC (permalink / raw)
To: Pierre Morel, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, alifm, mjrosato,
jjherne, thuth, pasic, fiuczy, buendgen
On 02/28/2018 11:15 AM, Pierre Morel wrote:
> On 27/02/2018 15:28, Tony Krowiak wrote:
>> Provides interfaces to assign AP adapters, usage domains
>> and control domains to a KVM guest.
>>
>> A KVM guest is started by executing the Start Interpretive Execution
>> (SIE)
>> instruction. The SIE state description is a control block that
>> contains the
>> state information for a KVM guest and is supplied as input to the SIE
>> instruction. The SIE state description has a satellite structure
>> called the
>> Crypto Control Block (CRYCB). The CRYCB contains three bitmask fields
>> identifying the adapters, queues (domains) and control domains
>> assigned to
>> the KVM guest:
>>
>> * The AP Adapter Mask (APM) field identifies the AP adapters assigned to
>> the KVM guest
>>
>> * The AP Queue Mask (AQM) field identifies the AP queues assigned to
>> the KVM guest. Each AP queue is connected to a usage domain within
>> an AP adapter.
>>
>> * The AP Domain Mask (ADM) field identifies the control domains
>> assigned to the KVM guest.
>>
>> Each adapter, queue (usage domain) and control domain are identified by
>> a number from 0 to 255. The bits in each mask, from most significant to
>> least significant bit, correspond to the numbers 0-255. When a bit is
>> set, the corresponding adapter, queue (usage domain) or control domain
>> is assigned to the KVM guest.
>
> ...snip...
>
>> static int kvm_ap_apxa_installed(void)
>> {
>> int ret;
>> @@ -50,3 +170,140 @@ void kvm_ap_set_crycb_format(struct kvm *kvm,
>> __u32 *crycbd)
>> *crycbd |= CRYCB_FORMAT1;
>> }
>> }
>> +
>> +static int kvm_ap_matrix_apm_create(struct kvm_ap_matrix *ap_matrix,
>> int apxa)
>> +{
>> + if (apxa)
>> + ap_matrix->apm_max = 256;
>
> AFAIK the number of possible bits in the masks for a system is not a
> generic value but is
> returned by the QCI instruction.
> Is there a reason to use a fix value?
Right you are! I'll initialize the value based on what is returned from
the QCI call.
>
>
>> + else
>> + ap_matrix->apm_max = 64;
>> +
>> + ap_matrix->apm = kzalloc(KVM_AP_MASK_BYTES(ap_matrix->apm_max),
>> + GFP_KERNEL);
>> + if (!ap_matrix->apm)
>> + return -ENOMEM;
>> +
>> + return 0;
>> +}
>> +
>> +static int kvm_ap_matrix_aqm_create(struct kvm_ap_matrix *ap_matrix,
>> int apxa)
>> +{
>> + if (apxa)
>> + ap_matrix->aqm_max = 256;
>
> same here
ditto
>
>> + else
>> + ap_matrix->aqm_max = 16;
>> +
>> + ap_matrix->aqm = kzalloc(KVM_AP_MASK_BYTES(ap_matrix->aqm_max),
>> + GFP_KERNEL);
>> + if (!ap_matrix->aqm)
>> + return -ENOMEM;
>> +
>> + return 0;
>> +}
>> +
>> +static int kvm_ap_matrix_adm_create(struct kvm_ap_matrix *ap_matrix,
>> int apxa)
>> +{
>> + if (apxa)
>> + ap_matrix->adm_max = 256;
>
> and here
ditto
>
>
> Pierre
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 07/15] KVM: s390: Interfaces to configure/deconfigure guest's AP matrix
2018-02-27 14:28 ` [PATCH v2 07/15] KVM: s390: Interfaces to configure/deconfigure guest's AP matrix Tony Krowiak
2018-02-28 16:15 ` Pierre Morel
2018-02-28 18:50 ` Tony Krowiak
@ 2018-02-28 19:38 ` Tony Krowiak
2018-03-08 23:03 ` Tony Krowiak
3 siblings, 0 replies; 56+ messages in thread
From: Tony Krowiak @ 2018-02-28 19:38 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen
On 02/27/2018 09:28 AM, Tony Krowiak wrote:
> Provides interfaces to assign AP adapters, usage domains
> and control domains to a KVM guest.
>
> A KVM guest is started by executing the Start Interpretive Execution (SIE)
> instruction. The SIE state description is a control block that contains the
> state information for a KVM guest and is supplied as input to the SIE
> instruction. The SIE state description has a satellite structure called the
> Crypto Control Block (CRYCB). The CRYCB contains three bitmask fields
> identifying the adapters, queues (domains) and control domains assigned to
> the KVM guest:
>
> * The AP Adapter Mask (APM) field identifies the AP adapters assigned to
> the KVM guest
>
> * The AP Queue Mask (AQM) field identifies the AP queues assigned to
> the KVM guest. Each AP queue is connected to a usage domain within
> an AP adapter.
>
> * The AP Domain Mask (ADM) field identifies the control domains
> assigned to the KVM guest.
>
> Each adapter, queue (usage domain) and control domain are identified by
> a number from 0 to 255. The bits in each mask, from most significant to
> least significant bit, correspond to the numbers 0-255. When a bit is
> set, the corresponding adapter, queue (usage domain) or control domain
> is assigned to the KVM guest.
>
> This patch will set the bits in the APM, AQM and ADM fields of the
> CRYCB referenced by the KVM guest's SIE state description. The process
> used is:
>
> 1. Verify that the bits to be set do not exceed the maximum bit
> number for the given mask.
>
> 2. Verify that the APQNs that can be derived from the intersection
> of the bits set in the APM and AQM fields of the KVM guest's CRYCB
> are not assigned to any other KVM guest running on the same linux
> host.
>
> 3. Set the APM, AQM and ADM in the CRYCB according to the matrix
> configured for the mediated matrix device via its sysfs
> adapter, domain and control domain attribute files respectively.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
> arch/s390/include/asm/kvm-ap.h | 36 +++++
> arch/s390/kvm/kvm-ap.c | 257 +++++++++++++++++++++++++++++++++
> drivers/s390/crypto/vfio_ap_ops.c | 19 +++
> drivers/s390/crypto/vfio_ap_private.h | 4 +
> 4 files changed, 316 insertions(+), 0 deletions(-)
>
> diff --git a/arch/s390/include/asm/kvm-ap.h b/arch/s390/include/asm/kvm-ap.h
> index ef749e7..46e7c5b 100644
> --- a/arch/s390/include/asm/kvm-ap.h
> +++ b/arch/s390/include/asm/kvm-ap.h
> @@ -10,9 +10,45 @@
> #define _ASM_KVM_AP
> #include <linux/types.h>
> #include <linux/kvm_host.h>
> +#include <linux/types.h>
> +#include <linux/kvm_host.h>
> +#include <linux/bitops.h>
> +
> +#define KVM_AP_MASK_BYTES(n)(n / BITS_PER_BYTE)
This macro is accurate only if (n % BITS_PER_BYTE) == 0.
There is a BITS_TO_BYTES macro in tools/include/linux/bitops.h that does
the
job, but that header file is not available to kvm-ap.h. I'm going to steal
the concept and make the following change:
#define KVM_AP_MASK_BYTES(n) DIV_ROUND_UP(n, BITS_PER_BYTE)
> +
> +/**
> + * The AP matrix is comprised of three bit masks identifying the adapters,
> + * queues (domains) and control domains that belong to an AP matrix. The bits in
> + * each mask, from least significant to most significant bit, correspond to IDs
> + * 0 to the maximum ID allowed for a given mask. When a bit is set, the
> + * corresponding ID belongs to the matrix.
> + *
> + * @apm_max: max number of bits in @apm
> + * @apm identifies the AP adapters in the matrix
> + * @aqm_max: max number of bits in @aqm
> + * @aqm identifies the AP queues (domains) in the matrix
> + * @adm_max: max number of bits in @adm
> + * @adm identifies the AP control domains in the matrix
> + */
> +struct kvm_ap_matrix {
> + int apm_max;
> + unsigned long *apm;
> + int aqm_max;
> + unsigned long *aqm;
> + int adm_max;
> + unsigned long *adm;
> +};
>
> void kvm_ap_set_crycb_format(struct kvm *kvm, __u32 *crycbd);
>
> int kvm_ap_get_crycb_format(struct kvm *kvm);
>
> +int kvm_ap_matrix_create(struct kvm_ap_matrix **ap_matrix);
> +
> +void kvm_ap_matrix_destroy(struct kvm_ap_matrix *ap_matrix);
> +
> +int kvm_ap_configure_matrix(struct kvm *kvm, struct kvm_ap_matrix *matrix);
> +
> +void kvm_ap_deconfigure_matrix(struct kvm *kvm);
> +
> #endif /* _ASM_KVM_AP */
> diff --git a/arch/s390/kvm/kvm-ap.c b/arch/s390/kvm/kvm-ap.c
> index bafe63b..bb29045 100644
> --- a/arch/s390/kvm/kvm-ap.c
> +++ b/arch/s390/kvm/kvm-ap.c
> @@ -8,6 +8,7 @@
>
> #include <asm/kvm-ap.h>
> #include <asm/ap.h>
> +#include <linux/bitops.h>
>
> #include "kvm-s390.h"
>
> @@ -16,6 +17,125 @@ int kvm_ap_get_crycb_format(struct kvm *kvm)
> return kvm->arch.crypto.crycbd & CRYCB_FORMAT_MASK;
> }
>
> +static inline void kvm_ap_clear_crycb_masks(struct kvm *kvm)
> +{
> + int crycb_fmt = kvm_ap_get_crycb_format(kvm);
> +
> + if (crycb_fmt == CRYCB_FORMAT2)
> + memset(&kvm->arch.crypto.crycb->apcb1, 0,
> + sizeof(kvm->arch.crypto.crycb->apcb1));
> + else
> + memset(&kvm->arch.crypto.crycb->apcb0, 0,
> + sizeof(kvm->arch.crypto.crycb->apcb0));
> +}
> +
> +static inline unsigned long *kvm_ap_get_crycb_apm(struct kvm *kvm)
> +{
> + unsigned long *apm;
> + int crycb_fmt = kvm_ap_get_crycb_format(kvm);
> +
> + if (crycb_fmt == CRYCB_FORMAT2)
> + apm = (unsigned long *)kvm->arch.crypto.crycb->apcb1.apm;
> + else
> + apm = (unsigned long *)kvm->arch.crypto.crycb->apcb0.apm;
> +
> + return apm;
> +}
> +
> +static inline unsigned long *kvm_ap_get_crycb_aqm(struct kvm *kvm)
> +{
> + unsigned long *aqm;
> + int crycb_fmt = kvm_ap_get_crycb_format(kvm);
> +
> + if (crycb_fmt == CRYCB_FORMAT2)
> + aqm = (unsigned long *)kvm->arch.crypto.crycb->apcb1.aqm;
> + else
> + aqm = (unsigned long *)kvm->arch.crypto.crycb->apcb0.aqm;
> +
> + return aqm;
> +}
> +
> +static inline unsigned long *kvm_ap_get_crycb_adm(struct kvm *kvm)
> +{
> + unsigned long *adm;
> + int crycb_fmt = kvm_ap_get_crycb_format(kvm);
> +
> + if (crycb_fmt == CRYCB_FORMAT2)
> + adm = (unsigned long *)kvm->arch.crypto.crycb->apcb1.adm;
> + else
> + adm = (unsigned long *)kvm->arch.crypto.crycb->apcb0.adm;
> +
> + return adm;
> +}
> +
> +static void kvm_ap_set_crycb_masks(struct kvm *kvm,
> + struct kvm_ap_matrix *matrix)
> +{
> + unsigned long *apm = kvm_ap_get_crycb_apm(kvm);
> + unsigned long *aqm = kvm_ap_get_crycb_aqm(kvm);
> + unsigned long *adm = kvm_ap_get_crycb_adm(kvm);
> +
> + kvm_ap_clear_crycb_masks(kvm);
> + memcpy(apm, matrix->apm, KVM_AP_MASK_BYTES(matrix->apm_max));
> + memcpy(aqm, matrix->aqm, KVM_AP_MASK_BYTES(matrix->aqm_max));
> +
> + /*
> + * Merge the AQM and ADM since the ADM is a superset of the
> + * AQM by architectural convention.
> + */
> + bitmap_or(adm, adm, aqm, matrix->adm_max);
> +}
> +
> +static void kvm_ap_log_sharing_err(struct kvm *kvm, unsigned long apid,
> + unsigned long apqi)
> +{
> + pr_err("%s: AP queue %02lx.%04lx is registered to guest %s", __func__,
> + apid, apqi, kvm->arch.dbf->name);
> +}
> +
> +/**
> + * kvm_ap_validate_queue_sharing
> + *
> + * Verifies that the APQNs derived from the intersection of the AP adapter IDs
> + * and AP queue indexes comprising the AP matrix are not configured for
> + * another guest. AP queue sharing is not allowed.
> + *
> + * @kvm: the KVM guest
> + * @matrix: the AP matrix
> + *
> + * Returns 0 if the APQNs are valid, otherwise; returns -EBUSY.
> + */
> +static int kvm_ap_validate_queue_sharing(struct kvm *kvm,
> + struct kvm_ap_matrix *matrix)
> +{
> + struct kvm *vm;
> + unsigned long *apm, *aqm;
> + unsigned long apid, apqi;
> +
> +
> + /* No other VM may share an AP Queue with the input VM */
> + list_for_each_entry(vm, &vm_list, vm_list) {
> + if (kvm == vm)
> + continue;
> +
> + apm = kvm_ap_get_crycb_apm(vm);
> + if (!bitmap_and(apm, apm, matrix->apm, matrix->apm_max))
> + continue;
> +
> + aqm = kvm_ap_get_crycb_aqm(vm);
> + if (!bitmap_and(aqm, aqm, matrix->aqm, matrix->aqm_max))
> + continue;
> +
> + for_each_set_bit_inv(apid, apm, matrix->apm_max)
> + for_each_set_bit_inv(apqi, aqm, matrix->aqm_max)
> + kvm_ap_log_sharing_err(kvm, apid, apqi);
> +
> + return -EBUSY;
> + }
> +
> + return 0;
> +}
> +
> static int kvm_ap_apxa_installed(void)
> {
> int ret;
> @@ -50,3 +170,140 @@ void kvm_ap_set_crycb_format(struct kvm *kvm, __u32 *crycbd)
> *crycbd |= CRYCB_FORMAT1;
> }
> }
> +
> +static int kvm_ap_matrix_apm_create(struct kvm_ap_matrix *ap_matrix, int apxa)
> +{
> + if (apxa)
> + ap_matrix->apm_max = 256;
> + else
> + ap_matrix->apm_max = 64;
> +
> + ap_matrix->apm = kzalloc(KVM_AP_MASK_BYTES(ap_matrix->apm_max),
> + GFP_KERNEL);
> + if (!ap_matrix->apm)
> + return -ENOMEM;
> +
> + return 0;
> +}
> +
> +static int kvm_ap_matrix_aqm_create(struct kvm_ap_matrix *ap_matrix, int apxa)
> +{
> + if (apxa)
> + ap_matrix->aqm_max = 256;
> + else
> + ap_matrix->aqm_max = 16;
> +
> + ap_matrix->aqm = kzalloc(KVM_AP_MASK_BYTES(ap_matrix->aqm_max),
> + GFP_KERNEL);
> + if (!ap_matrix->aqm)
> + return -ENOMEM;
> +
> + return 0;
> +}
> +
> +static int kvm_ap_matrix_adm_create(struct kvm_ap_matrix *ap_matrix, int apxa)
> +{
> + if (apxa)
> + ap_matrix->adm_max = 256;
> + else
> + ap_matrix->adm_max = 16;
> +
> + ap_matrix->adm = kzalloc(KVM_AP_MASK_BYTES(ap_matrix->adm_max),
> + GFP_KERNEL);
> + if (!ap_matrix->adm)
> + return -ENOMEM;
> +
> + return 0;
> +}
> +
> +static void kvm_ap_matrix_masks_destroy(struct kvm_ap_matrix *ap_matrix)
> +{
> + kfree(ap_matrix->apm);
> + kfree(ap_matrix->aqm);
> + kfree(ap_matrix->adm);
> +}
> +
> +int kvm_ap_matrix_create(struct kvm_ap_matrix **ap_matrix)
> +{
> + int ret;
> + int apxa = kvm_ap_apxa_installed();
> + struct kvm_ap_matrix *matrix;
> +
> + matrix = kzalloc(sizeof(*matrix), GFP_KERNEL);
> + if (!matrix)
> + return -ENOMEM;
> +
> + ret = kvm_ap_matrix_apm_create(matrix, apxa);
> + if (ret)
> + goto mask_create_err;
> +
> + ret = kvm_ap_matrix_aqm_create(matrix, apxa);
> + if (ret)
> + goto mask_create_err;
> +
> + ret = kvm_ap_matrix_adm_create(matrix, apxa);
> + if (ret)
> + goto mask_create_err;
> +
> + *ap_matrix = matrix;
> +
> + return 0;
> +
> +mask_create_err:
> + kvm_ap_matrix_masks_destroy(matrix);
> + kfree(matrix);
> + return ret;
> +}
> +EXPORT_SYMBOL(kvm_ap_matrix_create);
> +
> +void kvm_ap_matrix_destroy(struct kvm_ap_matrix *ap_matrix)
> +{
> + kvm_ap_matrix_masks_destroy(ap_matrix);
> + kfree(ap_matrix);
> +}
> +EXPORT_SYMBOL(kvm_ap_matrix_destroy);
> +
> +/**
> + * kvm_ap_configure_matrix
> + *
> + * Configure the AP matrix for a KVM guest.
> + *
> + * @kvm: the KVM guest
> + * @matrix: the matrix configuration information
> + *
> + * Returns 0 if the APQNs derived from the intersection of the set of adapter
> + * IDs (APM) and queue indexes (AQM) in @matrix are not configured for any
> + * other KVM guest running on the same linux host. Otherwise returns an error
> + * code.
> + */
> +int kvm_ap_configure_matrix(struct kvm *kvm, struct kvm_ap_matrix *matrix)
> +{
> + int ret = 0;
> +
> + mutex_lock(&kvm->lock);
> +
> + ret = kvm_ap_validate_queue_sharing(kvm, matrix);
> + if (ret)
> + return ret;
> +
> + kvm_ap_set_crycb_masks(kvm, matrix);
> +
> + mutex_unlock(&kvm->lock);
> +
> + return 0;
> +}
> +EXPORT_SYMBOL(kvm_ap_configure_matrix);
> +
> +/**
> + * kvm_ap_deconfigure_matrix
> + *
> + * Deconfigure the AP matrix for a KVM guest. Clears all of the bits in the
> + * APM, AQM and ADM in the guest's CRYCB.
> + *
> + * @kvm: the KVM guest
> + */
> +void kvm_ap_deconfigure_matrix(struct kvm *kvm)
> +{
> + kvm_ap_clear_crycb_masks(kvm);
> +}
> +EXPORT_SYMBOL(kvm_ap_deconfigure_matrix);
> diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
> index 4292a5e..4fda44e 100644
> --- a/drivers/s390/crypto/vfio_ap_ops.c
> +++ b/drivers/s390/crypto/vfio_ap_ops.c
> @@ -10,6 +10,7 @@
> #include <linux/device.h>
> #include <linux/list.h>
> #include <linux/ctype.h>
> +#include <asm/kvm-ap.h>
>
> #include "vfio_ap_private.h"
>
> @@ -18,8 +19,23 @@
>
> static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev)
> {
> + int ret;
> + struct ap_matrix_mdev *matrix_mdev;
> struct ap_matrix *ap_matrix = to_ap_matrix(mdev_parent_dev(mdev));
> + struct kvm_ap_matrix *matrix;
> +
> + ret = kvm_ap_matrix_create(&matrix);
> + if (ret)
> + return ret;
> +
> + matrix_mdev = kzalloc(sizeof(*matrix_mdev), GFP_KERNEL);
> + if (!matrix_mdev) {
> + kvm_ap_matrix_destroy(matrix);
> + return -ENOMEM;
> + }
>
> + matrix_mdev->matrix = matrix;
> + mdev_set_drvdata(mdev, matrix_mdev);
> ap_matrix->available_instances--;
>
> return 0;
> @@ -28,7 +44,10 @@ static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev)
> static int vfio_ap_mdev_remove(struct mdev_device *mdev)
> {
> struct ap_matrix *ap_matrix = to_ap_matrix(mdev_parent_dev(mdev));
> + struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
>
> + kvm_ap_matrix_destroy(matrix_mdev->matrix);
> + kfree(matrix_mdev);
> ap_matrix->available_instances++;
>
> return 0;
> diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
> index c264415..522564e 100644
> --- a/drivers/s390/crypto/vfio_ap_private.h
> +++ b/drivers/s390/crypto/vfio_ap_private.h
> @@ -27,6 +27,10 @@ struct ap_matrix {
> int available_instances;
> };
>
> +struct ap_matrix_mdev {
> + struct kvm_ap_matrix *matrix;
> +};
> +
> static inline struct ap_matrix *to_ap_matrix(struct device *dev)
> {
> return container_of(dev, struct ap_matrix, device);
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 05/15] s390: vfio-ap: base implementation of VFIO AP device driver
2018-02-28 18:10 ` Cornelia Huck
@ 2018-02-28 20:34 ` Tony Krowiak
0 siblings, 0 replies; 56+ messages in thread
From: Tony Krowiak @ 2018-02-28 20:34 UTC (permalink / raw)
To: Cornelia Huck
Cc: Pierre Morel, linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, alifm, mjrosato, jjherne, thuth, pasic, fiuczy,
buendgen
On 02/28/2018 01:10 PM, Cornelia Huck wrote:
> On Wed, 28 Feb 2018 11:43:37 -0500
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> On 02/28/2018 10:33 AM, Pierre Morel wrote:
>>> On 27/02/2018 15:28, Tony Krowiak wrote:
> (...)
>
>>>> diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
>>>> index cbe1d97..9f23caf 100644
>>>> --- a/arch/s390/Kconfig
>>>> +++ b/arch/s390/Kconfig
>>>> @@ -771,6 +771,14 @@ config VFIO_CCW
>>>> To compile this driver as a module, choose M here: the
>>>> module will be called vfio_ccw.
>>>>
>>>> +config VFIO_AP
>>>> + def_tristate m
> Any reason you default to m instead of n here?
None in particular, is there a good reason to change this to n?
>
>>>> + prompt "Support for virtual Adjunct Processor device interface"
>>> The VFIO AP devices are not virtual.
>>> What about
>>> "VFIO support for AP devices"
>> Sounds good.
> +1
>
>>>
>>>> + depends on ZCRYPT && VFIO_MDEV_DEVICE
>>>> + help
>>>> + driver grants access to Adjunct Processor (AP) devices
> s/driver/This driver/
Okay, I'll make the change
>
>>>> + via the VFIO mediated device interface.
> You also might want to add
>
> "To compile this driver as a module, choose M here: the
> module will be called..."
Will do
>
>>>> +
>>>> endmenu
> It's a tad confusing to find this in the I/O submenu, but I don't
> really have a better idea.
I wasn't sure either, but couldn't think of a better location. Anybody
else have any ideas?
>
>>>> menu "Dump support"
>>>> diff --git a/arch/s390/configs/default_defconfig
>>>> b/arch/s390/configs/default_defconfig
>>>> index 5af8458..40fa3f6 100644
>>>> --- a/arch/s390/configs/default_defconfig
>>>> +++ b/arch/s390/configs/default_defconfig
>>> Not sure that this file belongs to this patch
>> Neither am I, but at the time I inserted this here - well before August
>> of last year - I was using vfio-ccw as a model.
>> If someone can verify this does not belong here, I'd be more than happy
>> to remove it.
> I don't see any entry for VFIO_CCW in there?
There isn't now, but there was at the time I first started working on this.
My first pass was a lot of cut-and-paste followed by modification of the
vfio_ccw implementation, but that was long ago. This is a remnant of that
time. Like I said, I'd be happy to remove this.
>
>>>
>>>> @@ -719,3 +719,6 @@ CONFIG_APPLDATA_BASE=y
>>>> CONFIG_KVM=m
>>>> CONFIG_KVM_S390_UCONTROL=y
>>>> CONFIG_VHOST_NET=m
>>>> +VFIO_MDEV=m
>>>> +VFIO_MDEV_DEVICE=m
>>>> +CONFIG_VFIO_AP=m
>>> What is your goal when modifying this three files?
>>> Could you add a comment in the commit message?
>> As stated above, this was originally based on the vfio-ccw model and has
>> been in the
>> patch series since its inception. I'd be happy to remove it if it is not
>> necessary.
> I'd vote for removing it.
Consider them gone.
>
> (...)
>
>>>> +static int vfio_ap_matrix_dev_create(void)
>>>> +{
>>>> + int ret;
>>>> +
>>>> + vfio_ap_root_device = root_device_register(VFIO_AP_ROOT_NAME);
>>>> +
>>>> + ret = PTR_ERR_OR_ZERO(vfio_ap_root_device);
>>> IS_ERR() is enough, root_device_register() never return NULL.
>> I searched the kernel code to look at other places the
>> root_device_register()
>> function is called to see how the return value is handled. I've seen all
>> of the
>> following used:
>> if (IS_ERR())
>> ret = PTR_ERR()
>> PTR_ERR()
>> PTR_ERR_OR_ZERO()
>>
>> I'm not sure why this is a concern, but I'll use the first option above
>> since PTR_ERR_OR_ZERO() also embeds the first option.
> PTR_ERR_OR_ZERO() seems like the best choice for the way the return
> code is processed here. (It's just unfortunate that its name conjures
> up connotations of NULL-pointer handling.)
I changed it to:
ret = IS_ERR(vfio_ap_root_device);
if (ret) {
ret = PTR_ERR(vfio_ap_root_device);
goto done;
}
Hopefully everybody is happy with this.
>
>>>> + if (ret)
>>>> + goto done;
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 08/15] KVM: s390: interface to enable AP execution mode
2018-02-28 9:42 ` David Hildenbrand
@ 2018-02-28 20:37 ` Tony Krowiak
2018-03-01 9:32 ` David Hildenbrand
0 siblings, 1 reply; 56+ messages in thread
From: Tony Krowiak @ 2018-02-28 20:37 UTC (permalink / raw)
To: David Hildenbrand, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen
On 02/28/2018 04:42 AM, David Hildenbrand wrote:
> On 27.02.2018 15:28, Tony Krowiak wrote:
>> Introduces a new interface to enable AP interpretive
>> execution (IE) mode for the KVM guest. When running
>> with IE mode enabled, AP instructions executed on the
>> KVM guest will be interpreted by the firmware and
>> passed directly through to an AP device installed on
>> the system. The CPU model feature for AP must
>> be enabled for the KVM guest in order to enable
>> interpretive execution mode.
>>
>> This interface will be used in a subsequent patch
>> by the VFIO AP device driver.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>> arch/s390/include/asm/kvm-ap.h | 2 ++
>> arch/s390/include/asm/kvm_host.h | 1 +
>> arch/s390/kvm/kvm-ap.c | 27 +++++++++++++++++++++++++++
>> arch/s390/kvm/kvm-s390.h | 1 +
>> 4 files changed, 31 insertions(+), 0 deletions(-)
>>
>> diff --git a/arch/s390/include/asm/kvm-ap.h b/arch/s390/include/asm/kvm-ap.h
>> index 46e7c5b..6bd6bfb 100644
>> --- a/arch/s390/include/asm/kvm-ap.h
>> +++ b/arch/s390/include/asm/kvm-ap.h
>> @@ -51,4 +51,6 @@ struct kvm_ap_matrix {
>>
>> void kvm_ap_deconfigure_matrix(struct kvm *kvm);
>>
>> +int kvm_ap_enable_ie_mode(struct kvm *kvm);
>> +
>> #endif /* _ASM_KVM_AP */
>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>> index a4c77d3..1eebdd6 100644
>> --- a/arch/s390/include/asm/kvm_host.h
>> +++ b/arch/s390/include/asm/kvm_host.h
>> @@ -186,6 +186,7 @@ struct kvm_s390_sie_block {
>> #define ECA_AIV 0x00200000
>> #define ECA_VX 0x00020000
>> #define ECA_PROTEXCI 0x00002000
>> +#define ECA_APIE 0x00000008
>> #define ECA_SII 0x00000001
>> __u32 eca; /* 0x004c */
>> #define ICPT_INST 0x04
>> diff --git a/arch/s390/kvm/kvm-ap.c b/arch/s390/kvm/kvm-ap.c
>> index bb29045..862e54b 100644
>> --- a/arch/s390/kvm/kvm-ap.c
>> +++ b/arch/s390/kvm/kvm-ap.c
>> @@ -307,3 +307,30 @@ void kvm_ap_deconfigure_matrix(struct kvm *kvm)
>> kvm_ap_clear_crycb_masks(kvm);
>> }
>> EXPORT_SYMBOL(kvm_ap_deconfigure_matrix);
>> +
>> +/**
>> + * kvm_ap_enable_ie_mode
>> + *
>> + * Enable interpretrive execution of AP instructions for the guest. When
>> + * enabled, AP instructions executed on the guest will be interpreted and
>> + * passed through to an AP installed on the host system.
>> + *
>> + * Returns 0 if interpretrive execution is enabled. Returns -EOPNOTSUPP
>> + * if AP facilities are not installed for the guest.
>> + *
>> + * @kvm: the guest's kvm structure
>> + */
>> +int kvm_ap_enable_ie_mode(struct kvm *kvm)
>> +{
>> + int i;
>> + struct kvm_vcpu *vcpu;
>> +
>> + if (!test_kvm_cpu_feat(kvm, KVM_S390_VM_CPU_FEAT_AP))
>> + return -EOPNOTSUPP;
>> +
>> + kvm_for_each_vcpu(i, vcpu, kvm)
>> + vcpu->arch.sie_block->eca |= ECA_APIE;
>> +
>> + return 0;
>> +}
>> +EXPORT_SYMBOL(kvm_ap_enable_ie_mode);
>> diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
>> index 1b5621f..3142541 100644
>> --- a/arch/s390/kvm/kvm-s390.h
>> +++ b/arch/s390/kvm/kvm-s390.h
>> @@ -18,6 +18,7 @@
>> #include <asm/facility.h>
>> #include <asm/processor.h>
>> #include <asm/sclp.h>
>> +#include <asm/ap.h>
>>
>> /* Transactional Memory Execution related macros */
>> #define IS_TE_ENABLED(vcpu) ((vcpu->arch.sie_block->ecb & ECB_TE))
>>
> What about VSIE?
That is a good question. I'm beginning to think introducing AP on second
level guests
ought to be postponed for now. I'm going to look into this further.
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 08/15] KVM: s390: interface to enable AP execution mode
2018-02-28 9:44 ` David Hildenbrand
@ 2018-02-28 20:39 ` Tony Krowiak
2018-03-01 9:35 ` David Hildenbrand
0 siblings, 1 reply; 56+ messages in thread
From: Tony Krowiak @ 2018-02-28 20:39 UTC (permalink / raw)
To: David Hildenbrand, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen
On 02/28/2018 04:44 AM, David Hildenbrand wrote:
> On 27.02.2018 15:28, Tony Krowiak wrote:
>> Introduces a new interface to enable AP interpretive
>> execution (IE) mode for the KVM guest. When running
>> with IE mode enabled, AP instructions executed on the
>> KVM guest will be interpreted by the firmware and
>> passed directly through to an AP device installed on
>> the system. The CPU model feature for AP must
>> be enabled for the KVM guest in order to enable
>> interpretive execution mode.
>>
>> This interface will be used in a subsequent patch
>> by the VFIO AP device driver.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>> arch/s390/include/asm/kvm-ap.h | 2 ++
>> arch/s390/include/asm/kvm_host.h | 1 +
>> arch/s390/kvm/kvm-ap.c | 27 +++++++++++++++++++++++++++
>> arch/s390/kvm/kvm-s390.h | 1 +
>> 4 files changed, 31 insertions(+), 0 deletions(-)
>>
>> diff --git a/arch/s390/include/asm/kvm-ap.h b/arch/s390/include/asm/kvm-ap.h
>> index 46e7c5b..6bd6bfb 100644
>> --- a/arch/s390/include/asm/kvm-ap.h
>> +++ b/arch/s390/include/asm/kvm-ap.h
>> @@ -51,4 +51,6 @@ struct kvm_ap_matrix {
>>
>> void kvm_ap_deconfigure_matrix(struct kvm *kvm);
>>
>> +int kvm_ap_enable_ie_mode(struct kvm *kvm);
>> +
>> #endif /* _ASM_KVM_AP */
>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>> index a4c77d3..1eebdd6 100644
>> --- a/arch/s390/include/asm/kvm_host.h
>> +++ b/arch/s390/include/asm/kvm_host.h
>> @@ -186,6 +186,7 @@ struct kvm_s390_sie_block {
>> #define ECA_AIV 0x00200000
>> #define ECA_VX 0x00020000
>> #define ECA_PROTEXCI 0x00002000
>> +#define ECA_APIE 0x00000008
>> #define ECA_SII 0x00000001
>> __u32 eca; /* 0x004c */
>> #define ICPT_INST 0x04
>> diff --git a/arch/s390/kvm/kvm-ap.c b/arch/s390/kvm/kvm-ap.c
>> index bb29045..862e54b 100644
>> --- a/arch/s390/kvm/kvm-ap.c
>> +++ b/arch/s390/kvm/kvm-ap.c
>> @@ -307,3 +307,30 @@ void kvm_ap_deconfigure_matrix(struct kvm *kvm)
>> kvm_ap_clear_crycb_masks(kvm);
>> }
>> EXPORT_SYMBOL(kvm_ap_deconfigure_matrix);
>> +
>> +/**
>> + * kvm_ap_enable_ie_mode
>> + *
>> + * Enable interpretrive execution of AP instructions for the guest. When
>> + * enabled, AP instructions executed on the guest will be interpreted and
>> + * passed through to an AP installed on the host system.
>> + *
>> + * Returns 0 if interpretrive execution is enabled. Returns -EOPNOTSUPP
>> + * if AP facilities are not installed for the guest.
>> + *
>> + * @kvm: the guest's kvm structure
>> + */
>> +int kvm_ap_enable_ie_mode(struct kvm *kvm)
>> +{
>> + int i;
>> + struct kvm_vcpu *vcpu;
>> +
>> + if (!test_kvm_cpu_feat(kvm, KVM_S390_VM_CPU_FEAT_AP))
>> + return -EOPNOTSUPP;
>> +
>> + kvm_for_each_vcpu(i, vcpu, kvm)
>> + vcpu->arch.sie_block->eca |= ECA_APIE;
>> +
>> + return 0;
>> +}
>> +EXPORT_SYMBOL(kvm_ap_enable_ie_mode);
>> diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
>> index 1b5621f..3142541 100644
>> --- a/arch/s390/kvm/kvm-s390.h
>> +++ b/arch/s390/kvm/kvm-s390.h
>> @@ -18,6 +18,7 @@
>> #include <asm/facility.h>
>> #include <asm/processor.h>
>> #include <asm/sclp.h>
>> +#include <asm/ap.h>
>>
>> /* Transactional Memory Execution related macros */
>> #define IS_TE_ENABLED(vcpu) ((vcpu->arch.sie_block->ecb & ECB_TE))
>>
> And also, what about hot-plugged CPUs?
I haven't considered that, do you have any suggestions?
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 13/15] KVM: s390: Configure the guest's CRYCB
2018-02-28 9:49 ` David Hildenbrand
@ 2018-02-28 20:45 ` Tony Krowiak
2018-03-01 9:37 ` David Hildenbrand
0 siblings, 1 reply; 56+ messages in thread
From: Tony Krowiak @ 2018-02-28 20:45 UTC (permalink / raw)
To: David Hildenbrand, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen
On 02/28/2018 04:49 AM, David Hildenbrand wrote:
>> +static int vfio_ap_mdev_open(struct mdev_device *mdev)
>> +{
>> + struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
>> + unsigned long events;
>> + int ret;
>> +
>> + matrix_mdev->group_notifier.notifier_call = vfio_ap_mdev_group_notifier;
>> + events = VFIO_GROUP_NOTIFY_SET_KVM;
>> + ret = vfio_register_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
>> + &events, &matrix_mdev->group_notifier);
>> +
>> + ret = kvm_ap_configure_matrix(matrix_mdev->kvm,
>> + matrix_mdev->matrix);
>> + if (ret)
>> + return ret;
>> +
>> + ret = kvm_ap_enable_ie_mode(matrix_mdev->kvm);
> Can't this happen while the guest is already running? Or what hinders us
> from doing that?
I'm not sure exactly what you're asking here. Are you asking if the
vfio_ap_mdev_open()
function can be called multiple times while the guest is running? AFAIK
this will be
called only once when the mediated device's file descriptor is opened.
This happens in
QEMU when the -device vfio-ap device is realized.
>
>> +
>> + return ret;
>> +}
>> +
>> +static void vfio_ap_mdev_release(struct mdev_device *mdev)
> Thanks,
>
> David / dhildenb
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 01/15] KVM: s390: refactor crypto initialization
2018-02-28 17:37 ` Cornelia Huck
@ 2018-02-28 21:23 ` Tony Krowiak
2018-03-01 9:59 ` Cornelia Huck
0 siblings, 1 reply; 56+ messages in thread
From: Tony Krowiak @ 2018-02-28 21:23 UTC (permalink / raw)
To: Cornelia Huck
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
fiuczy, buendgen
On 02/28/2018 12:37 PM, Cornelia Huck wrote:
> On Tue, 27 Feb 2018 09:27:59 -0500
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> The crypto control block designation (CRYCBD) is a 32-bit
>> field in the KVM guest's SIE state description. The
>> contents of bits 1-28 of this field, with three zero bits
>> appended on the right, designate the host real 31-bit
>> address of a crypto control block (CRYCB). Bits 30-31
>> specify the format of the CRYCB. In the current
>> implementation, the address of the CRYCB is stored in
>> the CRYCBD only if the Message-Security-Assist extension
>> 3 (MSA3) facility is installed. Virtualization of AP
>> facilities, however, requires that a CRYCB of the
>> appropriate format be made available to SIE regardless
>> of whether MSA3 is installed or not.
>>
>> This patch introduces a new compilation unit to provide
>> all interfaces related to configuration of AP facilities.
>> Let's start by moving the function for setting the CRYCB
>> format from arch/s390/kvm/kvm-s390 to this new AP
>> configuration interface.
> Hm, I would tweak this patch description a bit. First, you talk about
> what the crycbd is; then, what needs to be done for vfio-ap support;
> then you simply state that you move some interfaces to a new file. I'd
> like to see a connection between those parts :)
>
> [It sounds a bit like you'd just introduce a new file and move some
> functions, while you do have more changes in there.]
I'll try to wordsmith the patch description.
>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>> MAINTAINERS | 10 ++++++
>> arch/s390/include/asm/kvm-ap.h | 16 ++++++++++
>> arch/s390/include/asm/kvm_host.h | 1 +
>> arch/s390/kvm/Makefile | 2 +-
>> arch/s390/kvm/kvm-ap.c | 47 ++++++++++++++++++++++++++++
>> arch/s390/kvm/kvm-s390.c | 62 +++++---------------------------------
>> 6 files changed, 83 insertions(+), 55 deletions(-)
>> create mode 100644 arch/s390/include/asm/kvm-ap.h
>> create mode 100644 arch/s390/kvm/kvm-ap.c
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index 0ec5881..4acf7c2 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -11875,6 +11875,16 @@ W: http://www.ibm.com/developerworks/linux/linux390/
>> S: Supported
>> F: drivers/s390/crypto/
>>
>> +S390 VFIO AP DRIVER
>> +M: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> +M: Christian BornTraeger <borntraeger@de.ibm.com>
> Typo.
Will fix
>
>> +M: Martin Schwidefsky <schwidefsky@de.ibm.com>
>> +L: linux-s390@vger.kernel.org
>> +W: http://www.ibm.com/developerworks/linux/linux390/
>> +S: Supported
>> +F: arch/s390/include/asm/kvm/kvm-ap.h
>> +F: arch/s390/kvm/kvm-ap.c
>> +
>> S390 ZFCP DRIVER
>> M: Steffen Maier <maier@linux.vnet.ibm.com>
>> M: Benjamin Block <bblock@linux.vnet.ibm.com>
> (...)
>
>> diff --git a/arch/s390/kvm/kvm-ap.c b/arch/s390/kvm/kvm-ap.c
>> new file mode 100644
>> index 0000000..5305f4c
>> --- /dev/null
>> +++ b/arch/s390/kvm/kvm-ap.c
>> @@ -0,0 +1,47 @@
>> +/*
>> + * Adjunct Processor (AP) configuration management for KVM guests
>> + *
>> + * Copyright IBM Corp. 2017
>> + *
>> + * Author(s): Tony Krowiak <akrowia@linux.vnet.ibm.com>
>> + */
>> +
>> +#include <asm/kvm-ap.h>
>> +#include <asm/ap.h>
>> +
>> +#include "kvm-s390.h"
>> +
>> +static int kvm_ap_apxa_installed(void)
>> +{
>> + int ret;
>> + struct ap_config_info config;
>> +
>> + ret = ap_query_configuration(&config);
> Doesn't that introduce a dependency on CONFIG_ZCRYPT?
It does, but AFAIK zcrypt is built into the kernel. Or is that not what
you are asking?
>
>> + if (ret)
>> + return 0;
>> +
>> + return (config.apxa == 1);
>> +}
>> +KVM guest's use.
>> +/**
>> + * kvm_ap_set_crycb_format
>> + *
>> + * Set the CRYCB format in the CRYCBD for the KVM guest.
> Spell out "crypto control block" somewhere?
Done
>
>> + *
>> + * @kvm: the KVM guest
>> + * @crycbd: the CRYCB descriptor
>> + */
>> +void kvm_ap_set_crycb_format(struct kvm *kvm, __u32 *crycbd)
>> +{
>> + *crycbd = (__u32)(unsigned long) kvm->arch.crypto.crycb;
>> +
>> + *crycbd &= ~(CRYCB_FORMAT_MASK);
>> +
>> + /* If the MSAX3 is installed */
> /* check whether MSAX3 is installed */ ?
Sure, why not
>
>> + if (test_kvm_facility(kvm, 76)) {
>> + if (kvm_ap_apxa_installed())
>> + *crycbd |= CRYCB_FORMAT2;
>> + else
>> + *crycbd |= CRYCB_FORMAT1;
>> + }
>> +}
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index 5f5a4cb..de1e299 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -1913,12 +1866,13 @@ static u64 kvm_s390_get_initial_cpuid(void)
>>
>> static void kvm_s390_crypto_init(struct kvm *kvm)
>> {
>> + kvm->arch.crypto.crycb = &kvm->arch.sie_page2->crycb;
>> + kvm->arch.crypto.crycbd = (__u32)(unsigned long) kvm->arch.crypto.crycb;
>> + kvm_ap_set_crycb_format(kvm, &kvm->arch.crypto.crycbd);
> Doesn't kvm_ap_set_crycb_format() already initialize its second
> parameter?
Yes it does. I'm going to have to rework this (see comment below)
>
> Would it make sense to do
>
> kvm->arch.crypto.crycbd = kvm_ap_build_crycbd(kvm);
>
> or so instead?
It would if this was the only place the function gets called. In patch
2, this is called
from VSIE and it wouldn't make sense in that context. I like your idea,
let me work on this
and figure out how best to make it happen.
>
>> +
>> if (!test_kvm_facility(kvm, 76))
>> return;
>>
>> - kvm->arch.crypto.crycb = &kvm->arch.sie_page2->crycb;
>> - kvm_s390_set_crycb_format(kvm);
>> -
>> /* Enable AES/DEA protected key functions by default */
>> kvm->arch.crypto.aes_kw = 1;
>> kvm->arch.crypto.dea_kw = 1;
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 03/15] s390: zcrypt: externalize AP instructions available function
2018-02-28 17:41 ` Cornelia Huck
@ 2018-03-01 8:38 ` Harald Freudenberger
0 siblings, 0 replies; 56+ messages in thread
From: Harald Freudenberger @ 2018-03-01 8:38 UTC (permalink / raw)
To: Cornelia Huck, Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
fiuczy, buendgen
On 02/28/2018 06:41 PM, Cornelia Huck wrote:
> On Tue, 27 Feb 2018 09:28:01 -0500
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> If the AP instructions are not available on the linux host, then
>> AP devices can not be interpreted by the SIE. The AP bus has a
>> function it uses to determine if the AP instructions are
>> available. This patch provides a new function that wraps the
>> AP bus's function to externalize it for use by KVM.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> Reviewed-by: Pierre Morel <pmorel@linux.vnet.ibm.com>
>> ---
>> arch/s390/include/asm/ap.h | 7 +++++++
>> drivers/s390/crypto/ap_bus.c | 6 ++++++
>> 2 files changed, 13 insertions(+), 0 deletions(-)
> While I don't see anything wrong with this patch as it stands, you need
> to be careful to call this only from vfio-ap code (which depends on
> CONFIG_ZCRYPT).
>
> Speaking of which: the ap_bus part cannot be a module, but only
> built-in, right? So we don't get additional module complications?
Right, the AP bus is not available as a module but statically build.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-s390" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 08/15] KVM: s390: interface to enable AP execution mode
2018-02-28 20:37 ` Tony Krowiak
@ 2018-03-01 9:32 ` David Hildenbrand
0 siblings, 0 replies; 56+ messages in thread
From: David Hildenbrand @ 2018-03-01 9:32 UTC (permalink / raw)
To: Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen
>>> +EXPORT_SYMBOL(kvm_ap_enable_ie_mode);
>>> diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
>>> index 1b5621f..3142541 100644
>>> --- a/arch/s390/kvm/kvm-s390.h
>>> +++ b/arch/s390/kvm/kvm-s390.h
>>> @@ -18,6 +18,7 @@
>>> #include <asm/facility.h>
>>> #include <asm/processor.h>
>>> #include <asm/sclp.h>
>>> +#include <asm/ap.h>
>>>
>>> /* Transactional Memory Execution related macros */
>>> #define IS_TE_ENABLED(vcpu) ((vcpu->arch.sie_block->ecb & ECB_TE))
>>>
>> What about VSIE?
> That is a good question. I'm beginning to think introducing AP on second
> level guests
> ought to be postponed for now. I'm going to look into this further.
>>
Once you provide AP support to the guest, vSIE also has to be in place.
But don't worry about vSIE, I think I know what we have to do.
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 08/15] KVM: s390: interface to enable AP execution mode
2018-02-28 20:39 ` Tony Krowiak
@ 2018-03-01 9:35 ` David Hildenbrand
2018-03-02 19:54 ` Tony Krowiak
2018-03-14 16:29 ` Tony Krowiak
0 siblings, 2 replies; 56+ messages in thread
From: David Hildenbrand @ 2018-03-01 9:35 UTC (permalink / raw)
To: Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen
On 28.02.2018 21:39, Tony Krowiak wrote:
> On 02/28/2018 04:44 AM, David Hildenbrand wrote:
>> On 27.02.2018 15:28, Tony Krowiak wrote:
>>> Introduces a new interface to enable AP interpretive
>>> execution (IE) mode for the KVM guest. When running
>>> with IE mode enabled, AP instructions executed on the
>>> KVM guest will be interpreted by the firmware and
>>> passed directly through to an AP device installed on
>>> the system. The CPU model feature for AP must
>>> be enabled for the KVM guest in order to enable
>>> interpretive execution mode.
>>>
>>> This interface will be used in a subsequent patch
>>> by the VFIO AP device driver.
>>>
>>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>>> ---
>>> arch/s390/include/asm/kvm-ap.h | 2 ++
>>> arch/s390/include/asm/kvm_host.h | 1 +
>>> arch/s390/kvm/kvm-ap.c | 27 +++++++++++++++++++++++++++
>>> arch/s390/kvm/kvm-s390.h | 1 +
>>> 4 files changed, 31 insertions(+), 0 deletions(-)
>>>
>>> diff --git a/arch/s390/include/asm/kvm-ap.h b/arch/s390/include/asm/kvm-ap.h
>>> index 46e7c5b..6bd6bfb 100644
>>> --- a/arch/s390/include/asm/kvm-ap.h
>>> +++ b/arch/s390/include/asm/kvm-ap.h
>>> @@ -51,4 +51,6 @@ struct kvm_ap_matrix {
>>>
>>> void kvm_ap_deconfigure_matrix(struct kvm *kvm);
>>>
>>> +int kvm_ap_enable_ie_mode(struct kvm *kvm);
>>> +
>>> #endif /* _ASM_KVM_AP */
>>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>>> index a4c77d3..1eebdd6 100644
>>> --- a/arch/s390/include/asm/kvm_host.h
>>> +++ b/arch/s390/include/asm/kvm_host.h
>>> @@ -186,6 +186,7 @@ struct kvm_s390_sie_block {
>>> #define ECA_AIV 0x00200000
>>> #define ECA_VX 0x00020000
>>> #define ECA_PROTEXCI 0x00002000
>>> +#define ECA_APIE 0x00000008
>>> #define ECA_SII 0x00000001
>>> __u32 eca; /* 0x004c */
>>> #define ICPT_INST 0x04
>>> diff --git a/arch/s390/kvm/kvm-ap.c b/arch/s390/kvm/kvm-ap.c
>>> index bb29045..862e54b 100644
>>> --- a/arch/s390/kvm/kvm-ap.c
>>> +++ b/arch/s390/kvm/kvm-ap.c
>>> @@ -307,3 +307,30 @@ void kvm_ap_deconfigure_matrix(struct kvm *kvm)
>>> kvm_ap_clear_crycb_masks(kvm);
>>> }
>>> EXPORT_SYMBOL(kvm_ap_deconfigure_matrix);
>>> +
>>> +/**
>>> + * kvm_ap_enable_ie_mode
>>> + *
>>> + * Enable interpretrive execution of AP instructions for the guest. When
>>> + * enabled, AP instructions executed on the guest will be interpreted and
>>> + * passed through to an AP installed on the host system.
>>> + *
>>> + * Returns 0 if interpretrive execution is enabled. Returns -EOPNOTSUPP
>>> + * if AP facilities are not installed for the guest.
>>> + *
>>> + * @kvm: the guest's kvm structure
>>> + */
>>> +int kvm_ap_enable_ie_mode(struct kvm *kvm)
>>> +{
>>> + int i;
>>> + struct kvm_vcpu *vcpu;
>>> +
>>> + if (!test_kvm_cpu_feat(kvm, KVM_S390_VM_CPU_FEAT_AP))
>>> + return -EOPNOTSUPP;
>>> +
>>> + kvm_for_each_vcpu(i, vcpu, kvm)
>>> + vcpu->arch.sie_block->eca |= ECA_APIE;
>>> +
>>> + return 0;
>>> +}
>>> +EXPORT_SYMBOL(kvm_ap_enable_ie_mode);
>>> diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
>>> index 1b5621f..3142541 100644
>>> --- a/arch/s390/kvm/kvm-s390.h
>>> +++ b/arch/s390/kvm/kvm-s390.h
>>> @@ -18,6 +18,7 @@
>>> #include <asm/facility.h>
>>> #include <asm/processor.h>
>>> #include <asm/sclp.h>
>>> +#include <asm/ap.h>
>>>
>>> /* Transactional Memory Execution related macros */
>>> #define IS_TE_ENABLED(vcpu) ((vcpu->arch.sie_block->ecb & ECB_TE))
>>>
>> And also, what about hot-plugged CPUs?
> I haven't considered that, do you have any suggestions?
You should handle the KVM_S390_VM_CPU_FEAT_AP feature instead during
kvm_arch_vcpu_setup(), independent of any configured AP devices.
This avoids the races I mentioned in regards to this series and also
will handle hotplugged CPUs properly.
If KVM_S390_VM_CPU_FEAT_AP is configured for a guest -> each CPU sets
ECA_APIE during kvm_arch_vcpu_setup().
(In the vSIE code, simply allow to set ECA_APIE in the shadow SCB in
case KVM_S390_VM_CPU_FEAT_AP is enabled)
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 13/15] KVM: s390: Configure the guest's CRYCB
2018-02-28 20:45 ` Tony Krowiak
@ 2018-03-01 9:37 ` David Hildenbrand
2018-03-01 20:42 ` Tony Krowiak
0 siblings, 1 reply; 56+ messages in thread
From: David Hildenbrand @ 2018-03-01 9:37 UTC (permalink / raw)
To: Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen
On 28.02.2018 21:45, Tony Krowiak wrote:
> On 02/28/2018 04:49 AM, David Hildenbrand wrote:
>>> +static int vfio_ap_mdev_open(struct mdev_device *mdev)
>>> +{
>>> + struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
>>> + unsigned long events;
>>> + int ret;
>>> +
>>> + matrix_mdev->group_notifier.notifier_call = vfio_ap_mdev_group_notifier;
>>> + events = VFIO_GROUP_NOTIFY_SET_KVM;
>>> + ret = vfio_register_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
>>> + &events, &matrix_mdev->group_notifier);
>>> +
>>> + ret = kvm_ap_configure_matrix(matrix_mdev->kvm,
>>> + matrix_mdev->matrix);
>>> + if (ret)
>>> + return ret;
>>> +
>>> + ret = kvm_ap_enable_ie_mode(matrix_mdev->kvm);
>> Can't this happen while the guest is already running? Or what hinders us
>> from doing that?
> I'm not sure exactly what you're asking here. Are you asking if the
> vfio_ap_mdev_open()
> function can be called multiple times while the guest is running? AFAIK
> this will be
> called only once when the mediated device's file descriptor is opened.
> This happens in
> QEMU when the -device vfio-ap device is realized.
Okay, but from a pure interface point of view, this could happen any
time, even while the guest is already running. Patching in the SCB of a
running VCPU is evil.
But I guess we don't have to worry about that when changing they way we
set ECA_APIE, as described in the other mail.
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 01/15] KVM: s390: refactor crypto initialization
2018-02-28 21:23 ` Tony Krowiak
@ 2018-03-01 9:59 ` Cornelia Huck
2018-03-14 16:02 ` Tony Krowiak
0 siblings, 1 reply; 56+ messages in thread
From: Cornelia Huck @ 2018-03-01 9:59 UTC (permalink / raw)
To: Tony Krowiak
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
fiuczy, buendgen
On Wed, 28 Feb 2018 16:23:29 -0500
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> On 02/28/2018 12:37 PM, Cornelia Huck wrote:
> > On Tue, 27 Feb 2018 09:27:59 -0500
> > Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> >> diff --git a/arch/s390/kvm/kvm-ap.c b/arch/s390/kvm/kvm-ap.c
> >> new file mode 100644
> >> index 0000000..5305f4c
> >> --- /dev/null
> >> +++ b/arch/s390/kvm/kvm-ap.c
> >> @@ -0,0 +1,47 @@
> >> +/*
> >> + * Adjunct Processor (AP) configuration management for KVM guests
> >> + *
> >> + * Copyright IBM Corp. 2017
> >> + *
> >> + * Author(s): Tony Krowiak <akrowia@linux.vnet.ibm.com>
> >> + */
> >> +
> >> +#include <asm/kvm-ap.h>
> >> +#include <asm/ap.h>
> >> +
> >> +#include "kvm-s390.h"
> >> +
> >> +static int kvm_ap_apxa_installed(void)
> >> +{
> >> + int ret;
> >> + struct ap_config_info config;
> >> +
> >> + ret = ap_query_configuration(&config);
> > Doesn't that introduce a dependency on CONFIG_ZCRYPT?
> It does, but AFAIK zcrypt is built into the kernel. Or is that not what
> you are asking?
It is built into the kernel (and not into a module) if CONFIG_ZCRYPT is
set. When I compile a kernel with CONFIG_KVM set and CONFIG_ZCRYPT
unset, with this patch applied I get
arch/s390/kvm/kvm-ap.o: In function `kvm_ap_apxa_installed':
/home/cohuck/git/linux/arch/s390/kvm/kvm-ap.c:19: undefined reference to `ap_query_configuration'
So I'm afraid you cannot use ap_query_configuration() in base s390 kvm
code unless you move that function to a place where it is always built
(or at least always built if either CONFIG_KVM or CONFIG_ZCRYPT are
set).
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 13/15] KVM: s390: Configure the guest's CRYCB
2018-03-01 9:37 ` David Hildenbrand
@ 2018-03-01 20:42 ` Tony Krowiak
2018-03-02 10:08 ` David Hildenbrand
0 siblings, 1 reply; 56+ messages in thread
From: Tony Krowiak @ 2018-03-01 20:42 UTC (permalink / raw)
To: David Hildenbrand, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen
On 03/01/2018 04:37 AM, David Hildenbrand wrote:
> On 28.02.2018 21:45, Tony Krowiak wrote:
>> On 02/28/2018 04:49 AM, David Hildenbrand wrote:
>>>> +static int vfio_ap_mdev_open(struct mdev_device *mdev)
>>>> +{
>>>> + struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
>>>> + unsigned long events;
>>>> + int ret;
>>>> +
>>>> + matrix_mdev->group_notifier.notifier_call = vfio_ap_mdev_group_notifier;
>>>> + events = VFIO_GROUP_NOTIFY_SET_KVM;
>>>> + ret = vfio_register_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
>>>> + &events, &matrix_mdev->group_notifier);
>>>> +
>>>> + ret = kvm_ap_configure_matrix(matrix_mdev->kvm,
>>>> + matrix_mdev->matrix);
>>>> + if (ret)
>>>> + return ret;
>>>> +
>>>> + ret = kvm_ap_enable_ie_mode(matrix_mdev->kvm);
>>> Can't this happen while the guest is already running? Or what hinders us
>>> from doing that?
>> I'm not sure exactly what you're asking here. Are you asking if the
>> vfio_ap_mdev_open()
>> function can be called multiple times while the guest is running? AFAIK
>> this will be
>> called only once when the mediated device's file descriptor is opened.
>> This happens in
>> QEMU when the -device vfio-ap device is realized.
> Okay, but from a pure interface point of view, this could happen any
> time, even while the guest is already running. Patching in the SCB of a
> running VCPU is evil.
How can this happen while the guest is running? QEMU opens the fd when the
device is realized and AFAIK vfio mdev will not allow any other process to
open it until the guest is terminated. What am I missing?
>
> But I guess we don't have to worry about that when changing they way we
> set ECA_APIE, as described in the other mail.
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 13/15] KVM: s390: Configure the guest's CRYCB
2018-03-01 20:42 ` Tony Krowiak
@ 2018-03-02 10:08 ` David Hildenbrand
2018-03-02 19:48 ` Tony Krowiak
0 siblings, 1 reply; 56+ messages in thread
From: David Hildenbrand @ 2018-03-02 10:08 UTC (permalink / raw)
To: Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen
On 01.03.2018 21:42, Tony Krowiak wrote:
> On 03/01/2018 04:37 AM, David Hildenbrand wrote:
>> On 28.02.2018 21:45, Tony Krowiak wrote:
>>> On 02/28/2018 04:49 AM, David Hildenbrand wrote:
>>>>> +static int vfio_ap_mdev_open(struct mdev_device *mdev)
>>>>> +{
>>>>> + struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
>>>>> + unsigned long events;
>>>>> + int ret;
>>>>> +
>>>>> + matrix_mdev->group_notifier.notifier_call = vfio_ap_mdev_group_notifier;
>>>>> + events = VFIO_GROUP_NOTIFY_SET_KVM;
>>>>> + ret = vfio_register_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
>>>>> + &events, &matrix_mdev->group_notifier);
>>>>> +
>>>>> + ret = kvm_ap_configure_matrix(matrix_mdev->kvm,
>>>>> + matrix_mdev->matrix);
>>>>> + if (ret)
>>>>> + return ret;
>>>>> +
>>>>> + ret = kvm_ap_enable_ie_mode(matrix_mdev->kvm);
>>>> Can't this happen while the guest is already running? Or what hinders us
>>>> from doing that?
>>> I'm not sure exactly what you're asking here. Are you asking if the
>>> vfio_ap_mdev_open()
>>> function can be called multiple times while the guest is running? AFAIK
>>> this will be
>>> called only once when the mediated device's file descriptor is opened.
>>> This happens in
>>> QEMU when the -device vfio-ap device is realized.
>> Okay, but from a pure interface point of view, this could happen any
>> time, even while the guest is already running. Patching in the SCB of a
>> running VCPU is evil.
> How can this happen while the guest is running? QEMU opens the fd when the
> device is realized and AFAIK vfio mdev will not allow any other process to
> open it until the guest is terminated. What am I missing?
It can't happen right now (the way QEMU uses it), but the kernel
interface allows it, no?
Anyhow, as discussed this should be handled directly while creating a
VCPU. Then also CPU hotplug is properly covered.
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 13/15] KVM: s390: Configure the guest's CRYCB
2018-03-02 10:08 ` David Hildenbrand
@ 2018-03-02 19:48 ` Tony Krowiak
0 siblings, 0 replies; 56+ messages in thread
From: Tony Krowiak @ 2018-03-02 19:48 UTC (permalink / raw)
To: David Hildenbrand, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen, akrowiak
On 03/02/2018 05:08 AM, David Hildenbrand wrote:
> On 01.03.2018 21:42, Tony Krowiak wrote:
>> On 03/01/2018 04:37 AM, David Hildenbrand wrote:
>>> On 28.02.2018 21:45, Tony Krowiak wrote:
>>>> On 02/28/2018 04:49 AM, David Hildenbrand wrote:
>>>>>> +static int vfio_ap_mdev_open(struct mdev_device *mdev)
>>>>>> +{
>>>>>> + struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
>>>>>> + unsigned long events;
>>>>>> + int ret;
>>>>>> +
>>>>>> + matrix_mdev->group_notifier.notifier_call = vfio_ap_mdev_group_notifier;
>>>>>> + events = VFIO_GROUP_NOTIFY_SET_KVM;
>>>>>> + ret = vfio_register_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
>>>>>> + &events, &matrix_mdev->group_notifier);
>>>>>> +
>>>>>> + ret = kvm_ap_configure_matrix(matrix_mdev->kvm,
>>>>>> + matrix_mdev->matrix);
>>>>>> + if (ret)
>>>>>> + return ret;
>>>>>> +
>>>>>> + ret = kvm_ap_enable_ie_mode(matrix_mdev->kvm);
>>>>> Can't this happen while the guest is already running? Or what hinders us
>>>>> from doing that?
>>>> I'm not sure exactly what you're asking here. Are you asking if the
>>>> vfio_ap_mdev_open()
>>>> function can be called multiple times while the guest is running? AFAIK
>>>> this will be
>>>> called only once when the mediated device's file descriptor is opened.
>>>> This happens in
>>>> QEMU when the -device vfio-ap device is realized.
>>> Okay, but from a pure interface point of view, this could happen any
>>> time, even while the guest is already running. Patching in the SCB of a
>>> running VCPU is evil.
>> How can this happen while the guest is running? QEMU opens the fd when the
>> device is realized and AFAIK vfio mdev will not allow any other process to
>> open it until the guest is terminated. What am I missing?
> It can't happen right now (the way QEMU uses it), but the kernel
> interface allows it, no?
>
> Anyhow, as discussed this should be handled directly while creating a
> VCPU. Then also CPU hotplug is properly covered.
Here is what I think we should do:
* Set ECA.28 in the VCPU setup function based on whether the CPU model
feature
has been turned on by user space as you suggest.
* Replace the kvm_ap_enable_ie_mode() call in the open() above with a
query of
the CPU model feature and return an error if it is not turned on.
Does this sound reasonable?
Would it be more appropriate in this case to rename the feature to
KVM_S390_VM_CPU_FEAT_APIE?
>
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 08/15] KVM: s390: interface to enable AP execution mode
2018-03-01 9:35 ` David Hildenbrand
@ 2018-03-02 19:54 ` Tony Krowiak
2018-03-14 16:29 ` Tony Krowiak
1 sibling, 0 replies; 56+ messages in thread
From: Tony Krowiak @ 2018-03-02 19:54 UTC (permalink / raw)
To: David Hildenbrand, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen, akrowiak
On 03/01/2018 04:35 AM, David Hildenbrand wrote:
> On 28.02.2018 21:39, Tony Krowiak wrote:
>> On 02/28/2018 04:44 AM, David Hildenbrand wrote:
>>> On 27.02.2018 15:28, Tony Krowiak wrote:
>>>> Introduces a new interface to enable AP interpretive
>>>> execution (IE) mode for the KVM guest. When running
>>>> with IE mode enabled, AP instructions executed on the
>>>> KVM guest will be interpreted by the firmware and
>>>> passed directly through to an AP device installed on
>>>> the system. The CPU model feature for AP must
>>>> be enabled for the KVM guest in order to enable
>>>> interpretive execution mode.
>>>>
>>>> This interface will be used in a subsequent patch
>>>> by the VFIO AP device driver.
>>>>
>>>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>>>> ---
>>>> arch/s390/include/asm/kvm-ap.h | 2 ++
>>>> arch/s390/include/asm/kvm_host.h | 1 +
>>>> arch/s390/kvm/kvm-ap.c | 27 +++++++++++++++++++++++++++
>>>> arch/s390/kvm/kvm-s390.h | 1 +
>>>> 4 files changed, 31 insertions(+), 0 deletions(-)
>>>>
>>>> diff --git a/arch/s390/include/asm/kvm-ap.h b/arch/s390/include/asm/kvm-ap.h
>>>> index 46e7c5b..6bd6bfb 100644
>>>> --- a/arch/s390/include/asm/kvm-ap.h
>>>> +++ b/arch/s390/include/asm/kvm-ap.h
>>>> @@ -51,4 +51,6 @@ struct kvm_ap_matrix {
>>>>
>>>> void kvm_ap_deconfigure_matrix(struct kvm *kvm);
>>>>
>>>> +int kvm_ap_enable_ie_mode(struct kvm *kvm);
>>>> +
>>>> #endif /* _ASM_KVM_AP */
>>>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>>>> index a4c77d3..1eebdd6 100644
>>>> --- a/arch/s390/include/asm/kvm_host.h
>>>> +++ b/arch/s390/include/asm/kvm_host.h
>>>> @@ -186,6 +186,7 @@ struct kvm_s390_sie_block {
>>>> #define ECA_AIV 0x00200000
>>>> #define ECA_VX 0x00020000
>>>> #define ECA_PROTEXCI 0x00002000
>>>> +#define ECA_APIE 0x00000008
>>>> #define ECA_SII 0x00000001
>>>> __u32 eca; /* 0x004c */
>>>> #define ICPT_INST 0x04
>>>> diff --git a/arch/s390/kvm/kvm-ap.c b/arch/s390/kvm/kvm-ap.c
>>>> index bb29045..862e54b 100644
>>>> --- a/arch/s390/kvm/kvm-ap.c
>>>> +++ b/arch/s390/kvm/kvm-ap.c
>>>> @@ -307,3 +307,30 @@ void kvm_ap_deconfigure_matrix(struct kvm *kvm)
>>>> kvm_ap_clear_crycb_masks(kvm);
>>>> }
>>>> EXPORT_SYMBOL(kvm_ap_deconfigure_matrix);
>>>> +
>>>> +/**
>>>> + * kvm_ap_enable_ie_mode
>>>> + *
>>>> + * Enable interpretrive execution of AP instructions for the guest. When
>>>> + * enabled, AP instructions executed on the guest will be interpreted and
>>>> + * passed through to an AP installed on the host system.
>>>> + *
>>>> + * Returns 0 if interpretrive execution is enabled. Returns -EOPNOTSUPP
>>>> + * if AP facilities are not installed for the guest.
>>>> + *
>>>> + * @kvm: the guest's kvm structure
>>>> + */
>>>> +int kvm_ap_enable_ie_mode(struct kvm *kvm)
>>>> +{
>>>> + int i;
>>>> + struct kvm_vcpu *vcpu;
>>>> +
>>>> + if (!test_kvm_cpu_feat(kvm, KVM_S390_VM_CPU_FEAT_AP))
>>>> + return -EOPNOTSUPP;
>>>> +
>>>> + kvm_for_each_vcpu(i, vcpu, kvm)
>>>> + vcpu->arch.sie_block->eca |= ECA_APIE;
>>>> +
>>>> + return 0;
>>>> +}
>>>> +EXPORT_SYMBOL(kvm_ap_enable_ie_mode);
>>>> diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
>>>> index 1b5621f..3142541 100644
>>>> --- a/arch/s390/kvm/kvm-s390.h
>>>> +++ b/arch/s390/kvm/kvm-s390.h
>>>> @@ -18,6 +18,7 @@
>>>> #include <asm/facility.h>
>>>> #include <asm/processor.h>
>>>> #include <asm/sclp.h>
>>>> +#include <asm/ap.h>
>>>>
>>>> /* Transactional Memory Execution related macros */
>>>> #define IS_TE_ENABLED(vcpu) ((vcpu->arch.sie_block->ecb & ECB_TE))
>>>>
>>> And also, what about hot-plugged CPUs?
>> I haven't considered that, do you have any suggestions?
> You should handle the KVM_S390_VM_CPU_FEAT_AP feature instead during
> kvm_arch_vcpu_setup(), independent of any configured AP devices.
>
> This avoids the races I mentioned in regards to this series and also
> will handle hotplugged CPUs properly.
>
> If KVM_S390_VM_CPU_FEAT_AP is configured for a guest -> each CPU sets
> ECA_APIE during kvm_arch_vcpu_setup().
We talked about this in a couple of other patches, but I am on
board with this. To make things clearer, Should we rename
the feature to KVM_S390_VM_CPU_FEAT_APIE so its purpose is more
clear?
>
>
> (In the vSIE code, simply allow to set ECA_APIE in the shadow SCB in
> case KVM_S390_VM_CPU_FEAT_AP is enabled)
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 07/15] KVM: s390: Interfaces to configure/deconfigure guest's AP matrix
2018-02-27 14:28 ` [PATCH v2 07/15] KVM: s390: Interfaces to configure/deconfigure guest's AP matrix Tony Krowiak
` (2 preceding siblings ...)
2018-02-28 19:38 ` Tony Krowiak
@ 2018-03-08 23:03 ` Tony Krowiak
3 siblings, 0 replies; 56+ messages in thread
From: Tony Krowiak @ 2018-03-08 23:03 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen, akrowiak
On 02/27/2018 09:28 AM, Tony Krowiak wrote:
> Provides interfaces to assign AP adapters, usage domains
> and control domains to a KVM guest.
>
> A KVM guest is started by executing the Start Interpretive Execution (SIE)
> instruction. The SIE state description is a control block that contains the
> state information for a KVM guest and is supplied as input to the SIE
> instruction. The SIE state description has a satellite structure called the
> Crypto Control Block (CRYCB). The CRYCB contains three bitmask fields
> identifying the adapters, queues (domains) and control domains assigned to
> the KVM guest:
>
> * The AP Adapter Mask (APM) field identifies the AP adapters assigned to
> the KVM guest
>
> * The AP Queue Mask (AQM) field identifies the AP queues assigned to
> the KVM guest. Each AP queue is connected to a usage domain within
> an AP adapter.
>
> * The AP Domain Mask (ADM) field identifies the control domains
> assigned to the KVM guest.
>
> Each adapter, queue (usage domain) and control domain are identified by
> a number from 0 to 255. The bits in each mask, from most significant to
> least significant bit, correspond to the numbers 0-255. When a bit is
> set, the corresponding adapter, queue (usage domain) or control domain
> is assigned to the KVM guest.
>
> This patch will set the bits in the APM, AQM and ADM fields of the
> CRYCB referenced by the KVM guest's SIE state description. The process
> used is:
>
> 1. Verify that the bits to be set do not exceed the maximum bit
> number for the given mask.
>
> 2. Verify that the APQNs that can be derived from the intersection
> of the bits set in the APM and AQM fields of the KVM guest's CRYCB
> are not assigned to any other KVM guest running on the same linux
> host.
>
> 3. Set the APM, AQM and ADM in the CRYCB according to the matrix
> configured for the mediated matrix device via its sysfs
> adapter, domain and control domain attribute files respectively.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
> arch/s390/include/asm/kvm-ap.h | 36 +++++
> arch/s390/kvm/kvm-ap.c | 257 +++++++++++++++++++++++++++++++++
> drivers/s390/crypto/vfio_ap_ops.c | 19 +++
> drivers/s390/crypto/vfio_ap_private.h | 4 +
> 4 files changed, 316 insertions(+), 0 deletions(-)
>
> diff --git a/arch/s390/include/asm/kvm-ap.h b/arch/s390/include/asm/kvm-ap.h
> index ef749e7..46e7c5b 100644
> --- a/arch/s390/include/asm/kvm-ap.h
> +++ b/arch/s390/include/asm/kvm-ap.h
> @@ -10,9 +10,45 @@
> #define _ASM_KVM_AP
> #include <linux/types.h>
> #include <linux/kvm_host.h>
> +#include <linux/types.h>
> +#include <linux/kvm_host.h>
> +#include <linux/bitops.h>
> +
> +#define KVM_AP_MASK_BYTES(n)(n / BITS_PER_BYTE)
> +
> +/**
> + * The AP matrix is comprised of three bit masks identifying the adapters,
> + * queues (domains) and control domains that belong to an AP matrix. The bits in
> + * each mask, from least significant to most significant bit, correspond to IDs
> + * 0 to the maximum ID allowed for a given mask. When a bit is set, the
> + * corresponding ID belongs to the matrix.
> + *
> + * @apm_max: max number of bits in @apm
> + * @apm identifies the AP adapters in the matrix
> + * @aqm_max: max number of bits in @aqm
> + * @aqm identifies the AP queues (domains) in the matrix
> + * @adm_max: max number of bits in @adm
> + * @adm identifies the AP control domains in the matrix
> + */
> +struct kvm_ap_matrix {
> + int apm_max;
> + unsigned long *apm;
> + int aqm_max;
> + unsigned long *aqm;
> + int adm_max;
> + unsigned long *adm;
> +};
>
> void kvm_ap_set_crycb_format(struct kvm *kvm, __u32 *crycbd);
>
> int kvm_ap_get_crycb_format(struct kvm *kvm);
>
> +int kvm_ap_matrix_create(struct kvm_ap_matrix **ap_matrix);
> +
> +void kvm_ap_matrix_destroy(struct kvm_ap_matrix *ap_matrix);
> +
> +int kvm_ap_configure_matrix(struct kvm *kvm, struct kvm_ap_matrix *matrix);
> +
> +void kvm_ap_deconfigure_matrix(struct kvm *kvm);
> +
> #endif /* _ASM_KVM_AP */
> diff --git a/arch/s390/kvm/kvm-ap.c b/arch/s390/kvm/kvm-ap.c
> index bafe63b..bb29045 100644
> --- a/arch/s390/kvm/kvm-ap.c
> +++ b/arch/s390/kvm/kvm-ap.c
> @@ -8,6 +8,7 @@
>
> #include <asm/kvm-ap.h>
> #include <asm/ap.h>
> +#include <linux/bitops.h>
>
> #include "kvm-s390.h"
>
> @@ -16,6 +17,125 @@ int kvm_ap_get_crycb_format(struct kvm *kvm)
> return kvm->arch.crypto.crycbd & CRYCB_FORMAT_MASK;
> }
>
> +static inline void kvm_ap_clear_crycb_masks(struct kvm *kvm)
> +{
> + int crycb_fmt = kvm_ap_get_crycb_format(kvm);
> +
> + if (crycb_fmt == CRYCB_FORMAT2)
> + memset(&kvm->arch.crypto.crycb->apcb1, 0,
> + sizeof(kvm->arch.crypto.crycb->apcb1));
> + else
> + memset(&kvm->arch.crypto.crycb->apcb0, 0,
> + sizeof(kvm->arch.crypto.crycb->apcb0));
> +}
> +
> +static inline unsigned long *kvm_ap_get_crycb_apm(struct kvm *kvm)
> +{
> + unsigned long *apm;
> + int crycb_fmt = kvm_ap_get_crycb_format(kvm);
> +
> + if (crycb_fmt == CRYCB_FORMAT2)
> + apm = (unsigned long *)kvm->arch.crypto.crycb->apcb1.apm;
> + else
> + apm = (unsigned long *)kvm->arch.crypto.crycb->apcb0.apm;
> +
> + return apm;
> +}
> +
> +static inline unsigned long *kvm_ap_get_crycb_aqm(struct kvm *kvm)
> +{
> + unsigned long *aqm;
> + int crycb_fmt = kvm_ap_get_crycb_format(kvm);
> +
> + if (crycb_fmt == CRYCB_FORMAT2)
> + aqm = (unsigned long *)kvm->arch.crypto.crycb->apcb1.aqm;
> + else
> + aqm = (unsigned long *)kvm->arch.crypto.crycb->apcb0.aqm;
> +
> + return aqm;
> +}
> +
> +static inline unsigned long *kvm_ap_get_crycb_adm(struct kvm *kvm)
> +{
> + unsigned long *adm;
> + int crycb_fmt = kvm_ap_get_crycb_format(kvm);
> +
> + if (crycb_fmt == CRYCB_FORMAT2)
> + adm = (unsigned long *)kvm->arch.crypto.crycb->apcb1.adm;
> + else
> + adm = (unsigned long *)kvm->arch.crypto.crycb->apcb0.adm;
> +
> + return adm;
> +}
> +
> +static void kvm_ap_set_crycb_masks(struct kvm *kvm,
> + struct kvm_ap_matrix *matrix)
> +{
> + unsigned long *apm = kvm_ap_get_crycb_apm(kvm);
> + unsigned long *aqm = kvm_ap_get_crycb_aqm(kvm);
> + unsigned long *adm = kvm_ap_get_crycb_adm(kvm);
> +
> + kvm_ap_clear_crycb_masks(kvm);
> + memcpy(apm, matrix->apm, KVM_AP_MASK_BYTES(matrix->apm_max));
> + memcpy(aqm, matrix->aqm, KVM_AP_MASK_BYTES(matrix->aqm_max));
> +
> + /*
> + * Merge the AQM and ADM since the ADM is a superset of the
> + * AQM by architectural convention.
> + */
> + bitmap_or(adm, adm, aqm, matrix->adm_max);
> +}
> +
> +static void kvm_ap_log_sharing_err(struct kvm *kvm, unsigned long apid,
> + unsigned long apqi)
> +{
> + pr_err("%s: AP queue %02lx.%04lx is registered to guest %s", __func__,
> + apid, apqi, kvm->arch.dbf->name);
> +}
> +
> +/**
> + * kvm_ap_validate_queue_sharing
> + *
> + * Verifies that the APQNs derived from the intersection of the AP adapter IDs
> + * and AP queue indexes comprising the AP matrix are not configured for
> + * another guest. AP queue sharing is not allowed.
> + *
> + * @kvm: the KVM guest
> + * @matrix: the AP matrix
> + *
> + * Returns 0 if the APQNs are valid, otherwise; returns -EBUSY.
> + */
> +static int kvm_ap_validate_queue_sharing(struct kvm *kvm,
> + struct kvm_ap_matrix *matrix)
> +{
> + struct kvm *vm;
> + unsigned long *apm, *aqm;
> + unsigned long apid, apqi;
> +
> +
> + /* No other VM may share an AP Queue with the input VM */
> + list_for_each_entry(vm, &vm_list, vm_list) {
> + if (kvm == vm)
> + continue;
> +
> + apm = kvm_ap_get_crycb_apm(vm);
> + if (!bitmap_and(apm, apm, matrix->apm, matrix->apm_max))
> + continue;
> +
> + aqm = kvm_ap_get_crycb_aqm(vm);
> + if (!bitmap_and(aqm, aqm, matrix->aqm, matrix->aqm_max))
> + continue;
> +
> + for_each_set_bit_inv(apid, apm, matrix->apm_max)
> + for_each_set_bit_inv(apqi, aqm, matrix->aqm_max)
> + kvm_ap_log_sharing_err(kvm, apid, apqi);
> +
> + return -EBUSY;
> + }
> +
> + return 0;
> +}
> +
> static int kvm_ap_apxa_installed(void)
> {
> int ret;
> @@ -50,3 +170,140 @@ void kvm_ap_set_crycb_format(struct kvm *kvm, __u32 *crycbd)
> *crycbd |= CRYCB_FORMAT1;
> }
> }
> +
> +static int kvm_ap_matrix_apm_create(struct kvm_ap_matrix *ap_matrix, int apxa)
> +{
> + if (apxa)
> + ap_matrix->apm_max = 256;
> + else
> + ap_matrix->apm_max = 64;
> +
> + ap_matrix->apm = kzalloc(KVM_AP_MASK_BYTES(ap_matrix->apm_max),
> + GFP_KERNEL);
> + if (!ap_matrix->apm)
> + return -ENOMEM;
> +
> + return 0;
> +}
> +
> +static int kvm_ap_matrix_aqm_create(struct kvm_ap_matrix *ap_matrix, int apxa)
> +{
> + if (apxa)
> + ap_matrix->aqm_max = 256;
> + else
> + ap_matrix->aqm_max = 16;
> +
> + ap_matrix->aqm = kzalloc(KVM_AP_MASK_BYTES(ap_matrix->aqm_max),
> + GFP_KERNEL);
> + if (!ap_matrix->aqm)
> + return -ENOMEM;
> +
> + return 0;
> +}
> +
> +static int kvm_ap_matrix_adm_create(struct kvm_ap_matrix *ap_matrix, int apxa)
> +{
> + if (apxa)
> + ap_matrix->adm_max = 256;
> + else
> + ap_matrix->adm_max = 16;
> +
> + ap_matrix->adm = kzalloc(KVM_AP_MASK_BYTES(ap_matrix->adm_max),
> + GFP_KERNEL);
> + if (!ap_matrix->adm)
> + return -ENOMEM;
> +
> + return 0;
> +}
> +
> +static void kvm_ap_matrix_masks_destroy(struct kvm_ap_matrix *ap_matrix)
> +{
> + kfree(ap_matrix->apm);
> + kfree(ap_matrix->aqm);
> + kfree(ap_matrix->adm);
> +}
> +
> +int kvm_ap_matrix_create(struct kvm_ap_matrix **ap_matrix)
> +{
> + int ret;
> + int apxa = kvm_ap_apxa_installed();
> + struct kvm_ap_matrix *matrix;
> +
> + matrix = kzalloc(sizeof(*matrix), GFP_KERNEL);
> + if (!matrix)
> + return -ENOMEM;
> +
> + ret = kvm_ap_matrix_apm_create(matrix, apxa);
> + if (ret)
> + goto mask_create_err;
> +
> + ret = kvm_ap_matrix_aqm_create(matrix, apxa);
> + if (ret)
> + goto mask_create_err;
> +
> + ret = kvm_ap_matrix_adm_create(matrix, apxa);
> + if (ret)
> + goto mask_create_err;
> +
> + *ap_matrix = matrix;
> +
> + return 0;
> +
> +mask_create_err:
> + kvm_ap_matrix_masks_destroy(matrix);
> + kfree(matrix);
> + return ret;
> +}
> +EXPORT_SYMBOL(kvm_ap_matrix_create);
> +
> +void kvm_ap_matrix_destroy(struct kvm_ap_matrix *ap_matrix)
> +{
> + kvm_ap_matrix_masks_destroy(ap_matrix);
> + kfree(ap_matrix);
> +}
> +EXPORT_SYMBOL(kvm_ap_matrix_destroy);
> +
> +/**
> + * kvm_ap_configure_matrix
> + *
> + * Configure the AP matrix for a KVM guest.
> + *
> + * @kvm: the KVM guest
> + * @matrix: the matrix configuration information
> + *
> + * Returns 0 if the APQNs derived from the intersection of the set of adapter
> + * IDs (APM) and queue indexes (AQM) in @matrix are not configured for any
> + * other KVM guest running on the same linux host. Otherwise returns an error
> + * code.
> + */
> +int kvm_ap_configure_matrix(struct kvm *kvm, struct kvm_ap_matrix *matrix)
> +{
> + int ret = 0;
> +
> + mutex_lock(&kvm->lock);
> +
> + ret = kvm_ap_validate_queue_sharing(kvm, matrix);
> + if (ret)
> + return ret;
The kvm->lock needs to be unlocked.
> +
> + kvm_ap_set_crycb_masks(kvm, matrix);
> +
> + mutex_unlock(&kvm->lock);
> +
> + return 0;
> +}
> +EXPORT_SYMBOL(kvm_ap_configure_matrix);
> +
> +/**
> + * kvm_ap_deconfigure_matrix
> + *
> + * Deconfigure the AP matrix for a KVM guest. Clears all of the bits in the
> + * APM, AQM and ADM in the guest's CRYCB.
> + *
> + * @kvm: the KVM guest
> + */
> +void kvm_ap_deconfigure_matrix(struct kvm *kvm)
> +{
> + kvm_ap_clear_crycb_masks(kvm);
> +}
> +EXPORT_SYMBOL(kvm_ap_deconfigure_matrix);
> diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
> index 4292a5e..4fda44e 100644
> --- a/drivers/s390/crypto/vfio_ap_ops.c
> +++ b/drivers/s390/crypto/vfio_ap_ops.c
> @@ -10,6 +10,7 @@
> #include <linux/device.h>
> #include <linux/list.h>
> #include <linux/ctype.h>
> +#include <asm/kvm-ap.h>
>
> #include "vfio_ap_private.h"
>
> @@ -18,8 +19,23 @@
>
> static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev)
> {
> + int ret;
> + struct ap_matrix_mdev *matrix_mdev;
> struct ap_matrix *ap_matrix = to_ap_matrix(mdev_parent_dev(mdev));
> + struct kvm_ap_matrix *matrix;
> +
> + ret = kvm_ap_matrix_create(&matrix);
> + if (ret)
> + return ret;
> +
> + matrix_mdev = kzalloc(sizeof(*matrix_mdev), GFP_KERNEL);
> + if (!matrix_mdev) {
> + kvm_ap_matrix_destroy(matrix);
> + return -ENOMEM;
> + }
>
> + matrix_mdev->matrix = matrix;
> + mdev_set_drvdata(mdev, matrix_mdev);
> ap_matrix->available_instances--;
>
> return 0;
> @@ -28,7 +44,10 @@ static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev)
> static int vfio_ap_mdev_remove(struct mdev_device *mdev)
> {
> struct ap_matrix *ap_matrix = to_ap_matrix(mdev_parent_dev(mdev));
> + struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
>
> + kvm_ap_matrix_destroy(matrix_mdev->matrix);
> + kfree(matrix_mdev);
> ap_matrix->available_instances++;
>
> return 0;
> diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
> index c264415..522564e 100644
> --- a/drivers/s390/crypto/vfio_ap_private.h
> +++ b/drivers/s390/crypto/vfio_ap_private.h
> @@ -27,6 +27,10 @@ struct ap_matrix {
> int available_instances;
> };
>
> +struct ap_matrix_mdev {
> + struct kvm_ap_matrix *matrix;
> +};
> +
> static inline struct ap_matrix *to_ap_matrix(struct device *dev)
> {
> return container_of(dev, struct ap_matrix, device);
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 01/15] KVM: s390: refactor crypto initialization
2018-03-01 9:59 ` Cornelia Huck
@ 2018-03-14 16:02 ` Tony Krowiak
0 siblings, 0 replies; 56+ messages in thread
From: Tony Krowiak @ 2018-03-14 16:02 UTC (permalink / raw)
To: Cornelia Huck
Cc: linux-s390, linux-kernel, kvm, freude, schwidefsky,
heiko.carstens, borntraeger, kwankhede, bjsdjshi, pbonzini,
alex.williamson, pmorel, alifm, mjrosato, jjherne, thuth, pasic,
fiuczy, buendgen
On 03/01/2018 04:59 AM, Cornelia Huck wrote:
> On Wed, 28 Feb 2018 16:23:29 -0500
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> On 02/28/2018 12:37 PM, Cornelia Huck wrote:
>>> On Tue, 27 Feb 2018 09:27:59 -0500
>>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>>> diff --git a/arch/s390/kvm/kvm-ap.c b/arch/s390/kvm/kvm-ap.c
>>>> new file mode 100644
>>>> index 0000000..5305f4c
>>>> --- /dev/null
>>>> +++ b/arch/s390/kvm/kvm-ap.c
>>>> @@ -0,0 +1,47 @@
>>>> +/*
>>>> + * Adjunct Processor (AP) configuration management for KVM guests
>>>> + *
>>>> + * Copyright IBM Corp. 2017
>>>> + *
>>>> + * Author(s): Tony Krowiak <akrowia@linux.vnet.ibm.com>
>>>> + */
>>>> +
>>>> +#include <asm/kvm-ap.h>
>>>> +#include <asm/ap.h>
>>>> +
>>>> +#include "kvm-s390.h"
>>>> +
>>>> +static int kvm_ap_apxa_installed(void)
>>>> +{
>>>> + int ret;
>>>> + struct ap_config_info config;
>>>> +
>>>> + ret = ap_query_configuration(&config);
>>> Doesn't that introduce a dependency on CONFIG_ZCRYPT?
>> It does, but AFAIK zcrypt is built into the kernel. Or is that not what
>> you are asking?
> It is built into the kernel (and not into a module) if CONFIG_ZCRYPT is
> set. When I compile a kernel with CONFIG_KVM set and CONFIG_ZCRYPT
> unset, with this patch applied I get
>
> arch/s390/kvm/kvm-ap.o: In function `kvm_ap_apxa_installed':
> /home/cohuck/git/linux/arch/s390/kvm/kvm-ap.c:19: undefined reference to `ap_query_configuration'
>
> So I'm afraid you cannot use ap_query_configuration() in base s390 kvm
> code unless you move that function to a place where it is always built
> (or at least always built if either CONFIG_KVM or CONFIG_ZCRYPT are
> set).
I was able to get around this by inserting the following into
arch/s390/kvm/Kconfig:
config KVM
def_tristate y
prompt "Kernel-based Virtual Machine (KVM) support"
depends on HAVE_KVM
select PREEMPT_NOTIFIERS
...
select ZCRYPT
If this is a problem, we can discuss it in the v3 which is forthcoming
very shortly
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 08/15] KVM: s390: interface to enable AP execution mode
2018-03-01 9:35 ` David Hildenbrand
2018-03-02 19:54 ` Tony Krowiak
@ 2018-03-14 16:29 ` Tony Krowiak
1 sibling, 0 replies; 56+ messages in thread
From: Tony Krowiak @ 2018-03-14 16:29 UTC (permalink / raw)
To: David Hildenbrand, linux-s390, linux-kernel, kvm
Cc: freude, schwidefsky, heiko.carstens, borntraeger, cohuck,
kwankhede, bjsdjshi, pbonzini, alex.williamson, pmorel, alifm,
mjrosato, jjherne, thuth, pasic, fiuczy, buendgen
On 03/01/2018 04:35 AM, David Hildenbrand wrote:
> On 28.02.2018 21:39, Tony Krowiak wrote:
>> On 02/28/2018 04:44 AM, David Hildenbrand wrote:
>>> On 27.02.2018 15:28, Tony Krowiak wrote:
>>>> Introduces a new interface to enable AP interpretive
>>>> execution (IE) mode for the KVM guest. When running
>>>> with IE mode enabled, AP instructions executed on the
>>>> KVM guest will be interpreted by the firmware and
>>>> passed directly through to an AP device installed on
>>>> the system. The CPU model feature for AP must
>>>> be enabled for the KVM guest in order to enable
>>>> interpretive execution mode.
>>>>
>>>> This interface will be used in a subsequent patch
>>>> by the VFIO AP device driver.
>>>>
>>>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>>>> ---
>>>> arch/s390/include/asm/kvm-ap.h | 2 ++
>>>> arch/s390/include/asm/kvm_host.h | 1 +
>>>> arch/s390/kvm/kvm-ap.c | 27 +++++++++++++++++++++++++++
>>>> arch/s390/kvm/kvm-s390.h | 1 +
>>>> 4 files changed, 31 insertions(+), 0 deletions(-)
>>>>
>>>> diff --git a/arch/s390/include/asm/kvm-ap.h b/arch/s390/include/asm/kvm-ap.h
>>>> index 46e7c5b..6bd6bfb 100644
>>>> --- a/arch/s390/include/asm/kvm-ap.h
>>>> +++ b/arch/s390/include/asm/kvm-ap.h
>>>> @@ -51,4 +51,6 @@ struct kvm_ap_matrix {
>>>>
>>>> void kvm_ap_deconfigure_matrix(struct kvm *kvm);
>>>>
>>>> +int kvm_ap_enable_ie_mode(struct kvm *kvm);
>>>> +
>>>> #endif /* _ASM_KVM_AP */
>>>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>>>> index a4c77d3..1eebdd6 100644
>>>> --- a/arch/s390/include/asm/kvm_host.h
>>>> +++ b/arch/s390/include/asm/kvm_host.h
>>>> @@ -186,6 +186,7 @@ struct kvm_s390_sie_block {
>>>> #define ECA_AIV 0x00200000
>>>> #define ECA_VX 0x00020000
>>>> #define ECA_PROTEXCI 0x00002000
>>>> +#define ECA_APIE 0x00000008
>>>> #define ECA_SII 0x00000001
>>>> __u32 eca; /* 0x004c */
>>>> #define ICPT_INST 0x04
>>>> diff --git a/arch/s390/kvm/kvm-ap.c b/arch/s390/kvm/kvm-ap.c
>>>> index bb29045..862e54b 100644
>>>> --- a/arch/s390/kvm/kvm-ap.c
>>>> +++ b/arch/s390/kvm/kvm-ap.c
>>>> @@ -307,3 +307,30 @@ void kvm_ap_deconfigure_matrix(struct kvm *kvm)
>>>> kvm_ap_clear_crycb_masks(kvm);
>>>> }
>>>> EXPORT_SYMBOL(kvm_ap_deconfigure_matrix);
>>>> +
>>>> +/**
>>>> + * kvm_ap_enable_ie_mode
>>>> + *
>>>> + * Enable interpretrive execution of AP instructions for the guest. When
>>>> + * enabled, AP instructions executed on the guest will be interpreted and
>>>> + * passed through to an AP installed on the host system.
>>>> + *
>>>> + * Returns 0 if interpretrive execution is enabled. Returns -EOPNOTSUPP
>>>> + * if AP facilities are not installed for the guest.
>>>> + *
>>>> + * @kvm: the guest's kvm structure
>>>> + */
>>>> +int kvm_ap_enable_ie_mode(struct kvm *kvm)
>>>> +{
>>>> + int i;
>>>> + struct kvm_vcpu *vcpu;
>>>> +
>>>> + if (!test_kvm_cpu_feat(kvm, KVM_S390_VM_CPU_FEAT_AP))
>>>> + return -EOPNOTSUPP;
>>>> +
>>>> + kvm_for_each_vcpu(i, vcpu, kvm)
>>>> + vcpu->arch.sie_block->eca |= ECA_APIE;
>>>> +
>>>> + return 0;
>>>> +}
>>>> +EXPORT_SYMBOL(kvm_ap_enable_ie_mode);
>>>> diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
>>>> index 1b5621f..3142541 100644
>>>> --- a/arch/s390/kvm/kvm-s390.h
>>>> +++ b/arch/s390/kvm/kvm-s390.h
>>>> @@ -18,6 +18,7 @@
>>>> #include <asm/facility.h>
>>>> #include <asm/processor.h>
>>>> #include <asm/sclp.h>
>>>> +#include <asm/ap.h>
>>>>
>>>> /* Transactional Memory Execution related macros */
>>>> #define IS_TE_ENABLED(vcpu) ((vcpu->arch.sie_block->ecb & ECB_TE))
>>>>
>>> And also, what about hot-plugged CPUs?
>> I haven't considered that, do you have any suggestions?
> You should handle the KVM_S390_VM_CPU_FEAT_AP feature instead during
> kvm_arch_vcpu_setup(), independent of any configured AP devices.
>
> This avoids the races I mentioned in regards to this series and also
> will handle hotplugged CPUs properly.
>
> If KVM_S390_VM_CPU_FEAT_AP is configured for a guest -> each CPU sets
> ECA_APIE during kvm_arch_vcpu_setup().
>
>
> (In the vSIE code, simply allow to set ECA_APIE in the shadow SCB in
> case KVM_S390_VM_CPU_FEAT_AP is enabled)
Patch series v3 will be posted very shortly, but I thought I'd give you a
heads up concerning what is forthcoming with regard to ECA_APIE. I'm
adding a device attribute to the KVM_S390_VM_CRYPTO group for setting a
flag via the KVM_SET_DEVICE_ATTR ioctl. The flag indicates whether
ECA_APIE should be set or not. The flag will be checked in the
test_kvm_cpu_feat() function and set or clear ECA_APIE accordingly.
You can comment on this in the v3 patch.
>
^ permalink raw reply [flat|nested] 56+ messages in thread
end of thread, other threads:[~2018-03-14 16:29 UTC | newest]
Thread overview: 56+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-02-27 14:27 [PATCH v2 00/15] s390: vfio-ap: guest dedicated crypto adapters Tony Krowiak
2018-02-27 14:27 ` [PATCH v2 01/15] KVM: s390: refactor crypto initialization Tony Krowiak
2018-02-28 17:37 ` Cornelia Huck
2018-02-28 21:23 ` Tony Krowiak
2018-03-01 9:59 ` Cornelia Huck
2018-03-14 16:02 ` Tony Krowiak
2018-02-27 14:28 ` [PATCH v2 02/15] s390: vsie: implement AP support for second level guest Tony Krowiak
2018-02-28 9:39 ` David Hildenbrand
2018-02-28 10:03 ` Pierre Morel
2018-02-27 14:28 ` [PATCH v2 03/15] s390: zcrypt: externalize AP instructions available function Tony Krowiak
2018-02-28 11:40 ` Harald Freudenberger
2018-02-28 17:41 ` Cornelia Huck
2018-03-01 8:38 ` Harald Freudenberger
2018-02-27 14:28 ` [PATCH v2 04/15] KVM: s390: CPU model support for AP virtualization Tony Krowiak
2018-02-28 9:48 ` David Hildenbrand
2018-02-28 11:40 ` Christian Borntraeger
2018-02-28 11:58 ` David Hildenbrand
2018-02-28 15:59 ` Tony Krowiak
2018-02-27 14:28 ` [PATCH v2 05/15] s390: vfio-ap: base implementation of VFIO AP device driver Tony Krowiak
2018-02-28 15:33 ` Pierre Morel
2018-02-28 16:43 ` Tony Krowiak
2018-02-28 18:10 ` Cornelia Huck
2018-02-28 20:34 ` Tony Krowiak
2018-02-27 14:28 ` [PATCH v2 06/15] s390: vfio-ap: register matrix device with VFIO mdev framework Tony Krowiak
2018-02-27 14:28 ` [PATCH v2 07/15] KVM: s390: Interfaces to configure/deconfigure guest's AP matrix Tony Krowiak
2018-02-28 16:15 ` Pierre Morel
2018-02-28 19:11 ` Tony Krowiak
2018-02-28 18:50 ` Tony Krowiak
2018-02-28 19:38 ` Tony Krowiak
2018-03-08 23:03 ` Tony Krowiak
2018-02-27 14:28 ` [PATCH v2 08/15] KVM: s390: interface to enable AP execution mode Tony Krowiak
2018-02-28 9:42 ` David Hildenbrand
2018-02-28 20:37 ` Tony Krowiak
2018-03-01 9:32 ` David Hildenbrand
2018-02-28 9:44 ` David Hildenbrand
2018-02-28 20:39 ` Tony Krowiak
2018-03-01 9:35 ` David Hildenbrand
2018-03-02 19:54 ` Tony Krowiak
2018-03-14 16:29 ` Tony Krowiak
2018-02-27 14:28 ` [PATCH v2 09/15] s390: vfio-ap: sysfs interfaces to configure adapters Tony Krowiak
2018-02-27 14:28 ` [PATCH v2 10/15] s390: vfio-ap: sysfs interfaces to configure domains Tony Krowiak
2018-02-27 14:28 ` [PATCH v2 11/15] s390: vfio-ap: sysfs interfaces to configure control domains Tony Krowiak
2018-02-27 14:28 ` [PATCH v2 12/15] s390: vfio-ap: sysfs interface to view matrix mdev matrix Tony Krowiak
2018-02-27 14:28 ` [PATCH v2 13/15] KVM: s390: Configure the guest's CRYCB Tony Krowiak
2018-02-28 9:49 ` David Hildenbrand
2018-02-28 20:45 ` Tony Krowiak
2018-03-01 9:37 ` David Hildenbrand
2018-03-01 20:42 ` Tony Krowiak
2018-03-02 10:08 ` David Hildenbrand
2018-03-02 19:48 ` Tony Krowiak
2018-02-27 14:28 ` [PATCH v2 14/15] s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl Tony Krowiak
2018-02-27 14:28 ` [PATCH v2 15/15] s390: doc: detailed specifications for AP virtualization Tony Krowiak
2018-02-27 15:57 ` Cornelia Huck
2018-02-27 18:12 ` Tony Krowiak
2018-02-27 14:58 ` [PATCH v2 00/15] s390: vfio-ap: guest dedicated crypto adapters Cornelia Huck
2018-02-27 15:57 ` Tony Krowiak
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).