[RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Tony Krowiak]

Overview:
--------
An adjunct processor (AP) facility is an IBM Z cryptographic facility. The 
AP facility is comprised of three AP instructions and from 1 to 256 AP 
adapter cards. The design takes advantage of the interpretive execution mode 
provided by the SIE architecture. With interpretive execution mode, the AP 
instructions executed on the guest are interpreted by the hardware. This 
allows guests direct access to AP adapter cards. The first goal of this 
patch series is to provide direct access by a KVM guest to an AP as a 
pass-through device. The second goal is to provide administrators with the
means to configure KVM guests to grant direct access to AP facilities 
assigned to the LPAR in which the host linux system is running. 

To facilitate the comprehension of the design, let's present an overview of
the AP architecture.

AP Architectural Overview
-------------------------
Let's start with some definitions:

* AP adapter

  An AP adapter is an IBM Z adapter card that can perform cryptographic 
  functionality. There can be from 0 to 256 adapters assigned to an LPAR.
  Each adapter is identified by a number from 0 to 255.   When 
  installed, an AP is accessed by AP instructions executed by any CPU. 

* AP domain

  An adapter can be partitioned into domains. An adapter can hold up to 256 
  domains. Each domain is identified by a number from 0 to 255. Domains can 
  be further classified into two types: 
  
    * Usage domains are domains that can be accessed directly to process AP 
      commands
  
    * Control domains are domains that are accessed indirectly by AP 
      commands sent to a usage domain to control or change the domain.

* AP Queue

  An AP queue is the means by which an AP command is sent to an 
  AP usage domain inside a specific AP. An AP queue is identified by a tuple 
  comprised of an AP adapter ID and a usage domain index corresponding
  to a given usage domain within the adapter. This tuple forms an AP Queue 
  Number (APQN) uniquely identifying an AP queue. AP instructions include 
  a field containing the APQN to identify the AP queue to which the AP 
  command is targetted.

* AP Instructions:

  There are three AP instructions:

  * NQAP: to enqueue an AP command-request message to a queue
  * DQAP: to dequeue an AP command-reply message from a queue
  * PQAP: to adminster the queues

Let's now see how AP instructions are interpreted by the hardware.

Start Interpretive Execution (SIE) Instruction
---------------------------------------------- 
A KVM guest is started by executing the Start Interpretive Execution (SIE)
instruction. The SIE state description is a control block that contains the
state information for a KVM guest and is supplied as input to the SIE 
instruction. The SIE state description contains a field that references 
a Crypto Control Block (CRYCB). The CRYCB contains three bitmask fields 
identifying the adapters, usage domains and control domains assigned to the 
KVM guest: 

* The AP Mask (APM) field specifies the AP adapters assigned to the 
  KVM guest. The APM controls which adapters are valid for the KVM guest. 
  The bits in the mask, from left to right, correspond to APIDs 
  0 up to the number of adapters that can be assigned to the LPAR. If a bit 
  is set, the corresponding adapter is valid for use by the KVM guest.

* The AP Queue Mask (AQM) field specifies the AP usage domains assigned 
  to the KVM guest. The bits in the mask, from left to right, correspond
  to the usage domains, from 0 up to the number of domains that can be 
  assigned to the LPAR. If a bit is set, the corresponding usage domain is 
  valid for use by the KVM guest. 

* The AP Domain Mask field specifies the AP control domains assigned to the 
  KVM guest. The ADM bitmask controls which domains can be changed by an AP 
  command-request message sent to a usage domain from the guest. The bits in 
  the mask, from left to right, correspond to domain 0 up to the number of 
  domains that can be assigned to the LPAR. If a bit is set, the 
  corresponding domain can be modified by an AP command-request message 
  sent to a usage domain configured for the KVM guest.

If you recall from the description of an AP Queue, AP instructions include
an APQN to identify the AP adapter and the specific usage domain within 
the adapter to which an AP command-request message is to be sent (NQAP 
and PQAP instructions), or from which a command-reply message is to be 
received (DQAP instruction). The validity of an APQN is defined by the 
matrix calculated from the APM and AQM; it is the intersection of all 
assigned adapter numbers (APM) with all assigned usage domain numbers (AQM).
For example, if adapters 1 and 2 and usage domains 5 and 6 are assigned to 
a guest, the APQNs (1,5), (1,6), (2,5) and (2,6) will be valid for the 
guest. 

The APQNs provide secure key functionality - i.e., the key is stored on the 
adapter card - so when the adapter card is not virtualized - i.e., the 
adapter is accessed directly by the guest - each APQN must be assigned to 
at most one guest.

   Example 1: Valid configuration:
   ------------------------------
   Guest1: adapters 1,2  domains 5,6
   Guest2: adapter  1,2  domain 7

   This is valid because both guests have a unique set of APQNs: Guest1 has
   APQNs (1,5), (1,6), (2,5) and (2,6); Guest2 has APQN (1,7) and (2,7).

   Example 2: Invalid configuration:
   --------------------------------
   Guest1: adapters 1,2  domains 5,6
   Guest2: adapter  1    domains 6,7

   This is an invalid configuration because both guests have access to 
   APQNs (1,6).  

Interruption architecture:

The AP interruption architecture may or may not generate interruptions to 
signal to the CPU the end of an AP transaction. The SIE interruption 
architecture, depending upon its configuration, may or may not redirect 
AP interrupts directly to a guest if the associated queue is valid for a 
guest, and may or may not report the interruption to the host.

Effective masking for guest level I and II:

A linux host running in the LPAR operates at guest-level 1 and has its own 
SIE state description. When operating at guest-level 1, the masks from the 
host's state description are used directly. A linux guest running in the 
host operates at guest-level 2. When operating at guest-level 2, the masks 
from the guest-level 1 (host) and guest-level 2 (guest) state descriptions 
are combined into a single description called an effective mask by 
performing a logical AND of the two state descriptions. 

The effective mask algorithm is used for the APM, AQM and ADM to create 
an EAPM, EAQM and EADM respectively. Use of the EAPM, EAQM and EADM 
precludes a guest-level 1 host program from passing to a guest-level 2
program APQNs to which it does not have access.

Linux cryptographic bus driver:

Linux already has a cryptographic bus driver that provides one AP device per
AP adapter and one device per AP queue. There is a device driver for each 
type of AP adapter device and each type of AP queue device. This design 
utilizes some of the interfaces and functionality provided by the AP bus 
driver.

Design Origin:
-------------

The original design was based on modelling AP Queue devices. The design
utilized the VFIO mediated device framework whereby a mediated AP queue
device would be created for each AP Queue bound to the VFIO AP Queue device
driver. This at first seemed like the most logical design choice for the 
following reasons:

* Securing access to an AP Queue device by unbinding it from its default 
  device driver and binding it to the VFIO device driver would not preclude 
  the host from having access to the other usage domains contained within 
  the same adapter card connected to the AP queue.

* An AP command is sent to a usage domain within a specific AP adapter via 
  an AP queue.

It became readily apparent that modelling the design on an AP queue was very 
convoluted for a number of reasons:

  * There is no convenient way to notify the VFIO device driver which guest 
    will have access to a given mediated AP queue device until the mediated 
    device's file descriptor is opened by the guest. Recall that the APQNs 
    configured for the guest are an intersection of all of the bits set in 
    both the APM and AQM, so the guest's APQNs can not be validated nor 
    its SIE state description configured until all of the guest's mediated 
    AP queue device file descriptors have been opened. 

    For example, suppose a guest opens file descriptors for mediated AP 
    queue devices representing APQNs 3,5 and 4,6. If bits 3 and 4 are set in 
    the guest's APM and bits 5 and 6 are set in the guest's AQM, then APQNs 
    (3,5), (3,6), (4,5) and (4,6) will be valid for the guest, but mediated 
    AP queue devices have been created only for APQNs (3,5) and (4,6). In 
    this case, APQNs still assigned to the host would also be available to 
    the guest which is a potential security breach.

  * Control domains are not devices and are not logically modelled as 
    mediated devices. In our original design, they were modelled as 
    attributes of a mediated AP queue device, but this was a clumsy use of
    the VFIO mediated device model.

  * The SIE state description models the assignment of AP resources as a
    matrix via the APM, AQM and ADM.   
     
The design we ultimately settled upon was modelled on the AP matrix as 
defined by the SIE state description. Supplying the complete AP matrix  
to SIE using bitmasks when starting a guest simplifies the code, is far 
easier to secure, and more closely matches the model employed by SIE. This 
is the design model implemented via this patch set. 

The Design
----------
This design introduces four new objects:

1. AP matrix bus

   The sysfs location of the AP matrix bus is /sys/bus/ap_matrix. This 
   bus will create a single AP matrix device (see below).

2. AP matrix device

   The AP matrix device is a singleton that hangs off of the AP matrix bus.
   This device holds the AP Queues that have been reserved for use by 
   KVM guests. The sysfs location of the AP matrix device is 
   /sys/devices/ap_matrix/matrix. It is also linked from the AP matrix
   bus at /sys/bus/ap_matrix/devices/matrix.  

3. VFIO AP matrix driver

   This driver is based on the VFIO mediated device framework. When the 
   driver is initialized, it will:

   * Get the AP matrix device created by AP matrix bus from the bus

   * Register with the AP bus to indicate that it can control AP Queue 
     devices. This allows AP Queue devices unbound from AP device drivers
     to be bound to the VFIO AP matrix driver. The AP Queues bound to the 
     VFIO AP matrix driver will be stored by the driver in the AP matrix 
     device. 

   * Register the AP matrix device with the VFIO mediated device 
     framework (MDEV). Registration with MDEV will create the sysfs 
     structures needed to create mediated matrix devices. Each MDEV matrix
     device is used to configure the AP matrix for a KVM guest. The MDEV
     matrix device's file descriptor can be used by QEMU to communicate
     with the VFIO AP matrix device driver.

   The VFIO AP matrix driver:

   * Provides the interfaces the administrator can use to secure AP Queues 
     for use by KVM guests. This is accomplished by unbinding the AP Queues 
     needed by each KVM guest from its AP device driver and binding it to 
     the VFIO AP queue driver. This prevents the host linux system from 
     using these Queues.

   * Provides an ioctl that can be used by QEMU to configure the 
     CRYCB referenced by the KVM guest's SIE state description. The ioctl 
     will 

     * Create an EAPM, EAQM and EADM by performing a logical AND of the 
       APM, AQM and ADM configured via the MDEV matrix device's sysfs 
       attributes files (see below) with the APM, AQM and ADM of the host's 
       SIE state description respectively. 

     * Configure the SIE state description for the KVM guest using the 
       effective masks created in the previous step.

4. VFIO MDEV matrix passthrough device

   An MDEV matrix passthrough device must be created for each KVM guest that 
   will need access to AP facilities. An MDEV matrix passthrough device is
   used by QEMU to configure the APM, AQM and ADM fields of the CRYCB 
   referenced by the KVM guest's SIE state description. The file descriptor
   for the MDEV matrix passthrough device provides the communication pathway
   between QEMU and the VFIO AP matrix device driver. 

   The MDEV matrix passthrough device, like the CRYCB, contains three 
   bitmasks - an APM, AQM and ADM - for specifying the AP matrix for the 
   KVM guest. Three sets of attributes files will be provided to allow an 
   administrator to set the bits in the MDEV matrix device's APM, AQM and
   ADM: 
 
   * A file to assign an AP adapter 
   * A file to unassign an AP adapter
   * A file to display the adapters assigned

   * A file to assign an AP domain 
   * A file to unassign an AP domain
   * A file to display the domains assigned

   * A file to assign an AP control domain 
   * A file to unassign an AP control domain
   * A file to display the control domains assigned

Example:
-------
Let's now provide an example to illustrate how KVM guests may be given
access to AP facilities. For this example, we will show how to configure 
two guests such that executing the lszcrypt command on the guests would 
look like this:

Guest1
------
CARD.DOMAIN TYPE  MODE        
------------------------------
05          CEX5C CCA-Coproc  
05.0004     CEX5C CCA-Coproc
05.00ab     CEX5C CCA-Coproc  
06          CEX5A Accelerator 
06.0004     CEX5A Accelerator 
06.00ab     CEX5C CCA-Coproc  

Guest2
------
CARD.DOMAIN TYPE  MODE        
------------------------------
05          CEX5A Accelerator 
05.0047     CEX5A Accelerator 
05.00ff     CEX5A Accelerator 

One thing to notice in this example is that each AP Queue set is identical.
For example, the two AP Queue sets for Guest1 both contain APQI 0004 and 
00ab. It would be an invalid condition if both queue sets did not contain
the same set of queues. We could not, for example, configure Guest1 with 
access to AP queue 05.00ff because the AP queue set for adapter 06 does not
contain AP queue 06.00ff. The point is, one must be careful to reserve 
a valid set of AP queues for a given guest.
a valid configuration. 

These are the steps for configuring the Guest1 and Guest2:
   
1. The first thing that needs to be done is to secure the AP queues to be 
   used by the two guests so that the host can not access them. This is done 
   by unbinding each AP Queue device from its respective AP driver. In our 
   example, these queues are bound to the cex4queue driver. This would be 
   the sysfs location of these devices: 

   /sys/bus/ap
   --- [drivers]
   ------ [cex4queue]
   --------- [05.0004]
   --------- [05.0047]
   --------- [05.00ab]
   --------- [05.00ff]
   --------- [06.0004]
   --------- [06.00ab]
   --------- unbind

   To unbind AP queue 05.0004 from the cex4queue device driver:

	echo 05.0004 > unbind

   This must also be done for AP queues 05.00ab, 05.0047, 05.00ff, 06.0004,
   and 06.00ab.

2. The next step is to reserve the queues for use by the two KVM guests. 
   This is accomplished by binding them to the VFIO AP matrix device driver. 
   This is the sysfs location of the VFIO AP matrix device driver:

   /sys/bus/ap
   ---[drivers]
   ------ [vfio_ap_matrix]
   ---------- bind

   To bind queue 05.0004 to the vfio_ap_matrix driver:

	echo 05.0004 > bind

   This must also be done for AP queues 05.00ab, 05.0047, 05.00ff, 06.0004,
   and 06.00ab.

3. Create the mediated devices needed to configure the AP matrices for the 
   two guests and to provide an interface to the vfio_ap_matrix driver for 
   use by the guests:

   /sys/devices/
   --- [ap_matrix]
   ------ [matrix] (this is the matrix device)
   --------- [mdev_supported_types]
   ------------ [ap_matrix-passthrough] (passthrough mediated device type)
   --------------- create
   --------------- [devices]

   To create the mediated devices for the two guests:

	uuidgen > create
	uuidgen > create

   This will create two mediated devices in the [devices] subdirectory named 
   with the UUID written to the create attribute file. We call them $uuid1
   and $uuid2:

   /sys/devices/
   --- [ap_matrix]
   ------ [matrix]
   --------- [mdev_supported_types]
   ------------ [ap_matrix-passthrough]
   --------------- [devices]
   ------------------ [$uuid1]
   --------------------- adapters
   --------------------- assign_adapter
   --------------------- assign_control_domain
   --------------------- assign_domain
   --------------------- control_domains
   --------------------- domains
   --------------------- unassign_adapter
   --------------------- unassign_control_domain
   --------------------- unassign_domain
   ------------------ [$uuid2]
   --------------------- adapters
   --------------------- assign_adapter
   --------------------- assign_control_domain
   --------------------- assign_domain
   --------------------- control_domains
   --------------------- domains
   --------------------- unassign_adapter
   --------------------- unassign_control_domain
   --------------------- unassign_domain

4. The administrator now needs to configure the matrices for mediated 
   devices $uuid1 (for Guest1) and $uuid2 (for Guest2). 

   This is how the matrix is configured for Guest1:

   echo 5 > assign_adapter
   echo 6 > assign_adapter 
   echo 4 > assign_domain
   echo ab > assign_domain

   When the assign.xxx file is written, the corresponding bit in the 
   respective MDEV matrix device's bitmask will be set. For example, when 
   adapter 5 is assigned, bit 5 - numbered from left to right starting with 
   bit 0 - will be set in the MDEV matrix device's APM. 

   By architectural convention, all usage domains - i.e., domains assigned 
   via the assign_domain attribute file - will also be configured in the ADM 
   field of the KVM guest's CRYCB, so there is no need to assign control 
   domains here unless you want to assign control domains that are not 
   assigned as usage domains. 

   If a mistake is made configuring an adapter, domain or control domain, 
   you can use the unassign_xxx files to unassign the adapter, domain or 
   control domain.

   To display the matrix configuration for Guest1:

   cat adapters
   cat domains
   cat control_domains

   This is how the matrix is configured for Guest2:

   echo 5 > assign_adapter 
   echo 47 > assign_domain
   echo ff > assign_domain

When a KVM guest is started, QEMU will open the file descriptor for its 
MDEV matrix device. The VFIO AP matrix device driver will be notified 
and will store the reference to the KVM guest's SIE state description. 
QEMU will then call the VFIO AP matrix ioctl requesting that the 
KVM guest's matrix be configured. The matrix driver will set the bits in the 
APM, AQM and ADM fields of the CRYCB referenced by the guest's SIE state 
description from the EAPM, EAQM and EADM created by performing a logical AND
of the AP masks configured in the MDEV matrix device and the masks 
configured in the host's SIE state description. When the guest comes up, it 
will have access to the APQNs identified in the AP matrix specified in the 
KVM guest's SIE state description. Programs running on the guest will then 
be able to use the cryptographic functions provided by the AP facilities 
configured for the guest.

Tony Krowiak (19):
  KVM: s390: SIE considerations for AP Queue virtualization
  KVM: s390: refactor crypto initialization
  s390/zcrypt: new AP matrix bus
  s390/zcrypt: create an AP matrix device on the AP matrix bus
  s390/zcrypt: base implementation of AP matrix device driver
  s390/zcrypt: register matrix device with VFIO mediated device
    framework
  KVM: s390: introduce AP matrix configuration interface
  s390/zcrypt: support for assigning adapters to matrix mdev
  s390/zcrypt: validate adapter assignment
  s390/zcrypt: sysfs interfaces supporting AP domain assignment
  s390/zcrypt: validate domain assignment
  s390/zcrypt: sysfs support for control domain assignment
  s390/zcrypt: validate control domain assignment
  KVM: s390: Connect the AP mediated matrix device to KVM
  s390/zcrypt: introduce ioctl access to VFIO AP Matrix driver
  KVM: s390: interface to configure KVM guest's AP matrix
  KVM: s390: validate input to AP matrix config interface
  KVM: s390: New ioctl to configure KVM guest's AP matrix
  s390/facilities: enable AP facilities needed by guest

 MAINTAINERS                                  |   13 +
 arch/s390/Kconfig                            |   13 +
 arch/s390/configs/default_defconfig          |    1 +
 arch/s390/configs/gcov_defconfig             |    1 +
 arch/s390/configs/performance_defconfig      |    1 +
 arch/s390/defconfig                          |    1 +
 arch/s390/include/asm/ap-config.h            |   32 +
 arch/s390/include/asm/kvm_host.h             |   26 +-
 arch/s390/kvm/Makefile                       |    2 +-
 arch/s390/kvm/ap-config.c                    |  224 ++++++++
 arch/s390/kvm/kvm-s390.c                     |   17 +-
 arch/s390/tools/gen_facilities.c             |    2 +
 drivers/s390/crypto/Makefile                 |    6 +-
 drivers/s390/crypto/ap_matrix_bus.c          |  115 ++++
 drivers/s390/crypto/ap_matrix_bus.h          |   25 +
 drivers/s390/crypto/vfio_ap_matrix_drv.c     |  107 ++++
 drivers/s390/crypto/vfio_ap_matrix_ops.c     |  790 ++++++++++++++++++++++++++
 drivers/s390/crypto/vfio_ap_matrix_private.h |   50 ++
 include/uapi/linux/vfio.h                    |   22 +
 19 files changed, 1438 insertions(+), 10 deletions(-)
 create mode 100644 arch/s390/include/asm/ap-config.h
 create mode 100644 arch/s390/kvm/ap-config.c
 create mode 100644 drivers/s390/crypto/ap_matrix_bus.c
 create mode 100644 drivers/s390/crypto/ap_matrix_bus.h
 create mode 100644 drivers/s390/crypto/vfio_ap_matrix_drv.c
 create mode 100644 drivers/s390/crypto/vfio_ap_matrix_ops.c
 create mode 100644 drivers/s390/crypto/vfio_ap_matrix_private.h

[RFC 01/19] KVM: s390: SIE considerations for AP Queue virtualization

[Tony Krowiak]

The Crypto Control Block (CRYCB) is referenced by the SIE state
description and controls KVM guest access to the Adjunct
Processor (AP) adapters, usage domains and control domains.
This patch defines the AP control blocks to be used for
controlling guest access to the AP adapters and domains.

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 arch/s390/include/asm/kvm_host.h |   25 +++++++++++++++++++++----
 1 files changed, 21 insertions(+), 4 deletions(-)

diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index 51375e7..50a6b25 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -685,11 +685,28 @@ struct kvm_s390_crypto {
 	__u8 dea_kw;
 };
 
+#define APCB0_MASK_SIZE 1
+struct kvm_s390_apcb0 {
+	__u64 apm[APCB0_MASK_SIZE];		/* 0x0000 */
+	__u64 aqm[APCB0_MASK_SIZE];		/* 0x0008 */
+	__u64 adm[APCB0_MASK_SIZE];		/* 0x0010 */
+	__u64 reserved18;			/* 0x0018 */
+};
+
+#define APCB1_MASK_SIZE 4
+struct kvm_s390_apcb1 {
+	__u64 apm[APCB1_MASK_SIZE];		/* 0x0000 */
+	__u64 aqm[APCB1_MASK_SIZE];		/* 0x0020 */
+	__u64 adm[APCB1_MASK_SIZE];		/* 0x0040 */
+	__u64 reserved60[4];			/* 0x0060 */
+};
+
 struct kvm_s390_crypto_cb {
-	__u8    reserved00[72];                 /* 0x0000 */
-	__u8    dea_wrapping_key_mask[24];      /* 0x0048 */
-	__u8    aes_wrapping_key_mask[32];      /* 0x0060 */
-	__u8    reserved80[128];                /* 0x0080 */
+	struct kvm_s390_apcb0 apcb0;		/* 0x0000 */
+	__u8    reserved20[40];			/* 0x0020 */
+	__u8    dea_wrapping_key_mask[24];	/* 0x0048 */
+	__u8    aes_wrapping_key_mask[32];	/* 0x0060 */
+	struct kvm_s390_apcb1 apcb1;		/* 0x0080 */
 };
 
 /*
-- 
1.7.1

[RFC 02/19] KVM: s390: refactor crypto initialization

[Tony Krowiak]

This patch introduces the following changes to crypto initialization.

* For key management operations support, the crypto control block
  (CRYCB) referenced by the KVM guest's SIE state description is
  formatted only if the Message-Security-Assist (MSA) extension 3
  facility is installed (STFLE.76 is set). Virtualization of AP
  facilities, however; requires that a CRYCB of the appropriate
  format be made available to SIE regardless of the value of STFLE.76.

* The Execution Controls A (ECA) field bit 28 in the SIE block needs
  to be set to enable interpretive execution mode of adjunct processor (AP)
  instructions.

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 arch/s390/include/asm/kvm_host.h |    1 +
 arch/s390/kvm/kvm-s390.c         |   17 +++++++++++++----
 2 files changed, 14 insertions(+), 4 deletions(-)

diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index 50a6b25..5683f18 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -188,6 +188,7 @@ struct kvm_s390_sie_block {
 #define ECA_MVPGI	0x01000000
 #define ECA_VX		0x00020000
 #define ECA_PROTEXCI	0x00002000
+#define ECA_AP		0x00000008
 #define ECA_SII		0x00000001
 	__u32	eca;			/* 0x004c */
 #define ICPT_INST	0x04
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 40d0a1a..e57fc9b 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -1819,7 +1819,9 @@ static void kvm_s390_set_crycb_format(struct kvm *kvm)
 {
 	kvm->arch.crypto.crycbd = (__u32)(unsigned long) kvm->arch.crypto.crycb;
 
-	if (kvm_s390_apxa_installed())
+	if (!test_kvm_facility(kvm, 76))
+		kvm->arch.crypto.crycbd &= ~(CRYCB_FORMAT2); /* format 0 */
+	else if (kvm_s390_apxa_installed())
 		kvm->arch.crypto.crycbd |= CRYCB_FORMAT2;
 	else
 		kvm->arch.crypto.crycbd |= CRYCB_FORMAT1;
@@ -1836,12 +1838,12 @@ static u64 kvm_s390_get_initial_cpuid(void)
 
 static void kvm_s390_crypto_init(struct kvm *kvm)
 {
-	if (!test_kvm_facility(kvm, 76))
-		return;
-
 	kvm->arch.crypto.crycb = &kvm->arch.sie_page2->crycb;
 	kvm_s390_set_crycb_format(kvm);
 
+	if (!test_kvm_facility(kvm, 76))
+		return;
+
 	/* Enable AES/DEA protected key functions by default */
 	kvm->arch.crypto.aes_kw = 1;
 	kvm->arch.crypto.dea_kw = 1;
@@ -2366,8 +2368,15 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)
 	vcpu->arch.enabled_gmap = vcpu->arch.gmap;
 }
 
+static void kvm_s390_vcpu_set_crypto_exec_mode(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.sie_block->eca |= ECA_AP;
+}
+
 static void kvm_s390_vcpu_crypto_setup(struct kvm_vcpu *vcpu)
 {
+	kvm_s390_vcpu_set_crypto_exec_mode(vcpu);
+
 	if (!test_kvm_facility(vcpu->kvm, 76))
 		return;
 
-- 
1.7.1

[RFC 03/19] s390/zcrypt: new AP matrix bus

[Tony Krowiak]

Introduces an AP matrix bus. The sysfs location of the
AP matrix bus is:

/sys/bus/ap_matrix

The AP matrix bus will create an AP matrix device that will
hold the AP queues reserved for use by KVM guests.

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 MAINTAINERS                         |    8 +++++
 drivers/s390/crypto/Makefile        |    4 +-
 drivers/s390/crypto/ap_matrix_bus.c |   52 +++++++++++++++++++++++++++++++++++
 drivers/s390/crypto/ap_matrix_bus.h |   15 ++++++++++
 4 files changed, 77 insertions(+), 2 deletions(-)
 create mode 100644 drivers/s390/crypto/ap_matrix_bus.c
 create mode 100644 drivers/s390/crypto/ap_matrix_bus.h

diff --git a/MAINTAINERS b/MAINTAINERS
index ef23cf5..cbd6f7c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -11711,6 +11711,14 @@ W:	http://www.ibm.com/developerworks/linux/linux390/
 S:	Supported
 F:	drivers/s390/crypto/
 
+S390 VFIO AP MATRIX DRIVER
+M:	Tony Krowiak <akrowiak@linux.vnet.ibm.com>
+L:	linux-s390@vger.kernel.org
+W:	http://www.ibm.com/developerworks/linux/linux390/
+S:	Supported
+F:	drivers/s390/crypto/ap_matrix_bus.h
+F:	drivers/s390/crypto/ap_matrix_bus.c
+
 S390 ZFCP DRIVER
 M:	Steffen Maier <maier@linux.vnet.ibm.com>
 M:	Benjamin Block <bblock@linux.vnet.ibm.com>
diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile
index be36f10..87646ca 100644
--- a/drivers/s390/crypto/Makefile
+++ b/drivers/s390/crypto/Makefile
@@ -2,7 +2,7 @@
 # S/390 crypto devices
 #
 
-ap-objs := ap_bus.o ap_card.o ap_queue.o
+ap-objs := ap_bus.o ap_card.o ap_queue.o ap_matrix_bus.o
 obj-$(subst m,y,$(CONFIG_ZCRYPT)) += ap.o
 # zcrypt_api.o and zcrypt_msgtype*.o depend on ap.o
 zcrypt-objs := zcrypt_api.o zcrypt_card.o zcrypt_queue.o
@@ -13,4 +13,4 @@ obj-$(CONFIG_ZCRYPT) += zcrypt_pcixcc.o zcrypt_cex2a.o zcrypt_cex4.o
 
 # pkey kernel module
 pkey-objs := pkey_api.o
-obj-$(CONFIG_PKEY) += pkey.o
+obj-$(CONFIG_PKEY) += pkey.o
\ No newline at end of file
diff --git a/drivers/s390/crypto/ap_matrix_bus.c b/drivers/s390/crypto/ap_matrix_bus.c
new file mode 100644
index 0000000..fbae175
--- /dev/null
+++ b/drivers/s390/crypto/ap_matrix_bus.c
@@ -0,0 +1,52 @@
+/*
+ * Copyright IBM Corp. 2017
+ * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
+ *
+ * Adjunct processor matrix bus.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/module.h>
+#include <linux/device.h>
+#include <asm/ap.h>
+
+#include "ap_matrix_bus.h"
+
+MODULE_AUTHOR("Tony Krowiak <akrowiak@linux.vnet.ibm.com");
+MODULE_DESCRIPTION("AP Matrix, Copyright IBM Corp. 2017");
+MODULE_LICENSE("GPL v2");
+
+#define AP_MATRIX_BUS_NAME "ap_matrix"
+
+static struct device *ap_matrix_root_device;
+
+static struct bus_type ap_matrix_bus_type = {
+	.name = AP_MATRIX_BUS_NAME,
+};
+
+int __init ap_matrix_init(void)
+{
+	int ret;
+
+	ap_matrix_root_device = root_device_register(AP_MATRIX_BUS_NAME);
+	ret = PTR_RET(ap_matrix_root_device);
+	if (ret)
+		goto done;
+
+	ret = bus_register(&ap_matrix_bus_type);
+	if (ret)
+		goto bus_reg_err;
+
+	return 0;
+
+bus_reg_err:
+	root_device_unregister(ap_matrix_root_device);
+
+done:
+	return ret;
+}
+device_initcall(ap_matrix_init);
diff --git a/drivers/s390/crypto/ap_matrix_bus.h b/drivers/s390/crypto/ap_matrix_bus.h
new file mode 100644
index 0000000..58e116f
--- /dev/null
+++ b/drivers/s390/crypto/ap_matrix_bus.h
@@ -0,0 +1,15 @@
+/*
+ * Adjunct processor matrix bus header file
+ *
+ * Copyright IBM Corp. 2017
+ * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
+ *
+ * Adjunct processor bus header file.
+ */
+
+#ifndef _AP_MATRIX_BUS_H_
+#define _AP_MATRIX_BUS_H_
+
+int ap_matrix_init(void);
+
+#endif /* _AP_MATRIX_BUS_H_ */
-- 
1.7.1

[RFC 04/19] s390/zcrypt: create an AP matrix device on the AP matrix bus

[Tony Krowiak]

Creates a single AP matrix device on the AP matrix bus.
The matrix device will be created as part of the AP matrix bus
initialization. The matrix device will hold the AP queues that
have been reserved for use by KVM guests. Mediated matrix devices
can then be created for each guest. The mediated matrix devices can
then be configured with a matrix of AP adapters, usage and
control domains that will be made accessible to the guest.

The sysfs location of the matrix device is:

/sys/bus/ap_matrix
... [devices]
...... [matrix]

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 drivers/s390/crypto/ap_matrix_bus.c |   54 +++++++++++++++++++++++++++++++++++
 drivers/s390/crypto/ap_matrix_bus.h |    6 ++++
 2 files changed, 60 insertions(+), 0 deletions(-)

diff --git a/drivers/s390/crypto/ap_matrix_bus.c b/drivers/s390/crypto/ap_matrix_bus.c
index fbae175..4eb1e3c 100644
--- a/drivers/s390/crypto/ap_matrix_bus.c
+++ b/drivers/s390/crypto/ap_matrix_bus.c
@@ -12,6 +12,7 @@
 
 #include <linux/module.h>
 #include <linux/device.h>
+#include <linux/slab.h>
 #include <asm/ap.h>
 
 #include "ap_matrix_bus.h"
@@ -21,13 +22,59 @@
 MODULE_LICENSE("GPL v2");
 
 #define AP_MATRIX_BUS_NAME "ap_matrix"
+#define AP_MATRIX_DEV_TYPE_NAME "ap_matrix"
+#define AP_MATRIX_DEV_NAME "matrix"
 
 static struct device *ap_matrix_root_device;
+static struct ap_matrix *matrix;
 
 static struct bus_type ap_matrix_bus_type = {
 	.name = AP_MATRIX_BUS_NAME,
 };
 
+static struct device_type ap_matrix_type = {
+	.name = AP_MATRIX_DEV_TYPE_NAME,
+};
+
+static void ap_matrix_dev_release(struct device *dev)
+{
+	struct ap_matrix *ap_matrix;
+
+	ap_matrix = container_of(dev, struct ap_matrix, device);
+
+	if (matrix == ap_matrix)
+		kfree(matrix);
+
+	matrix = NULL;
+}
+
+static int ap_matrix_dev_create(void)
+{
+	int ret;
+
+	matrix = kzalloc(sizeof(*matrix), GFP_KERNEL);
+	if (!matrix)
+		return -ENOMEM;
+
+	matrix->device.type = &ap_matrix_type;
+	dev_set_name(&matrix->device, "%s", AP_MATRIX_DEV_NAME);
+	matrix->device.bus = &ap_matrix_bus_type;
+	matrix->device.parent = ap_matrix_root_device;
+	matrix->device.release = ap_matrix_dev_release;
+
+	ret = device_register(&matrix->device);
+	if (ret) {
+		put_device(&matrix->device);
+		kfree(matrix);
+		matrix = NULL;
+		return ret;
+	}
+
+	get_device(&matrix->device);
+
+	return 0;
+}
+
 int __init ap_matrix_init(void)
 {
 	int ret;
@@ -41,8 +88,15 @@ int __init ap_matrix_init(void)
 	if (ret)
 		goto bus_reg_err;
 
+	ret = ap_matrix_dev_create();
+	if (ret)
+		goto matrix_create_err;
+
 	return 0;
 
+matrix_create_err:
+	bus_unregister(&ap_matrix_bus_type);
+
 bus_reg_err:
 	root_device_unregister(ap_matrix_root_device);
 
diff --git a/drivers/s390/crypto/ap_matrix_bus.h b/drivers/s390/crypto/ap_matrix_bus.h
index 58e116f..225db4f 100644
--- a/drivers/s390/crypto/ap_matrix_bus.h
+++ b/drivers/s390/crypto/ap_matrix_bus.h
@@ -10,6 +10,12 @@
 #ifndef _AP_MATRIX_BUS_H_
 #define _AP_MATRIX_BUS_H_
 
+#include <linux/device.h>
+
+struct ap_matrix {
+	struct device device;
+};
+
 int ap_matrix_init(void);
 
 #endif /* _AP_MATRIX_BUS_H_ */
-- 
1.7.1

[RFC 05/19] s390/zcrypt: base implementation of AP matrix device driver

[Tony Krowiak]

Introduces a new AP matrix device driver. This device driver
will ultimately perform the following functions:

* Register with the AP bus to let it know that the matrix
  driver can control AP queue devices. This will allow
  an administrator to unbind an AP queue device from its
  device driver and bind it to the matrix device driver.
  This is how AP queue devices will be reserved for use
  by guest machines.

* Register the matrix device created by the AP matrix bus
  with the VFIO mediated device framework. This will create
  the sysfs entries needed to create mediated matrix devices.
  Each mediated matrix device can be configured with a matrix
  of adapters, usage domains and control domains that can be
  accessed by a guest machine.

* Process requests via ioctl calls defined for the mediated
  matrix device. The guest can access the ioctl calls via
  the mediated device's file descriptor to:

    * Grant access to the adapters, usage domains and
      control domains configured for the mediated matrix
      device.

This device driver
is built on the VFIO mediated device framework. The VFIO mediated
device framework allows a mediated device to be dedicated exclusively
to a single guest VM.

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 MAINTAINERS                                  |    2 +
 arch/s390/Kconfig                            |   13 +++
 arch/s390/configs/default_defconfig          |    1 +
 arch/s390/configs/gcov_defconfig             |    1 +
 arch/s390/configs/performance_defconfig      |    1 +
 arch/s390/defconfig                          |    1 +
 drivers/s390/crypto/Makefile                 |    6 +-
 drivers/s390/crypto/ap_matrix_bus.c          |    8 ++
 drivers/s390/crypto/ap_matrix_bus.h          |    2 +-
 drivers/s390/crypto/vfio_ap_matrix_drv.c     |  102 ++++++++++++++++++++++++++
 drivers/s390/crypto/vfio_ap_matrix_private.h |   47 ++++++++++++
 11 files changed, 182 insertions(+), 2 deletions(-)
 create mode 100644 drivers/s390/crypto/vfio_ap_matrix_drv.c
 create mode 100644 drivers/s390/crypto/vfio_ap_matrix_private.h

diff --git a/MAINTAINERS b/MAINTAINERS
index cbd6f7c..115965e 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -11718,6 +11718,8 @@ W:	http://www.ibm.com/developerworks/linux/linux390/
 S:	Supported
 F:	drivers/s390/crypto/ap_matrix_bus.h
 F:	drivers/s390/crypto/ap_matrix_bus.c
+F:	drivers/s390/crypto/vfio_ap_matrix_drv.c
+F:	drivers/s390/crypto/vfio_ap_matrix_private.h
 
 S390 ZFCP DRIVER
 M:	Steffen Maier <maier@linux.vnet.ibm.com>
diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
index 48af970..411c19a 100644
--- a/arch/s390/Kconfig
+++ b/arch/s390/Kconfig
@@ -722,6 +722,19 @@ config VFIO_CCW
 	  To compile this driver as a module, choose M here: the
 	  module will be called vfio_ccw.
 
+config VFIO_AP_MATRIX
+	def_tristate m
+	prompt "Support for Adjunct Processor Matrix device interface"
+	depends on ZCRYPT
+	select VFIO
+	select MDEV
+	select VFIO_MDEV
+	select VFIO_MDEV_DEVICE
+	select IOMMU_API
+	help
+		driver grants access to Adjunct Processor (AP) devices
+		via the VFIO mediated device interface.
+
 endmenu
 
 menu "Dump support"
diff --git a/arch/s390/configs/default_defconfig b/arch/s390/configs/default_defconfig
index 2820722..d913042 100644
--- a/arch/s390/configs/default_defconfig
+++ b/arch/s390/configs/default_defconfig
@@ -725,3 +725,4 @@ CONFIG_APPLDATA_BASE=y
 CONFIG_KVM=m
 CONFIG_KVM_S390_UCONTROL=y
 CONFIG_VHOST_NET=m
+CONFIG_VFIO_AP_MATRIX=m
diff --git a/arch/s390/configs/gcov_defconfig b/arch/s390/configs/gcov_defconfig
index 3c6b781..5f08278 100644
--- a/arch/s390/configs/gcov_defconfig
+++ b/arch/s390/configs/gcov_defconfig
@@ -666,3 +666,4 @@ CONFIG_APPLDATA_BASE=y
 CONFIG_KVM=m
 CONFIG_KVM_S390_UCONTROL=y
 CONFIG_VHOST_NET=m
+CONFIG_VFIO_AP_MATRIX=m
diff --git a/arch/s390/configs/performance_defconfig b/arch/s390/configs/performance_defconfig
index 653d72b..4b77519 100644
--- a/arch/s390/configs/performance_defconfig
+++ b/arch/s390/configs/performance_defconfig
@@ -664,3 +664,4 @@ CONFIG_APPLDATA_BASE=y
 CONFIG_KVM=m
 CONFIG_KVM_S390_UCONTROL=y
 CONFIG_VHOST_NET=m
+CONFIG_VFIO_AP_MATRIX=m
diff --git a/arch/s390/defconfig b/arch/s390/defconfig
index 20244a3..aa46cb5 100644
--- a/arch/s390/defconfig
+++ b/arch/s390/defconfig
@@ -242,3 +242,4 @@ CONFIG_CRC7=m
 # CONFIG_XZ_DEC_ARMTHUMB is not set
 # CONFIG_XZ_DEC_SPARC is not set
 CONFIG_CMM=m
+CONFIG_VFIO_AP_MATRIX=m
diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile
index 87646ca..1983afa 100644
--- a/drivers/s390/crypto/Makefile
+++ b/drivers/s390/crypto/Makefile
@@ -13,4 +13,8 @@ obj-$(CONFIG_ZCRYPT) += zcrypt_pcixcc.o zcrypt_cex2a.o zcrypt_cex4.o
 
 # pkey kernel module
 pkey-objs := pkey_api.o
-obj-$(CONFIG_PKEY) += pkey.o
\ No newline at end of file
+obj-$(CONFIG_PKEY) += pkey.o
+
+# adjunct processor matrix
+vfio_ap_matrix-objs := vfio_ap_matrix_drv.o
+obj-$(CONFIG_VFIO_AP_MATRIX) += vfio_ap_matrix.o
diff --git a/drivers/s390/crypto/ap_matrix_bus.c b/drivers/s390/crypto/ap_matrix_bus.c
index 4eb1e3c..66bfa54 100644
--- a/drivers/s390/crypto/ap_matrix_bus.c
+++ b/drivers/s390/crypto/ap_matrix_bus.c
@@ -75,10 +75,18 @@ static int ap_matrix_dev_create(void)
 	return 0;
 }
 
+struct ap_matrix *ap_matrix_get_device(void)
+{
+	return matrix;
+}
+EXPORT_SYMBOL(ap_matrix_get_device);
+
 int __init ap_matrix_init(void)
 {
 	int ret;
 
+	matrix = NULL;
+
 	ap_matrix_root_device = root_device_register(AP_MATRIX_BUS_NAME);
 	ret = PTR_RET(ap_matrix_root_device);
 	if (ret)
diff --git a/drivers/s390/crypto/ap_matrix_bus.h b/drivers/s390/crypto/ap_matrix_bus.h
index 225db4f..c2aff23 100644
--- a/drivers/s390/crypto/ap_matrix_bus.h
+++ b/drivers/s390/crypto/ap_matrix_bus.h
@@ -16,6 +16,6 @@ struct ap_matrix {
 	struct device device;
 };
 
-int ap_matrix_init(void);
+struct ap_matrix *ap_matrix_get_device(void);
 
 #endif /* _AP_MATRIX_BUS_H_ */
diff --git a/drivers/s390/crypto/vfio_ap_matrix_drv.c b/drivers/s390/crypto/vfio_ap_matrix_drv.c
new file mode 100644
index 0000000..760ed56
--- /dev/null
+++ b/drivers/s390/crypto/vfio_ap_matrix_drv.c
@@ -0,0 +1,102 @@
+/*
+ * VFIO based AP Matrix device driver
+ *
+ * Copyright IBM Corp. 2017
+ *
+ * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
+ */
+
+#include <linux/module.h>
+#include <linux/mod_devicetable.h>
+#include <linux/slab.h>
+
+#include "ap_bus.h"
+#include "ap_matrix_bus.h"
+#include "vfio_ap_matrix_private.h"
+
+#define VFIO_AP_MATRIX_DRV_NAME "vfio_ap_queue"
+
+MODULE_AUTHOR("IBM Corporation");
+MODULE_DESCRIPTION("AP Matrix device driver, Copyright IBM Corp. 2017");
+MODULE_LICENSE("GPL v2");
+
+static struct ap_device_id ap_queue_ids[] = {
+	{ .dev_type = AP_DEVICE_TYPE_CEX4,
+	  .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
+	{ .dev_type = AP_DEVICE_TYPE_CEX5,
+	  .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
+	{ .dev_type = AP_DEVICE_TYPE_CEX6,
+	  .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
+	{ /* end of list */ },
+};
+
+MODULE_DEVICE_TABLE(ap_matrix, ap_queue_ids);
+
+static struct ap_matrix_driver {
+	struct ap_driver ap_drv;
+	struct ap_matrix *ap_matrix;
+} vfio_ap_matrix_drv;
+
+static int ap_matrix_queue_dev_probe(struct ap_device *apdev)
+{
+	struct vfio_ap_queue *vapq;
+	struct ap_queue *apq = to_ap_queue(&apdev->device);
+	struct ap_matrix *ap_matrix = vfio_ap_matrix_drv.ap_matrix;
+
+	vapq = kzalloc(sizeof(*vapq), GFP_KERNEL);
+	if (!vapq)
+		return -ENOMEM;
+
+	INIT_LIST_HEAD(&vapq->list);
+	vapq->queue = apq;
+	spin_lock_bh(&ap_matrix->qlock);
+	list_add_tail(&vapq->list, &ap_matrix->queues);
+	spin_unlock_bh(&ap_matrix->qlock);
+
+	return 0;
+}
+
+static void ap_matrix_queue_dev_remove(struct ap_device *apdev)
+{
+	struct vfio_ap_queue *vapq;
+	struct ap_queue *apq = to_ap_queue(&apdev->device);
+	struct ap_matrix *ap_matrix = vfio_ap_matrix_drv.ap_matrix;
+
+	vapq = find_vapq(ap_matrix, apq->qid);
+
+	if (vapq) {
+		spin_lock_bh(&ap_matrix->qlock);
+		list_del_init(&vapq->list);
+		spin_unlock_bh(&ap_matrix->qlock);
+		kfree(vapq);
+	}
+}
+
+int __init ap_matrix_init(void)
+{
+
+	int ret;
+
+	vfio_ap_matrix_drv.ap_matrix = ap_matrix_get_device();
+	if (!vfio_ap_matrix_drv.ap_matrix)
+		return -ENODEV;
+
+	vfio_ap_matrix_drv.ap_drv.probe = ap_matrix_queue_dev_probe;
+	vfio_ap_matrix_drv.ap_drv.remove = ap_matrix_queue_dev_remove;
+	vfio_ap_matrix_drv.ap_drv.ids = ap_queue_ids;
+
+	ret = ap_driver_register(&vfio_ap_matrix_drv.ap_drv,
+				 THIS_MODULE, VFIO_AP_MATRIX_DRV_NAME);
+	if (ret)
+		return ret;
+
+	return ret;
+}
+
+void __exit ap_matrix_exit(void)
+{
+	ap_driver_unregister(&vfio_ap_matrix_drv.ap_drv);
+}
+
+module_init(ap_matrix_init);
+module_exit(ap_matrix_exit);
diff --git a/drivers/s390/crypto/vfio_ap_matrix_private.h b/drivers/s390/crypto/vfio_ap_matrix_private.h
new file mode 100644
index 0000000..11c5e02
--- /dev/null
+++ b/drivers/s390/crypto/vfio_ap_matrix_private.h
@@ -0,0 +1,47 @@
+/*
+ * Private data and functions for adjunct processor VFIO matrix driver.
+ *
+ * Copyright IBM Corp. 2016
+ * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
+ */
+
+#ifndef _VFIO_AP_PRIVATE_H_
+#define _VFIO_AP_PRIVATE_H_
+
+#include <linux/types.h>
+
+#include "ap_bus.h"
+#include "ap_matrix_bus.h"
+
+#define VFIO_AP_MATRIX_MODULE_NAME "vfio_ap_matrix"
+
+struct vfio_ap_queue {
+	struct ap_queue *queue;
+	struct list_head list;
+};
+
+static inline struct vfio_ap_queue *to_vapq(struct ap_device *ap_dev)
+{
+	struct ap_queue *ap_queue = to_ap_queue(&ap_dev->device);
+	struct vfio_ap_queue *vapq;
+
+	vapq = container_of(&ap_queue, struct vfio_ap_queue, queue);
+
+	return vapq;
+}
+
+static inline struct vfio_ap_queue *find_vapq(struct ap_matrix *ap_matrix,
+					      ap_qid_t qid)
+{
+	struct vfio_ap_queue *vapq;
+
+	if (!list_empty(&ap_matrix->queues)) {
+		list_for_each_entry(vapq, &ap_matrix->queues, list)
+			if (vapq->queue->qid == qid)
+				return vapq;
+	}
+
+	return NULL;
+}
+
+#endif /* _VFIO_AP_PRIVATE_H_ */
-- 
1.7.1

[RFC 06/19] s390/zcrypt: register matrix device with VFIO mediated device framework

[Tony Krowiak]

Registers the matrix device created by the AP matrix bus with
the VFIO mediated device framework. Registering the matrix
device will create the sysfs structures needed to create
mediated matrix devices that can be configured with a matrix
of adapters, usage domains and control domains for a guest
machine.

Registering the matrix device with the VFIO mediated device
framework will create the following sysfs structures:

/sys/devices/ap_matrix
... [matrix]
...... [mdev_supported_types]
......... [ap_matrix-passthrough]
............ available_instances
............ create
............ device_api
............ [devices]
............ name

To create a mediated device for the AP matrix device, write a UUID
to the create file:

    uuidgen > create

A symbolic link to the mediated device's directory will be created in the
devices subdirectory named after the generated $uuid:

/sys/devices/ap_matrix
... [matrix]
...... [mdev_supported_types]
......... [ap_matrix-passthrough]
............ [devices]
............... [$uuid]

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 MAINTAINERS                                  |    1 +
 drivers/s390/crypto/Makefile                 |    2 +-
 drivers/s390/crypto/ap_matrix_bus.c          |    1 +
 drivers/s390/crypto/ap_matrix_bus.h          |    4 +
 drivers/s390/crypto/vfio_ap_matrix_drv.c     |    5 +
 drivers/s390/crypto/vfio_ap_matrix_ops.c     |  172 ++++++++++++++++++++++++++
 drivers/s390/crypto/vfio_ap_matrix_private.h |    3 +
 include/uapi/linux/vfio.h                    |    1 +
 8 files changed, 188 insertions(+), 1 deletions(-)
 create mode 100644 drivers/s390/crypto/vfio_ap_matrix_ops.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 115965e..0c8855a 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -11720,6 +11720,7 @@ F:	drivers/s390/crypto/ap_matrix_bus.h
 F:	drivers/s390/crypto/ap_matrix_bus.c
 F:	drivers/s390/crypto/vfio_ap_matrix_drv.c
 F:	drivers/s390/crypto/vfio_ap_matrix_private.h
+F:	drivers/s390/crypto/vfio_ap_matrix_ops.c
 
 S390 ZFCP DRIVER
 M:	Steffen Maier <maier@linux.vnet.ibm.com>
diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile
index 1983afa..b2545c8 100644
--- a/drivers/s390/crypto/Makefile
+++ b/drivers/s390/crypto/Makefile
@@ -16,5 +16,5 @@ pkey-objs := pkey_api.o
 obj-$(CONFIG_PKEY) += pkey.o
 
 # adjunct processor matrix
-vfio_ap_matrix-objs := vfio_ap_matrix_drv.o
+vfio_ap_matrix-objs := vfio_ap_matrix_drv.o vfio_ap_matrix_ops.o
 obj-$(CONFIG_VFIO_AP_MATRIX) += vfio_ap_matrix.o
diff --git a/drivers/s390/crypto/ap_matrix_bus.c b/drivers/s390/crypto/ap_matrix_bus.c
index 66bfa54..418c23b 100644
--- a/drivers/s390/crypto/ap_matrix_bus.c
+++ b/drivers/s390/crypto/ap_matrix_bus.c
@@ -61,6 +61,7 @@ static int ap_matrix_dev_create(void)
 	matrix->device.bus = &ap_matrix_bus_type;
 	matrix->device.parent = ap_matrix_root_device;
 	matrix->device.release = ap_matrix_dev_release;
+	INIT_LIST_HEAD(&matrix->queues);
 
 	ret = device_register(&matrix->device);
 	if (ret) {
diff --git a/drivers/s390/crypto/ap_matrix_bus.h b/drivers/s390/crypto/ap_matrix_bus.h
index c2aff23..3eccc36 100644
--- a/drivers/s390/crypto/ap_matrix_bus.h
+++ b/drivers/s390/crypto/ap_matrix_bus.h
@@ -12,8 +12,12 @@
 
 #include <linux/device.h>
 
+#include "ap_bus.h"
+
 struct ap_matrix {
 	struct device device;
+	spinlock_t qlock;
+	struct list_head queues;
 };
 
 struct ap_matrix *ap_matrix_get_device(void);
diff --git a/drivers/s390/crypto/vfio_ap_matrix_drv.c b/drivers/s390/crypto/vfio_ap_matrix_drv.c
index 760ed56..664d2f7 100644
--- a/drivers/s390/crypto/vfio_ap_matrix_drv.c
+++ b/drivers/s390/crypto/vfio_ap_matrix_drv.c
@@ -90,11 +90,16 @@ int __init ap_matrix_init(void)
 	if (ret)
 		return ret;
 
+	ret = ap_matrix_mdev_register(vfio_ap_matrix_drv.ap_matrix);
+	if (ret)
+		ap_driver_unregister(&vfio_ap_matrix_drv.ap_drv);
+
 	return ret;
 }
 
 void __exit ap_matrix_exit(void)
 {
+	ap_matrix_mdev_unregister(vfio_ap_matrix_drv.ap_matrix);
 	ap_driver_unregister(&vfio_ap_matrix_drv.ap_drv);
 }
 
diff --git a/drivers/s390/crypto/vfio_ap_matrix_ops.c b/drivers/s390/crypto/vfio_ap_matrix_ops.c
new file mode 100644
index 0000000..7d01f18
--- /dev/null
+++ b/drivers/s390/crypto/vfio_ap_matrix_ops.c
@@ -0,0 +1,172 @@
+/*
+ * Adjunct processor matrix VFIO device driver callbacks.
+ *
+ * Copyright IBM Corp. 2017
+ * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
+ *
+ */
+#include <asm/ap-config.h>
+#include <linux/string.h>
+#include <linux/vfio.h>
+#include <linux/mdev.h>
+#include <linux/device.h>
+#include <linux/list.h>
+#include <linux/ctype.h>
+
+#include "vfio_ap_matrix_private.h"
+
+#define AP_MATRIX_MDEV_TYPE_HWVIRT "passthrough"
+#define AP_MATRIX_MDEV_NAME_HWVIRT "AP Matrix Passthrough Device"
+
+#define AP_MATRIX_MAX_AVAILABLE_INSTANCES	255
+
+struct ap_matrix_mdev {
+	struct mdev_device *mdev;
+	struct list_head node;
+};
+
+struct ap_matrix	*matrix;
+struct mutex		mdev_devices_lock;
+struct list_head	mdev_devices;
+int			available_instances;
+
+static struct ap_matrix_mdev *ap_matrix_mdev_find_by_uuid(uuid_le uuid)
+{
+	struct ap_matrix_mdev *matrix_mdev;
+
+	list_for_each_entry(matrix_mdev, &mdev_devices, node) {
+		if (uuid_le_cmp(mdev_uuid(matrix_mdev->mdev), uuid) == 0)
+			return matrix_mdev;
+	}
+
+	return NULL;
+}
+
+static int ap_matrix_mdev_create(struct kobject *kobj,
+				 struct mdev_device *mdev)
+{
+	struct ap_matrix_mdev *matrix_mdev;
+	uuid_le uuid;
+
+	if (available_instances == 0) {
+		pr_err("%s: Only %d mediated devices allowed for %s-%s device",
+		       VFIO_AP_MATRIX_MODULE_NAME,
+		       AP_MATRIX_MAX_AVAILABLE_INSTANCES,
+		       VFIO_AP_MATRIX_MODULE_NAME,
+		       AP_MATRIX_MDEV_TYPE_HWVIRT);
+		return -EPERM;
+	}
+
+	uuid = mdev_uuid(mdev);
+	matrix_mdev = ap_matrix_mdev_find_by_uuid(uuid);
+	if (matrix_mdev) {
+		pr_err("%s: Mediated device %pU already exists",
+		       VFIO_AP_MATRIX_MODULE_NAME, uuid.b);
+		return -EEXIST;
+	}
+
+	matrix_mdev = kzalloc(sizeof(*matrix_mdev), GFP_KERNEL);
+	if (!matrix_mdev)
+		return -ENOMEM;
+
+	mdev_set_drvdata(mdev, matrix_mdev);
+	mutex_lock(&mdev_devices_lock);
+	list_add_tail(&matrix_mdev->node, &mdev_devices);
+	mutex_unlock(&mdev_devices_lock);
+
+	return 0;
+}
+
+static int ap_matrix_mdev_remove(struct mdev_device *mdev)
+{
+	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+	uuid_le uuid = mdev_uuid(mdev);
+
+	matrix_mdev = ap_matrix_mdev_find_by_uuid(uuid);
+
+	if (matrix_mdev) {
+		mutex_lock(&mdev_devices_lock);
+		list_del(&matrix_mdev->node);
+		mutex_unlock(&mdev_devices_lock);
+		mdev_set_drvdata(mdev, NULL);
+		kfree(matrix_mdev);
+		available_instances--;
+	}
+
+	return 0;
+}
+
+static ssize_t name_show(struct kobject *kobj, struct device *dev, char *buf)
+{
+	return sprintf(buf, "%s\n", AP_MATRIX_MDEV_NAME_HWVIRT);
+}
+
+MDEV_TYPE_ATTR_RO(name);
+
+static ssize_t available_instances_show(struct kobject *kobj,
+					struct device *dev, char *buf)
+{
+	return sprintf(buf, "%d\n", available_instances);
+}
+
+MDEV_TYPE_ATTR_RO(available_instances);
+
+static ssize_t device_api_show(struct kobject *kobj, struct device *dev,
+			       char *buf)
+{
+	return sprintf(buf, "%s\n", VFIO_DEVICE_API_AP_MATRIX_STRING);
+}
+
+MDEV_TYPE_ATTR_RO(device_api);
+
+static struct attribute *ap_matrix_mdev_type_attrs[] = {
+	&mdev_type_attr_name.attr,
+	&mdev_type_attr_device_api.attr,
+	&mdev_type_attr_available_instances.attr,
+	NULL,
+};
+
+static struct attribute_group ap_matrix_mdev_hwvirt_type_group = {
+	.name = AP_MATRIX_MDEV_TYPE_HWVIRT,
+	.attrs = ap_matrix_mdev_type_attrs,
+};
+
+static struct attribute_group *ap_matrix_mdev_type_groups[] = {
+	&ap_matrix_mdev_hwvirt_type_group,
+	NULL,
+};
+
+static const struct mdev_parent_ops vfio_ap_matrix_ops = {
+	.owner                  = THIS_MODULE,
+	.supported_type_groups  = ap_matrix_mdev_type_groups,
+	.create                 = ap_matrix_mdev_create,
+	.remove                 = ap_matrix_mdev_remove,
+};
+
+int ap_matrix_mdev_register(struct ap_matrix *ap_matrix)
+{
+	int ret;
+	struct device *dev = &ap_matrix->device;
+
+	ret = mdev_register_device(dev, &vfio_ap_matrix_ops);
+	if (ret)
+		return ret;
+
+	matrix = ap_matrix;
+	mutex_init(&mdev_devices_lock);
+	INIT_LIST_HEAD(&mdev_devices);
+	available_instances = AP_MATRIX_MAX_AVAILABLE_INSTANCES;
+
+	return 0;
+}
+
+void ap_matrix_mdev_unregister(struct ap_matrix *ap_matrix)
+{
+	struct device *dev;
+
+	if (ap_matrix == matrix) {
+		dev = &matrix->device;
+		available_instances--;
+		mdev_unregister_device(dev);
+	}
+}
diff --git a/drivers/s390/crypto/vfio_ap_matrix_private.h b/drivers/s390/crypto/vfio_ap_matrix_private.h
index 11c5e02..5c82bea 100644
--- a/drivers/s390/crypto/vfio_ap_matrix_private.h
+++ b/drivers/s390/crypto/vfio_ap_matrix_private.h
@@ -44,4 +44,7 @@ struct vfio_ap_queue {
 	return NULL;
 }
 
+extern int ap_matrix_mdev_register(struct ap_matrix *ap_matrix);
+extern void ap_matrix_mdev_unregister(struct ap_matrix *ap_matrix);
+
 #endif /* _VFIO_AP_PRIVATE_H_ */
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index ae46105..743456f 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -214,6 +214,7 @@ struct vfio_device_info {
 #define VFIO_DEVICE_API_PLATFORM_STRING		"vfio-platform"
 #define VFIO_DEVICE_API_AMBA_STRING		"vfio-amba"
 #define VFIO_DEVICE_API_CCW_STRING		"vfio-ccw"
+#define VFIO_DEVICE_API_AP_MATRIX_STRING	"vfio-ap-matrix"
 
 /**
  * VFIO_DEVICE_GET_REGION_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 8,
-- 
1.7.1

[RFC 07/19] KVM: s390: introduce AP matrix configuration interface

[Tony Krowiak]

This patch introduces a new compilation unit that will contain
all of the data structures and logic for configuring AP adapters,
usage domains and control domains for a KVM guest.

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 MAINTAINERS                       |    2 ++
 arch/s390/include/asm/ap-config.h |   12 ++++++++++++
 arch/s390/kvm/Makefile            |    2 +-
 arch/s390/kvm/ap-config.c         |    9 +++++++++
 4 files changed, 24 insertions(+), 1 deletions(-)
 create mode 100644 arch/s390/include/asm/ap-config.h
 create mode 100644 arch/s390/kvm/ap-config.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 0c8855a..4de0904 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -11721,6 +11721,8 @@ F:	drivers/s390/crypto/ap_matrix_bus.c
 F:	drivers/s390/crypto/vfio_ap_matrix_drv.c
 F:	drivers/s390/crypto/vfio_ap_matrix_private.h
 F:	drivers/s390/crypto/vfio_ap_matrix_ops.c
+F:	arch/s390/include/asm/kvm/ap-config.h
+F:	arch/s390/kvm/ap-config.c
 
 S390 ZFCP DRIVER
 M:	Steffen Maier <maier@linux.vnet.ibm.com>
diff --git a/arch/s390/include/asm/ap-config.h b/arch/s390/include/asm/ap-config.h
new file mode 100644
index 0000000..8491b5f
--- /dev/null
+++ b/arch/s390/include/asm/ap-config.h
@@ -0,0 +1,12 @@
+/*
+ * Adjunct Processor (AP) configuration management for KVM guests
+ *
+ * Copyright IBM Corp. 2017
+ *
+ * Author(s): Tony Krowiak <akrowia@linux.vnet.ibm.com>
+ */
+
+#ifndef _ASM_KVM_AP_CONFIG_H_
+#define _ASM_KVM_AP_CONFIG_H_
+
+#endif /* _ASM_KVM_AP_CONFIG_H_ */
diff --git a/arch/s390/kvm/Makefile b/arch/s390/kvm/Makefile
index 09a9e6d..0397b9c 100644
--- a/arch/s390/kvm/Makefile
+++ b/arch/s390/kvm/Makefile
@@ -12,6 +12,6 @@ common-objs = $(KVM)/kvm_main.o $(KVM)/eventfd.o  $(KVM)/async_pf.o $(KVM)/irqch
 ccflags-y := -Ivirt/kvm -Iarch/s390/kvm
 
 kvm-objs := $(common-objs) kvm-s390.o intercept.o interrupt.o priv.o sigp.o
-kvm-objs += diag.o gaccess.o guestdbg.o sthyi.o vsie.o
+kvm-objs += diag.o gaccess.o guestdbg.o sthyi.o vsie.o ap-config.o
 
 obj-$(CONFIG_KVM) += kvm.o
diff --git a/arch/s390/kvm/ap-config.c b/arch/s390/kvm/ap-config.c
new file mode 100644
index 0000000..84fdf43
--- /dev/null
+++ b/arch/s390/kvm/ap-config.c
@@ -0,0 +1,9 @@
+/*
+ * Adjunct Processor (AP) configuration management for KVM guests
+ *
+ * Copyright IBM Corp. 2017
+ *
+ * Author(s): Tony Krowiak <akrowia@linux.vnet.ibm.com>
+ */
+
+#include <asm/ap-config.h>
-- 
1.7.1

[RFC 08/19] s390/zcrypt: support for assigning adapters to matrix mdev

[Tony Krowiak]

Provides the sysfs interfaces for assigning an adapter to
and unassigning an AP adapter from a mediated matrix device.

The IDs of the AP adapters assigned to the mediated matrix
device are stored in a bit mask. The bits in the mask, from
left to right, correspond to AP adapter numbers 0 to 255. The
bit corresponding to the ID of the adapter being assigned will
be set in the bit mask.

The relevant sysfs structures are:

/sys/devices/ap_matrix
... [matrix]
...... [mdev_supported_types]
......... [ap_matrix-passthrough]
............ [devices]
...............[$uuid]
.................. adapters
.................. assign_adapter
.................. unassign_adapter

To assign an adapter to the $uuid mediated matrix device, write
ID of the adapter (hex value) to the assign_adapter file. To
unassign an adapter, write the ID of the adapter (hex value)
to the unassign_adapter file. The list of adapters that have
been assigned can be viewed by displaying the contents of the
adapters file.

For example, to assign adapter 0xad to mediated matrix device
$uuid:

	echo ad > assign_adapter

To unassign adapter 0xad:

	echo ad > unassign_adapter

To see the list of adapters assigned:

	cat adapters

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 arch/s390/include/asm/ap-config.h        |   13 +++
 drivers/s390/crypto/vfio_ap_matrix_ops.c |  147 ++++++++++++++++++++++++++++++
 2 files changed, 160 insertions(+), 0 deletions(-)

diff --git a/arch/s390/include/asm/ap-config.h b/arch/s390/include/asm/ap-config.h
index 8491b5f..3064215 100644
--- a/arch/s390/include/asm/ap-config.h
+++ b/arch/s390/include/asm/ap-config.h
@@ -9,4 +9,17 @@
 #ifndef _ASM_KVM_AP_CONFIG_H_
 #define _ASM_KVM_AP_CONFIG_H_
 
+#include <linux/types.h>
+
+#define AP_MATRIX_MAX_MASK_BITS		256
+#define AP_MATRIX_MASK_INDICES		(AP_MATRIX_MAX_MASK_BITS / \
+					(sizeof(u64) * 8))
+#define AP_MATRIX_MAX_MASK_BYTES	(AP_MATRIX_MASK_INDICES * sizeof(u64))
+
+struct ap_config_masks {
+	u64 apm[AP_MATRIX_MASK_INDICES];
+	u64 aqm[AP_MATRIX_MASK_INDICES];
+	u64 adm[AP_MATRIX_MASK_INDICES];
+};
+
 #endif /* _ASM_KVM_AP_CONFIG_H_ */
diff --git a/drivers/s390/crypto/vfio_ap_matrix_ops.c b/drivers/s390/crypto/vfio_ap_matrix_ops.c
index 7d01f18..e4b1236 100644
--- a/drivers/s390/crypto/vfio_ap_matrix_ops.c
+++ b/drivers/s390/crypto/vfio_ap_matrix_ops.c
@@ -23,6 +23,7 @@
 struct ap_matrix_mdev {
 	struct mdev_device *mdev;
 	struct list_head node;
+	struct ap_config_masks masks;
 };
 
 struct ap_matrix	*matrix;
@@ -136,9 +137,155 @@ static ssize_t device_api_show(struct kobject *kobj, struct device *dev,
 	NULL,
 };
 
+static int ap_matrix_id_is_xnum(char *id)
+{
+	size_t i;
+
+	for (i = 0; i < strlen(id); i++) {
+		if (!isxdigit(id[i]))
+			return 0;
+	}
+
+	return 1;
+}
+
+static int ap_matrix_parse_id(const char *buf, unsigned int *id)
+{
+	int ret;
+	char *bufcpy;
+	char *id_str;
+
+	if ((strlen(buf) == 0) || (*buf == '\n')) {
+		pr_err("%s: input buffer '%s' contains no valid hex values",
+		       VFIO_AP_MATRIX_MODULE_NAME, buf);
+		return -EINVAL;
+	}
+
+	bufcpy = kzalloc(strlen(buf) + 1, GFP_KERNEL);
+	if (!bufcpy)
+		return -ENOMEM;
+
+	strcpy(bufcpy, buf);
+	id_str = strim(bufcpy);
+
+	if (!ap_matrix_id_is_xnum(id_str)) {
+		pr_err("%s: input id '%s' contains an invalid hex value '%s'",
+		       VFIO_AP_MATRIX_MODULE_NAME, buf, id_str);
+		ret = -EINVAL;
+		goto done;
+	}
+
+	ret = kstrtouint (id_str, 16, id);
+	if (ret || (*id >= AP_MATRIX_MAX_MASK_BITS)) {
+		pr_err("%s: input id '%s' is not a value from 0 to %x",
+		       VFIO_AP_MATRIX_MODULE_NAME, buf,
+		       AP_MATRIX_MAX_MASK_BITS);
+		ret = -EINVAL;
+		goto done;
+	}
+
+	ret = 0;
+
+done:
+	kfree(bufcpy);
+	return ret;
+}
+
+static ssize_t ap_matrix_adapters_assign(struct device *dev,
+					 struct device_attribute *attr,
+					 const char *buf, size_t count)
+{
+	int ret;
+	unsigned int apid;
+	struct mdev_device *mdev = mdev_from_dev(dev);
+	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+	ret = ap_matrix_parse_id(buf, &apid);
+	if (ret)
+		return ret;
+
+	set_bit_inv((unsigned long)apid,
+		    (unsigned long *)matrix_mdev->masks.apm);
+
+	return count;
+}
+static DEVICE_ATTR(assign_adapter, 0644, NULL, ap_matrix_adapters_assign);
+
+static ssize_t ap_matrix_adapters_unassign(struct device *dev,
+					   struct device_attribute *attr,
+					   const char *buf, size_t count)
+{
+	int ret;
+	unsigned int apid;
+	struct mdev_device *mdev = mdev_from_dev(dev);
+	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+	ret = ap_matrix_parse_id(buf, &apid);
+	if (ret)
+		return ret;
+
+	clear_bit_inv((unsigned long)apid,
+		      (unsigned long *)matrix_mdev->masks.apm);
+
+	return count;
+}
+static DEVICE_ATTR(unassign_adapter, 0644, NULL, ap_matrix_adapters_unassign);
+
+static ssize_t ap_matrix_adapters_show(struct device *dev,
+				       struct device_attribute *attr,
+				       char *buf)
+{
+	struct mdev_device *mdev = mdev_from_dev(dev);
+	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+	unsigned long *apm = (unsigned long *)matrix_mdev->masks.apm;
+	unsigned long id;
+	unsigned long nbits = 256;
+	char *bufpos = buf;
+	int nchars = 0;
+	int n;
+
+	id = find_first_bit_inv(apm, nbits);
+	while (id < nbits) {
+		if (nchars) {
+			n = sprintf(bufpos, ",");
+			bufpos += n;
+			nchars += n;
+		}
+
+		n = sprintf(bufpos, "%02lx", id);
+		bufpos += n;
+		nchars += n;
+		id = find_next_bit_inv(apm, nbits, id + 1);
+	}
+
+	n = sprintf(bufpos, "\n");
+	bufpos += n;
+	nchars += n;
+
+	return nchars;
+}
+static DEVICE_ATTR(adapters, 0644, ap_matrix_adapters_show, NULL);
+
+static struct attribute *ap_matrix_mdev_attrs[] = {
+	&dev_attr_assign_adapter.attr,
+	&dev_attr_unassign_adapter.attr,
+	&dev_attr_adapters.attr,
+	NULL
+};
+
+static struct attribute_group ap_matrix_mdev_attr_group = {
+	.attrs = ap_matrix_mdev_attrs
+};
+
+static const struct attribute_group *ap_matrix_mdev_attr_groups[] = {
+	&ap_matrix_mdev_attr_group,
+	NULL
+};
+
 static const struct mdev_parent_ops vfio_ap_matrix_ops = {
 	.owner                  = THIS_MODULE,
 	.supported_type_groups  = ap_matrix_mdev_type_groups,
+	.mdev_attr_groups	= ap_matrix_mdev_attr_groups,
 	.create                 = ap_matrix_mdev_create,
 	.remove                 = ap_matrix_mdev_remove,
 };
-- 
1.7.1

[RFC 09/19] s390/zcrypt: validate adapter assignment

[Tony Krowiak]

Each adapter assigned to the mediated matrix device
must be validated to ensure that the adapter is accessible
from one of the AP queues bound to the AP matrix device
driver. Each AP queue bound to the matrix device is
identified by its AP Queue Number (APQN). An APQN is
comprised of an adapter ID (APID) and an AP queue index
(APQI) which corresponds to a domain ID. Consequently,
each adapter being assigned will be validated as follows:

* If no domains have been assigned to the mediated
  matrix device:

    * The ID of the adapter being assigned must correspond
      to the APID of at least one of the AP queue devices
      bound to the AP matrix device driver.

* If domains have been assigned to the mediated
  matrix device:

    * Each APQN that can be derived from the ID of the adapter
      being assigned and the ID of each domain previously
      assigned must match the APQN of an AP queue bound to
      the AP matrix device driver.

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 drivers/s390/crypto/vfio_ap_matrix_ops.c |   63 ++++++++++++++++++++++++++++++
 1 files changed, 63 insertions(+), 0 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_matrix_ops.c b/drivers/s390/crypto/vfio_ap_matrix_ops.c
index e4b1236..2e63d1b 100644
--- a/drivers/s390/crypto/vfio_ap_matrix_ops.c
+++ b/drivers/s390/crypto/vfio_ap_matrix_ops.c
@@ -191,6 +191,65 @@ static int ap_matrix_parse_id(const char *buf, unsigned int *id)
 	return ret;
 }
 
+static int ap_matrix_validate_adapter_queues(unsigned long apid,
+					     unsigned long *aqm)
+{
+	unsigned long apqi;
+	unsigned long nbits = AP_MATRIX_MAX_MASK_BITS;
+	ap_qid_t qid;
+
+	apqi = find_first_bit_inv(aqm, nbits);
+	while (apqi < nbits) {
+		qid = AP_MKQID(apid, apqi);
+
+		if (find_vapq(matrix, qid)) {
+			apqi = find_next_bit_inv(aqm, nbits, apqi + 1);
+			continue;
+		}
+
+		pr_err("%s: AP queue %02lx.%04lx not bound to driver",
+		       VFIO_AP_MATRIX_MODULE_NAME, apid, apqi);
+		return -ENODEV;
+	}
+
+	return 0;
+}
+
+static int ap_matrix_validate_apid(unsigned long apid)
+{
+
+	struct vfio_ap_queue *vapq;
+
+	if (!list_empty(&matrix->queues)) {
+		list_for_each_entry(vapq, &matrix->queues, list)
+			if (AP_QID_CARD(vapq->queue->qid) == apid)
+				return 0;
+	}
+
+	pr_err("%s: AP queue with adapter %02lx is not bound to driver",
+	       VFIO_AP_MATRIX_MODULE_NAME, apid);
+
+	return -ENODEV;
+}
+
+static int ap_matrix_validate_adapter(struct ap_matrix_mdev *matrix_mdev,
+				      unsigned long apid)
+{
+	int ret = 0;
+	unsigned long nbits = AP_MATRIX_MAX_MASK_BITS;
+	unsigned long *aqm = (unsigned long *)matrix_mdev->masks.aqm;
+	unsigned long apqi;
+
+	apqi = find_first_bit_inv(aqm, nbits);
+	/* If no queues have yet been assigned */
+	if (apqi < nbits)
+		ret = ap_matrix_validate_adapter_queues(apid, aqm);
+	else
+		ret = ap_matrix_validate_apid(apid);
+
+	return ret;
+}
+
 static ssize_t ap_matrix_adapters_assign(struct device *dev,
 					 struct device_attribute *attr,
 					 const char *buf, size_t count)
@@ -204,6 +263,10 @@ static ssize_t ap_matrix_adapters_assign(struct device *dev,
 	if (ret)
 		return ret;
 
+	ret = ap_matrix_validate_adapter(matrix_mdev, apid);
+	if (ret)
+		return ret;
+
 	set_bit_inv((unsigned long)apid,
 		    (unsigned long *)matrix_mdev->masks.apm);
 
-- 
1.7.1

[RFC 10/19] s390/zcrypt: sysfs interfaces supporting AP domain assignment

[Tony Krowiak]

Provides the sysfs interfaces for assigning an AP usage domain
to and unassigning a usage domain from a mediated matrix device.

The relevant sysfs structures are:

/sys/devices/ap_matrix
... [matrix]
...... [mdev_supported_types]
......... [ap_matrix-passthrough]
............ [devices]
...............[$uuid]
.................. assign_domain
.................. domains
.................. unassign_domain

To assign a domain to the $uuid mediated matrix device, write
its domain ID (hex value) to the assign_domain file. To
unassign a domain, write its domain ID (hex value) to the
unassign_domain file. To view a list of domains which have
been assigned, print the contents of the domains file

For example, to assign domain 0x04 to the $uuid mediated
matrix device:

	echo 4 > assign_domain

To unassign domain 0x04:

	echo 4 > unassign_domain

To see the list of domains assigned:

	cat domains

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 drivers/s390/crypto/vfio_ap_matrix_ops.c |   80 ++++++++++++++++++++++++++++++
 1 files changed, 80 insertions(+), 0 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_matrix_ops.c b/drivers/s390/crypto/vfio_ap_matrix_ops.c
index 2e63d1b..f54285c 100644
--- a/drivers/s390/crypto/vfio_ap_matrix_ops.c
+++ b/drivers/s390/crypto/vfio_ap_matrix_ops.c
@@ -329,10 +329,90 @@ static ssize_t ap_matrix_adapters_show(struct device *dev,
 }
 static DEVICE_ATTR(adapters, 0644, ap_matrix_adapters_show, NULL);
 
+static ssize_t ap_matrix_domain_assign(struct device *dev,
+				       struct device_attribute *attr,
+				       const char *buf, size_t count)
+{
+	int ret;
+	unsigned int apqi;
+	struct mdev_device *mdev = mdev_from_dev(dev);
+	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+	ret = ap_matrix_parse_id(buf, &apqi);
+	if (ret)
+		return ret;
+
+	set_bit_inv((unsigned long)apqi,
+		    (unsigned long *)matrix_mdev->masks.aqm);
+
+	return count;
+}
+static DEVICE_ATTR(assign_domain, 0644, NULL, ap_matrix_domain_assign);
+
+static ssize_t ap_matrix_domain_unassign(struct device *dev,
+					 struct device_attribute *attr,
+					 const char *buf, size_t count)
+{
+	int ret;
+	unsigned int apqi;
+	struct mdev_device *mdev = mdev_from_dev(dev);
+	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+	ret = ap_matrix_parse_id(buf, &apqi);
+	if (ret)
+		return ret;
+
+	clear_bit_inv((unsigned long)apqi,
+		      (unsigned long *)matrix_mdev->masks.aqm);
+
+	return count;
+}
+static DEVICE_ATTR(unassign_domain, 0644, NULL, ap_matrix_domain_unassign);
+
+static ssize_t ap_matrix_domains_show(struct device *dev,
+				       struct device_attribute *attr,
+				       char *buf)
+{
+	struct mdev_device *mdev = mdev_from_dev(dev);
+	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+	unsigned long *aqm = (unsigned long *)matrix_mdev->masks.aqm;
+	unsigned long id;
+	unsigned long nbits = 256;
+	char *bufpos = buf;
+	int nchars = 0;
+	int n;
+
+	id = find_first_bit_inv(aqm, nbits);
+	while (id < nbits) {
+		if (nchars) {
+			n = sprintf(bufpos, ",");
+			bufpos += n;
+			nchars += n;
+		}
+
+		n = sprintf(bufpos, "%04lx", id);
+		bufpos += n;
+		nchars += n;
+		id = find_next_bit_inv(aqm, nbits, id + 1);
+	}
+
+	n = sprintf(bufpos, "\n");
+	bufpos += n;
+	nchars += n;
+
+	return nchars;
+}
+
+static DEVICE_ATTR(domains, 0644, ap_matrix_domains_show,
+		   NULL);
+
 static struct attribute *ap_matrix_mdev_attrs[] = {
 	&dev_attr_assign_adapter.attr,
 	&dev_attr_unassign_adapter.attr,
 	&dev_attr_adapters.attr,
+	&dev_attr_assign_domain.attr,
+	&dev_attr_unassign_domain.attr,
+	&dev_attr_domains.attr,
 	NULL
 };
 
-- 
1.7.1

[RFC 11/19] s390/zcrypt: validate domain assignment

[Tony Krowiak]

Each domain assigned to the mediated matrix device
must be validated to ensure that the domain is accessible
from one of the AP queues bound to the AP matrix device
driver.

Each AP queue bound to the device driver is identified by
an AP Queue Number (APQN). An APQN is comprised of an
adapter ID (APID) and an AP queue index (APQI) which
corresponds to a domain ID. Consequently, each domain
being assigned will be validated as follows:

* If no adapters have been assigned to the mediated
  matrix device:

    * The ID of the domain being assigned must correspond
      to the APQI of at least one of the AP queue devices
      bound to the AP matrix device driver.

* If adapters have been assigned to the mediated matrix
  device:

     * Each APQN that can be derived from the ID of the domain
       being assigned and the ID of each adapter previously
       assigned must match the APQN of an AP queue bound to
       the AP matrix device driver.

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 drivers/s390/crypto/vfio_ap_matrix_ops.c |   63 ++++++++++++++++++++++++++++++
 1 files changed, 63 insertions(+), 0 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_matrix_ops.c b/drivers/s390/crypto/vfio_ap_matrix_ops.c
index f54285c..74d7feb 100644
--- a/drivers/s390/crypto/vfio_ap_matrix_ops.c
+++ b/drivers/s390/crypto/vfio_ap_matrix_ops.c
@@ -329,6 +329,65 @@ static ssize_t ap_matrix_adapters_show(struct device *dev,
 }
 static DEVICE_ATTR(adapters, 0644, ap_matrix_adapters_show, NULL);
 
+static int ap_matrix_validate_domain_queues(unsigned long apqi,
+					    unsigned long *apm)
+{
+	unsigned long apid;
+	unsigned long nbits = AP_MATRIX_MAX_MASK_BITS;
+	ap_qid_t qid;
+
+	apid = find_first_bit_inv(apm, nbits);
+	while (apid < nbits) {
+		qid = AP_MKQID(apid, apqi);
+
+		if (find_vapq(matrix, qid)) {
+			apid = find_next_bit_inv(apm, nbits, apid + 1);
+			continue;
+		}
+
+		pr_err("%s: AP queue %02lx.%04lx not bound to driver",
+		       VFIO_AP_MATRIX_MODULE_NAME, apid, apid);
+		return -ENODEV;
+	}
+
+	return 0;
+}
+
+static int ap_matrix_validate_apqi(unsigned long apqi)
+{
+
+	struct vfio_ap_queue *vapq;
+
+	if (!list_empty(&matrix->queues)) {
+		list_for_each_entry(vapq, &matrix->queues, list)
+			if (AP_QID_QUEUE(vapq->queue->qid) == apqi)
+				return 0;
+	}
+
+	pr_err("%s: AP queue with domain %02lx is not bound to driver",
+	       VFIO_AP_MATRIX_MODULE_NAME, apqi);
+
+	return -ENODEV;
+}
+
+static int ap_matrix_validate_domain(struct ap_matrix_mdev *matrix_mdev,
+				     unsigned long apqi)
+{
+	int ret = 0;
+	unsigned long nbits = AP_MATRIX_MAX_MASK_BITS;
+	unsigned long *apm = (unsigned long *)matrix_mdev->masks.apm;
+	unsigned long apid;
+
+	apid = find_first_bit_inv(apm, nbits);
+	/* If no adapters have yet been assigned */
+	if (apid < nbits)
+		ret = ap_matrix_validate_apqi(apqi);
+	else
+		ret = ap_matrix_validate_domain_queues(apqi, apm);
+
+	return ret;
+}
+
 static ssize_t ap_matrix_domain_assign(struct device *dev,
 				       struct device_attribute *attr,
 				       const char *buf, size_t count)
@@ -342,6 +401,10 @@ static ssize_t ap_matrix_domain_assign(struct device *dev,
 	if (ret)
 		return ret;
 
+	ret = ap_matrix_validate_domain(matrix_mdev, apqi);
+	if (ret)
+		return ret;
+
 	set_bit_inv((unsigned long)apqi,
 		    (unsigned long *)matrix_mdev->masks.aqm);
 
-- 
1.7.1

[RFC 12/19] s390/zcrypt: sysfs support for control domain assignment

[Tony Krowiak]

Provides the sysfs interfaces for assigning an AP control
domain to and unassigning a control domain from a mediated
matrix device. The relevant sysfs structures are:

/sys/devices/ap_matrix
... [matrix]
...... [mdev_supported_types]
......... [ap_matrix-passthrough]
............ [devices]
...............[$uuid]
.................. assign_control_domain
.................. unassign_control_domain
.................. control_domains

To assign a control domain to the $uuid mediated matrix device,
write its domain ID (hex value) to the assign_control_domain file.
To unassign a control domain, write its domain ID to the
unassign_control_domain file. To display the list of control
domains assigned, print the domains file.

For example, to assign control domain 0xff to the $uuid mediated
matrix device:

	echo ff > assign_control_domain

To see the list of control domains assigned:

	cat control_domains

To unassign control domain 0xff:

	echo ff > unassign_control_domain

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 drivers/s390/crypto/vfio_ap_matrix_ops.c |   82 ++++++++++++++++++++++++++++++
 1 files changed, 82 insertions(+), 0 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_matrix_ops.c b/drivers/s390/crypto/vfio_ap_matrix_ops.c
index 74d7feb..2387916 100644
--- a/drivers/s390/crypto/vfio_ap_matrix_ops.c
+++ b/drivers/s390/crypto/vfio_ap_matrix_ops.c
@@ -469,6 +469,85 @@ static ssize_t ap_matrix_domains_show(struct device *dev,
 static DEVICE_ATTR(domains, 0644, ap_matrix_domains_show,
 		   NULL);
 
+static ssize_t ap_matrix_control_domain_assign(struct device *dev,
+					       struct device_attribute *attr,
+					       const char *buf, size_t count)
+{
+	int ret;
+	unsigned int id;
+	struct mdev_device *mdev = mdev_from_dev(dev);
+	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+	ret = ap_matrix_parse_id(buf, &id);
+	if (ret)
+		return ret;
+
+	set_bit_inv((unsigned long)id,
+		    (unsigned long *)matrix_mdev->masks.adm);
+
+	return count;
+}
+static DEVICE_ATTR(assign_control_domain, 0644, NULL,
+		   ap_matrix_control_domain_assign);
+
+static ssize_t ap_matrix_control_domain_unassign(struct device *dev,
+						  struct device_attribute *attr,
+						  const char *buf, size_t count)
+{
+	int ret;
+	unsigned int id;
+	struct mdev_device *mdev = mdev_from_dev(dev);
+	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+	ret = ap_matrix_parse_id(buf, &id);
+	if (ret)
+		return ret;
+
+	clear_bit_inv((unsigned long)id,
+		      (unsigned long *)matrix_mdev->masks.adm);
+
+	return count;
+}
+static DEVICE_ATTR(unassign_control_domain, 0644, NULL,
+		   ap_matrix_control_domain_unassign);
+
+static ssize_t ap_matrix_control_domains_show(struct device *dev,
+					      struct device_attribute *attr,
+					      char *buf)
+{
+	struct mdev_device *mdev = mdev_from_dev(dev);
+	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+	unsigned long *adm = (unsigned long *)matrix_mdev->masks.adm;
+	unsigned long id;
+	unsigned long nbits = 256;
+	char *bufpos = buf;
+	int nchars = 0;
+	int n;
+
+	id = find_first_bit_inv(adm, nbits);
+	while (id < nbits) {
+		if (nchars) {
+			n = sprintf(bufpos, ",");
+			bufpos += n;
+			nchars += n;
+		}
+
+		n = sprintf(bufpos, "%04lx", id);
+		bufpos += n;
+		nchars += n;
+		id = find_next_bit_inv(adm, nbits, id + 1);
+	}
+
+	n = sprintf(bufpos, "\n");
+	bufpos += n;
+	nchars += n;
+
+	return nchars;
+}
+
+static DEVICE_ATTR(control_domains, 0644, ap_matrix_control_domains_show,
+		   NULL);
+
 static struct attribute *ap_matrix_mdev_attrs[] = {
 	&dev_attr_assign_adapter.attr,
 	&dev_attr_unassign_adapter.attr,
@@ -476,6 +555,9 @@ static DEVICE_ATTR(domains, 0644, ap_matrix_domains_show,
 	&dev_attr_assign_domain.attr,
 	&dev_attr_unassign_domain.attr,
 	&dev_attr_domains.attr,
+	&dev_attr_assign_control_domain.attr,
+	&dev_attr_unassign_control_domain.attr,
+	&dev_attr_control_domains.attr,
 	NULL
 };
 
-- 
1.7.1

[RFC 13/19] s390/zcrypt: validate control domain assignment

[Tony Krowiak]

The AP control domain being assigned to the mediated
matrix driver must be assigned to the LPAR.

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 drivers/s390/crypto/vfio_ap_matrix_ops.c |   33 ++++++++++++++++++++++++++++++
 1 files changed, 33 insertions(+), 0 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_matrix_ops.c b/drivers/s390/crypto/vfio_ap_matrix_ops.c
index 2387916..10a006c 100644
--- a/drivers/s390/crypto/vfio_ap_matrix_ops.c
+++ b/drivers/s390/crypto/vfio_ap_matrix_ops.c
@@ -469,12 +469,41 @@ static ssize_t ap_matrix_domains_show(struct device *dev,
 static DEVICE_ATTR(domains, 0644, ap_matrix_domains_show,
 		   NULL);
 
+static int
+ap_matrix_validate_control_domains(struct ap_matrix_mdev *matrix_mdev,
+				   unsigned long id)
+{
+	int ret;
+	struct ap_config_info config;
+
+	memset(&config, 0, sizeof(config));
+
+	ret = ap_query_configuration(&config);
+	if (ret == -EOPNOTSUPP)
+		return 0;
+
+	if (ret) {
+		pr_err("%s: query AP configuration failed with rc %d",
+		       VFIO_AP_MATRIX_MODULE_NAME, ret);
+		return ret;
+	}
+
+	if (!test_bit_inv(id, (unsigned long *)config.adm)) {
+		pr_err("%s: control domain %04lx is not installed on the lpar",
+		       VFIO_AP_MATRIX_MODULE_NAME, id);
+		return -ENODEV;
+	}
+
+	return 0;
+}
+
 static ssize_t ap_matrix_control_domain_assign(struct device *dev,
 					       struct device_attribute *attr,
 					       const char *buf, size_t count)
 {
 	int ret;
 	unsigned int id;
+
 	struct mdev_device *mdev = mdev_from_dev(dev);
 	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
 
@@ -482,6 +511,10 @@ static ssize_t ap_matrix_control_domain_assign(struct device *dev,
 	if (ret)
 		return ret;
 
+	ret = ap_matrix_validate_control_domains(matrix_mdev, id);
+	if (ret)
+		return ret;
+
 	set_bit_inv((unsigned long)id,
 		    (unsigned long *)matrix_mdev->masks.adm);
 
-- 
1.7.1

[RFC 14/19] KVM: s390: Connect the AP mediated matrix device to KVM

[Tony Krowiak]

Registers a group notifier during the open of the mediated
device to get information on KVM presence through the
VFIO_GROUP_NOTIFY_SET_KVM event.

When notified, save the pointer to KVM inside mediated matrix device
structure.

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 drivers/s390/crypto/vfio_ap_matrix_ops.c |   43 ++++++++++++++++++++++++++++++
 1 files changed, 43 insertions(+), 0 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_matrix_ops.c b/drivers/s390/crypto/vfio_ap_matrix_ops.c
index 10a006c..fd36074 100644
--- a/drivers/s390/crypto/vfio_ap_matrix_ops.c
+++ b/drivers/s390/crypto/vfio_ap_matrix_ops.c
@@ -12,6 +12,9 @@
 #include <linux/device.h>
 #include <linux/list.h>
 #include <linux/ctype.h>
+#include <linux/kvm_host.h>
+#include <linux/eventfd.h>
+#include <linux/notifier.h>
 
 #include "vfio_ap_matrix_private.h"
 
@@ -24,6 +27,8 @@ struct ap_matrix_mdev {
 	struct mdev_device *mdev;
 	struct list_head node;
 	struct ap_config_masks masks;
+	struct notifier_block group_notifier;
+	struct kvm *kvm;
 };
 
 struct ap_matrix	*matrix;
@@ -97,6 +102,41 @@ static int ap_matrix_mdev_remove(struct mdev_device *mdev)
 	return 0;
 }
 
+static int ap_matrix_mdev_group_notifier(struct notifier_block *nb,
+					 unsigned long action, void *data)
+{
+	struct ap_matrix_mdev *matrix_mdev;
+
+	matrix_mdev = container_of(nb, struct ap_matrix_mdev, group_notifier);
+
+	if (action == VFIO_GROUP_NOTIFY_SET_KVM)
+		matrix_mdev->kvm = data;
+
+	return NOTIFY_OK;
+}
+
+static int ap_matrix_mdev_open(struct mdev_device *mdev)
+{
+	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+	unsigned long events;
+	int ret;
+
+	matrix_mdev->group_notifier.notifier_call =
+		ap_matrix_mdev_group_notifier;
+	events = VFIO_GROUP_NOTIFY_SET_KVM;
+	ret = vfio_register_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
+				     &events, &matrix_mdev->group_notifier);
+	return ret;
+}
+
+static void ap_matrix_mdev_release(struct mdev_device *mdev)
+{
+	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+	vfio_unregister_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
+				 &matrix_mdev->group_notifier);
+}
+
 static ssize_t name_show(struct kobject *kobj, struct device *dev, char *buf)
 {
 	return sprintf(buf, "%s\n", AP_MATRIX_MDEV_NAME_HWVIRT);
@@ -609,6 +649,9 @@ static DEVICE_ATTR(control_domains, 0644, ap_matrix_control_domains_show,
 	.mdev_attr_groups	= ap_matrix_mdev_attr_groups,
 	.create                 = ap_matrix_mdev_create,
 	.remove                 = ap_matrix_mdev_remove,
+	.open                   = ap_matrix_mdev_open,
+	.release                = ap_matrix_mdev_release,
+
 };
 
 int ap_matrix_mdev_register(struct ap_matrix *ap_matrix)
-- 
1.7.1

[RFC 15/19] s390/zcrypt: introduce ioctl access to VFIO AP Matrix driver

[Tony Krowiak]

Introduces ioctl access to the VFIO AP Matrix device driver
by implementing the VFIO_DEVICE_GET_INFO ioctl. This ioctl
provides the VFIO AP Matrix device driver information to the
guest machine.

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 drivers/s390/crypto/vfio_ap_matrix_ops.c |   43 ++++++++++++++++++++++++++++++
 include/uapi/linux/vfio.h                |    1 +
 2 files changed, 44 insertions(+), 0 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_matrix_ops.c b/drivers/s390/crypto/vfio_ap_matrix_ops.c
index fd36074..74ea8db 100644
--- a/drivers/s390/crypto/vfio_ap_matrix_ops.c
+++ b/drivers/s390/crypto/vfio_ap_matrix_ops.c
@@ -137,6 +137,48 @@ static void ap_matrix_mdev_release(struct mdev_device *mdev)
 				 &matrix_mdev->group_notifier);
 }
 
+static int ap_matrix_mdev_get_device_info(unsigned long arg)
+{
+	unsigned long minsz;
+	struct vfio_device_info info;
+
+	minsz = offsetofend(struct vfio_device_info, num_irqs);
+
+	if (copy_from_user(&info, (void __user *)arg, minsz))
+		return -EFAULT;
+
+	if (info.argsz < minsz) {
+		pr_err("%s: Argument size %u less than min size %li",
+		       VFIO_AP_MATRIX_MODULE_NAME, info.argsz, minsz);
+		return -EINVAL;
+	}
+
+	info.flags = VFIO_DEVICE_FLAGS_AP_MATRIX;
+	info.num_regions = 0;
+	info.num_irqs = 0;
+
+	return copy_to_user((void __user *)arg, &info, minsz);
+}
+
+static ssize_t ap_matrix_mdev_ioctl(struct mdev_device *mdev,
+				    unsigned int cmd, unsigned long arg)
+{
+	int ret;
+
+	switch (cmd) {
+	case VFIO_DEVICE_GET_INFO:
+		ret = ap_matrix_mdev_get_device_info(arg);
+		break;
+	default:
+		pr_err("%s: ioctl command %d is not a supported command",
+		       VFIO_AP_MATRIX_MODULE_NAME, cmd);
+		ret = -EOPNOTSUPP;
+		break;
+	}
+
+	return ret;
+}
+
 static ssize_t name_show(struct kobject *kobj, struct device *dev, char *buf)
 {
 	return sprintf(buf, "%s\n", AP_MATRIX_MDEV_NAME_HWVIRT);
@@ -651,6 +693,7 @@ static DEVICE_ATTR(control_domains, 0644, ap_matrix_control_domains_show,
 	.remove                 = ap_matrix_mdev_remove,
 	.open                   = ap_matrix_mdev_open,
 	.release                = ap_matrix_mdev_release,
+	.ioctl                  = ap_matrix_mdev_ioctl,
 
 };
 
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index 743456f..08f1ab4 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -199,6 +199,7 @@ struct vfio_device_info {
 #define VFIO_DEVICE_FLAGS_PLATFORM (1 << 2)	/* vfio-platform device */
 #define VFIO_DEVICE_FLAGS_AMBA  (1 << 3)	/* vfio-amba device */
 #define VFIO_DEVICE_FLAGS_CCW	(1 << 4)	/* vfio-ccw device */
+#define VFIO_DEVICE_FLAGS_AP_MATRIX (1 << 5)	/* vfio-ap-matrix device */
 	__u32	num_regions;	/* Max region index + 1 */
 	__u32	num_irqs;	/* Max IRQ index + 1 */
 };
-- 
1.7.1

[RFC 16/19] KVM: s390: interface to configure KVM guest's AP matrix

[Tony Krowiak]

Provides an interface to assign AP adapters, usage domains
and control domains to a KVM guest.

A KVM guest is started by executing the Start Interpretive Execution (SIE)
instruction. The SIE state description is a control block that contains the
state information for a KVM guest and is supplied as input to the SIE
instruction. The SIE state description contains a field that references
a Crypto Control Block (CRYCB). The CRYCB contains three bitmask fields
identifying the adapters, usage domains and control domains assigned to the
KVM guest:

* The AP Adapter Matrix (APM) field identifies the AP adapters assigned to
  the KVM guest

* The AP Queue Matrix (AQM) field identifies the usage domains assigned to
  the KVM guest

* The AP Domain matrix (ADM) field identifies the control domains
  assigned to the KVM guest.

Each adapter, usage domain and control domain are identified by a number
from 0 to 255. The bits in each mask, from left to right, correspond to
the numbers 0-255. When a bit is set, the corresponding adapter,
usage domain or control domain will be assigned to the KVM guest.

This patch will set the bits in the APM, AQM and ADM fields of the
CRYCB referenced by the KVM guest's SIE state description. The process
used is:

1. Perform a logical AND of the AP matrix masks configured for the
   mediated AP matrix device via its sysfs attributes files with
   the matrix masks assigned to the LPAR in which the host linux
   system is running to create the effective masks, EAPM, EAQM and
   EADM.

2. Set the APM, AQM and ADM in the CRYCB from the EAPM, EAQM and
   EADM calculated in step 1.

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 arch/s390/include/asm/ap-config.h |    7 ++
 arch/s390/kvm/ap-config.c         |  144 +++++++++++++++++++++++++++++++++++++
 2 files changed, 151 insertions(+), 0 deletions(-)

diff --git a/arch/s390/include/asm/ap-config.h b/arch/s390/include/asm/ap-config.h
index 3064215..866f008 100644
--- a/arch/s390/include/asm/ap-config.h
+++ b/arch/s390/include/asm/ap-config.h
@@ -10,16 +10,23 @@
 #define _ASM_KVM_AP_CONFIG_H_
 
 #include <linux/types.h>
+#include <linux/kvm_host.h>
 
 #define AP_MATRIX_MAX_MASK_BITS		256
 #define AP_MATRIX_MASK_INDICES		(AP_MATRIX_MAX_MASK_BITS / \
 					(sizeof(u64) * 8))
 #define AP_MATRIX_MAX_MASK_BYTES	(AP_MATRIX_MASK_INDICES * sizeof(u64))
 
+#define AP_MATRIX_MASK_TYPE_ADAPTER	"adapter"
+#define AP_MATRIX_MASK_TYPE_DOMAIN	"domain"
+#define AP_MATRIX_MASK_TYPE_CONTROL	"control domain"
+
 struct ap_config_masks {
 	u64 apm[AP_MATRIX_MASK_INDICES];
 	u64 aqm[AP_MATRIX_MASK_INDICES];
 	u64 adm[AP_MATRIX_MASK_INDICES];
 };
 
+extern int ap_config_matrix(struct kvm *kvm, struct ap_config_masks *masks);
+
 #endif /* _ASM_KVM_AP_CONFIG_H_ */
diff --git a/arch/s390/kvm/ap-config.c b/arch/s390/kvm/ap-config.c
index 84fdf43..dc79798 100644
--- a/arch/s390/kvm/ap-config.c
+++ b/arch/s390/kvm/ap-config.c
@@ -7,3 +7,147 @@
  */
 
 #include <asm/ap-config.h>
+#include <asm/ap.h>
+#include <linux/bitops.h>
+
+static inline int is_format2_crycb(struct kvm *kvm)
+{
+	int fmt2_mask = kvm->arch.crypto.crycbd & CRYCB_FORMAT2;
+
+	return (fmt2_mask == CRYCB_FORMAT2);
+}
+
+static inline u64 *ap_config_get_crycb_apm(struct kvm *kvm)
+{
+	u64 *apm;
+
+	if (is_format2_crycb(kvm))
+		apm = kvm->arch.crypto.crycb->apcb1.apm;
+	else
+		apm = kvm->arch.crypto.crycb->apcb0.apm;
+
+	return apm;
+}
+
+static inline u64 *ap_config_get_crycb_aqm(struct kvm *kvm)
+{
+	u64 *aqm;
+
+	if (is_format2_crycb(kvm))
+		aqm = kvm->arch.crypto.crycb->apcb1.aqm;
+	else
+		aqm = kvm->arch.crypto.crycb->apcb0.aqm;
+
+	return aqm;
+}
+
+static inline u64 *ap_config_get_crycb_adm(struct kvm *kvm)
+{
+	u64 *adm;
+
+	if (is_format2_crycb(kvm))
+		adm = kvm->arch.crypto.crycb->apcb1.adm;
+	else
+		adm = kvm->arch.crypto.crycb->apcb0.adm;
+
+	return adm;
+}
+
+static void ap_config_set_crycb_masks(struct kvm *kvm,
+				    struct ap_config_masks *masks)
+{
+	size_t i;
+	size_t masksz;
+	u64 *mask = ap_config_get_crycb_apm(kvm);
+
+	masksz = (is_format2_crycb(kvm)) ? APCB1_MASK_SIZE : APCB0_MASK_SIZE;
+
+	for (i = 0; i < masksz; i++)
+		mask[i] = masks->apm[i];
+
+	mask = ap_config_get_crycb_aqm(kvm);
+
+	for (i = 0; i < masksz; i++)
+		mask[i] = masks->aqm[i];
+
+	mask = ap_config_get_crycb_adm(kvm);
+
+	for (i = 0; i < masksz; i++)
+		mask[i] = masks->adm[i];
+}
+
+static int ap_config_set_emask(const char *mask_type, unsigned long *mask,
+			       unsigned long *cfgmask)
+{
+	unsigned long id;
+	unsigned long nbits = AP_MATRIX_MAX_MASK_BITS;
+
+	id = find_first_bit_inv(mask, nbits);
+	while (id < nbits) {
+		if (!test_bit_inv(id, cfgmask)) {
+			clear_bit_inv(id, mask);
+			pr_err("%s: %s %02lx is not installed on the host system",
+			       __func__, mask_type, id);
+			return -ENODEV;
+		}
+
+		id = find_next_bit_inv(mask, nbits, id + 1);
+	}
+
+	return 0;
+}
+
+static int ap_config_get_emasks(struct ap_config_masks *masks)
+{
+	int ret;
+	struct ap_config_info config;
+
+	ret = ap_query_configuration(&config);
+	if (ret) {
+		if (ret == -EOPNOTSUPP) {
+			pr_err("%s: Query AP configuration not supported",
+			       __func__);
+
+			return ret;
+		}
+
+		pr_err("%s: Query AP configuration failed with rc=%d",
+		       __func__, ret);
+
+		return ret;
+	}
+
+	ret = ap_config_set_emask(AP_MATRIX_MASK_TYPE_ADAPTER,
+				  (unsigned long *)masks->apm,
+				  (unsigned long *)config.apm);
+	if (ret)
+		return ret;
+
+	ret = ap_config_set_emask(AP_MATRIX_MASK_TYPE_DOMAIN,
+				  (unsigned long *)masks->aqm,
+				  (unsigned long *)config.aqm);
+	if (ret)
+		return ret;
+
+	ret = ap_config_set_emask(AP_MATRIX_MASK_TYPE_CONTROL,
+				  (unsigned long *)masks->adm,
+				  (unsigned long *)config.adm);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+int ap_config_matrix(struct kvm *kvm, struct ap_config_masks *masks)
+{
+	int ret;
+
+	ret = ap_config_get_emasks(masks);
+	if (ret)
+		return ret;
+
+	ap_config_set_crycb_masks(kvm, masks);
+
+	return 0;
+}
+EXPORT_SYMBOL(ap_config_matrix);
-- 
1.7.1

[RFC 17/19] KVM: s390: validate input to AP matrix config interface

[Tony Krowiak]

Verifies that the AP matrix assigned to the KVM guest is
not shared by any other KVM guest running on the same
system.

The Crypto Control Block referenced by a KVM guest's SIE
state description contains two bit mask fields that identify
the AP adapters and usage domains to assigned to the
guest: The AP Matrix (APM) identifies the AP adapters assigned
to the KVM guest; and the AP Queue Matrix (AQM) identifies
the usage domains assigned to the KVM guest. Each adapter and
usage domain is identified by a number from 0 to 255. The
bits in each mask, from left to right, correspond to the
numbers 0-255. When a bit is set, the corresponding adapter
or usage domain is assigned to the KVM guest.

AP instructions identify the AP device to use to perform the
cryptographic function contained in the instruction's
payload. The AP device is identified by and AP Queue
Number (APQN). The APQN is comprised of two fields: The
AP Identifier (APID) that specifies an adapter ID; and
an AP Queue Identifier (APQI) that specifies a domain ID.

The bits in the APM and AQM fields of the KVM guest's CRYCB
specify the list of APQNs that are valid for instructions
submitted from the KVM guest. When an AP instruction is
executed by the KVM guest, if the bit in the APM corresponding
to the APID contained in the APQN specified in the AP
instruction is not set, the instruction will fail. Likewise,
if the bit in the AQM corresponding to the APQI contained in
the APQN specified in the AP instruction is not set, the
instruction will fail.

The APQNs that can be derived from the bits set in the
APM and AQM fields of the KVM guest's CRYCB must not be
available to any other KVM guest running on the same
system. If any APQN is not unique to the KVM guest,
the ioctl will fail.

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 arch/s390/kvm/ap-config.c |   71 +++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 71 insertions(+), 0 deletions(-)

diff --git a/arch/s390/kvm/ap-config.c b/arch/s390/kvm/ap-config.c
index dc79798..4b0794d 100644
--- a/arch/s390/kvm/ap-config.c
+++ b/arch/s390/kvm/ap-config.c
@@ -138,6 +138,73 @@ static int ap_config_get_emasks(struct ap_config_masks *masks)
 	return 0;
 }
 
+static unsigned long ap_config_get_num_mask_bits(struct kvm *kvm)
+{
+	return is_format2_crycb(kvm) ? sizeof(u64) * APCB1_MASK_SIZE * 8 :
+				       sizeof(u64) * APCB0_MASK_SIZE * 8;
+}
+
+static int ap_config_validate_queue(struct kvm *kvm, unsigned long apid,
+				    unsigned long apqi)
+{
+	int ret = 0;
+	struct kvm *vm;
+	u64 *mask;
+
+	mutex_lock(&kvm->lock);
+
+	/* No other VM may share an AP Queue with the input VM */
+	list_for_each_entry(vm, &vm_list, vm_list) {
+		if (kvm == vm)
+			continue;
+
+		mask = ap_config_get_crycb_apm(vm);
+		if (!test_bit_inv(apid, (unsigned long *)mask))
+			continue;
+
+		mask = ap_config_get_crycb_aqm(vm);
+		if (!test_bit_inv(apqi, (unsigned long *)mask))
+			continue;
+
+		pr_err("%s: AP queue %02lx.%04lx is already registered to %s",
+		       __func__, apid, apqi, kvm->arch.dbf->name);
+		ret = -EBUSY;
+
+		goto done;
+	}
+
+done:
+	mutex_unlock(&kvm->lock);
+	return ret;
+}
+
+static int ap_config_validate_queues(struct kvm *kvm,
+				     struct ap_config_masks *masks)
+{
+	int ret;
+	const unsigned long *apm = (unsigned long *)masks->apm;
+	const unsigned long *aqm = (unsigned long *)masks->aqm;
+	unsigned long nbits = ap_config_get_num_mask_bits(kvm);
+	unsigned long apid;
+	unsigned long apqi;
+
+	apid = find_first_bit_inv(apm, nbits);
+	while (apid < nbits) {
+		apqi = find_first_bit_inv(aqm, nbits);
+		while (apqi < nbits) {
+			ret = ap_config_validate_queue(kvm, apid, apqi);
+			if (ret)
+				return ret;
+
+			apqi = find_next_bit_inv(aqm, nbits, apqi + 1);
+		}
+
+		apid = find_next_bit_inv(apm, nbits, apid + 1);
+	}
+
+	return 0;
+}
+
 int ap_config_matrix(struct kvm *kvm, struct ap_config_masks *masks)
 {
 	int ret;
@@ -146,6 +213,10 @@ int ap_config_matrix(struct kvm *kvm, struct ap_config_masks *masks)
 	if (ret)
 		return ret;
 
+	ret = ap_config_validate_queues(kvm, masks);
+	if (ret)
+		return ret;
+
 	ap_config_set_crycb_masks(kvm, masks);
 
 	return 0;
-- 
1.7.1

[RFC 18/19] KVM: s390: New ioctl to configure KVM guest's AP matrix

[Tony Krowiak]

Implements an ioctl to configure the adapters, usage domains
and control domains to which a KVM guest will be granted
access. The ioctl is invoked using the VFIO mediated matrix
device's file descriptor.

Guest access to AP adapters, usage domains and control domains
is controlled by three bit masks referenced from the
guest's SIE state description: The AP Mask (APM) controls
access to the AP adapters; the AP Queue Mask (AQM) controls
access to the AP usage domains; and the AP Domain Mask (ADM)
controls access to the AP control domains. Each bit
in the APM represents an adapter, from left to right,
starting with adapter 0. Each bit in the AQM represents
a usage domain, from left to right, starting with domain 0.
Each bit in the ADM represents a control domain, from left
to right, starting with domain 0.

The APM, AQM and ADM bit masks will be populated from the
bit masks stored in the corresponding mediated matrix device's
sysfs files. The bit masks will also be returned by the ioctl
call to the caller.

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 drivers/s390/crypto/vfio_ap_matrix_ops.c |   68 +++++++++++++++++++++++++++++-
 include/uapi/linux/vfio.h                |   20 +++++++++
 2 files changed, 86 insertions(+), 2 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_matrix_ops.c b/drivers/s390/crypto/vfio_ap_matrix_ops.c
index 74ea8db..d81e3ab 100644
--- a/drivers/s390/crypto/vfio_ap_matrix_ops.c
+++ b/drivers/s390/crypto/vfio_ap_matrix_ops.c
@@ -160,8 +160,69 @@ static int ap_matrix_mdev_get_device_info(unsigned long arg)
 	return copy_to_user((void __user *)arg, &info, minsz);
 }
 
-static ssize_t ap_matrix_mdev_ioctl(struct mdev_device *mdev,
-				    unsigned int cmd, unsigned long arg)
+static int ap_matrix_configure(struct mdev_device *mdev)
+{
+	int ret;
+	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+	ret = ap_config_matrix(matrix_mdev->kvm, &matrix_mdev->masks);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+static int ap_matrix_copy_to_user(struct mdev_device *mdev,
+				  struct vfio_ap_matrix_config *mcfg,
+				  unsigned long arg)
+{
+	size_t i;
+	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+
+	for (i = 0; i < AP_MATRIX_MASK_INDICES; i++) {
+		if (i < VFIO_AP_MATRIX_MASK_INDICES) {
+			mcfg->apm[i] = matrix_mdev->masks.apm[i];
+			mcfg->aqm[i] = matrix_mdev->masks.aqm[i];
+			mcfg->adm[i] = matrix_mdev->masks.adm[i];
+		}
+	}
+
+	if (copy_to_user((void __user *)arg, &mcfg, sizeof(mcfg)))
+		return -EFAULT;
+
+	return 0;
+}
+
+static ssize_t ap_matrix_mdev_configure(struct mdev_device *mdev,
+					unsigned long arg)
+{
+	int ret;
+	unsigned long minsz;
+	struct vfio_ap_matrix_config mcfg;
+
+	minsz = offsetofend(struct vfio_ap_matrix_config, adm);
+
+	if (copy_from_user(&mcfg, (void __user *)arg, minsz))
+		return -EFAULT;
+
+	if (mcfg.argsz < minsz) {
+		pr_err("%s: Argument size %u less than min size %li",
+		       VFIO_AP_MATRIX_MODULE_NAME, mcfg.argsz, minsz);
+		return -EINVAL;
+	}
+
+	ret = ap_matrix_configure(mdev);
+	if (ret)
+		return ret;
+
+	ret = ap_matrix_copy_to_user(mdev, &mcfg, arg);
+
+	return 0;
+}
+
+static long ap_matrix_mdev_ioctl(struct mdev_device *mdev,
+				 unsigned int cmd, unsigned long arg)
 {
 	int ret;
 
@@ -169,6 +230,9 @@ static ssize_t ap_matrix_mdev_ioctl(struct mdev_device *mdev,
 	case VFIO_DEVICE_GET_INFO:
 		ret = ap_matrix_mdev_get_device_info(arg);
 		break;
+	case VFIO_AP_MATRIX_CONFIGURE:
+		ret = ap_matrix_mdev_configure(mdev, arg);
+		break;
 	default:
 		pr_err("%s: ioctl command %d is not a supported command",
 		       VFIO_AP_MATRIX_MODULE_NAME, cmd);
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index 08f1ab4..2d96c57 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -716,6 +716,26 @@ struct vfio_iommu_spapr_tce_remove {
 };
 #define VFIO_IOMMU_SPAPR_TCE_REMOVE	_IO(VFIO_TYPE, VFIO_BASE + 20)
 
+/**
+ * VFIO_AP_MATRIX_CONFIGURE		_IO(VFIO_TYPE, VFIO_BASE + 21
+ *
+ * Configure the AP matrix for a KVM guest.
+ */
+#define VFIO_AP_MATRIX_CONFIGURE	_IO(VFIO_TYPE, VFIO_BASE + 21)
+
+#define VFIO_AP_MATRIX_MASK_INDICES	4
+#define VFIO_AP_MATTRIX_MASK_BYTES	(VFIO_AP_MATRIX_MASK_INDICES * \
+					 sizeof(__u64))
+#define VFIO_AP_MATRIX_MASK_BITS	(VFIO_AP_MATTRIX_MASK_BYTES * 8)
+
+struct vfio_ap_matrix_config {
+	__u32 argsz;
+	__u32 flags;
+	/* out */
+	__u64 apm[VFIO_AP_MATRIX_MASK_INDICES];
+	__u64 aqm[VFIO_AP_MATRIX_MASK_INDICES];
+	__u64 adm[VFIO_AP_MATRIX_MASK_INDICES];
+};
 /* ***************************************************************** */
 
 #endif /* _UAPIVFIO_H */
-- 
1.7.1

[RFC 19/19] s390/facilities: enable AP facilities needed by guest

[Tony Krowiak]

Sets up the following facilities bits to enable the specified AP
facilities for the guest VM:
    * STFLE.12: Enables the AP Query Configuration Information
                facility. The AP bus running in the guest uses
                the information returned from this instruction
                to configure AP adapters and domains for the
                guest machine.
    * STFLE.15: Indicates the AP facilities test is available.
                The AP bus running in the guest uses the
                information.

Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
---
 arch/s390/tools/gen_facilities.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/arch/s390/tools/gen_facilities.c b/arch/s390/tools/gen_facilities.c
index 70dd8f1..eeaa7db 100644
--- a/arch/s390/tools/gen_facilities.c
+++ b/arch/s390/tools/gen_facilities.c
@@ -74,8 +74,10 @@ struct facility_def {
 			8,  /* enhanced-DAT 1 */
 			9,  /* sense-running-status */
 			10, /* conditional sske */
+			12, /* AP query configuration */
 			13, /* ipte-range */
 			14, /* nonquiescing key-setting */
+			15, /* AP special-command facility */
 			73, /* transactional execution */
 			75, /* access-exception-fetch/store indication */
 			76, /* msa extension 3 */
-- 
1.7.1

Re: [RFC 03/19] s390/zcrypt: new AP matrix bus

[Martin Schwidefsky]

On Fri, 13 Oct 2017 13:38:48 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> Introduces an AP matrix bus. The sysfs location of the
> AP matrix bus is:
> 
> /sys/bus/ap_matrix
> 
> The AP matrix bus will create an AP matrix device that will
> hold the AP queues reserved for use by KVM guests.
> 
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>  MAINTAINERS                         |    8 +++++
>  drivers/s390/crypto/Makefile        |    4 +-
>  drivers/s390/crypto/ap_matrix_bus.c |   52 +++++++++++++++++++++++++++++++++++
>  drivers/s390/crypto/ap_matrix_bus.h |   15 ++++++++++
>  4 files changed, 77 insertions(+), 2 deletions(-)
>  create mode 100644 drivers/s390/crypto/ap_matrix_bus.c
>  create mode 100644 drivers/s390/crypto/ap_matrix_bus.h
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index ef23cf5..cbd6f7c 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -11711,6 +11711,14 @@ W:	http://www.ibm.com/developerworks/linux/linux390/
>  S:	Supported
>  F:	drivers/s390/crypto/
> 
> +S390 VFIO AP MATRIX DRIVER
> +M:	Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> +L:	linux-s390@vger.kernel.org
> +W:	http://www.ibm.com/developerworks/linux/linux390/
> +S:	Supported
> +F:	drivers/s390/crypto/ap_matrix_bus.h
> +F:	drivers/s390/crypto/ap_matrix_bus.c
> +
>  S390 ZFCP DRIVER
>  M:	Steffen Maier <maier@linux.vnet.ibm.com>
>  M:	Benjamin Block <bblock@linux.vnet.ibm.com>
> diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile
> index be36f10..87646ca 100644
> --- a/drivers/s390/crypto/Makefile
> +++ b/drivers/s390/crypto/Makefile
> @@ -2,7 +2,7 @@
>  # S/390 crypto devices
>  #
> 
> -ap-objs := ap_bus.o ap_card.o ap_queue.o
> +ap-objs := ap_bus.o ap_card.o ap_queue.o ap_matrix_bus.o
>  obj-$(subst m,y,$(CONFIG_ZCRYPT)) += ap.o
>  # zcrypt_api.o and zcrypt_msgtype*.o depend on ap.o
>  zcrypt-objs := zcrypt_api.o zcrypt_card.o zcrypt_queue.o
> @@ -13,4 +13,4 @@ obj-$(CONFIG_ZCRYPT) += zcrypt_pcixcc.o zcrypt_cex2a.o zcrypt_cex4.o
> 
>  # pkey kernel module
>  pkey-objs := pkey_api.o
> -obj-$(CONFIG_PKEY) += pkey.o
> +obj-$(CONFIG_PKEY) += pkey.o
> \ No newline at end of file

Try to avoid this newline hunk please. It is removed again with patch #5.

> diff --git a/drivers/s390/crypto/ap_matrix_bus.c b/drivers/s390/crypto/ap_matrix_bus.c
> new file mode 100644
> index 0000000..fbae175
> --- /dev/null
> +++ b/drivers/s390/crypto/ap_matrix_bus.c
> @@ -0,0 +1,52 @@
> +/*
> + * Copyright IBM Corp. 2017
> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> + *
> + * Adjunct processor matrix bus.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +
> +#include <linux/module.h>
> +#include <linux/device.h>
> +#include <asm/ap.h>
> +
> +#include "ap_matrix_bus.h"
> +
> +MODULE_AUTHOR("Tony Krowiak <akrowiak@linux.vnet.ibm.com");
> +MODULE_DESCRIPTION("AP Matrix, Copyright IBM Corp. 2017");
> +MODULE_LICENSE("GPL v2");
> +
> +#define AP_MATRIX_BUS_NAME "ap_matrix"
> +
> +static struct device *ap_matrix_root_device;
> +
> +static struct bus_type ap_matrix_bus_type = {
> +	.name = AP_MATRIX_BUS_NAME,
> +};
> +
> +int __init ap_matrix_init(void)
> +{
> +	int ret;
> +
> +	ap_matrix_root_device = root_device_register(AP_MATRIX_BUS_NAME);
> +	ret = PTR_RET(ap_matrix_root_device);
> +	if (ret)
> +		goto done;
> +
> +	ret = bus_register(&ap_matrix_bus_type);
> +	if (ret)
> +		goto bus_reg_err;
> +
> +	return 0;
> +
> +bus_reg_err:
> +	root_device_unregister(ap_matrix_root_device);
> +
> +done:
> +	return ret;
> +}
> +device_initcall(ap_matrix_init);
> diff --git a/drivers/s390/crypto/ap_matrix_bus.h b/drivers/s390/crypto/ap_matrix_bus.h
> new file mode 100644
> index 0000000..58e116f
> --- /dev/null
> +++ b/drivers/s390/crypto/ap_matrix_bus.h
> @@ -0,0 +1,15 @@
> +/*
> + * Adjunct processor matrix bus header file
> + *
> + * Copyright IBM Corp. 2017
> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> + *
> + * Adjunct processor bus header file.
> + */
> +
> +#ifndef _AP_MATRIX_BUS_H_
> +#define _AP_MATRIX_BUS_H_
> +
> +int ap_matrix_init(void);
> +
> +#endif /* _AP_MATRIX_BUS_H_ */

While I understand the approach to split the patches into meaningful
parts, there is always a balance between too coarse and too fine granular.

I would merge patch #3, patch #4 and parts of patch #5 that affect
ap_matrix_bus.c & ap_matrix_bus.h.


-- 
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.

Re: [RFC 05/19] s390/zcrypt: base implementation of AP matrix device driver

[Martin Schwidefsky]

On Fri, 13 Oct 2017 13:38:50 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile
> index 87646ca..1983afa 100644
> --- a/drivers/s390/crypto/Makefile
> +++ b/drivers/s390/crypto/Makefile
> @@ -13,4 +13,8 @@ obj-$(CONFIG_ZCRYPT) += zcrypt_pcixcc.o zcrypt_cex2a.o zcrypt_cex4.o
> 
>  # pkey kernel module
>  pkey-objs := pkey_api.o
> -obj-$(CONFIG_PKEY) += pkey.o
> \ No newline at end of file
> +obj-$(CONFIG_PKEY) += pkey.o
> +
> +# adjunct processor matrix
> +vfio_ap_matrix-objs := vfio_ap_matrix_drv.o
> +obj-$(CONFIG_VFIO_AP_MATRIX) += vfio_ap_matrix.o

With the fix to patch #3 the newline quirk will be gone.

> diff --git a/drivers/s390/crypto/ap_matrix_bus.c b/drivers/s390/crypto/ap_matrix_bus.c
> index 4eb1e3c..66bfa54 100644
> --- a/drivers/s390/crypto/ap_matrix_bus.c
> +++ b/drivers/s390/crypto/ap_matrix_bus.c
> @@ -75,10 +75,18 @@ static int ap_matrix_dev_create(void)
>  	return 0;
>  }
> 
> +struct ap_matrix *ap_matrix_get_device(void)
> +{
> +	return matrix;
> +}
> +EXPORT_SYMBOL(ap_matrix_get_device);
> +
>  int __init ap_matrix_init(void)
>  {
>  	int ret;
> 
> +	matrix = NULL;
> +
>  	ap_matrix_root_device = root_device_register(AP_MATRIX_BUS_NAME);
>  	ret = PTR_RET(ap_matrix_root_device);
>  	if (ret)

The global variable "matrix" is already NULL at the time ap_matrix_init is
called. No need to set it again. And I would appreciate if the global
variable would be named "ap_matrix".

> diff --git a/drivers/s390/crypto/ap_matrix_bus.h b/drivers/s390/crypto/ap_matrix_bus.h
> index 225db4f..c2aff23 100644
> --- a/drivers/s390/crypto/ap_matrix_bus.h
> +++ b/drivers/s390/crypto/ap_matrix_bus.h
> @@ -16,6 +16,6 @@ struct ap_matrix {
>  	struct device device;
>  };
> 
> -int ap_matrix_init(void);
> +struct ap_matrix *ap_matrix_get_device(void);
> 
>  #endif /* _AP_MATRIX_BUS_H_ */
> diff --git a/drivers/s390/crypto/vfio_ap_matrix_drv.c b/drivers/s390/crypto/vfio_ap_matrix_drv.c
> new file mode 100644
> index 0000000..760ed56
> --- /dev/null
> +++ b/drivers/s390/crypto/vfio_ap_matrix_drv.c
> @@ -0,0 +1,102 @@
> +/*
> + * VFIO based AP Matrix device driver
> + *
> + * Copyright IBM Corp. 2017
> + *
> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> + */
> +
> +#include <linux/module.h>
> +#include <linux/mod_devicetable.h>
> +#include <linux/slab.h>
> +
> +#include "ap_bus.h"
> +#include "ap_matrix_bus.h"
> +#include "vfio_ap_matrix_private.h"
> +
> +#define VFIO_AP_MATRIX_DRV_NAME "vfio_ap_queue"
> +
> +MODULE_AUTHOR("IBM Corporation");
> +MODULE_DESCRIPTION("AP Matrix device driver, Copyright IBM Corp. 2017");
> +MODULE_LICENSE("GPL v2");
> +
> +static struct ap_device_id ap_queue_ids[] = {
> +	{ .dev_type = AP_DEVICE_TYPE_CEX4,
> +	  .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
> +	{ .dev_type = AP_DEVICE_TYPE_CEX5,
> +	  .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
> +	{ .dev_type = AP_DEVICE_TYPE_CEX6,
> +	  .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
> +	{ /* end of list */ },
> +};
> +
> +MODULE_DEVICE_TABLE(ap_matrix, ap_queue_ids);
> +
> +static struct ap_matrix_driver {
> +	struct ap_driver ap_drv;
> +	struct ap_matrix *ap_matrix;
> +} vfio_ap_matrix_drv;
> +
> +static int ap_matrix_queue_dev_probe(struct ap_device *apdev)
> +{
> +	struct vfio_ap_queue *vapq;
> +	struct ap_queue *apq = to_ap_queue(&apdev->device);
> +	struct ap_matrix *ap_matrix = vfio_ap_matrix_drv.ap_matrix;
> +
> +	vapq = kzalloc(sizeof(*vapq), GFP_KERNEL);
> +	if (!vapq)
> +		return -ENOMEM;
> +
> +	INIT_LIST_HEAD(&vapq->list);
> +	vapq->queue = apq;
> +	spin_lock_bh(&ap_matrix->qlock);
> +	list_add_tail(&vapq->list, &ap_matrix->queues);
> +	spin_unlock_bh(&ap_matrix->qlock);
> +
> +	return 0;
> +}
> +
> +static void ap_matrix_queue_dev_remove(struct ap_device *apdev)
> +{
> +	struct vfio_ap_queue *vapq;
> +	struct ap_queue *apq = to_ap_queue(&apdev->device);
> +	struct ap_matrix *ap_matrix = vfio_ap_matrix_drv.ap_matrix;
> +
> +	vapq = find_vapq(ap_matrix, apq->qid);
> +
> +	if (vapq) {
> +		spin_lock_bh(&ap_matrix->qlock);
> +		list_del_init(&vapq->list);
> +		spin_unlock_bh(&ap_matrix->qlock);
> +		kfree(vapq);
> +	}
> +}

I would opt for a quick exit if find_vapq can not find the device.
The un-indent the actual remove code.

> +
> +int __init ap_matrix_init(void)
> +{
> +
> +	int ret;
> +
> +	vfio_ap_matrix_drv.ap_matrix = ap_matrix_get_device();
> +	if (!vfio_ap_matrix_drv.ap_matrix)
> +		return -ENODEV;
> +
> +	vfio_ap_matrix_drv.ap_drv.probe = ap_matrix_queue_dev_probe;
> +	vfio_ap_matrix_drv.ap_drv.remove = ap_matrix_queue_dev_remove;
> +	vfio_ap_matrix_drv.ap_drv.ids = ap_queue_ids;
> +
> +	ret = ap_driver_register(&vfio_ap_matrix_drv.ap_drv,
> +				 THIS_MODULE, VFIO_AP_MATRIX_DRV_NAME);
> +	if (ret)
> +		return ret;
> +
> +	return ret;
> +}
> +
> +void __exit ap_matrix_exit(void)
> +{
> +	ap_driver_unregister(&vfio_ap_matrix_drv.ap_drv);
> +}
> +
> +module_init(ap_matrix_init);
> +module_exit(ap_matrix_exit);
> diff --git a/drivers/s390/crypto/vfio_ap_matrix_private.h b/drivers/s390/crypto/vfio_ap_matrix_private.h
> new file mode 100644
> index 0000000..11c5e02
> --- /dev/null
> +++ b/drivers/s390/crypto/vfio_ap_matrix_private.h
> @@ -0,0 +1,47 @@
> +/*
> + * Private data and functions for adjunct processor VFIO matrix driver.
> + *
> + * Copyright IBM Corp. 2016
> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> + */
> +
> +#ifndef _VFIO_AP_PRIVATE_H_
> +#define _VFIO_AP_PRIVATE_H_
> +
> +#include <linux/types.h>
> +
> +#include "ap_bus.h"
> +#include "ap_matrix_bus.h"
> +
> +#define VFIO_AP_MATRIX_MODULE_NAME "vfio_ap_matrix"
> +
> +struct vfio_ap_queue {
> +	struct ap_queue *queue;
> +	struct list_head list;
> +};
> +
> +static inline struct vfio_ap_queue *to_vapq(struct ap_device *ap_dev)
> +{
> +	struct ap_queue *ap_queue = to_ap_queue(&ap_dev->device);
> +	struct vfio_ap_queue *vapq;
> +
> +	vapq = container_of(&ap_queue, struct vfio_ap_queue, queue);
> +
> +	return vapq;
> +}

This can be shortened a bit

	return container_of(&ap_queue, struct vfio_ap_queue, queue);

> +
> +static inline struct vfio_ap_queue *find_vapq(struct ap_matrix *ap_matrix,
> +					      ap_qid_t qid)
> +{
> +	struct vfio_ap_queue *vapq;
> +
> +	if (!list_empty(&ap_matrix->queues)) {
> +		list_for_each_entry(vapq, &ap_matrix->queues, list)
> +			if (vapq->queue->qid == qid)
> +				return vapq;
> +	}
> +
> +	return NULL;
> +}
> +
> +#endif /* _VFIO_AP_PRIVATE_H_ */


-- 
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.

Re: [RFC 06/19] s390/zcrypt: register matrix device with VFIO mediated device framework

[Martin Schwidefsky]

On Fri, 13 Oct 2017 13:38:51 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> diff --git a/drivers/s390/crypto/ap_matrix_bus.c b/drivers/s390/crypto/ap_matrix_bus.c
> index 66bfa54..418c23b 100644
> --- a/drivers/s390/crypto/ap_matrix_bus.c
> +++ b/drivers/s390/crypto/ap_matrix_bus.c
> @@ -61,6 +61,7 @@ static int ap_matrix_dev_create(void)
>  	matrix->device.bus = &ap_matrix_bus_type;
>  	matrix->device.parent = ap_matrix_root_device;
>  	matrix->device.release = ap_matrix_dev_release;
> +	INIT_LIST_HEAD(&matrix->queues);
> 
>  	ret = device_register(&matrix->device);
>  	if (ret) {
> diff --git a/drivers/s390/crypto/ap_matrix_bus.h b/drivers/s390/crypto/ap_matrix_bus.h
> index c2aff23..3eccc36 100644
> --- a/drivers/s390/crypto/ap_matrix_bus.h
> +++ b/drivers/s390/crypto/ap_matrix_bus.h
> @@ -12,8 +12,12 @@
> 
>  #include <linux/device.h>
> 
> +#include "ap_bus.h"
> +
>  struct ap_matrix {
>  	struct device device;
> +	spinlock_t qlock;
> +	struct list_head queues;
>  };
> 
>  struct ap_matrix *ap_matrix_get_device(void);

Move these two hunks into patch #5 please. Yes, strictly speaking the two elements
in the struct ap_matrix are needed only with patch #6, but it is fine to introduce
an element with a new driver that is only exploited with a later patch.

-- 
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.

Re: [RFC 07/19] KVM: s390: introduce AP matrix configuration interface

[Martin Schwidefsky]

On Fri, 13 Oct 2017 13:38:52 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> This patch introduces a new compilation unit that will contain
> all of the data structures and logic for configuring AP adapters,
> usage domains and control domains for a KVM guest.
> 
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>  MAINTAINERS                       |    2 ++
>  arch/s390/include/asm/ap-config.h |   12 ++++++++++++
>  arch/s390/kvm/Makefile            |    2 +-
>  arch/s390/kvm/ap-config.c         |    9 +++++++++
>  4 files changed, 24 insertions(+), 1 deletions(-)
>  create mode 100644 arch/s390/include/asm/ap-config.h
>  create mode 100644 arch/s390/kvm/ap-config.c
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 0c8855a..4de0904 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -11721,6 +11721,8 @@ F:	drivers/s390/crypto/ap_matrix_bus.c
>  F:	drivers/s390/crypto/vfio_ap_matrix_drv.c
>  F:	drivers/s390/crypto/vfio_ap_matrix_private.h
>  F:	drivers/s390/crypto/vfio_ap_matrix_ops.c
> +F:	arch/s390/include/asm/kvm/ap-config.h
> +F:	arch/s390/kvm/ap-config.c
> 
>  S390 ZFCP DRIVER
>  M:	Steffen Maier <maier@linux.vnet.ibm.com>
> diff --git a/arch/s390/include/asm/ap-config.h b/arch/s390/include/asm/ap-config.h
> new file mode 100644
> index 0000000..8491b5f
> --- /dev/null
> +++ b/arch/s390/include/asm/ap-config.h
> @@ -0,0 +1,12 @@
> +/*
> + * Adjunct Processor (AP) configuration management for KVM guests
> + *
> + * Copyright IBM Corp. 2017
> + *
> + * Author(s): Tony Krowiak <akrowia@linux.vnet.ibm.com>
> + */
> +
> +#ifndef _ASM_KVM_AP_CONFIG_H_
> +#define _ASM_KVM_AP_CONFIG_H_
> +
> +#endif /* _ASM_KVM_AP_CONFIG_H_ */
> diff --git a/arch/s390/kvm/Makefile b/arch/s390/kvm/Makefile
> index 09a9e6d..0397b9c 100644
> --- a/arch/s390/kvm/Makefile
> +++ b/arch/s390/kvm/Makefile
> @@ -12,6 +12,6 @@ common-objs = $(KVM)/kvm_main.o $(KVM)/eventfd.o  $(KVM)/async_pf.o $(KVM)/irqch
>  ccflags-y := -Ivirt/kvm -Iarch/s390/kvm
> 
>  kvm-objs := $(common-objs) kvm-s390.o intercept.o interrupt.o priv.o sigp.o
> -kvm-objs += diag.o gaccess.o guestdbg.o sthyi.o vsie.o
> +kvm-objs += diag.o gaccess.o guestdbg.o sthyi.o vsie.o ap-config.o
> 
>  obj-$(CONFIG_KVM) += kvm.o
> diff --git a/arch/s390/kvm/ap-config.c b/arch/s390/kvm/ap-config.c
> new file mode 100644
> index 0000000..84fdf43
> --- /dev/null
> +++ b/arch/s390/kvm/ap-config.c
> @@ -0,0 +1,9 @@
> +/*
> + * Adjunct Processor (AP) configuration management for KVM guests
> + *
> + * Copyright IBM Corp. 2017
> + *
> + * Author(s): Tony Krowiak <akrowia@linux.vnet.ibm.com>
> + */
> +
> +#include <asm/ap-config.h>

Another patch that introduces empty files. Please don't do that. Merge
patch #7 with patch #16. With the dependency in patch #8 on the ap-config.h
header the new place for the combined patch would be prior to patch #8.

-- 
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.

Re: [RFC 13/19] s390/zcrypt: validate control domain assignment

[Martin Schwidefsky]

On Fri, 13 Oct 2017 13:38:58 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> The AP control domain being assigned to the mediated
> matrix driver must be assigned to the LPAR.
> 
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>  drivers/s390/crypto/vfio_ap_matrix_ops.c |   33 ++++++++++++++++++++++++++++++
>  1 files changed, 33 insertions(+), 0 deletions(-)
> 
> diff --git a/drivers/s390/crypto/vfio_ap_matrix_ops.c b/drivers/s390/crypto/vfio_ap_matrix_ops.c
> index 2387916..10a006c 100644
> --- a/drivers/s390/crypto/vfio_ap_matrix_ops.c
> +++ b/drivers/s390/crypto/vfio_ap_matrix_ops.c
> @@ -469,12 +469,41 @@ static ssize_t ap_matrix_domains_show(struct device *dev,
>  static DEVICE_ATTR(domains, 0644, ap_matrix_domains_show,
>  		   NULL);
> 
> +static int
> +ap_matrix_validate_control_domains(struct ap_matrix_mdev *matrix_mdev,
> +				   unsigned long id)
> +{
> +	int ret;
> +	struct ap_config_info config;
> +
> +	memset(&config, 0, sizeof(config));
> +
> +	ret = ap_query_configuration(&config);
> +	if (ret == -EOPNOTSUPP)
> +		return 0;
> +
> +	if (ret) {
> +		pr_err("%s: query AP configuration failed with rc %d",
> +		       VFIO_AP_MATRIX_MODULE_NAME, ret);
> +		return ret;
> +	}
> +
> +	if (!test_bit_inv(id, (unsigned long *)config.adm)) {
> +		pr_err("%s: control domain %04lx is not installed on the lpar",
> +		       VFIO_AP_MATRIX_MODULE_NAME, id);
> +		return -ENODEV;
> +	}
> +
> +	return 0;
> +}
> +
>  static ssize_t ap_matrix_control_domain_assign(struct device *dev,
>  					       struct device_attribute *attr,
>  					       const char *buf, size_t count)
>  {
>  	int ret;
>  	unsigned int id;
> +
>  	struct mdev_device *mdev = mdev_from_dev(dev);
>  	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
> 
> @@ -482,6 +511,10 @@ static ssize_t ap_matrix_control_domain_assign(struct device *dev,
>  	if (ret)
>  		return ret;
> 
> +	ret = ap_matrix_validate_control_domains(matrix_mdev, id);
> +	if (ret)
> +		return ret;
> +
>  	set_bit_inv((unsigned long)id,
>  		    (unsigned long *)matrix_mdev->masks.adm);
> 

Is there a specific reason why patch #13 is separate from patch #12 that
introduces the domain assignment? The assignment code does not make much
sense with the check if the domain is assigned to the LPAR, no?

-- 
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.

Re: [RFC 19/19] s390/facilities: enable AP facilities needed by guest

[Martin Schwidefsky]

On Fri, 13 Oct 2017 13:39:04 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> Sets up the following facilities bits to enable the specified AP
> facilities for the guest VM:
>     * STFLE.12: Enables the AP Query Configuration Information
>                 facility. The AP bus running in the guest uses
>                 the information returned from this instruction
>                 to configure AP adapters and domains for the
>                 guest machine.
>     * STFLE.15: Indicates the AP facilities test is available.
>                 The AP bus running in the guest uses the
>                 information.
> 
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>  arch/s390/tools/gen_facilities.c |    2 ++
>  1 files changed, 2 insertions(+), 0 deletions(-)
> 
> diff --git a/arch/s390/tools/gen_facilities.c b/arch/s390/tools/gen_facilities.c
> index 70dd8f1..eeaa7db 100644
> --- a/arch/s390/tools/gen_facilities.c
> +++ b/arch/s390/tools/gen_facilities.c
> @@ -74,8 +74,10 @@ struct facility_def {
>  			8,  /* enhanced-DAT 1 */
>  			9,  /* sense-running-status */
>  			10, /* conditional sske */
> +			12, /* AP query configuration */
>  			13, /* ipte-range */
>  			14, /* nonquiescing key-setting */
> +			15, /* AP special-command facility */
>  			73, /* transactional execution */
>  			75, /* access-exception-fetch/store indication */
>  			76, /* msa extension 3 */

With this all KVM guests will always have the AP instructions available, no?
In principles I like this approach, but it differs from the way z/VM does things,
there the guest will get an exception if it tries to execute an AP instruction
if there are no AP devices assigned to the guest. I wonder if there is a reason
why z/VM does it the way it does.

-- 
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Martin Schwidefsky]

On Fri, 13 Oct 2017 13:38:45 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> Tony Krowiak (19):
>   KVM: s390: SIE considerations for AP Queue virtualization
>   KVM: s390: refactor crypto initialization
>   s390/zcrypt: new AP matrix bus
>   s390/zcrypt: create an AP matrix device on the AP matrix bus
>   s390/zcrypt: base implementation of AP matrix device driver
>   s390/zcrypt: register matrix device with VFIO mediated device
>     framework
>   KVM: s390: introduce AP matrix configuration interface
>   s390/zcrypt: support for assigning adapters to matrix mdev
>   s390/zcrypt: validate adapter assignment
>   s390/zcrypt: sysfs interfaces supporting AP domain assignment
>   s390/zcrypt: validate domain assignment
>   s390/zcrypt: sysfs support for control domain assignment
>   s390/zcrypt: validate control domain assignment
>   KVM: s390: Connect the AP mediated matrix device to KVM
>   s390/zcrypt: introduce ioctl access to VFIO AP Matrix driver
>   KVM: s390: interface to configure KVM guest's AP matrix
>   KVM: s390: validate input to AP matrix config interface
>   KVM: s390: New ioctl to configure KVM guest's AP matrix
>   s390/facilities: enable AP facilities needed by guest

Overall I am quite happy with the patches, only minor things I would
like to see improved. We have to agree how we want to approach the
upstream process between my s390/linux tree and the s390/kvm tree.
Probably a tip branch again for the both tress to pull from.

-- 
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Cornelia Huck]

On Fri, 13 Oct 2017 13:38:45 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> Overview:
> --------
> An adjunct processor (AP) facility is an IBM Z cryptographic facility. The 
> AP facility is comprised of three AP instructions and from 1 to 256 AP 
> adapter cards. The design takes advantage of the interpretive execution mode 
> provided by the SIE architecture. With interpretive execution mode, the AP 
> instructions executed on the guest are interpreted by the hardware. This 
> allows guests direct access to AP adapter cards. The first goal of this 
> patch series is to provide direct access by a KVM guest to an AP as a 
> pass-through device. The second goal is to provide administrators with the
> means to configure KVM guests to grant direct access to AP facilities 
> assigned to the LPAR in which the host linux system is running. 
> 
> To facilitate the comprehension of the design, let's present an overview of
> the AP architecture.

I have not yet looked at the patches (and I'm not sure whether I'll be
able to do so until after the KVM Forum), but I think this also needs
some explanation under Documentation/, so people can still make sense
of it in the future.

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Christian Borntraeger]

On 10/16/2017 11:27 AM, Martin Schwidefsky wrote:
> On Fri, 13 Oct 2017 13:38:45 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> 
>> Tony Krowiak (19):
>>   KVM: s390: SIE considerations for AP Queue virtualization
>>   KVM: s390: refactor crypto initialization
>>   s390/zcrypt: new AP matrix bus
>>   s390/zcrypt: create an AP matrix device on the AP matrix bus
>>   s390/zcrypt: base implementation of AP matrix device driver
>>   s390/zcrypt: register matrix device with VFIO mediated device
>>     framework
>>   KVM: s390: introduce AP matrix configuration interface
>>   s390/zcrypt: support for assigning adapters to matrix mdev
>>   s390/zcrypt: validate adapter assignment
>>   s390/zcrypt: sysfs interfaces supporting AP domain assignment
>>   s390/zcrypt: validate domain assignment
>>   s390/zcrypt: sysfs support for control domain assignment
>>   s390/zcrypt: validate control domain assignment
>>   KVM: s390: Connect the AP mediated matrix device to KVM
>>   s390/zcrypt: introduce ioctl access to VFIO AP Matrix driver
>>   KVM: s390: interface to configure KVM guest's AP matrix
>>   KVM: s390: validate input to AP matrix config interface
>>   KVM: s390: New ioctl to configure KVM guest's AP matrix
>>   s390/facilities: enable AP facilities needed by guest
> 
> Overall I am quite happy with the patches, only minor things I would
> like to see improved. We have to agree how we want to approach the
> upstream process between my s390/linux tree and the s390/kvm tree.
> Probably a tip branch again for the both tress to pull from.

Thanks a lot for the review of the zcrypt patches. Its very good to know
that these base changes are ok with you.

Regarding merging: yes, this should be a tip branch then. Before that, we
have to finalize the interface and design to make sure that the changes
will work throughout the whole stack (kvm/vfio/qemu/libvirt).

Re: [RFC 03/19] s390/zcrypt: new AP matrix bus

[Tony Krowiak]

On 10/16/2017 04:47 AM, Martin Schwidefsky wrote:
> On Fri, 13 Oct 2017 13:38:48 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> Introduces an AP matrix bus. The sysfs location of the
>> AP matrix bus is:
>>
>> /sys/bus/ap_matrix
>>
>> The AP matrix bus will create an AP matrix device that will
>> hold the AP queues reserved for use by KVM guests.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>>   MAINTAINERS                         |    8 +++++
>>   drivers/s390/crypto/Makefile        |    4 +-
>>   drivers/s390/crypto/ap_matrix_bus.c |   52 +++++++++++++++++++++++++++++++++++
>>   drivers/s390/crypto/ap_matrix_bus.h |   15 ++++++++++
>>   4 files changed, 77 insertions(+), 2 deletions(-)
>>   create mode 100644 drivers/s390/crypto/ap_matrix_bus.c
>>   create mode 100644 drivers/s390/crypto/ap_matrix_bus.h
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index ef23cf5..cbd6f7c 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -11711,6 +11711,14 @@ W:	http://www.ibm.com/developerworks/linux/linux390/
>>   S:	Supported
>>   F:	drivers/s390/crypto/
>>
>> +S390 VFIO AP MATRIX DRIVER
>> +M:	Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> +L:	linux-s390@vger.kernel.org
>> +W:	http://www.ibm.com/developerworks/linux/linux390/
>> +S:	Supported
>> +F:	drivers/s390/crypto/ap_matrix_bus.h
>> +F:	drivers/s390/crypto/ap_matrix_bus.c
>> +
>>   S390 ZFCP DRIVER
>>   M:	Steffen Maier <maier@linux.vnet.ibm.com>
>>   M:	Benjamin Block <bblock@linux.vnet.ibm.com>
>> diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile
>> index be36f10..87646ca 100644
>> --- a/drivers/s390/crypto/Makefile
>> +++ b/drivers/s390/crypto/Makefile
>> @@ -2,7 +2,7 @@
>>   # S/390 crypto devices
>>   #
>>
>> -ap-objs := ap_bus.o ap_card.o ap_queue.o
>> +ap-objs := ap_bus.o ap_card.o ap_queue.o ap_matrix_bus.o
>>   obj-$(subst m,y,$(CONFIG_ZCRYPT)) += ap.o
>>   # zcrypt_api.o and zcrypt_msgtype*.o depend on ap.o
>>   zcrypt-objs := zcrypt_api.o zcrypt_card.o zcrypt_queue.o
>> @@ -13,4 +13,4 @@ obj-$(CONFIG_ZCRYPT) += zcrypt_pcixcc.o zcrypt_cex2a.o zcrypt_cex4.o
>>
>>   # pkey kernel module
>>   pkey-objs := pkey_api.o
>> -obj-$(CONFIG_PKEY) += pkey.o
>> +obj-$(CONFIG_PKEY) += pkey.o
>> \ No newline at end of file
> Try to avoid this newline hunk please. It is removed again with patch #5.
Will do.
>
>> diff --git a/drivers/s390/crypto/ap_matrix_bus.c b/drivers/s390/crypto/ap_matrix_bus.c
>> new file mode 100644
>> index 0000000..fbae175
>> --- /dev/null
>> +++ b/drivers/s390/crypto/ap_matrix_bus.c
>> @@ -0,0 +1,52 @@
>> +/*
>> + * Copyright IBM Corp. 2017
>> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> + *
>> + * Adjunct processor matrix bus.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + */
>> +
>> +#include <linux/module.h>
>> +#include <linux/device.h>
>> +#include <asm/ap.h>
>> +
>> +#include "ap_matrix_bus.h"
>> +
>> +MODULE_AUTHOR("Tony Krowiak <akrowiak@linux.vnet.ibm.com");
>> +MODULE_DESCRIPTION("AP Matrix, Copyright IBM Corp. 2017");
>> +MODULE_LICENSE("GPL v2");
>> +
>> +#define AP_MATRIX_BUS_NAME "ap_matrix"
>> +
>> +static struct device *ap_matrix_root_device;
>> +
>> +static struct bus_type ap_matrix_bus_type = {
>> +	.name = AP_MATRIX_BUS_NAME,
>> +};
>> +
>> +int __init ap_matrix_init(void)
>> +{
>> +	int ret;
>> +
>> +	ap_matrix_root_device = root_device_register(AP_MATRIX_BUS_NAME);
>> +	ret = PTR_RET(ap_matrix_root_device);
>> +	if (ret)
>> +		goto done;
>> +
>> +	ret = bus_register(&ap_matrix_bus_type);
>> +	if (ret)
>> +		goto bus_reg_err;
>> +
>> +	return 0;
>> +
>> +bus_reg_err:
>> +	root_device_unregister(ap_matrix_root_device);
>> +
>> +done:
>> +	return ret;
>> +}
>> +device_initcall(ap_matrix_init);
>> diff --git a/drivers/s390/crypto/ap_matrix_bus.h b/drivers/s390/crypto/ap_matrix_bus.h
>> new file mode 100644
>> index 0000000..58e116f
>> --- /dev/null
>> +++ b/drivers/s390/crypto/ap_matrix_bus.h
>> @@ -0,0 +1,15 @@
>> +/*
>> + * Adjunct processor matrix bus header file
>> + *
>> + * Copyright IBM Corp. 2017
>> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> + *
>> + * Adjunct processor bus header file.
>> + */
>> +
>> +#ifndef _AP_MATRIX_BUS_H_
>> +#define _AP_MATRIX_BUS_H_
>> +
>> +int ap_matrix_init(void);
>> +
>> +#endif /* _AP_MATRIX_BUS_H_ */
> While I understand the approach to split the patches into meaningful
> parts, there is always a balance between too coarse and too fine granular.
>
> I would merge patch #3, patch #4 and parts of patch #5 that affect
> ap_matrix_bus.c & ap_matrix_bus.h.
Will do.
>
>

Re: [RFC 05/19] s390/zcrypt: base implementation of AP matrix device driver

[Tony Krowiak]

On 10/16/2017 04:59 AM, Martin Schwidefsky wrote:
> On Fri, 13 Oct 2017 13:38:50 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile
>> index 87646ca..1983afa 100644
>> --- a/drivers/s390/crypto/Makefile
>> +++ b/drivers/s390/crypto/Makefile
>> @@ -13,4 +13,8 @@ obj-$(CONFIG_ZCRYPT) += zcrypt_pcixcc.o zcrypt_cex2a.o zcrypt_cex4.o
>>
>>   # pkey kernel module
>>   pkey-objs := pkey_api.o
>> -obj-$(CONFIG_PKEY) += pkey.o
>> \ No newline at end of file
>> +obj-$(CONFIG_PKEY) += pkey.o
>> +
>> +# adjunct processor matrix
>> +vfio_ap_matrix-objs := vfio_ap_matrix_drv.o
>> +obj-$(CONFIG_VFIO_AP_MATRIX) += vfio_ap_matrix.o
> With the fix to patch #3 the newline quirk will be gone.
Yes it will.
>
>> diff --git a/drivers/s390/crypto/ap_matrix_bus.c b/drivers/s390/crypto/ap_matrix_bus.c
>> index 4eb1e3c..66bfa54 100644
>> --- a/drivers/s390/crypto/ap_matrix_bus.c
>> +++ b/drivers/s390/crypto/ap_matrix_bus.c
>> @@ -75,10 +75,18 @@ static int ap_matrix_dev_create(void)
>>   	return 0;
>>   }
>>
>> +struct ap_matrix *ap_matrix_get_device(void)
>> +{
>> +	return matrix;
>> +}
>> +EXPORT_SYMBOL(ap_matrix_get_device);
>> +
>>   int __init ap_matrix_init(void)
>>   {
>>   	int ret;
>>
>> +	matrix = NULL;
>> +
>>   	ap_matrix_root_device = root_device_register(AP_MATRIX_BUS_NAME);
>>   	ret = PTR_RET(ap_matrix_root_device);
>>   	if (ret)
> The global variable "matrix" is already NULL at the time ap_matrix_init is
> called. No need to set it again. And I would appreciate if the global
> variable would be named "ap_matrix".
Will make those changes.
>
>> diff --git a/drivers/s390/crypto/ap_matrix_bus.h b/drivers/s390/crypto/ap_matrix_bus.h
>> index 225db4f..c2aff23 100644
>> --- a/drivers/s390/crypto/ap_matrix_bus.h
>> +++ b/drivers/s390/crypto/ap_matrix_bus.h
>> @@ -16,6 +16,6 @@ struct ap_matrix {
>>   	struct device device;
>>   };
>>
>> -int ap_matrix_init(void);
>> +struct ap_matrix *ap_matrix_get_device(void);
>>
>>   #endif /* _AP_MATRIX_BUS_H_ */
>> diff --git a/drivers/s390/crypto/vfio_ap_matrix_drv.c b/drivers/s390/crypto/vfio_ap_matrix_drv.c
>> new file mode 100644
>> index 0000000..760ed56
>> --- /dev/null
>> +++ b/drivers/s390/crypto/vfio_ap_matrix_drv.c
>> @@ -0,0 +1,102 @@
>> +/*
>> + * VFIO based AP Matrix device driver
>> + *
>> + * Copyright IBM Corp. 2017
>> + *
>> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> + */
>> +
>> +#include <linux/module.h>
>> +#include <linux/mod_devicetable.h>
>> +#include <linux/slab.h>
>> +
>> +#include "ap_bus.h"
>> +#include "ap_matrix_bus.h"
>> +#include "vfio_ap_matrix_private.h"
>> +
>> +#define VFIO_AP_MATRIX_DRV_NAME "vfio_ap_queue"
>> +
>> +MODULE_AUTHOR("IBM Corporation");
>> +MODULE_DESCRIPTION("AP Matrix device driver, Copyright IBM Corp. 2017");
>> +MODULE_LICENSE("GPL v2");
>> +
>> +static struct ap_device_id ap_queue_ids[] = {
>> +	{ .dev_type = AP_DEVICE_TYPE_CEX4,
>> +	  .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
>> +	{ .dev_type = AP_DEVICE_TYPE_CEX5,
>> +	  .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
>> +	{ .dev_type = AP_DEVICE_TYPE_CEX6,
>> +	  .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
>> +	{ /* end of list */ },
>> +};
>> +
>> +MODULE_DEVICE_TABLE(ap_matrix, ap_queue_ids);
>> +
>> +static struct ap_matrix_driver {
>> +	struct ap_driver ap_drv;
>> +	struct ap_matrix *ap_matrix;
>> +} vfio_ap_matrix_drv;
>> +
>> +static int ap_matrix_queue_dev_probe(struct ap_device *apdev)
>> +{
>> +	struct vfio_ap_queue *vapq;
>> +	struct ap_queue *apq = to_ap_queue(&apdev->device);
>> +	struct ap_matrix *ap_matrix = vfio_ap_matrix_drv.ap_matrix;
>> +
>> +	vapq = kzalloc(sizeof(*vapq), GFP_KERNEL);
>> +	if (!vapq)
>> +		return -ENOMEM;
>> +
>> +	INIT_LIST_HEAD(&vapq->list);
>> +	vapq->queue = apq;
>> +	spin_lock_bh(&ap_matrix->qlock);
>> +	list_add_tail(&vapq->list, &ap_matrix->queues);
>> +	spin_unlock_bh(&ap_matrix->qlock);
>> +
>> +	return 0;
>> +}
>> +
>> +static void ap_matrix_queue_dev_remove(struct ap_device *apdev)
>> +{
>> +	struct vfio_ap_queue *vapq;
>> +	struct ap_queue *apq = to_ap_queue(&apdev->device);
>> +	struct ap_matrix *ap_matrix = vfio_ap_matrix_drv.ap_matrix;
>> +
>> +	vapq = find_vapq(ap_matrix, apq->qid);
>> +
>> +	if (vapq) {
>> +		spin_lock_bh(&ap_matrix->qlock);
>> +		list_del_init(&vapq->list);
>> +		spin_unlock_bh(&ap_matrix->qlock);
>> +		kfree(vapq);
>> +	}
>> +}
> I would opt for a quick exit if find_vapq can not find the device.
> The un-indent the actual remove code.
Okay, will do.
>
>> +
>> +int __init ap_matrix_init(void)
>> +{
>> +
>> +	int ret;
>> +
>> +	vfio_ap_matrix_drv.ap_matrix = ap_matrix_get_device();
>> +	if (!vfio_ap_matrix_drv.ap_matrix)
>> +		return -ENODEV;
>> +
>> +	vfio_ap_matrix_drv.ap_drv.probe = ap_matrix_queue_dev_probe;
>> +	vfio_ap_matrix_drv.ap_drv.remove = ap_matrix_queue_dev_remove;
>> +	vfio_ap_matrix_drv.ap_drv.ids = ap_queue_ids;
>> +
>> +	ret = ap_driver_register(&vfio_ap_matrix_drv.ap_drv,
>> +				 THIS_MODULE, VFIO_AP_MATRIX_DRV_NAME);
>> +	if (ret)
>> +		return ret;
>> +
>> +	return ret;
>> +}
>> +
>> +void __exit ap_matrix_exit(void)
>> +{
>> +	ap_driver_unregister(&vfio_ap_matrix_drv.ap_drv);
>> +}
>> +
>> +module_init(ap_matrix_init);
>> +module_exit(ap_matrix_exit);
>> diff --git a/drivers/s390/crypto/vfio_ap_matrix_private.h b/drivers/s390/crypto/vfio_ap_matrix_private.h
>> new file mode 100644
>> index 0000000..11c5e02
>> --- /dev/null
>> +++ b/drivers/s390/crypto/vfio_ap_matrix_private.h
>> @@ -0,0 +1,47 @@
>> +/*
>> + * Private data and functions for adjunct processor VFIO matrix driver.
>> + *
>> + * Copyright IBM Corp. 2016
>> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> + */
>> +
>> +#ifndef _VFIO_AP_PRIVATE_H_
>> +#define _VFIO_AP_PRIVATE_H_
>> +
>> +#include <linux/types.h>
>> +
>> +#include "ap_bus.h"
>> +#include "ap_matrix_bus.h"
>> +
>> +#define VFIO_AP_MATRIX_MODULE_NAME "vfio_ap_matrix"
>> +
>> +struct vfio_ap_queue {
>> +	struct ap_queue *queue;
>> +	struct list_head list;
>> +};
>> +
>> +static inline struct vfio_ap_queue *to_vapq(struct ap_device *ap_dev)
>> +{
>> +	struct ap_queue *ap_queue = to_ap_queue(&ap_dev->device);
>> +	struct vfio_ap_queue *vapq;
>> +
>> +	vapq = container_of(&ap_queue, struct vfio_ap_queue, queue);
>> +
>> +	return vapq;
>> +}
> This can be shortened a bit
>
> 	return container_of(&ap_queue, struct vfio_ap_queue, queue);
Okay, will do.
>
>> +
>> +static inline struct vfio_ap_queue *find_vapq(struct ap_matrix *ap_matrix,
>> +					      ap_qid_t qid)
>> +{
>> +	struct vfio_ap_queue *vapq;
>> +
>> +	if (!list_empty(&ap_matrix->queues)) {
>> +		list_for_each_entry(vapq, &ap_matrix->queues, list)
>> +			if (vapq->queue->qid == qid)
>> +				return vapq;
>> +	}
>> +
>> +	return NULL;
>> +}
>> +
>> +#endif /* _VFIO_AP_PRIVATE_H_ */
>

Re: [RFC 06/19] s390/zcrypt: register matrix device with VFIO mediated device framework

[Tony Krowiak]

On 10/16/2017 05:03 AM, Martin Schwidefsky wrote:
> On Fri, 13 Oct 2017 13:38:51 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> diff --git a/drivers/s390/crypto/ap_matrix_bus.c b/drivers/s390/crypto/ap_matrix_bus.c
>> index 66bfa54..418c23b 100644
>> --- a/drivers/s390/crypto/ap_matrix_bus.c
>> +++ b/drivers/s390/crypto/ap_matrix_bus.c
>> @@ -61,6 +61,7 @@ static int ap_matrix_dev_create(void)
>>   	matrix->device.bus = &ap_matrix_bus_type;
>>   	matrix->device.parent = ap_matrix_root_device;
>>   	matrix->device.release = ap_matrix_dev_release;
>> +	INIT_LIST_HEAD(&matrix->queues);
>>
>>   	ret = device_register(&matrix->device);
>>   	if (ret) {
>> diff --git a/drivers/s390/crypto/ap_matrix_bus.h b/drivers/s390/crypto/ap_matrix_bus.h
>> index c2aff23..3eccc36 100644
>> --- a/drivers/s390/crypto/ap_matrix_bus.h
>> +++ b/drivers/s390/crypto/ap_matrix_bus.h
>> @@ -12,8 +12,12 @@
>>
>>   #include <linux/device.h>
>>
>> +#include "ap_bus.h"
>> +
>>   struct ap_matrix {
>>   	struct device device;
>> +	spinlock_t qlock;
>> +	struct list_head queues;
>>   };
>>
>>   struct ap_matrix *ap_matrix_get_device(void);
> Move these two hunks into patch #5 please. Yes, strictly speaking the two elements
> in the struct ap_matrix are needed only with patch #6, but it is fine to introduce
> an element with a new driver that is only exploited with a later patch.
Will do.
>

Re: [RFC 07/19] KVM: s390: introduce AP matrix configuration interface

[Tony Krowiak]

On 10/16/2017 05:10 AM, Martin Schwidefsky wrote:
> On Fri, 13 Oct 2017 13:38:52 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> This patch introduces a new compilation unit that will contain
>> all of the data structures and logic for configuring AP adapters,
>> usage domains and control domains for a KVM guest.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>>   MAINTAINERS                       |    2 ++
>>   arch/s390/include/asm/ap-config.h |   12 ++++++++++++
>>   arch/s390/kvm/Makefile            |    2 +-
>>   arch/s390/kvm/ap-config.c         |    9 +++++++++
>>   4 files changed, 24 insertions(+), 1 deletions(-)
>>   create mode 100644 arch/s390/include/asm/ap-config.h
>>   create mode 100644 arch/s390/kvm/ap-config.c
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index 0c8855a..4de0904 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -11721,6 +11721,8 @@ F:	drivers/s390/crypto/ap_matrix_bus.c
>>   F:	drivers/s390/crypto/vfio_ap_matrix_drv.c
>>   F:	drivers/s390/crypto/vfio_ap_matrix_private.h
>>   F:	drivers/s390/crypto/vfio_ap_matrix_ops.c
>> +F:	arch/s390/include/asm/kvm/ap-config.h
>> +F:	arch/s390/kvm/ap-config.c
>>
>>   S390 ZFCP DRIVER
>>   M:	Steffen Maier <maier@linux.vnet.ibm.com>
>> diff --git a/arch/s390/include/asm/ap-config.h b/arch/s390/include/asm/ap-config.h
>> new file mode 100644
>> index 0000000..8491b5f
>> --- /dev/null
>> +++ b/arch/s390/include/asm/ap-config.h
>> @@ -0,0 +1,12 @@
>> +/*
>> + * Adjunct Processor (AP) configuration management for KVM guests
>> + *
>> + * Copyright IBM Corp. 2017
>> + *
>> + * Author(s): Tony Krowiak <akrowia@linux.vnet.ibm.com>
>> + */
>> +
>> +#ifndef _ASM_KVM_AP_CONFIG_H_
>> +#define _ASM_KVM_AP_CONFIG_H_
>> +
>> +#endif /* _ASM_KVM_AP_CONFIG_H_ */
>> diff --git a/arch/s390/kvm/Makefile b/arch/s390/kvm/Makefile
>> index 09a9e6d..0397b9c 100644
>> --- a/arch/s390/kvm/Makefile
>> +++ b/arch/s390/kvm/Makefile
>> @@ -12,6 +12,6 @@ common-objs = $(KVM)/kvm_main.o $(KVM)/eventfd.o  $(KVM)/async_pf.o $(KVM)/irqch
>>   ccflags-y := -Ivirt/kvm -Iarch/s390/kvm
>>
>>   kvm-objs := $(common-objs) kvm-s390.o intercept.o interrupt.o priv.o sigp.o
>> -kvm-objs += diag.o gaccess.o guestdbg.o sthyi.o vsie.o
>> +kvm-objs += diag.o gaccess.o guestdbg.o sthyi.o vsie.o ap-config.o
>>
>>   obj-$(CONFIG_KVM) += kvm.o
>> diff --git a/arch/s390/kvm/ap-config.c b/arch/s390/kvm/ap-config.c
>> new file mode 100644
>> index 0000000..84fdf43
>> --- /dev/null
>> +++ b/arch/s390/kvm/ap-config.c
>> @@ -0,0 +1,9 @@
>> +/*
>> + * Adjunct Processor (AP) configuration management for KVM guests
>> + *
>> + * Copyright IBM Corp. 2017
>> + *
>> + * Author(s): Tony Krowiak <akrowia@linux.vnet.ibm.com>
>> + */
>> +
>> +#include <asm/ap-config.h>
> Another patch that introduces empty files. Please don't do that. Merge
> patch #7 with patch #16. With the dependency in patch #8 on the ap-config.h
> header the new place for the combined patch would be prior to patch #8.
Sounds reasonable, will do.
>

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Tony Krowiak]

On 10/16/2017 06:05 AM, Cornelia Huck wrote:
> On Fri, 13 Oct 2017 13:38:45 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> Overview:
>> --------
>> An adjunct processor (AP) facility is an IBM Z cryptographic facility. The
>> AP facility is comprised of three AP instructions and from 1 to 256 AP
>> adapter cards. The design takes advantage of the interpretive execution mode
>> provided by the SIE architecture. With interpretive execution mode, the AP
>> instructions executed on the guest are interpreted by the hardware. This
>> allows guests direct access to AP adapter cards. The first goal of this
>> patch series is to provide direct access by a KVM guest to an AP as a
>> pass-through device. The second goal is to provide administrators with the
>> means to configure KVM guests to grant direct access to AP facilities
>> assigned to the LPAR in which the host linux system is running.
>>
>> To facilitate the comprehension of the design, let's present an overview of
>> the AP architecture.
> I have not yet looked at the patches (and I'm not sure whether I'll be
> able to do so until after the KVM Forum), but I think this also needs
> some explanation under Documentation/, so people can still make sense
> of it in the future.
That makes sense, I will create a document.
>

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Tony Krowiak]

On 10/16/2017 06:06 AM, Christian Borntraeger wrote:
>
> On 10/16/2017 11:27 AM, Martin Schwidefsky wrote:
>> On Fri, 13 Oct 2017 13:38:45 -0400
>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>
>>> Tony Krowiak (19):
>>>    KVM: s390: SIE considerations for AP Queue virtualization
>>>    KVM: s390: refactor crypto initialization
>>>    s390/zcrypt: new AP matrix bus
>>>    s390/zcrypt: create an AP matrix device on the AP matrix bus
>>>    s390/zcrypt: base implementation of AP matrix device driver
>>>    s390/zcrypt: register matrix device with VFIO mediated device
>>>      framework
>>>    KVM: s390: introduce AP matrix configuration interface
>>>    s390/zcrypt: support for assigning adapters to matrix mdev
>>>    s390/zcrypt: validate adapter assignment
>>>    s390/zcrypt: sysfs interfaces supporting AP domain assignment
>>>    s390/zcrypt: validate domain assignment
>>>    s390/zcrypt: sysfs support for control domain assignment
>>>    s390/zcrypt: validate control domain assignment
>>>    KVM: s390: Connect the AP mediated matrix device to KVM
>>>    s390/zcrypt: introduce ioctl access to VFIO AP Matrix driver
>>>    KVM: s390: interface to configure KVM guest's AP matrix
>>>    KVM: s390: validate input to AP matrix config interface
>>>    KVM: s390: New ioctl to configure KVM guest's AP matrix
>>>    s390/facilities: enable AP facilities needed by guest
>> Overall I am quite happy with the patches, only minor things I would
>> like to see improved. We have to agree how we want to approach the
>> upstream process between my s390/linux tree and the s390/kvm tree.
>> Probably a tip branch again for the both tress to pull from.
> Thanks a lot for the review of the zcrypt patches. Its very good to know
> that these base changes are ok with you.
>
> Regarding merging: yes, this should be a tip branch then. Before that, we
> have to finalize the interface and design to make sure that the changes
> will work throughout the whole stack (kvm/vfio/qemu/libvirt).
I would also like to add my thank you for the review comments. I am 
currently
debugging the QEMU code I've written and Farhan Ali is in the
review process for the libvirt component.
>
>
>

Re: [RFC 16/19] KVM: s390: interface to configure KVM guest's AP matrix

[Tony Krowiak]

On 10/13/2017 01:39 PM, Tony Krowiak wrote:
> Provides an interface to assign AP adapters, usage domains
> and control domains to a KVM guest.
>
> A KVM guest is started by executing the Start Interpretive Execution (SIE)
> instruction. The SIE state description is a control block that contains the
> state information for a KVM guest and is supplied as input to the SIE
> instruction. The SIE state description contains a field that references
> a Crypto Control Block (CRYCB). The CRYCB contains three bitmask fields
> identifying the adapters, usage domains and control domains assigned to the
> KVM guest:
>
> * The AP Adapter Matrix (APM) field identifies the AP adapters assigned to
>    the KVM guest
>
> * The AP Queue Matrix (AQM) field identifies the usage domains assigned to
>    the KVM guest
>
> * The AP Domain matrix (ADM) field identifies the control domains
>    assigned to the KVM guest.
>
> Each adapter, usage domain and control domain are identified by a number
> from 0 to 255. The bits in each mask, from left to right, correspond to
> the numbers 0-255. When a bit is set, the corresponding adapter,
> usage domain or control domain will be assigned to the KVM guest.
>
> This patch will set the bits in the APM, AQM and ADM fields of the
> CRYCB referenced by the KVM guest's SIE state description. The process
> used is:
There is a step missing here and is also missing from the code.
By architectural convention, all bits set in the ADM must also
be set in the AQM, but the ADM may have bits set that are not
set in the AQM. In other words, all usage domains are control
domains, but some control domains may not be usage usage domains.
So, the ADM contained in the CRYCB will be set with the bits
resulting from the logical OR of the ADM and AQM configured for
the mediated AP matrix device via its sysfs attributes files.
>
> 1. Perform a logical AND of the AP matrix masks configured for the
>     mediated AP matrix device via its sysfs attributes files with
>     the matrix masks assigned to the LPAR in which the host linux
>     system is running to create the effective masks, EAPM, EAQM and
>     EADM.
>
> 2. Set the APM, AQM and ADM in the CRYCB from the EAPM, EAQM and
>     EADM calculated in step 1.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>   arch/s390/include/asm/ap-config.h |    7 ++
>   arch/s390/kvm/ap-config.c         |  144 +++++++++++++++++++++++++++++++++++++
>   2 files changed, 151 insertions(+), 0 deletions(-)
>
> diff --git a/arch/s390/include/asm/ap-config.h b/arch/s390/include/asm/ap-config.h
> index 3064215..866f008 100644
> --- a/arch/s390/include/asm/ap-config.h
> +++ b/arch/s390/include/asm/ap-config.h
> @@ -10,16 +10,23 @@
>   #define _ASM_KVM_AP_CONFIG_H_
>
>   #include <linux/types.h>
> +#include <linux/kvm_host.h>
>
>   #define AP_MATRIX_MAX_MASK_BITS		256
>   #define AP_MATRIX_MASK_INDICES		(AP_MATRIX_MAX_MASK_BITS / \
>   					(sizeof(u64) * 8))
>   #define AP_MATRIX_MAX_MASK_BYTES	(AP_MATRIX_MASK_INDICES * sizeof(u64))
>
> +#define AP_MATRIX_MASK_TYPE_ADAPTER	"adapter"
> +#define AP_MATRIX_MASK_TYPE_DOMAIN	"domain"
> +#define AP_MATRIX_MASK_TYPE_CONTROL	"control domain"
> +
>   struct ap_config_masks {
>   	u64 apm[AP_MATRIX_MASK_INDICES];
>   	u64 aqm[AP_MATRIX_MASK_INDICES];
>   	u64 adm[AP_MATRIX_MASK_INDICES];
>   };
>
> +extern int ap_config_matrix(struct kvm *kvm, struct ap_config_masks *masks);
> +
>   #endif /* _ASM_KVM_AP_CONFIG_H_ */
> diff --git a/arch/s390/kvm/ap-config.c b/arch/s390/kvm/ap-config.c
> index 84fdf43..dc79798 100644
> --- a/arch/s390/kvm/ap-config.c
> +++ b/arch/s390/kvm/ap-config.c
> @@ -7,3 +7,147 @@
>    */
>
>   #include <asm/ap-config.h>
> +#include <asm/ap.h>
> +#include <linux/bitops.h>
> +
> +static inline int is_format2_crycb(struct kvm *kvm)
> +{
> +	int fmt2_mask = kvm->arch.crypto.crycbd & CRYCB_FORMAT2;
> +
> +	return (fmt2_mask == CRYCB_FORMAT2);
> +}
> +
> +static inline u64 *ap_config_get_crycb_apm(struct kvm *kvm)
> +{
> +	u64 *apm;
> +
> +	if (is_format2_crycb(kvm))
> +		apm = kvm->arch.crypto.crycb->apcb1.apm;
> +	else
> +		apm = kvm->arch.crypto.crycb->apcb0.apm;
> +
> +	return apm;
> +}
> +
> +static inline u64 *ap_config_get_crycb_aqm(struct kvm *kvm)
> +{
> +	u64 *aqm;
> +
> +	if (is_format2_crycb(kvm))
> +		aqm = kvm->arch.crypto.crycb->apcb1.aqm;
> +	else
> +		aqm = kvm->arch.crypto.crycb->apcb0.aqm;
> +
> +	return aqm;
> +}
> +
> +static inline u64 *ap_config_get_crycb_adm(struct kvm *kvm)
> +{
> +	u64 *adm;
> +
> +	if (is_format2_crycb(kvm))
> +		adm = kvm->arch.crypto.crycb->apcb1.adm;
> +	else
> +		adm = kvm->arch.crypto.crycb->apcb0.adm;
> +
> +	return adm;
> +}
> +
> +static void ap_config_set_crycb_masks(struct kvm *kvm,
> +				    struct ap_config_masks *masks)
> +{
> +	size_t i;
> +	size_t masksz;
> +	u64 *mask = ap_config_get_crycb_apm(kvm);
> +
> +	masksz = (is_format2_crycb(kvm)) ? APCB1_MASK_SIZE : APCB0_MASK_SIZE;
> +
> +	for (i = 0; i < masksz; i++)
> +		mask[i] = masks->apm[i];
> +
> +	mask = ap_config_get_crycb_aqm(kvm);
> +
> +	for (i = 0; i < masksz; i++)
> +		mask[i] = masks->aqm[i];
> +
> +	mask = ap_config_get_crycb_adm(kvm);
> +
> +	for (i = 0; i < masksz; i++)
> +		mask[i] = masks->adm[i];
> +}
> +
> +static int ap_config_set_emask(const char *mask_type, unsigned long *mask,
> +			       unsigned long *cfgmask)
> +{
> +	unsigned long id;
> +	unsigned long nbits = AP_MATRIX_MAX_MASK_BITS;
> +
> +	id = find_first_bit_inv(mask, nbits);
> +	while (id < nbits) {
> +		if (!test_bit_inv(id, cfgmask)) {
> +			clear_bit_inv(id, mask);
> +			pr_err("%s: %s %02lx is not installed on the host system",
> +			       __func__, mask_type, id);
> +			return -ENODEV;
> +		}
> +
> +		id = find_next_bit_inv(mask, nbits, id + 1);
> +	}
> +
> +	return 0;
> +}
> +
> +static int ap_config_get_emasks(struct ap_config_masks *masks)
> +{
> +	int ret;
> +	struct ap_config_info config;
> +
> +	ret = ap_query_configuration(&config);
> +	if (ret) {
> +		if (ret == -EOPNOTSUPP) {
> +			pr_err("%s: Query AP configuration not supported",
> +			       __func__);
> +
> +			return ret;
> +		}
> +
> +		pr_err("%s: Query AP configuration failed with rc=%d",
> +		       __func__, ret);
> +
> +		return ret;
> +	}
> +
> +	ret = ap_config_set_emask(AP_MATRIX_MASK_TYPE_ADAPTER,
> +				  (unsigned long *)masks->apm,
> +				  (unsigned long *)config.apm);
> +	if (ret)
> +		return ret;
> +
> +	ret = ap_config_set_emask(AP_MATRIX_MASK_TYPE_DOMAIN,
> +				  (unsigned long *)masks->aqm,
> +				  (unsigned long *)config.aqm);
> +	if (ret)
> +		return ret;
> +
> +	ret = ap_config_set_emask(AP_MATRIX_MASK_TYPE_CONTROL,
> +				  (unsigned long *)masks->adm,
> +				  (unsigned long *)config.adm);
> +	if (ret)
> +		return ret;
> +
> +	return 0;
> +}
> +
> +int ap_config_matrix(struct kvm *kvm, struct ap_config_masks *masks)
> +{
> +	int ret;
> +
For the reasons stated earlier in this response, we need to do a
logical OR of the bits in the ADM with those in the AQM here:

size_t i;

for (i = 0; i < AP_MATRIX_MASK_INDICES; i++)
     masks->adm[i] |= masks->aqm[i];
> +	ret = ap_config_get_emasks(masks);
> +	if (ret)
> +		return ret;
> +
> +	ap_config_set_crycb_masks(kvm, masks);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL(ap_config_matrix);

Re: [RFC 04/19] s390/zcrypt: create an AP matrix device on the AP matrix bus

[Cornelia Huck]

On Fri, 13 Oct 2017 13:38:49 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

[Please take with a grain of salt as I did not yet have time to take
more than a very superficial glance at the whole structure.]

> Creates a single AP matrix device on the AP matrix bus.
> The matrix device will be created as part of the AP matrix bus
> initialization. The matrix device will hold the AP queues that
> have been reserved for use by KVM guests. Mediated matrix devices
> can then be created for each guest. The mediated matrix devices can
> then be configured with a matrix of AP adapters, usage and
> control domains that will be made accessible to the guest.
> 
> The sysfs location of the matrix device is:
> 
> /sys/bus/ap_matrix
> ... [devices]
> ...... [matrix]

Also /sys/devices/ap_matrix/matrix, isn't it?

> 
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>  drivers/s390/crypto/ap_matrix_bus.c |   54 +++++++++++++++++++++++++++++++++++
>  drivers/s390/crypto/ap_matrix_bus.h |    6 ++++
>  2 files changed, 60 insertions(+), 0 deletions(-)
> 
> diff --git a/drivers/s390/crypto/ap_matrix_bus.c b/drivers/s390/crypto/ap_matrix_bus.c
> index fbae175..4eb1e3c 100644
> --- a/drivers/s390/crypto/ap_matrix_bus.c
> +++ b/drivers/s390/crypto/ap_matrix_bus.c
> @@ -12,6 +12,7 @@
>  
>  #include <linux/module.h>
>  #include <linux/device.h>
> +#include <linux/slab.h>
>  #include <asm/ap.h>
>  
>  #include "ap_matrix_bus.h"
> @@ -21,13 +22,59 @@
>  MODULE_LICENSE("GPL v2");
>  
>  #define AP_MATRIX_BUS_NAME "ap_matrix"
> +#define AP_MATRIX_DEV_TYPE_NAME "ap_matrix"
> +#define AP_MATRIX_DEV_NAME "matrix"
>  
>  static struct device *ap_matrix_root_device;
> +static struct ap_matrix *matrix;
>  
>  static struct bus_type ap_matrix_bus_type = {
>  	.name = AP_MATRIX_BUS_NAME,
>  };
>  
> +static struct device_type ap_matrix_type = {
> +	.name = AP_MATRIX_DEV_TYPE_NAME,
> +};
> +
> +static void ap_matrix_dev_release(struct device *dev)
> +{
> +	struct ap_matrix *ap_matrix;
> +
> +	ap_matrix = container_of(dev, struct ap_matrix, device);
> +
> +	if (matrix == ap_matrix)
> +		kfree(matrix);
> +
> +	matrix = NULL;

This looks very, very odd. Refcounting should take care that the
release function is only invoked if the device is really gone.

Also, I don't think you need to keep a pointer to the device as it is a
singleton: It's simply the only device on the list and you should be
able to easily pick it that way. If your code does not register further
devices on the bus, there won't be ambiguities.

> +}
> +
> +static int ap_matrix_dev_create(void)
> +{
> +	int ret;
> +
> +	matrix = kzalloc(sizeof(*matrix), GFP_KERNEL);
> +	if (!matrix)
> +		return -ENOMEM;
> +
> +	matrix->device.type = &ap_matrix_type;
> +	dev_set_name(&matrix->device, "%s", AP_MATRIX_DEV_NAME);
> +	matrix->device.bus = &ap_matrix_bus_type;
> +	matrix->device.parent = ap_matrix_root_device;
> +	matrix->device.release = ap_matrix_dev_release;
> +
> +	ret = device_register(&matrix->device);
> +	if (ret) {
> +		put_device(&matrix->device);
> +		kfree(matrix);
> +		matrix = NULL;

That's wrong. The release function needs to clean up the embedding
structure, so that kfree is at best useless and at worst wrong, if
something else grabbed a reference.

> +		return ret;
> +	}
> +
> +	get_device(&matrix->device);

Why should you need an extra reference here? I'd expect the code
hanging devices off this one to properly grab a reference, so you
should be all good?

> +
> +	return 0;
> +}
> +
>  int __init ap_matrix_init(void)
>  {
>  	int ret;
> @@ -41,8 +88,15 @@ int __init ap_matrix_init(void)
>  	if (ret)
>  		goto bus_reg_err;
>  
> +	ret = ap_matrix_dev_create();
> +	if (ret)
> +		goto matrix_create_err;
> +
>  	return 0;
>  
> +matrix_create_err:
> +	bus_unregister(&ap_matrix_bus_type);
> +
>  bus_reg_err:
>  	root_device_unregister(ap_matrix_root_device);

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Christian Borntraeger]

As a remark. I think it makes more sense to concentrate on the whole concept 
before doing an depth review (of course, if you have the cycles - feel free
to go ahead :-) ).
Let me just summarize the HW idea in a very simplified fashion, so that we can
try to let everybody understand what is going on here.

Implementation-wise (in LPAR and as supported by the HW-virtualization for KVM) the 
implementation itself is really really simple. As a satellite block of the
state descriptor (sie control block) there is the crypto control block.
(usually one per guest). This contains 3 256bit bitfields which allow/disallow
1. adapters by number (0-255)
2. usage domains by number (0-255)
3. control domains by number (0-255)

For simplicity lets ignore the control domains, the mechanism is similar to
the usage domain.

The guest will then have access to ALL elements of the cross product.
So adapter 1,2,3 and domain 10,11,12 will result in 9 tuples accessible by 
the guest (1,10) ; (1,11) ; (1,12) ; (2,10) ; (2,11) ; (2,12) ; (3,10) ;
(3,11) ; (3,12)
the admin (or the management instance) must ensure that each tuple
is only in use by 1 guest.
Setting the bits is basically all what is needed (plus some minimal glue
and handling), everything else will be handled by hardware/firmware.

Now if we do not need to care about security, we could just use some kvm
ioctl and be done with it. As we learned for PCI, this is not going to be
secure as qemu is untrusted in svirt/appamor contexts.

So this is basically using vfio and mdev as a means to orchestrate things.
And this is where things get complicated and multiple options are possible.

Christian

Re: [RFC 04/19] s390/zcrypt: create an AP matrix device on the AP matrix bus

[Tony Krowiak]

On 10/18/2017 12:20 PM, Cornelia Huck wrote:
> On Fri, 13 Oct 2017 13:38:49 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
> [Please take with a grain of salt as I did not yet have time to take
> more than a very superficial glance at the whole structure.]
>
>> Creates a single AP matrix device on the AP matrix bus.
>> The matrix device will be created as part of the AP matrix bus
>> initialization. The matrix device will hold the AP queues that
>> have been reserved for use by KVM guests. Mediated matrix devices
>> can then be created for each guest. The mediated matrix devices can
>> then be configured with a matrix of AP adapters, usage and
>> control domains that will be made accessible to the guest.
>>
>> The sysfs location of the matrix device is:
>>
>> /sys/bus/ap_matrix
>> ... [devices]
>> ...... [matrix]
> Also /sys/devices/ap_matrix/matrix, isn't it?
Yes
>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>>   drivers/s390/crypto/ap_matrix_bus.c |   54 +++++++++++++++++++++++++++++++++++
>>   drivers/s390/crypto/ap_matrix_bus.h |    6 ++++
>>   2 files changed, 60 insertions(+), 0 deletions(-)
>>
>> diff --git a/drivers/s390/crypto/ap_matrix_bus.c b/drivers/s390/crypto/ap_matrix_bus.c
>> index fbae175..4eb1e3c 100644
>> --- a/drivers/s390/crypto/ap_matrix_bus.c
>> +++ b/drivers/s390/crypto/ap_matrix_bus.c
>> @@ -12,6 +12,7 @@
>>   
>>   #include <linux/module.h>
>>   #include <linux/device.h>
>> +#include <linux/slab.h>
>>   #include <asm/ap.h>
>>   
>>   #include "ap_matrix_bus.h"
>> @@ -21,13 +22,59 @@
>>   MODULE_LICENSE("GPL v2");
>>   
>>   #define AP_MATRIX_BUS_NAME "ap_matrix"
>> +#define AP_MATRIX_DEV_TYPE_NAME "ap_matrix"
>> +#define AP_MATRIX_DEV_NAME "matrix"
>>   
>>   static struct device *ap_matrix_root_device;
>> +static struct ap_matrix *matrix;
>>   
>>   static struct bus_type ap_matrix_bus_type = {
>>   	.name = AP_MATRIX_BUS_NAME,
>>   };
>>   
>> +static struct device_type ap_matrix_type = {
>> +	.name = AP_MATRIX_DEV_TYPE_NAME,
>> +};
>> +
>> +static void ap_matrix_dev_release(struct device *dev)
>> +{
>> +	struct ap_matrix *ap_matrix;
>> +
>> +	ap_matrix = container_of(dev, struct ap_matrix, device);
>> +
>> +	if (matrix == ap_matrix)
>> +		kfree(matrix);
>> +
>> +	matrix = NULL;
> This looks very, very odd. Refcounting should take care that the
> release function is only invoked if the device is really gone.
>
> Also, I don't think you need to keep a pointer to the device as it is a
> singleton: It's simply the only device on the list and you should be
> able to easily pick it that way. If your code does not register further
> devices on the bus, there won't be ambiguities.
Okay, I'll make that change
>
>> +}
>> +
>> +static int ap_matrix_dev_create(void)
>> +{
>> +	int ret;
>> +
>> +	matrix = kzalloc(sizeof(*matrix), GFP_KERNEL);
>> +	if (!matrix)
>> +		return -ENOMEM;
>> +
>> +	matrix->device.type = &ap_matrix_type;
>> +	dev_set_name(&matrix->device, "%s", AP_MATRIX_DEV_NAME);
>> +	matrix->device.bus = &ap_matrix_bus_type;
>> +	matrix->device.parent = ap_matrix_root_device;
>> +	matrix->device.release = ap_matrix_dev_release;
>> +
>> +	ret = device_register(&matrix->device);
>> +	if (ret) {
>> +		put_device(&matrix->device);
>> +		kfree(matrix);
>> +		matrix = NULL;
> That's wrong. The release function needs to clean up the embedding
> structure, so that kfree is at best useless and at worst wrong, if
> something else grabbed a reference.
Good point.
>
>> +		return ret;
>> +	}
>> +
>> +	get_device(&matrix->device);
> Why should you need an extra reference here? I'd expect the code
> hanging devices off this one to properly grab a reference, so you
> should be all good?
I'll remove this.
>
>> +
>> +	return 0;
>> +}
>> +
>>   int __init ap_matrix_init(void)
>>   {
>>   	int ret;
>> @@ -41,8 +88,15 @@ int __init ap_matrix_init(void)
>>   	if (ret)
>>   		goto bus_reg_err;
>>   
>> +	ret = ap_matrix_dev_create();
>> +	if (ret)
>> +		goto matrix_create_err;
>> +
>>   	return 0;
>>   
>> +matrix_create_err:
>> +	bus_unregister(&ap_matrix_bus_type);
>> +
>>   bus_reg_err:
>>   	root_device_unregister(ap_matrix_root_device);

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Cornelia Huck]

On Fri, 13 Oct 2017 13:38:45 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> Tony Krowiak (19):
>   KVM: s390: SIE considerations for AP Queue virtualization
>   KVM: s390: refactor crypto initialization
>   s390/zcrypt: new AP matrix bus
>   s390/zcrypt: create an AP matrix device on the AP matrix bus
>   s390/zcrypt: base implementation of AP matrix device driver
>   s390/zcrypt: register matrix device with VFIO mediated device
>     framework
>   KVM: s390: introduce AP matrix configuration interface
>   s390/zcrypt: support for assigning adapters to matrix mdev
>   s390/zcrypt: validate adapter assignment
>   s390/zcrypt: sysfs interfaces supporting AP domain assignment
>   s390/zcrypt: validate domain assignment
>   s390/zcrypt: sysfs support for control domain assignment
>   s390/zcrypt: validate control domain assignment
>   KVM: s390: Connect the AP mediated matrix device to KVM
>   s390/zcrypt: introduce ioctl access to VFIO AP Matrix driver
>   KVM: s390: interface to configure KVM guest's AP matrix
>   KVM: s390: validate input to AP matrix config interface
>   KVM: s390: New ioctl to configure KVM guest's AP matrix
>   s390/facilities: enable AP facilities needed by guest

I'll try to summarize all of this in my own words, both to make sure I
understand the design correctly and to give others a different view on
this.

[I'm completely disregarding control domains here.]

On s390, we have cryptographic coprocessor cards, which are modeled on
Linux as devices on the AP bus. There's also a concept called domains,
which means an individual queue of a crypto device is basically a
(card,domain) tuple. We model this something like the following
(assuming we have access to cards 3 and 4 and domains 1 and 2):

AP -> card3 -> queue (3,1)
            -> queue (3,2)
   -> card4 -> queue (4,1)
            -> queue (4,2)

(The AP bus is a bit different for backwards compat.)

If we want to virtualize this, we can use a feature provided by the
hardware. We basically attach a satellite control block to our main
hardware virtualization control block and the hardware takes care of
(mostly) everything.

For this control block, we don't specify explicit tuples, but a list of
cards and a list of domains. The guest will get access to the cross
product.

Because of this, we need to take care that the lists provided to
different guests don't overlap; i.e., we need to enforce sane
configurations. Otherwise, one guest may get access to things like
secret keys for another guest.

The idea of this patch set is to introduce a new device, the matrix
device. This matrix device hangs off a different root and acts as the
node where mdev devices hang off.

If you now want to give the tuples (4,1) and (4,2), you need to do the
following:

- Unbind the (4,1) and (4,2) tuples from their ap bus driver.
- Bind the (4,1) and (4,2) tuples to the ap matrix driver.
- Create the mediated device.
- Assign card 4 and domains 1 and 2.

QEMU will now simply consume the mediated device and things should work.

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Christian Borntraeger]

adding qemu devel and add Daniel and Erik from libvirt to keep them in the loop. 

On 10/29/2017 12:11 PM, Cornelia Huck wrote:
> On Fri, 13 Oct 2017 13:38:45 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> 
>> Tony Krowiak (19):
>>   KVM: s390: SIE considerations for AP Queue virtualization
>>   KVM: s390: refactor crypto initialization
>>   s390/zcrypt: new AP matrix bus
>>   s390/zcrypt: create an AP matrix device on the AP matrix bus
>>   s390/zcrypt: base implementation of AP matrix device driver
>>   s390/zcrypt: register matrix device with VFIO mediated device
>>     framework
>>   KVM: s390: introduce AP matrix configuration interface
>>   s390/zcrypt: support for assigning adapters to matrix mdev
>>   s390/zcrypt: validate adapter assignment
>>   s390/zcrypt: sysfs interfaces supporting AP domain assignment
>>   s390/zcrypt: validate domain assignment
>>   s390/zcrypt: sysfs support for control domain assignment
>>   s390/zcrypt: validate control domain assignment
>>   KVM: s390: Connect the AP mediated matrix device to KVM
>>   s390/zcrypt: introduce ioctl access to VFIO AP Matrix driver
>>   KVM: s390: interface to configure KVM guest's AP matrix
>>   KVM: s390: validate input to AP matrix config interface
>>   KVM: s390: New ioctl to configure KVM guest's AP matrix
>>   s390/facilities: enable AP facilities needed by guest
> 
> I'll try to summarize all of this in my own words, both to make sure I
> understand the design correctly and to give others a different view on
> this.
> 
> [I'm completely disregarding control domains here.]
> 
> On s390, we have cryptographic coprocessor cards, which are modeled on
> Linux as devices on the AP bus. There's also a concept called domains,
> which means an individual queue of a crypto device is basically a
> (card,domain) tuple. We model this something like the following
> (assuming we have access to cards 3 and 4 and domains 1 and 2):
> 
> AP -> card3 -> queue (3,1)
>             -> queue (3,2)
>    -> card4 -> queue (4,1)
>             -> queue (4,2)
> 
> (The AP bus is a bit different for backwards compat.)
> 
> If we want to virtualize this, we can use a feature provided by the
> hardware. We basically attach a satellite control block to our main
> hardware virtualization control block and the hardware takes care of
> (mostly) everything.
> 
> For this control block, we don't specify explicit tuples, but a list of
> cards and a list of domains. The guest will get access to the cross
> product.
> 
> Because of this, we need to take care that the lists provided to
> different guests don't overlap; i.e., we need to enforce sane
> configurations. Otherwise, one guest may get access to things like
> secret keys for another guest.
> 
> The idea of this patch set is to introduce a new device, the matrix
> device. This matrix device hangs off a different root and acts as the
> node where mdev devices hang off.
> 
> If you now want to give the tuples (4,1) and (4,2), you need to do the
> following:
> 
> - Unbind the (4,1) and (4,2) tuples from their ap bus driver.
> - Bind the (4,1) and (4,2) tuples to the ap matrix driver.
> - Create the mediated device.
> - Assign card 4 and domains 1 and 2.
> 
> QEMU will now simply consume the mediated device and things should work.
> 

This is probably the shortest possible summary I can imagine.
Tony can you double check if it matches your understanding as well?

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Tony Krowiak]

On 10/30/2017 04:57 AM, Christian Borntraeger wrote:
> adding qemu devel and add Daniel and Erik from libvirt to keep them in the loop.
>
> On 10/29/2017 12:11 PM, Cornelia Huck wrote:
>> On Fri, 13 Oct 2017 13:38:45 -0400
>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>
>>> Tony Krowiak (19):
>>>    KVM: s390: SIE considerations for AP Queue virtualization
>>>    KVM: s390: refactor crypto initialization
>>>    s390/zcrypt: new AP matrix bus
>>>    s390/zcrypt: create an AP matrix device on the AP matrix bus
>>>    s390/zcrypt: base implementation of AP matrix device driver
>>>    s390/zcrypt: register matrix device with VFIO mediated device
>>>      framework
>>>    KVM: s390: introduce AP matrix configuration interface
>>>    s390/zcrypt: support for assigning adapters to matrix mdev
>>>    s390/zcrypt: validate adapter assignment
>>>    s390/zcrypt: sysfs interfaces supporting AP domain assignment
>>>    s390/zcrypt: validate domain assignment
>>>    s390/zcrypt: sysfs support for control domain assignment
>>>    s390/zcrypt: validate control domain assignment
>>>    KVM: s390: Connect the AP mediated matrix device to KVM
>>>    s390/zcrypt: introduce ioctl access to VFIO AP Matrix driver
>>>    KVM: s390: interface to configure KVM guest's AP matrix
>>>    KVM: s390: validate input to AP matrix config interface
>>>    KVM: s390: New ioctl to configure KVM guest's AP matrix
>>>    s390/facilities: enable AP facilities needed by guest
>> I'll try to summarize all of this in my own words, both to make sure I
>> understand the design correctly and to give others a different view on
>> this.
>>
>> [I'm completely disregarding control domains here.]
>>
>> On s390, we have cryptographic coprocessor cards, which are modeled on
>> Linux as devices on the AP bus. There's also a concept called domains,
>> which means an individual queue of a crypto device is basically a
>> (card,domain) tuple. We model this something like the following
>> (assuming we have access to cards 3 and 4 and domains 1 and 2):
>>
>> AP -> card3 -> queue (3,1)
>>              -> queue (3,2)
>>     -> card4 -> queue (4,1)
>>              -> queue (4,2)
>>
>> (The AP bus is a bit different for backwards compat.)
>>
>> If we want to virtualize this, we can use a feature provided by the
>> hardware. We basically attach a satellite control block to our main
>> hardware virtualization control block and the hardware takes care of
>> (mostly) everything.
>>
>> For this control block, we don't specify explicit tuples, but a list of
>> cards and a list of domains. The guest will get access to the cross
>> product.
>>
>> Because of this, we need to take care that the lists provided to
>> different guests don't overlap; i.e., we need to enforce sane
>> configurations. Otherwise, one guest may get access to things like
>> secret keys for another guest.
>>
>> The idea of this patch set is to introduce a new device, the matrix
>> device. This matrix device hangs off a different root and acts as the
>> node where mdev devices hang off.
>>
>> If you now want to give the tuples (4,1) and (4,2), you need to do the
>> following:
>>
>> - Unbind the (4,1) and (4,2) tuples from their ap bus driver.
>> - Bind the (4,1) and (4,2) tuples to the ap matrix driver.
>> - Create the mediated device.
>> - Assign card 4 and domains 1 and 2.
>>
>> QEMU will now simply consume the mediated device and things should work.
>>
> This is probably the shortest possible summary I can imagine.
> Tony can you double check if it matches your understanding as well?
This is a concise and accurate summary.
>

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Tony Krowiak]

On 10/13/2017 01:38 PM, Tony Krowiak wrote:
Ping
> Overview:
> --------
> An adjunct processor (AP) facility is an IBM Z cryptographic facility. The
> AP facility is comprised of three AP instructions and from 1 to 256 AP
> adapter cards. The design takes advantage of the interpretive execution mode
> provided by the SIE architecture. With interpretive execution mode, the AP
> instructions executed on the guest are interpreted by the hardware. This
> allows guests direct access to AP adapter cards. The first goal of this
> patch series is to provide direct access by a KVM guest to an AP as a
> pass-through device. The second goal is to provide administrators with the
> means to configure KVM guests to grant direct access to AP facilities
> assigned to the LPAR in which the host linux system is running.
>
> To facilitate the comprehension of the design, let's present an overview of
> the AP architecture.
>
> AP Architectural Overview
> -------------------------
> Let's start with some definitions:
>
> * AP adapter
>
>    An AP adapter is an IBM Z adapter card that can perform cryptographic
>    functionality. There can be from 0 to 256 adapters assigned to an LPAR.
>    Each adapter is identified by a number from 0 to 255.   When
>    installed, an AP is accessed by AP instructions executed by any CPU.
>
> * AP domain
>
>    An adapter can be partitioned into domains. An adapter can hold up to 256
>    domains. Each domain is identified by a number from 0 to 255. Domains can
>    be further classified into two types:
>    
>      * Usage domains are domains that can be accessed directly to process AP
>        commands
>    
>      * Control domains are domains that are accessed indirectly by AP
>        commands sent to a usage domain to control or change the domain.
>
> * AP Queue
>
>    An AP queue is the means by which an AP command is sent to an
>    AP usage domain inside a specific AP. An AP queue is identified by a tuple
>    comprised of an AP adapter ID and a usage domain index corresponding
>    to a given usage domain within the adapter. This tuple forms an AP Queue
>    Number (APQN) uniquely identifying an AP queue. AP instructions include
>    a field containing the APQN to identify the AP queue to which the AP
>    command is targetted.
>
> * AP Instructions:
>
>    There are three AP instructions:
>
>    * NQAP: to enqueue an AP command-request message to a queue
>    * DQAP: to dequeue an AP command-reply message from a queue
>    * PQAP: to adminster the queues
>
> Let's now see how AP instructions are interpreted by the hardware.
>
> Start Interpretive Execution (SIE) Instruction
> ----------------------------------------------
> A KVM guest is started by executing the Start Interpretive Execution (SIE)
> instruction. The SIE state description is a control block that contains the
> state information for a KVM guest and is supplied as input to the SIE
> instruction. The SIE state description contains a field that references
> a Crypto Control Block (CRYCB). The CRYCB contains three bitmask fields
> identifying the adapters, usage domains and control domains assigned to the
> KVM guest:
>
> * The AP Mask (APM) field specifies the AP adapters assigned to the
>    KVM guest. The APM controls which adapters are valid for the KVM guest.
>    The bits in the mask, from left to right, correspond to APIDs
>    0 up to the number of adapters that can be assigned to the LPAR. If a bit
>    is set, the corresponding adapter is valid for use by the KVM guest.
>
> * The AP Queue Mask (AQM) field specifies the AP usage domains assigned
>    to the KVM guest. The bits in the mask, from left to right, correspond
>    to the usage domains, from 0 up to the number of domains that can be
>    assigned to the LPAR. If a bit is set, the corresponding usage domain is
>    valid for use by the KVM guest.
>
> * The AP Domain Mask field specifies the AP control domains assigned to the
>    KVM guest. The ADM bitmask controls which domains can be changed by an AP
>    command-request message sent to a usage domain from the guest. The bits in
>    the mask, from left to right, correspond to domain 0 up to the number of
>    domains that can be assigned to the LPAR. If a bit is set, the
>    corresponding domain can be modified by an AP command-request message
>    sent to a usage domain configured for the KVM guest.
>
> If you recall from the description of an AP Queue, AP instructions include
> an APQN to identify the AP adapter and the specific usage domain within
> the adapter to which an AP command-request message is to be sent (NQAP
> and PQAP instructions), or from which a command-reply message is to be
> received (DQAP instruction). The validity of an APQN is defined by the
> matrix calculated from the APM and AQM; it is the intersection of all
> assigned adapter numbers (APM) with all assigned usage domain numbers (AQM).
> For example, if adapters 1 and 2 and usage domains 5 and 6 are assigned to
> a guest, the APQNs (1,5), (1,6), (2,5) and (2,6) will be valid for the
> guest.
>
> The APQNs provide secure key functionality - i.e., the key is stored on the
> adapter card - so when the adapter card is not virtualized - i.e., the
> adapter is accessed directly by the guest - each APQN must be assigned to
> at most one guest.
>
>     Example 1: Valid configuration:
>     ------------------------------
>     Guest1: adapters 1,2  domains 5,6
>     Guest2: adapter  1,2  domain 7
>
>     This is valid because both guests have a unique set of APQNs: Guest1 has
>     APQNs (1,5), (1,6), (2,5) and (2,6); Guest2 has APQN (1,7) and (2,7).
>
>     Example 2: Invalid configuration:
>     --------------------------------
>     Guest1: adapters 1,2  domains 5,6
>     Guest2: adapter  1    domains 6,7
>
>     This is an invalid configuration because both guests have access to
>     APQNs (1,6).
>
> Interruption architecture:
>
> The AP interruption architecture may or may not generate interruptions to
> signal to the CPU the end of an AP transaction. The SIE interruption
> architecture, depending upon its configuration, may or may not redirect
> AP interrupts directly to a guest if the associated queue is valid for a
> guest, and may or may not report the interruption to the host.
>
> Effective masking for guest level I and II:
>
> A linux host running in the LPAR operates at guest-level 1 and has its own
> SIE state description. When operating at guest-level 1, the masks from the
> host's state description are used directly. A linux guest running in the
> host operates at guest-level 2. When operating at guest-level 2, the masks
> from the guest-level 1 (host) and guest-level 2 (guest) state descriptions
> are combined into a single description called an effective mask by
> performing a logical AND of the two state descriptions.
>
> The effective mask algorithm is used for the APM, AQM and ADM to create
> an EAPM, EAQM and EADM respectively. Use of the EAPM, EAQM and EADM
> precludes a guest-level 1 host program from passing to a guest-level 2
> program APQNs to which it does not have access.
>
> Linux cryptographic bus driver:
>
> Linux already has a cryptographic bus driver that provides one AP device per
> AP adapter and one device per AP queue. There is a device driver for each
> type of AP adapter device and each type of AP queue device. This design
> utilizes some of the interfaces and functionality provided by the AP bus
> driver.
>
> Design Origin:
> -------------
>
> The original design was based on modelling AP Queue devices. The design
> utilized the VFIO mediated device framework whereby a mediated AP queue
> device would be created for each AP Queue bound to the VFIO AP Queue device
> driver. This at first seemed like the most logical design choice for the
> following reasons:
>
> * Securing access to an AP Queue device by unbinding it from its default
>    device driver and binding it to the VFIO device driver would not preclude
>    the host from having access to the other usage domains contained within
>    the same adapter card connected to the AP queue.
>
> * An AP command is sent to a usage domain within a specific AP adapter via
>    an AP queue.
>
> It became readily apparent that modelling the design on an AP queue was very
> convoluted for a number of reasons:
>
>    * There is no convenient way to notify the VFIO device driver which guest
>      will have access to a given mediated AP queue device until the mediated
>      device's file descriptor is opened by the guest. Recall that the APQNs
>      configured for the guest are an intersection of all of the bits set in
>      both the APM and AQM, so the guest's APQNs can not be validated nor
>      its SIE state description configured until all of the guest's mediated
>      AP queue device file descriptors have been opened.
>
>      For example, suppose a guest opens file descriptors for mediated AP
>      queue devices representing APQNs 3,5 and 4,6. If bits 3 and 4 are set in
>      the guest's APM and bits 5 and 6 are set in the guest's AQM, then APQNs
>      (3,5), (3,6), (4,5) and (4,6) will be valid for the guest, but mediated
>      AP queue devices have been created only for APQNs (3,5) and (4,6). In
>      this case, APQNs still assigned to the host would also be available to
>      the guest which is a potential security breach.
>
>    * Control domains are not devices and are not logically modelled as
>      mediated devices. In our original design, they were modelled as
>      attributes of a mediated AP queue device, but this was a clumsy use of
>      the VFIO mediated device model.
>
>    * The SIE state description models the assignment of AP resources as a
>      matrix via the APM, AQM and ADM.
>       
> The design we ultimately settled upon was modelled on the AP matrix as
> defined by the SIE state description. Supplying the complete AP matrix
> to SIE using bitmasks when starting a guest simplifies the code, is far
> easier to secure, and more closely matches the model employed by SIE. This
> is the design model implemented via this patch set.
>
> The Design
> ----------
> This design introduces four new objects:
>
> 1. AP matrix bus
>
>     The sysfs location of the AP matrix bus is /sys/bus/ap_matrix. This
>     bus will create a single AP matrix device (see below).
>
> 2. AP matrix device
>
>     The AP matrix device is a singleton that hangs off of the AP matrix bus.
>     This device holds the AP Queues that have been reserved for use by
>     KVM guests. The sysfs location of the AP matrix device is
>     /sys/devices/ap_matrix/matrix. It is also linked from the AP matrix
>     bus at /sys/bus/ap_matrix/devices/matrix.
>
> 3. VFIO AP matrix driver
>
>     This driver is based on the VFIO mediated device framework. When the
>     driver is initialized, it will:
>
>     * Get the AP matrix device created by AP matrix bus from the bus
>
>     * Register with the AP bus to indicate that it can control AP Queue
>       devices. This allows AP Queue devices unbound from AP device drivers
>       to be bound to the VFIO AP matrix driver. The AP Queues bound to the
>       VFIO AP matrix driver will be stored by the driver in the AP matrix
>       device.
>
>     * Register the AP matrix device with the VFIO mediated device
>       framework (MDEV). Registration with MDEV will create the sysfs
>       structures needed to create mediated matrix devices. Each MDEV matrix
>       device is used to configure the AP matrix for a KVM guest. The MDEV
>       matrix device's file descriptor can be used by QEMU to communicate
>       with the VFIO AP matrix device driver.
>
>     The VFIO AP matrix driver:
>
>     * Provides the interfaces the administrator can use to secure AP Queues
>       for use by KVM guests. This is accomplished by unbinding the AP Queues
>       needed by each KVM guest from its AP device driver and binding it to
>       the VFIO AP queue driver. This prevents the host linux system from
>       using these Queues.
>
>     * Provides an ioctl that can be used by QEMU to configure the
>       CRYCB referenced by the KVM guest's SIE state description. The ioctl
>       will
>
>       * Create an EAPM, EAQM and EADM by performing a logical AND of the
>         APM, AQM and ADM configured via the MDEV matrix device's sysfs
>         attributes files (see below) with the APM, AQM and ADM of the host's
>         SIE state description respectively.
>
>       * Configure the SIE state description for the KVM guest using the
>         effective masks created in the previous step.
>
> 4. VFIO MDEV matrix passthrough device
>
>     An MDEV matrix passthrough device must be created for each KVM guest that
>     will need access to AP facilities. An MDEV matrix passthrough device is
>     used by QEMU to configure the APM, AQM and ADM fields of the CRYCB
>     referenced by the KVM guest's SIE state description. The file descriptor
>     for the MDEV matrix passthrough device provides the communication pathway
>     between QEMU and the VFIO AP matrix device driver.
>
>     The MDEV matrix passthrough device, like the CRYCB, contains three
>     bitmasks - an APM, AQM and ADM - for specifying the AP matrix for the
>     KVM guest. Three sets of attributes files will be provided to allow an
>     administrator to set the bits in the MDEV matrix device's APM, AQM and
>     ADM:
>
>     * A file to assign an AP adapter
>     * A file to unassign an AP adapter
>     * A file to display the adapters assigned
>
>     * A file to assign an AP domain
>     * A file to unassign an AP domain
>     * A file to display the domains assigned
>
>     * A file to assign an AP control domain
>     * A file to unassign an AP control domain
>     * A file to display the control domains assigned
>
> Example:
> -------
> Let's now provide an example to illustrate how KVM guests may be given
> access to AP facilities. For this example, we will show how to configure
> two guests such that executing the lszcrypt command on the guests would
> look like this:
>
> Guest1
> ------
> CARD.DOMAIN TYPE  MODE
> ------------------------------
> 05          CEX5C CCA-Coproc
> 05.0004     CEX5C CCA-Coproc
> 05.00ab     CEX5C CCA-Coproc
> 06          CEX5A Accelerator
> 06.0004     CEX5A Accelerator
> 06.00ab     CEX5C CCA-Coproc
>
> Guest2
> ------
> CARD.DOMAIN TYPE  MODE
> ------------------------------
> 05          CEX5A Accelerator
> 05.0047     CEX5A Accelerator
> 05.00ff     CEX5A Accelerator
>
> One thing to notice in this example is that each AP Queue set is identical.
> For example, the two AP Queue sets for Guest1 both contain APQI 0004 and
> 00ab. It would be an invalid condition if both queue sets did not contain
> the same set of queues. We could not, for example, configure Guest1 with
> access to AP queue 05.00ff because the AP queue set for adapter 06 does not
> contain AP queue 06.00ff. The point is, one must be careful to reserve
> a valid set of AP queues for a given guest.
> a valid configuration.
>
> These are the steps for configuring the Guest1 and Guest2:
>     
> 1. The first thing that needs to be done is to secure the AP queues to be
>     used by the two guests so that the host can not access them. This is done
>     by unbinding each AP Queue device from its respective AP driver. In our
>     example, these queues are bound to the cex4queue driver. This would be
>     the sysfs location of these devices:
>
>     /sys/bus/ap
>     --- [drivers]
>     ------ [cex4queue]
>     --------- [05.0004]
>     --------- [05.0047]
>     --------- [05.00ab]
>     --------- [05.00ff]
>     --------- [06.0004]
>     --------- [06.00ab]
>     --------- unbind
>
>     To unbind AP queue 05.0004 from the cex4queue device driver:
>
> 	echo 05.0004 > unbind
>
>     This must also be done for AP queues 05.00ab, 05.0047, 05.00ff, 06.0004,
>     and 06.00ab.
>
> 2. The next step is to reserve the queues for use by the two KVM guests.
>     This is accomplished by binding them to the VFIO AP matrix device driver.
>     This is the sysfs location of the VFIO AP matrix device driver:
>
>     /sys/bus/ap
>     ---[drivers]
>     ------ [vfio_ap_matrix]
>     ---------- bind
>
>     To bind queue 05.0004 to the vfio_ap_matrix driver:
>
> 	echo 05.0004 > bind
>
>     This must also be done for AP queues 05.00ab, 05.0047, 05.00ff, 06.0004,
>     and 06.00ab.
>
> 3. Create the mediated devices needed to configure the AP matrices for the
>     two guests and to provide an interface to the vfio_ap_matrix driver for
>     use by the guests:
>
>     /sys/devices/
>     --- [ap_matrix]
>     ------ [matrix] (this is the matrix device)
>     --------- [mdev_supported_types]
>     ------------ [ap_matrix-passthrough] (passthrough mediated device type)
>     --------------- create
>     --------------- [devices]
>
>     To create the mediated devices for the two guests:
>
> 	uuidgen > create
> 	uuidgen > create
>
>     This will create two mediated devices in the [devices] subdirectory named
>     with the UUID written to the create attribute file. We call them $uuid1
>     and $uuid2:
>
>     /sys/devices/
>     --- [ap_matrix]
>     ------ [matrix]
>     --------- [mdev_supported_types]
>     ------------ [ap_matrix-passthrough]
>     --------------- [devices]
>     ------------------ [$uuid1]
>     --------------------- adapters
>     --------------------- assign_adapter
>     --------------------- assign_control_domain
>     --------------------- assign_domain
>     --------------------- control_domains
>     --------------------- domains
>     --------------------- unassign_adapter
>     --------------------- unassign_control_domain
>     --------------------- unassign_domain
>     ------------------ [$uuid2]
>     --------------------- adapters
>     --------------------- assign_adapter
>     --------------------- assign_control_domain
>     --------------------- assign_domain
>     --------------------- control_domains
>     --------------------- domains
>     --------------------- unassign_adapter
>     --------------------- unassign_control_domain
>     --------------------- unassign_domain
>
> 4. The administrator now needs to configure the matrices for mediated
>     devices $uuid1 (for Guest1) and $uuid2 (for Guest2).
>
>     This is how the matrix is configured for Guest1:
>
>     echo 5 > assign_adapter
>     echo 6 > assign_adapter
>     echo 4 > assign_domain
>     echo ab > assign_domain
>
>     When the assign.xxx file is written, the corresponding bit in the
>     respective MDEV matrix device's bitmask will be set. For example, when
>     adapter 5 is assigned, bit 5 - numbered from left to right starting with
>     bit 0 - will be set in the MDEV matrix device's APM.
>
>     By architectural convention, all usage domains - i.e., domains assigned
>     via the assign_domain attribute file - will also be configured in the ADM
>     field of the KVM guest's CRYCB, so there is no need to assign control
>     domains here unless you want to assign control domains that are not
>     assigned as usage domains.
>
>     If a mistake is made configuring an adapter, domain or control domain,
>     you can use the unassign_xxx files to unassign the adapter, domain or
>     control domain.
>
>     To display the matrix configuration for Guest1:
>
>     cat adapters
>     cat domains
>     cat control_domains
>
>     This is how the matrix is configured for Guest2:
>
>     echo 5 > assign_adapter
>     echo 47 > assign_domain
>     echo ff > assign_domain
>
> When a KVM guest is started, QEMU will open the file descriptor for its
> MDEV matrix device. The VFIO AP matrix device driver will be notified
> and will store the reference to the KVM guest's SIE state description.
> QEMU will then call the VFIO AP matrix ioctl requesting that the
> KVM guest's matrix be configured. The matrix driver will set the bits in the
> APM, AQM and ADM fields of the CRYCB referenced by the guest's SIE state
> description from the EAPM, EAQM and EADM created by performing a logical AND
> of the AP masks configured in the MDEV matrix device and the masks
> configured in the host's SIE state description. When the guest comes up, it
> will have access to the APQNs identified in the AP matrix specified in the
> KVM guest's SIE state description. Programs running on the guest will then
> be able to use the cryptographic functions provided by the AP facilities
> configured for the guest.
>
> Tony Krowiak (19):
>    KVM: s390: SIE considerations for AP Queue virtualization
>    KVM: s390: refactor crypto initialization
>    s390/zcrypt: new AP matrix bus
>    s390/zcrypt: create an AP matrix device on the AP matrix bus
>    s390/zcrypt: base implementation of AP matrix device driver
>    s390/zcrypt: register matrix device with VFIO mediated device
>      framework
>    KVM: s390: introduce AP matrix configuration interface
>    s390/zcrypt: support for assigning adapters to matrix mdev
>    s390/zcrypt: validate adapter assignment
>    s390/zcrypt: sysfs interfaces supporting AP domain assignment
>    s390/zcrypt: validate domain assignment
>    s390/zcrypt: sysfs support for control domain assignment
>    s390/zcrypt: validate control domain assignment
>    KVM: s390: Connect the AP mediated matrix device to KVM
>    s390/zcrypt: introduce ioctl access to VFIO AP Matrix driver
>    KVM: s390: interface to configure KVM guest's AP matrix
>    KVM: s390: validate input to AP matrix config interface
>    KVM: s390: New ioctl to configure KVM guest's AP matrix
>    s390/facilities: enable AP facilities needed by guest
>
>   MAINTAINERS                                  |   13 +
>   arch/s390/Kconfig                            |   13 +
>   arch/s390/configs/default_defconfig          |    1 +
>   arch/s390/configs/gcov_defconfig             |    1 +
>   arch/s390/configs/performance_defconfig      |    1 +
>   arch/s390/defconfig                          |    1 +
>   arch/s390/include/asm/ap-config.h            |   32 +
>   arch/s390/include/asm/kvm_host.h             |   26 +-
>   arch/s390/kvm/Makefile                       |    2 +-
>   arch/s390/kvm/ap-config.c                    |  224 ++++++++
>   arch/s390/kvm/kvm-s390.c                     |   17 +-
>   arch/s390/tools/gen_facilities.c             |    2 +
>   drivers/s390/crypto/Makefile                 |    6 +-
>   drivers/s390/crypto/ap_matrix_bus.c          |  115 ++++
>   drivers/s390/crypto/ap_matrix_bus.h          |   25 +
>   drivers/s390/crypto/vfio_ap_matrix_drv.c     |  107 ++++
>   drivers/s390/crypto/vfio_ap_matrix_ops.c     |  790 ++++++++++++++++++++++++++
>   drivers/s390/crypto/vfio_ap_matrix_private.h |   50 ++
>   include/uapi/linux/vfio.h                    |   22 +
>   19 files changed, 1438 insertions(+), 10 deletions(-)
>   create mode 100644 arch/s390/include/asm/ap-config.h
>   create mode 100644 arch/s390/kvm/ap-config.c
>   create mode 100644 drivers/s390/crypto/ap_matrix_bus.c
>   create mode 100644 drivers/s390/crypto/ap_matrix_bus.h
>   create mode 100644 drivers/s390/crypto/vfio_ap_matrix_drv.c
>   create mode 100644 drivers/s390/crypto/vfio_ap_matrix_ops.c
>   create mode 100644 drivers/s390/crypto/vfio_ap_matrix_private.h
>

Re: [RFC 01/19] KVM: s390: SIE considerations for AP Queue virtualization

[Christian Borntraeger]

To make the whole series smaller I will apply some of these patches
directly (those of which are "obvious" and touch kvm/s390 only)


On 10/13/2017 07:38 PM, Tony Krowiak wrote:
> The Crypto Control Block (CRYCB) is referenced by the SIE state
> description and controls KVM guest access to the Adjunct
> Processor (AP) adapters, usage domains and control domains.
> This patch defines the AP control blocks to be used for
> controlling guest access to the AP adapters and domains.
> 
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>

thanks applied.
> ---
>  arch/s390/include/asm/kvm_host.h |   25 +++++++++++++++++++++----
>  1 files changed, 21 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index 51375e7..50a6b25 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -685,11 +685,28 @@ struct kvm_s390_crypto {
>  	__u8 dea_kw;
>  };
> 
> +#define APCB0_MASK_SIZE 1
> +struct kvm_s390_apcb0 {
> +	__u64 apm[APCB0_MASK_SIZE];		/* 0x0000 */
> +	__u64 aqm[APCB0_MASK_SIZE];		/* 0x0008 */
> +	__u64 adm[APCB0_MASK_SIZE];		/* 0x0010 */
> +	__u64 reserved18;			/* 0x0018 */
> +};
> +
> +#define APCB1_MASK_SIZE 4
> +struct kvm_s390_apcb1 {
> +	__u64 apm[APCB1_MASK_SIZE];		/* 0x0000 */
> +	__u64 aqm[APCB1_MASK_SIZE];		/* 0x0020 */
> +	__u64 adm[APCB1_MASK_SIZE];		/* 0x0040 */
> +	__u64 reserved60[4];			/* 0x0060 */
> +};
> +
>  struct kvm_s390_crypto_cb {
> -	__u8    reserved00[72];                 /* 0x0000 */
> -	__u8    dea_wrapping_key_mask[24];      /* 0x0048 */
> -	__u8    aes_wrapping_key_mask[32];      /* 0x0060 */
> -	__u8    reserved80[128];                /* 0x0080 */
> +	struct kvm_s390_apcb0 apcb0;		/* 0x0000 */
> +	__u8    reserved20[40];			/* 0x0020 */
> +	__u8    dea_wrapping_key_mask[24];	/* 0x0048 */
> +	__u8    aes_wrapping_key_mask[32];	/* 0x0060 */
> +	struct kvm_s390_apcb1 apcb1;		/* 0x0080 */
>  };
> 
>  /*
> 

Re: [RFC 19/19] s390/facilities: enable AP facilities needed by guest

[Christian Borntraeger]

On 10/16/2017 11:25 AM, Martin Schwidefsky wrote:
> On Fri, 13 Oct 2017 13:39:04 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> 
>> Sets up the following facilities bits to enable the specified AP
>> facilities for the guest VM:
>>     * STFLE.12: Enables the AP Query Configuration Information
>>                 facility. The AP bus running in the guest uses
>>                 the information returned from this instruction
>>                 to configure AP adapters and domains for the
>>                 guest machine.
>>     * STFLE.15: Indicates the AP facilities test is available.
>>                 The AP bus running in the guest uses the
>>                 information.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>>  arch/s390/tools/gen_facilities.c |    2 ++
>>  1 files changed, 2 insertions(+), 0 deletions(-)
>>
>> diff --git a/arch/s390/tools/gen_facilities.c b/arch/s390/tools/gen_facilities.c
>> index 70dd8f1..eeaa7db 100644
>> --- a/arch/s390/tools/gen_facilities.c
>> +++ b/arch/s390/tools/gen_facilities.c
>> @@ -74,8 +74,10 @@ struct facility_def {
>>  			8,  /* enhanced-DAT 1 */
>>  			9,  /* sense-running-status */
>>  			10, /* conditional sske */
>> +			12, /* AP query configuration */
>>  			13, /* ipte-range */
>>  			14, /* nonquiescing key-setting */
>> +			15, /* AP special-command facility */
>>  			73, /* transactional execution */
>>  			75, /* access-exception-fetch/store indication */
>>  			76, /* msa extension 3 */
> 
> With this all KVM guests will always have the AP instructions available, no?
> In principles I like this approach, but it differs from the way z/VM does things,
> there the guest will get an exception if it tries to execute an AP instruction
> if there are no AP devices assigned to the guest. I wonder if there is a reason
> why z/VM does it the way it does.

A good question. For LPAR it seems that you have AP instructions even if you have
no crypto cards.

Re: [RFC 19/19] s390/facilities: enable AP facilities needed by guest

[Halil Pasic]

On 11/02/2017 01:08 PM, Christian Borntraeger wrote:
> 
> 
> On 10/16/2017 11:25 AM, Martin Schwidefsky wrote:
>> On Fri, 13 Oct 2017 13:39:04 -0400
>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>
>>> Sets up the following facilities bits to enable the specified AP
>>> facilities for the guest VM:
>>>     * STFLE.12: Enables the AP Query Configuration Information
>>>                 facility. The AP bus running in the guest uses
>>>                 the information returned from this instruction
>>>                 to configure AP adapters and domains for the
>>>                 guest machine.
>>>     * STFLE.15: Indicates the AP facilities test is available.
>>>                 The AP bus running in the guest uses the
>>>                 information.
>>>
>>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>>> ---
>>>  arch/s390/tools/gen_facilities.c |    2 ++
>>>  1 files changed, 2 insertions(+), 0 deletions(-)
>>>
>>> diff --git a/arch/s390/tools/gen_facilities.c b/arch/s390/tools/gen_facilities.c
>>> index 70dd8f1..eeaa7db 100644
>>> --- a/arch/s390/tools/gen_facilities.c
>>> +++ b/arch/s390/tools/gen_facilities.c
>>> @@ -74,8 +74,10 @@ struct facility_def {
>>>  			8,  /* enhanced-DAT 1 */
>>>  			9,  /* sense-running-status */
>>>  			10, /* conditional sske */
>>> +			12, /* AP query configuration */
>>>  			13, /* ipte-range */
>>>  			14, /* nonquiescing key-setting */
>>> +			15, /* AP special-command facility */
>>>  			73, /* transactional execution */
>>>  			75, /* access-exception-fetch/store indication */
>>>  			76, /* msa extension 3 */
>>
>> With this all KVM guests will always have the AP instructions available, no?
>> In principles I like this approach, but it differs from the way z/VM does things,
>> there the guest will get an exception if it tries to execute an AP instruction
>> if there are no AP devices assigned to the guest. I wonder if there is a reason
>> why z/VM does it the way it does.
> 
> A good question. For LPAR it seems that you have AP instructions even if you have
> no crypto cards.
> 

I've tried to figure these things out last week but I've failed.

Right at the beginning of  AR-10334-03-POK we have this text:
"An adjunct processor (AP) facility consists of the three AP
instructions, and one to sixty-four APs.". This reads like if we
have AP facility we have to have at least one AP.

But when I've tried to get a better understanding how the
presence/absence of the AP facility is indicated and what facility
bits do we have in this context I got confused.

Tony, could you please give us a detailed summary on this (best with
references, and focusing on the (guest) program perspective)?

Regards,
Halil

Re: [RFC 02/19] KVM: s390: refactor crypto initialization

[Christian Borntraeger]

On 10/13/2017 07:38 PM, Tony Krowiak wrote:
> This patch introduces the following changes to crypto initialization.
> 
> * For key management operations support, the crypto control block
>   (CRYCB) referenced by the KVM guest's SIE state description is
>   formatted only if the Message-Security-Assist (MSA) extension 3
>   facility is installed (STFLE.76 is set). Virtualization of AP
>   facilities, however; requires that a CRYCB of the appropriate
>   format be made available to SIE regardless of the value of STFLE.76.
> 
> * The Execution Controls A (ECA) field bit 28 in the SIE block needs
>   to be set to enable interpretive execution mode of adjunct processor (AP)
>   instructions.

We should fence setting ECA to cases where we have virtualization capability 
for crypto. In addition we need to bind this somehow to the CPU model, so
I guess we need to add some CRYPTO feature e.g. add KVM_S390_VM_CPU_FEAT_AP to the
list of know features
(see arch/s390/include/uapi/asm/kvm.h)
---snip---
#define KVM_S390_VM_CPU_FEAT_ESOP       0
#define KVM_S390_VM_CPU_FEAT_SIEF2      1
#define KVM_S390_VM_CPU_FEAT_64BSCAO    2
#define KVM_S390_VM_CPU_FEAT_SIIF       3
#define KVM_S390_VM_CPU_FEAT_GPERE      4
#define KVM_S390_VM_CPU_FEAT_GSLS       5
#define KVM_S390_VM_CPU_FEAT_IB         6
#define KVM_S390_VM_CPU_FEAT_CEI        7
#define KVM_S390_VM_CPU_FEAT_IBS        8
#define KVM_S390_VM_CPU_FEAT_SKEY       9
#define KVM_S390_VM_CPU_FEAT_CMMA       10
#define KVM_S390_VM_CPU_FEAT_PFMFI      11
#define KVM_S390_VM_CPU_FEAT_SIGPIF     12
#define KVM_S390_VM_CPU_FEAT_KSS        13
---snip---


I will try to find out a way to properly detect that.



> 
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>  arch/s390/include/asm/kvm_host.h |    1 +
>  arch/s390/kvm/kvm-s390.c         |   17 +++++++++++++----
>  2 files changed, 14 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index 50a6b25..5683f18 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -188,6 +188,7 @@ struct kvm_s390_sie_block {
>  #define ECA_MVPGI	0x01000000
>  #define ECA_VX		0x00020000
>  #define ECA_PROTEXCI	0x00002000
> +#define ECA_AP		0x00000008
>  #define ECA_SII		0x00000001
>  	__u32	eca;			/* 0x004c */
>  #define ICPT_INST	0x04
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 40d0a1a..e57fc9b 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -1819,7 +1819,9 @@ static void kvm_s390_set_crycb_format(struct kvm *kvm)
>  {
>  	kvm->arch.crypto.crycbd = (__u32)(unsigned long) kvm->arch.crypto.crycb;
> 
> -	if (kvm_s390_apxa_installed())
> +	if (!test_kvm_facility(kvm, 76))
> +		kvm->arch.crypto.crycbd &= ~(CRYCB_FORMAT2); /* format 0 */
> +	else if (kvm_s390_apxa_installed())
>  		kvm->arch.crypto.crycbd |= CRYCB_FORMAT2;
>  	else
>  		kvm->arch.crypto.crycbd |= CRYCB_FORMAT1;
> @@ -1836,12 +1838,12 @@ static u64 kvm_s390_get_initial_cpuid(void)
> 
>  static void kvm_s390_crypto_init(struct kvm *kvm)
>  {
> -	if (!test_kvm_facility(kvm, 76))
> -		return;
> -
>  	kvm->arch.crypto.crycb = &kvm->arch.sie_page2->crycb;
>  	kvm_s390_set_crycb_format(kvm);
> 
> +	if (!test_kvm_facility(kvm, 76))
> +		return;
> +
>  	/* Enable AES/DEA protected key functions by default */
>  	kvm->arch.crypto.aes_kw = 1;
>  	kvm->arch.crypto.dea_kw = 1;
> @@ -2366,8 +2368,15 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)
>  	vcpu->arch.enabled_gmap = vcpu->arch.gmap;
>  }
> 
> +static void kvm_s390_vcpu_set_crypto_exec_mode(struct kvm_vcpu *vcpu)
> +{
> +	vcpu->arch.sie_block->eca |= ECA_AP;
> +}
> +
>  static void kvm_s390_vcpu_crypto_setup(struct kvm_vcpu *vcpu)
>  {
> +	kvm_s390_vcpu_set_crypto_exec_mode(vcpu);
> +
>  	if (!test_kvm_facility(vcpu->kvm, 76))
>  		return;
> 

Re: [RFC 19/19] s390/facilities: enable AP facilities needed by guest

[Christian Borntraeger]

On 11/02/2017 04:36 PM, Tony Krowiak wrote:
> On 11/02/2017 08:08 AM, Christian Borntraeger wrote:
>>
>> On 10/16/2017 11:25 AM, Martin Schwidefsky wrote:
>>> On Fri, 13 Oct 2017 13:39:04 -0400
>>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>>
>>>> Sets up the following facilities bits to enable the specified AP
>>>> facilities for the guest VM:
>>>>     * STFLE.12: Enables the AP Query Configuration Information
>>>>                 facility. The AP bus running in the guest uses
>>>>                 the information returned from this instruction
>>>>                 to configure AP adapters and domains for the
>>>>                 guest machine.
>>>>     * STFLE.15: Indicates the AP facilities test is available.
>>>>                 The AP bus running in the guest uses the
>>>>                 information.
>>>>
>>>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>>>> ---
>>>>  arch/s390/tools/gen_facilities.c |    2 ++
>>>>  1 files changed, 2 insertions(+), 0 deletions(-)
>>>>
>>>> diff --git a/arch/s390/tools/gen_facilities.c b/arch/s390/tools/gen_facilities.c
>>>> index 70dd8f1..eeaa7db 100644
>>>> --- a/arch/s390/tools/gen_facilities.c
>>>> +++ b/arch/s390/tools/gen_facilities.c
>>>> @@ -74,8 +74,10 @@ struct facility_def {
>>>>  			8,  /* enhanced-DAT 1 */
>>>>  			9,  /* sense-running-status */
>>>>  			10, /* conditional sske */
>>>> +			12, /* AP query configuration */
>>>>  			13, /* ipte-range */
>>>>  			14, /* nonquiescing key-setting */
>>>> +			15, /* AP special-command facility */
>>>>  			73, /* transactional execution */
>>>>  			75, /* access-exception-fetch/store indication */
>>>>  			76, /* msa extension 3 */
>>> With this all KVM guests will always have the AP instructions available, no?
>>> In principles I like this approach, but it differs from the way z/VM does things,
>>> there the guest will get an exception if it tries to execute an AP instruction
>>> if there are no AP devices assigned to the guest. I wonder if there is a reason
>>> why z/VM does it the way it does.
>> A good question. For LPAR it seems that you have AP instructions even if you have
>> no crypto cards.
>>
> I don't believe these facilities control whether or not AP instructions will be available
> 
> to the guest.

This is actually handled by your patch2 enabling the ECA bit.
I think we must decide if we want to be able to disable these instructions
via the cpu model. If yes we must then couple the facilities with the enablement.

Re: [RFC 19/19] s390/facilities: enable AP facilities needed by guest

[Tony Krowiak]

On 11/02/2017 11:53 AM, Christian Borntraeger wrote:
>
> On 11/02/2017 04:36 PM, Tony Krowiak wrote:
>> On 11/02/2017 08:08 AM, Christian Borntraeger wrote:
>>> On 10/16/2017 11:25 AM, Martin Schwidefsky wrote:
>>>> On Fri, 13 Oct 2017 13:39:04 -0400
>>>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>>>
>>>>> Sets up the following facilities bits to enable the specified AP
>>>>> facilities for the guest VM:
>>>>>      * STFLE.12: Enables the AP Query Configuration Information
>>>>>                  facility. The AP bus running in the guest uses
>>>>>                  the information returned from this instruction
>>>>>                  to configure AP adapters and domains for the
>>>>>                  guest machine.
>>>>>      * STFLE.15: Indicates the AP facilities test is available.
>>>>>                  The AP bus running in the guest uses the
>>>>>                  information.
>>>>>
>>>>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>>>>> ---
>>>>>   arch/s390/tools/gen_facilities.c |    2 ++
>>>>>   1 files changed, 2 insertions(+), 0 deletions(-)
>>>>>
>>>>> diff --git a/arch/s390/tools/gen_facilities.c b/arch/s390/tools/gen_facilities.c
>>>>> index 70dd8f1..eeaa7db 100644
>>>>> --- a/arch/s390/tools/gen_facilities.c
>>>>> +++ b/arch/s390/tools/gen_facilities.c
>>>>> @@ -74,8 +74,10 @@ struct facility_def {
>>>>>   			8,  /* enhanced-DAT 1 */
>>>>>   			9,  /* sense-running-status */
>>>>>   			10, /* conditional sske */
>>>>> +			12, /* AP query configuration */
>>>>>   			13, /* ipte-range */
>>>>>   			14, /* nonquiescing key-setting */
>>>>> +			15, /* AP special-command facility */
>>>>>   			73, /* transactional execution */
>>>>>   			75, /* access-exception-fetch/store indication */
>>>>>   			76, /* msa extension 3 */
>>>> With this all KVM guests will always have the AP instructions available, no?
>>>> In principles I like this approach, but it differs from the way z/VM does things,
>>>> there the guest will get an exception if it tries to execute an AP instruction
>>>> if there are no AP devices assigned to the guest. I wonder if there is a reason
>>>> why z/VM does it the way it does.
>>> A good question. For LPAR it seems that you have AP instructions even if you have
>>> no crypto cards.
>>>
>> I don't believe these facilities control whether or not AP instructions will be available
>>
>> to the guest.
> This is actually handled by your patch2 enabling the ECA bit.
> I think we must decide if we want to be able to disable these instructions
> via the cpu model. If yes we must then couple the facilities with the enablement.
The ECA.28 bit controls whether instructions are intercepted or 
interpreted - i.e., handled via hardware
virtualization. If set, as is done in patch2, then instructions will be 
interpreted. I don't see how
that affects enabling or disabling AP instructions, unless we don't set 
ECA.28, intercept every instruction
and program check. Am I missing something here?
>

Re: [RFC 18/19] KVM: s390: New ioctl to configure KVM guest's AP matrix

[Tony Krowiak]

On 10/13/2017 01:39 PM, Tony Krowiak wrote:
> Implements an ioctl to configure the adapters, usage domains
> and control domains to which a KVM guest will be granted
> access. The ioctl is invoked using the VFIO mediated matrix
> device's file descriptor.
>
> Guest access to AP adapters, usage domains and control domains
> is controlled by three bit masks referenced from the
> guest's SIE state description: The AP Mask (APM) controls
> access to the AP adapters; the AP Queue Mask (AQM) controls
> access to the AP usage domains; and the AP Domain Mask (ADM)
> controls access to the AP control domains. Each bit
> in the APM represents an adapter, from left to right,
> starting with adapter 0. Each bit in the AQM represents
> a usage domain, from left to right, starting with domain 0.
> Each bit in the ADM represents a control domain, from left
> to right, starting with domain 0.
>
> The APM, AQM and ADM bit masks will be populated from the
> bit masks stored in the corresponding mediated matrix device's
> sysfs files. The bit masks will also be returned by the ioctl
> call to the caller.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>   drivers/s390/crypto/vfio_ap_matrix_ops.c |   68 +++++++++++++++++++++++++++++-
>   include/uapi/linux/vfio.h                |   20 +++++++++
>   2 files changed, 86 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/s390/crypto/vfio_ap_matrix_ops.c b/drivers/s390/crypto/vfio_ap_matrix_ops.c
> index 74ea8db..d81e3ab 100644
> --- a/drivers/s390/crypto/vfio_ap_matrix_ops.c
> +++ b/drivers/s390/crypto/vfio_ap_matrix_ops.c
> @@ -160,8 +160,69 @@ static int ap_matrix_mdev_get_device_info(unsigned long arg)
>   	return copy_to_user((void __user *)arg, &info, minsz);
>   }
>
> -static ssize_t ap_matrix_mdev_ioctl(struct mdev_device *mdev,
> -				    unsigned int cmd, unsigned long arg)
> +static int ap_matrix_configure(struct mdev_device *mdev)
> +{
> +	int ret;
> +	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
> +
> +	ret = ap_config_matrix(matrix_mdev->kvm, &matrix_mdev->masks);
> +	if (ret)
> +		return ret;
> +
> +	return 0;
> +}
> +
> +static int ap_matrix_copy_to_user(struct mdev_device *mdev,
> +				  struct vfio_ap_matrix_config *mcfg,
> +				  unsigned long arg)
> +{
> +	size_t i;
> +	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
> +
> +
> +	for (i = 0; i < AP_MATRIX_MASK_INDICES; i++) {
> +		if (i < VFIO_AP_MATRIX_MASK_INDICES) {
> +			mcfg->apm[i] = matrix_mdev->masks.apm[i];
> +			mcfg->aqm[i] = matrix_mdev->masks.aqm[i];
> +			mcfg->adm[i] = matrix_mdev->masks.adm[i];
> +		}
> +	}
> +
> +	if (copy_to_user((void __user *)arg, &mcfg, sizeof(mcfg)))
> +		return -EFAULT;
This is in error and needs to be:

if (copy_to_user((void __user *)arg, &mcfg, sizeof(*mcfg)))
	return -EFAULT;

> +
> +	return 0;
> +}
> +
> +static ssize_t ap_matrix_mdev_configure(struct mdev_device *mdev,
> +					unsigned long arg)
> +{
> +	int ret;
> +	unsigned long minsz;
> +	struct vfio_ap_matrix_config mcfg;
> +
> +	minsz = offsetofend(struct vfio_ap_matrix_config, adm);
> +
> +	if (copy_from_user(&mcfg, (void __user *)arg, minsz))
> +		return -EFAULT;
> +
> +	if (mcfg.argsz < minsz) {
> +		pr_err("%s: Argument size %u less than min size %li",
> +		       VFIO_AP_MATRIX_MODULE_NAME, mcfg.argsz, minsz);
> +		return -EINVAL;
> +	}
> +
> +	ret = ap_matrix_configure(mdev);
> +	if (ret)
> +		return ret;
> +
> +	ret = ap_matrix_copy_to_user(mdev, &mcfg, arg);
> +
> +	return 0;
> +}
> +
> +static long ap_matrix_mdev_ioctl(struct mdev_device *mdev,
> +				 unsigned int cmd, unsigned long arg)
>   {
>   	int ret;
>
> @@ -169,6 +230,9 @@ static ssize_t ap_matrix_mdev_ioctl(struct mdev_device *mdev,
>   	case VFIO_DEVICE_GET_INFO:
>   		ret = ap_matrix_mdev_get_device_info(arg);
>   		break;
> +	case VFIO_AP_MATRIX_CONFIGURE:
> +		ret = ap_matrix_mdev_configure(mdev, arg);
> +		break;
>   	default:
>   		pr_err("%s: ioctl command %d is not a supported command",
>   		       VFIO_AP_MATRIX_MODULE_NAME, cmd);
> diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
> index 08f1ab4..2d96c57 100644
> --- a/include/uapi/linux/vfio.h
> +++ b/include/uapi/linux/vfio.h
> @@ -716,6 +716,26 @@ struct vfio_iommu_spapr_tce_remove {
>   };
>   #define VFIO_IOMMU_SPAPR_TCE_REMOVE	_IO(VFIO_TYPE, VFIO_BASE + 20)
>
> +/**
> + * VFIO_AP_MATRIX_CONFIGURE		_IO(VFIO_TYPE, VFIO_BASE + 21
> + *
> + * Configure the AP matrix for a KVM guest.
> + */
> +#define VFIO_AP_MATRIX_CONFIGURE	_IO(VFIO_TYPE, VFIO_BASE + 21)
> +
> +#define VFIO_AP_MATRIX_MASK_INDICES	4
> +#define VFIO_AP_MATTRIX_MASK_BYTES	(VFIO_AP_MATRIX_MASK_INDICES * \
> +					 sizeof(__u64))
> +#define VFIO_AP_MATRIX_MASK_BITS	(VFIO_AP_MATTRIX_MASK_BYTES * 8)
> +
> +struct vfio_ap_matrix_config {
> +	__u32 argsz;
> +	__u32 flags;
> +	/* out */
> +	__u64 apm[VFIO_AP_MATRIX_MASK_INDICES];
> +	__u64 aqm[VFIO_AP_MATRIX_MASK_INDICES];
> +	__u64 adm[VFIO_AP_MATRIX_MASK_INDICES];
> +};
>   /* ***************************************************************** */
>
>   #endif /* _UAPIVFIO_H */

Re: [RFC 01/19] KVM: s390: SIE considerations for AP Queue virtualization

[Tony Krowiak]

On 11/02/2017 07:54 AM, Christian Borntraeger wrote:
> To make the whole series smaller I will apply some of these patches
> directly (those of which are "obvious" and touch kvm/s390 only)
Okay
>
>
> On 10/13/2017 07:38 PM, Tony Krowiak wrote:
>> The Crypto Control Block (CRYCB) is referenced by the SIE state
>> description and controls KVM guest access to the Adjunct
>> Processor (AP) adapters, usage domains and control domains.
>> This patch defines the AP control blocks to be used for
>> controlling guest access to the AP adapters and domains.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> thanks applied.
>> ---
>>   arch/s390/include/asm/kvm_host.h |   25 +++++++++++++++++++++----
>>   1 files changed, 21 insertions(+), 4 deletions(-)
>>
>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>> index 51375e7..50a6b25 100644
>> --- a/arch/s390/include/asm/kvm_host.h
>> +++ b/arch/s390/include/asm/kvm_host.h
>> @@ -685,11 +685,28 @@ struct kvm_s390_crypto {
>>   	__u8 dea_kw;
>>   };
>>
>> +#define APCB0_MASK_SIZE 1
>> +struct kvm_s390_apcb0 {
>> +	__u64 apm[APCB0_MASK_SIZE];		/* 0x0000 */
>> +	__u64 aqm[APCB0_MASK_SIZE];		/* 0x0008 */
>> +	__u64 adm[APCB0_MASK_SIZE];		/* 0x0010 */
>> +	__u64 reserved18;			/* 0x0018 */
>> +};
>> +
>> +#define APCB1_MASK_SIZE 4
>> +struct kvm_s390_apcb1 {
>> +	__u64 apm[APCB1_MASK_SIZE];		/* 0x0000 */
>> +	__u64 aqm[APCB1_MASK_SIZE];		/* 0x0020 */
>> +	__u64 adm[APCB1_MASK_SIZE];		/* 0x0040 */
>> +	__u64 reserved60[4];			/* 0x0060 */
>> +};
>> +
>>   struct kvm_s390_crypto_cb {
>> -	__u8    reserved00[72];                 /* 0x0000 */
>> -	__u8    dea_wrapping_key_mask[24];      /* 0x0048 */
>> -	__u8    aes_wrapping_key_mask[32];      /* 0x0060 */
>> -	__u8    reserved80[128];                /* 0x0080 */
>> +	struct kvm_s390_apcb0 apcb0;		/* 0x0000 */
>> +	__u8    reserved20[40];			/* 0x0020 */
>> +	__u8    dea_wrapping_key_mask[24];	/* 0x0048 */
>> +	__u8    aes_wrapping_key_mask[32];	/* 0x0060 */
>> +	struct kvm_s390_apcb1 apcb1;		/* 0x0080 */
>>   };
>>
>>   /*
>>

Re: [RFC 19/19] s390/facilities: enable AP facilities needed by guest

[Christian Borntraeger]

On 11/02/2017 07:49 PM, Tony Krowiak wrote:
> On 11/02/2017 11:53 AM, Christian Borntraeger wrote:
>>
>> On 11/02/2017 04:36 PM, Tony Krowiak wrote:
>>> On 11/02/2017 08:08 AM, Christian Borntraeger wrote:
>>>> On 10/16/2017 11:25 AM, Martin Schwidefsky wrote:
>>>>> On Fri, 13 Oct 2017 13:39:04 -0400
>>>>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>>>>
>>>>>> Sets up the following facilities bits to enable the specified AP
>>>>>> facilities for the guest VM:
>>>>>>      * STFLE.12: Enables the AP Query Configuration Information
>>>>>>                  facility. The AP bus running in the guest uses
>>>>>>                  the information returned from this instruction
>>>>>>                  to configure AP adapters and domains for the
>>>>>>                  guest machine.
>>>>>>      * STFLE.15: Indicates the AP facilities test is available.
>>>>>>                  The AP bus running in the guest uses the
>>>>>>                  information.
>>>>>>
>>>>>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>>>>>> ---
>>>>>>   arch/s390/tools/gen_facilities.c |    2 ++
>>>>>>   1 files changed, 2 insertions(+), 0 deletions(-)
>>>>>>
>>>>>> diff --git a/arch/s390/tools/gen_facilities.c b/arch/s390/tools/gen_facilities.c
>>>>>> index 70dd8f1..eeaa7db 100644
>>>>>> --- a/arch/s390/tools/gen_facilities.c
>>>>>> +++ b/arch/s390/tools/gen_facilities.c
>>>>>> @@ -74,8 +74,10 @@ struct facility_def {
>>>>>>               8,  /* enhanced-DAT 1 */
>>>>>>               9,  /* sense-running-status */
>>>>>>               10, /* conditional sske */
>>>>>> +            12, /* AP query configuration */
>>>>>>               13, /* ipte-range */
>>>>>>               14, /* nonquiescing key-setting */
>>>>>> +            15, /* AP special-command facility */
>>>>>>               73, /* transactional execution */
>>>>>>               75, /* access-exception-fetch/store indication */
>>>>>>               76, /* msa extension 3 */
>>>>> With this all KVM guests will always have the AP instructions available, no?
>>>>> In principles I like this approach, but it differs from the way z/VM does things,
>>>>> there the guest will get an exception if it tries to execute an AP instruction
>>>>> if there are no AP devices assigned to the guest. I wonder if there is a reason
>>>>> why z/VM does it the way it does.
>>>> A good question. For LPAR it seems that you have AP instructions even if you have
>>>> no crypto cards.
>>>>
>>> I don't believe these facilities control whether or not AP instructions will be available
>>>
>>> to the guest.
>> This is actually handled by your patch2 enabling the ECA bit.
>> I think we must decide if we want to be able to disable these instructions
>> via the cpu model. If yes we must then couple the facilities with the enablement.
> The ECA.28 bit controls whether instructions are intercepted or interpreted - i.e., handled via hardware
> virtualization. If set, as is done in patch2, then instructions will be interpreted. I don't see how
> that affects enabling or disabling AP instructions, unless we don't set ECA.28, intercept every instruction
> and program check. Am I missing something here?

If we do not set ECA.28 these instructions intercept and we (the hypervisor) can then
decide what to do. For example we can give an PIC01 operation exception (illegal
instruction) - thats what we do today.

Now: if we want to be able to migrate a guest from a new kernel back to an old kernel,
there must be a way to disable the new behaviour so that the user can configure a guest
that does NOT have these 3 instructions. That means, I want to bind the ap instruction
to a cpu model feature, so that we only enable ECA.28 and the facility bits, if the
feature is enabled in the CPU model. Otherwise we have no control on what happens
when the guest issues these instructions.

Imagine what happens if we not do this and you migrate from an identical hw with an
identical libvirt/qemu but from a new kernel to an old kernel:

The guest boots starts up on the new kernel
guest kernel: drivers/s390/crypto/ap_bus.c  ap_module_init -> ap_instructions_available
checks if the instructions work. They do and now the guest driver assumes that all
instructions will continue to work.

Now the guest is migrated back to an old kernel
sooner or later the ap_scan_bus kthread will run to scan the bus (or some crypto operation
is started) and the instruction will be rejected with a PIC01. kernel oops.

Re: [RFC 19/19] s390/facilities: enable AP facilities needed by guest

[Christian Borntraeger]

On 11/03/2017 09:14 AM, Harald Freudenberger wrote:
> I talked with the firmware guys about this.
> The answer is that the AP 3 instructions (NQAP, DQAP, PQAP) are always available.

That is true for basic mode and LPAR, but certainly not for z/VM or KVM guests.


> Tony is right with the facility bits.
> facility bit 15 tells if the T bit is supported with the PQAP(TAPQ) subcommand,
> facility bit 12 tells if the PQAP(QCI) subcommand is available.
> 
> As far as I could discover the AP instructions where optional on some older machines.
> This may be the reason why zVM throws an exception if there is no AP device assigned.
> However, for all machines still in service the AP instructions are available at al time.

We have to distinguish LPARs and 2nd level guests under KVM/z/VM. For those 2nd level
guests there are configurations out there TODAY that do not support PQAP and friends.

Re: [RFC 02/19] KVM: s390: refactor crypto initialization

[Cornelia Huck]

On Thu, 2 Nov 2017 13:41:18 +0100
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> On 10/13/2017 07:38 PM, Tony Krowiak wrote:
> > This patch introduces the following changes to crypto initialization.
> > 
> > * For key management operations support, the crypto control block
> >   (CRYCB) referenced by the KVM guest's SIE state description is
> >   formatted only if the Message-Security-Assist (MSA) extension 3
> >   facility is installed (STFLE.76 is set). Virtualization of AP
> >   facilities, however; requires that a CRYCB of the appropriate
> >   format be made available to SIE regardless of the value of STFLE.76.
> > 
> > * The Execution Controls A (ECA) field bit 28 in the SIE block needs
> >   to be set to enable interpretive execution mode of adjunct processor (AP)
> >   instructions.  
> 
> We should fence setting ECA to cases where we have virtualization capability 
> for crypto. In addition we need to bind this somehow to the CPU model, so
> I guess we need to add some CRYPTO feature e.g. add KVM_S390_VM_CPU_FEAT_AP to the
> list of know features
> (see arch/s390/include/uapi/asm/kvm.h)
> ---snip---
> #define KVM_S390_VM_CPU_FEAT_ESOP       0
> #define KVM_S390_VM_CPU_FEAT_SIEF2      1
> #define KVM_S390_VM_CPU_FEAT_64BSCAO    2
> #define KVM_S390_VM_CPU_FEAT_SIIF       3
> #define KVM_S390_VM_CPU_FEAT_GPERE      4
> #define KVM_S390_VM_CPU_FEAT_GSLS       5
> #define KVM_S390_VM_CPU_FEAT_IB         6
> #define KVM_S390_VM_CPU_FEAT_CEI        7
> #define KVM_S390_VM_CPU_FEAT_IBS        8
> #define KVM_S390_VM_CPU_FEAT_SKEY       9
> #define KVM_S390_VM_CPU_FEAT_CMMA       10
> #define KVM_S390_VM_CPU_FEAT_PFMFI      11
> #define KVM_S390_VM_CPU_FEAT_SIGPIF     12
> #define KVM_S390_VM_CPU_FEAT_KSS        13
> ---snip---

So, we need this so userspace can add the appropriate flags, right?

> 
> 
> I will try to find out a way to properly detect that.

Did you manage to find out?

> 
> 
> 
> > 
> > Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> > ---
> >  arch/s390/include/asm/kvm_host.h |    1 +
> >  arch/s390/kvm/kvm-s390.c         |   17 +++++++++++++----
> >  2 files changed, 14 insertions(+), 4 deletions(-)
> > 
> > diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> > index 50a6b25..5683f18 100644
> > --- a/arch/s390/include/asm/kvm_host.h
> > +++ b/arch/s390/include/asm/kvm_host.h
> > @@ -188,6 +188,7 @@ struct kvm_s390_sie_block {
> >  #define ECA_MVPGI	0x01000000
> >  #define ECA_VX		0x00020000
> >  #define ECA_PROTEXCI	0x00002000
> > +#define ECA_AP		0x00000008
> >  #define ECA_SII		0x00000001
> >  	__u32	eca;			/* 0x004c */
> >  #define ICPT_INST	0x04
> > diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> > index 40d0a1a..e57fc9b 100644
> > --- a/arch/s390/kvm/kvm-s390.c
> > +++ b/arch/s390/kvm/kvm-s390.c
> > @@ -1819,7 +1819,9 @@ static void kvm_s390_set_crycb_format(struct kvm *kvm)
> >  {
> >  	kvm->arch.crypto.crycbd = (__u32)(unsigned long) kvm->arch.crypto.crycb;
> > 
> > -	if (kvm_s390_apxa_installed())
> > +	if (!test_kvm_facility(kvm, 76))
> > +		kvm->arch.crypto.crycbd &= ~(CRYCB_FORMAT2); /* format 0 */
> > +	else if (kvm_s390_apxa_installed())
> >  		kvm->arch.crypto.crycbd |= CRYCB_FORMAT2;
> >  	else
> >  		kvm->arch.crypto.crycbd |= CRYCB_FORMAT1;
> > @@ -1836,12 +1838,12 @@ static u64 kvm_s390_get_initial_cpuid(void)
> > 
> >  static void kvm_s390_crypto_init(struct kvm *kvm)
> >  {
> > -	if (!test_kvm_facility(kvm, 76))
> > -		return;
> > -
> >  	kvm->arch.crypto.crycb = &kvm->arch.sie_page2->crycb;
> >  	kvm_s390_set_crycb_format(kvm);
> > 
> > +	if (!test_kvm_facility(kvm, 76))
> > +		return;
> > +
> >  	/* Enable AES/DEA protected key functions by default */
> >  	kvm->arch.crypto.aes_kw = 1;
> >  	kvm->arch.crypto.dea_kw = 1;
> > @@ -2366,8 +2368,15 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)
> >  	vcpu->arch.enabled_gmap = vcpu->arch.gmap;
> >  }
> > 
> > +static void kvm_s390_vcpu_set_crypto_exec_mode(struct kvm_vcpu *vcpu)
> > +{
> > +	vcpu->arch.sie_block->eca |= ECA_AP;
> > +}
> > +
> >  static void kvm_s390_vcpu_crypto_setup(struct kvm_vcpu *vcpu)
> >  {
> > +	kvm_s390_vcpu_set_crypto_exec_mode(vcpu);
> > +
> >  	if (!test_kvm_facility(vcpu->kvm, 76))
> >  		return;
> >   
> 

Re: [RFC 03/19] s390/zcrypt: new AP matrix bus

[Cornelia Huck]

On Fri, 13 Oct 2017 13:38:48 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> Introduces an AP matrix bus. The sysfs location of the
> AP matrix bus is:
> 
> /sys/bus/ap_matrix
> 
> The AP matrix bus will create an AP matrix device that will
> hold the AP queues reserved for use by KVM guests.
> 
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>  MAINTAINERS                         |    8 +++++
>  drivers/s390/crypto/Makefile        |    4 +-
>  drivers/s390/crypto/ap_matrix_bus.c |   52 +++++++++++++++++++++++++++++++++++
>  drivers/s390/crypto/ap_matrix_bus.h |   15 ++++++++++
>  4 files changed, 77 insertions(+), 2 deletions(-)
>  create mode 100644 drivers/s390/crypto/ap_matrix_bus.c
>  create mode 100644 drivers/s390/crypto/ap_matrix_bus.h
> 

> diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile
> index be36f10..87646ca 100644
> --- a/drivers/s390/crypto/Makefile
> +++ b/drivers/s390/crypto/Makefile
> @@ -2,7 +2,7 @@
>  # S/390 crypto devices
>  #
>  
> -ap-objs := ap_bus.o ap_card.o ap_queue.o
> +ap-objs := ap_bus.o ap_card.o ap_queue.o ap_matrix_bus.o
>  obj-$(subst m,y,$(CONFIG_ZCRYPT)) += ap.o
>  # zcrypt_api.o and zcrypt_msgtype*.o depend on ap.o
>  zcrypt-objs := zcrypt_api.o zcrypt_card.o zcrypt_queue.o
> @@ -13,4 +13,4 @@ obj-$(CONFIG_ZCRYPT) += zcrypt_pcixcc.o zcrypt_cex2a.o zcrypt_cex4.o
>  
>  # pkey kernel module
>  pkey-objs := pkey_api.o
> -obj-$(CONFIG_PKEY) += pkey.o
> +obj-$(CONFIG_PKEY) += pkey.o

Unrelated change.

> \ No newline at end of file

Re: [RFC 05/19] s390/zcrypt: base implementation of AP matrix device driver

[Cornelia Huck]

On Fri, 13 Oct 2017 13:38:50 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> Introduces a new AP matrix device driver. This device driver
> will ultimately perform the following functions:
> 
> * Register with the AP bus to let it know that the matrix
>   driver can control AP queue devices. This will allow
>   an administrator to unbind an AP queue device from its
>   device driver and bind it to the matrix device driver.
>   This is how AP queue devices will be reserved for use
>   by guest machines.
> 
> * Register the matrix device created by the AP matrix bus
>   with the VFIO mediated device framework. This will create
>   the sysfs entries needed to create mediated matrix devices.
>   Each mediated matrix device can be configured with a matrix
>   of adapters, usage domains and control domains that can be
>   accessed by a guest machine.
> 
> * Process requests via ioctl calls defined for the mediated
>   matrix device. The guest can access the ioctl calls via
>   the mediated device's file descriptor to:
> 
>     * Grant access to the adapters, usage domains and
>       control domains configured for the mediated matrix
>       device.
> 
> This device driver
> is built on the VFIO mediated device framework. The VFIO mediated
> device framework allows a mediated device to be dedicated exclusively
> to a single guest VM.
> 
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>  MAINTAINERS                                  |    2 +
>  arch/s390/Kconfig                            |   13 +++
>  arch/s390/configs/default_defconfig          |    1 +
>  arch/s390/configs/gcov_defconfig             |    1 +
>  arch/s390/configs/performance_defconfig      |    1 +
>  arch/s390/defconfig                          |    1 +
>  drivers/s390/crypto/Makefile                 |    6 +-
>  drivers/s390/crypto/ap_matrix_bus.c          |    8 ++
>  drivers/s390/crypto/ap_matrix_bus.h          |    2 +-
>  drivers/s390/crypto/vfio_ap_matrix_drv.c     |  102 ++++++++++++++++++++++++++
>  drivers/s390/crypto/vfio_ap_matrix_private.h |   47 ++++++++++++
>  11 files changed, 182 insertions(+), 2 deletions(-)
>  create mode 100644 drivers/s390/crypto/vfio_ap_matrix_drv.c
>  create mode 100644 drivers/s390/crypto/vfio_ap_matrix_private.h
> 

> diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
> index 48af970..411c19a 100644
> --- a/arch/s390/Kconfig
> +++ b/arch/s390/Kconfig
> @@ -722,6 +722,19 @@ config VFIO_CCW
>  	  To compile this driver as a module, choose M here: the
>  	  module will be called vfio_ccw.
>  
> +config VFIO_AP_MATRIX
> +	def_tristate m
> +	prompt "Support for Adjunct Processor Matrix device interface"
> +	depends on ZCRYPT
> +	select VFIO
> +	select MDEV
> +	select VFIO_MDEV
> +	select VFIO_MDEV_DEVICE
> +	select IOMMU_API

I think the more common pattern is to depend on the VFIO configs
instead of selecting them.

> +	help
> +		driver grants access to Adjunct Processor (AP) devices
> +		via the VFIO mediated device interface.
> +
>  endmenu
>  
>  menu "Dump support"

> diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile
> index 87646ca..1983afa 100644
> --- a/drivers/s390/crypto/Makefile
> +++ b/drivers/s390/crypto/Makefile
> @@ -13,4 +13,8 @@ obj-$(CONFIG_ZCRYPT) += zcrypt_pcixcc.o zcrypt_cex2a.o zcrypt_cex4.o
>  
>  # pkey kernel module
>  pkey-objs := pkey_api.o
> -obj-$(CONFIG_PKEY) += pkey.o
> \ No newline at end of file
> +obj-$(CONFIG_PKEY) += pkey.o

Another change of that line.

> +
> +# adjunct processor matrix
> +vfio_ap_matrix-objs := vfio_ap_matrix_drv.o
> +obj-$(CONFIG_VFIO_AP_MATRIX) += vfio_ap_matrix.o
> diff --git a/drivers/s390/crypto/ap_matrix_bus.c b/drivers/s390/crypto/ap_matrix_bus.c
> index 4eb1e3c..66bfa54 100644
> --- a/drivers/s390/crypto/ap_matrix_bus.c
> +++ b/drivers/s390/crypto/ap_matrix_bus.c
> @@ -75,10 +75,18 @@ static int ap_matrix_dev_create(void)
>  	return 0;
>  }
>  
> +struct ap_matrix *ap_matrix_get_device(void)
> +{
> +	return matrix;

See the comments I had for the previous patch. In particular, I think
it is better to retrieve a pointer to the matrix device via driver core
methods.

> +}
> +EXPORT_SYMBOL(ap_matrix_get_device);
> +
>  int __init ap_matrix_init(void)
>  {
>  	int ret;
>  
> +	matrix = NULL;
> +
>  	ap_matrix_root_device = root_device_register(AP_MATRIX_BUS_NAME);
>  	ret = PTR_RET(ap_matrix_root_device);
>  	if (ret)
> diff --git a/drivers/s390/crypto/ap_matrix_bus.h b/drivers/s390/crypto/ap_matrix_bus.h
> index 225db4f..c2aff23 100644
> --- a/drivers/s390/crypto/ap_matrix_bus.h
> +++ b/drivers/s390/crypto/ap_matrix_bus.h
> @@ -16,6 +16,6 @@ struct ap_matrix {
>  	struct device device;
>  };
>  
> -int ap_matrix_init(void);

So, was that not needed before?

> +struct ap_matrix *ap_matrix_get_device(void);
>  
>  #endif /* _AP_MATRIX_BUS_H_ */
> diff --git a/drivers/s390/crypto/vfio_ap_matrix_drv.c b/drivers/s390/crypto/vfio_ap_matrix_drv.c
> new file mode 100644
> index 0000000..760ed56
> --- /dev/null
> +++ b/drivers/s390/crypto/vfio_ap_matrix_drv.c
> @@ -0,0 +1,102 @@
> +/*
> + * VFIO based AP Matrix device driver
> + *
> + * Copyright IBM Corp. 2017
> + *
> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> + */
> +
> +#include <linux/module.h>
> +#include <linux/mod_devicetable.h>
> +#include <linux/slab.h>
> +
> +#include "ap_bus.h"
> +#include "ap_matrix_bus.h"
> +#include "vfio_ap_matrix_private.h"
> +
> +#define VFIO_AP_MATRIX_DRV_NAME "vfio_ap_queue"
> +
> +MODULE_AUTHOR("IBM Corporation");
> +MODULE_DESCRIPTION("AP Matrix device driver, Copyright IBM Corp. 2017");
> +MODULE_LICENSE("GPL v2");
> +
> +static struct ap_device_id ap_queue_ids[] = {
> +	{ .dev_type = AP_DEVICE_TYPE_CEX4,
> +	  .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
> +	{ .dev_type = AP_DEVICE_TYPE_CEX5,
> +	  .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
> +	{ .dev_type = AP_DEVICE_TYPE_CEX6,
> +	  .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },

So, you explicitly don't match to all devices, but only to the newer
ones? This needs an explaining comment.

> +	{ /* end of list */ },
> +};
> +
> +MODULE_DEVICE_TABLE(ap_matrix, ap_queue_ids);
> +
> +static struct ap_matrix_driver {
> +	struct ap_driver ap_drv;
> +	struct ap_matrix *ap_matrix;

Do you actually need that pointer to the matrix device? One usage is to
pass it as an parameter to the mdev registration in the next patch. As
you only support one matrix device, would it be better to move getting
that device into the mdev code?

For the other usage, move getting it into find_vapq()?

> +} vfio_ap_matrix_drv;
> +
> +static int ap_matrix_queue_dev_probe(struct ap_device *apdev)
> +{
> +	struct vfio_ap_queue *vapq;
> +	struct ap_queue *apq = to_ap_queue(&apdev->device);
> +	struct ap_matrix *ap_matrix = vfio_ap_matrix_drv.ap_matrix;
> +
> +	vapq = kzalloc(sizeof(*vapq), GFP_KERNEL);
> +	if (!vapq)
> +		return -ENOMEM;
> +
> +	INIT_LIST_HEAD(&vapq->list);
> +	vapq->queue = apq;
> +	spin_lock_bh(&ap_matrix->qlock);
> +	list_add_tail(&vapq->list, &ap_matrix->queues);
> +	spin_unlock_bh(&ap_matrix->qlock);
> +
> +	return 0;
> +}
> +
> +static void ap_matrix_queue_dev_remove(struct ap_device *apdev)
> +{
> +	struct vfio_ap_queue *vapq;
> +	struct ap_queue *apq = to_ap_queue(&apdev->device);
> +	struct ap_matrix *ap_matrix = vfio_ap_matrix_drv.ap_matrix;
> +
> +	vapq = find_vapq(ap_matrix, apq->qid);
> +
> +	if (vapq) {
> +		spin_lock_bh(&ap_matrix->qlock);
> +		list_del_init(&vapq->list);
> +		spin_unlock_bh(&ap_matrix->qlock);
> +		kfree(vapq);
> +	}
> +}
> +
> +int __init ap_matrix_init(void)
> +{
> +
> +	int ret;
> +
> +	vfio_ap_matrix_drv.ap_matrix = ap_matrix_get_device();
> +	if (!vfio_ap_matrix_drv.ap_matrix)
> +		return -ENODEV;
> +
> +	vfio_ap_matrix_drv.ap_drv.probe = ap_matrix_queue_dev_probe;
> +	vfio_ap_matrix_drv.ap_drv.remove = ap_matrix_queue_dev_remove;
> +	vfio_ap_matrix_drv.ap_drv.ids = ap_queue_ids;

Can you use an static initializer for that?

> +
> +	ret = ap_driver_register(&vfio_ap_matrix_drv.ap_drv,
> +				 THIS_MODULE, VFIO_AP_MATRIX_DRV_NAME);
> +	if (ret)
> +		return ret;
> +
> +	return ret;
> +}
> +
> +void __exit ap_matrix_exit(void)
> +{
> +	ap_driver_unregister(&vfio_ap_matrix_drv.ap_drv);
> +}
> +
> +module_init(ap_matrix_init);
> +module_exit(ap_matrix_exit);
> diff --git a/drivers/s390/crypto/vfio_ap_matrix_private.h b/drivers/s390/crypto/vfio_ap_matrix_private.h
> new file mode 100644
> index 0000000..11c5e02
> --- /dev/null
> +++ b/drivers/s390/crypto/vfio_ap_matrix_private.h
> @@ -0,0 +1,47 @@
> +/*
> + * Private data and functions for adjunct processor VFIO matrix driver.
> + *
> + * Copyright IBM Corp. 2016
> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> + */
> +
> +#ifndef _VFIO_AP_PRIVATE_H_
> +#define _VFIO_AP_PRIVATE_H_
> +
> +#include <linux/types.h>
> +
> +#include "ap_bus.h"
> +#include "ap_matrix_bus.h"
> +
> +#define VFIO_AP_MATRIX_MODULE_NAME "vfio_ap_matrix"
> +
> +struct vfio_ap_queue {
> +	struct ap_queue *queue;
> +	struct list_head list;
> +};
> +
> +static inline struct vfio_ap_queue *to_vapq(struct ap_device *ap_dev)
> +{
> +	struct ap_queue *ap_queue = to_ap_queue(&ap_dev->device);
> +	struct vfio_ap_queue *vapq;
> +
> +	vapq = container_of(&ap_queue, struct vfio_ap_queue, queue);

Why not just return container_of(...); ?

> +
> +	return vapq;
> +}
> +
> +static inline struct vfio_ap_queue *find_vapq(struct ap_matrix *ap_matrix,
> +					      ap_qid_t qid)
> +{
> +	struct vfio_ap_queue *vapq;
> +
> +	if (!list_empty(&ap_matrix->queues)) {
> +		list_for_each_entry(vapq, &ap_matrix->queues, list)
> +			if (vapq->queue->qid == qid)
> +				return vapq;
> +	}
> +
> +	return NULL;
> +}
> +
> +#endif /* _VFIO_AP_PRIVATE_H_ */

Re: [RFC 06/19] s390/zcrypt: register matrix device with VFIO mediated device framework

[Cornelia Huck]

On Fri, 13 Oct 2017 13:38:51 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> Registers the matrix device created by the AP matrix bus with
> the VFIO mediated device framework. Registering the matrix
> device will create the sysfs structures needed to create
> mediated matrix devices that can be configured with a matrix
> of adapters, usage domains and control domains for a guest
> machine.
> 
> Registering the matrix device with the VFIO mediated device
> framework will create the following sysfs structures:
> 
> /sys/devices/ap_matrix
> ... [matrix]
> ...... [mdev_supported_types]
> ......... [ap_matrix-passthrough]
> ............ available_instances
> ............ create
> ............ device_api
> ............ [devices]
> ............ name
> 
> To create a mediated device for the AP matrix device, write a UUID
> to the create file:
> 
>     uuidgen > create
> 
> A symbolic link to the mediated device's directory will be created in the
> devices subdirectory named after the generated $uuid:
> 
> /sys/devices/ap_matrix
> ... [matrix]
> ...... [mdev_supported_types]
> ......... [ap_matrix-passthrough]
> ............ [devices]
> ............... [$uuid]
> 
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>  MAINTAINERS                                  |    1 +
>  drivers/s390/crypto/Makefile                 |    2 +-
>  drivers/s390/crypto/ap_matrix_bus.c          |    1 +
>  drivers/s390/crypto/ap_matrix_bus.h          |    4 +
>  drivers/s390/crypto/vfio_ap_matrix_drv.c     |    5 +
>  drivers/s390/crypto/vfio_ap_matrix_ops.c     |  172 ++++++++++++++++++++++++++
>  drivers/s390/crypto/vfio_ap_matrix_private.h |    3 +
>  include/uapi/linux/vfio.h                    |    1 +
>  8 files changed, 188 insertions(+), 1 deletions(-)
>  create mode 100644 drivers/s390/crypto/vfio_ap_matrix_ops.c
> 

> diff --git a/drivers/s390/crypto/ap_matrix_bus.c b/drivers/s390/crypto/ap_matrix_bus.c
> index 66bfa54..418c23b 100644
> --- a/drivers/s390/crypto/ap_matrix_bus.c
> +++ b/drivers/s390/crypto/ap_matrix_bus.c
> @@ -61,6 +61,7 @@ static int ap_matrix_dev_create(void)
>  	matrix->device.bus = &ap_matrix_bus_type;
>  	matrix->device.parent = ap_matrix_root_device;
>  	matrix->device.release = ap_matrix_dev_release;
> +	INIT_LIST_HEAD(&matrix->queues);

Don't you also need to init the spin lock?

>  
>  	ret = device_register(&matrix->device);
>  	if (ret) {
> diff --git a/drivers/s390/crypto/ap_matrix_bus.h b/drivers/s390/crypto/ap_matrix_bus.h
> index c2aff23..3eccc36 100644
> --- a/drivers/s390/crypto/ap_matrix_bus.h
> +++ b/drivers/s390/crypto/ap_matrix_bus.h
> @@ -12,8 +12,12 @@
>  
>  #include <linux/device.h>
>  
> +#include "ap_bus.h"

Why do you need this include now? (IOW, does that need to go into
another patch?)

> +
>  struct ap_matrix {
>  	struct device device;
> +	spinlock_t qlock;
> +	struct list_head queues;
>  };
>  
>  struct ap_matrix *ap_matrix_get_device(void);

> diff --git a/drivers/s390/crypto/vfio_ap_matrix_ops.c b/drivers/s390/crypto/vfio_ap_matrix_ops.c
> new file mode 100644
> index 0000000..7d01f18
> --- /dev/null
> +++ b/drivers/s390/crypto/vfio_ap_matrix_ops.c
> @@ -0,0 +1,172 @@
> +/*
> + * Adjunct processor matrix VFIO device driver callbacks.
> + *
> + * Copyright IBM Corp. 2017
> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> + *
> + */
> +#include <asm/ap-config.h>
> +#include <linux/string.h>
> +#include <linux/vfio.h>
> +#include <linux/mdev.h>
> +#include <linux/device.h>
> +#include <linux/list.h>
> +#include <linux/ctype.h>
> +
> +#include "vfio_ap_matrix_private.h"
> +
> +#define AP_MATRIX_MDEV_TYPE_HWVIRT "passthrough"
> +#define AP_MATRIX_MDEV_NAME_HWVIRT "AP Matrix Passthrough Device"
> +
> +#define AP_MATRIX_MAX_AVAILABLE_INSTANCES	255
> +
> +struct ap_matrix_mdev {
> +	struct mdev_device *mdev;
> +	struct list_head node;
> +};
> +
> +struct ap_matrix	*matrix;
> +struct mutex		mdev_devices_lock;
> +struct list_head	mdev_devices;
> +int			available_instances;

Having this as a global variable feels wrong. I see that vfio-ccw keeps
things like this in driver data attached to its parent. Can you do
something like that as well?

> +
> +static struct ap_matrix_mdev *ap_matrix_mdev_find_by_uuid(uuid_le uuid)
> +{
> +	struct ap_matrix_mdev *matrix_mdev;
> +
> +	list_for_each_entry(matrix_mdev, &mdev_devices, node) {
> +		if (uuid_le_cmp(mdev_uuid(matrix_mdev->mdev), uuid) == 0)
> +			return matrix_mdev;
> +	}
> +
> +	return NULL;
> +}

(...)

> +int ap_matrix_mdev_register(struct ap_matrix *ap_matrix)
> +{
> +	int ret;
> +	struct device *dev = &ap_matrix->device;
> +
> +	ret = mdev_register_device(dev, &vfio_ap_matrix_ops);
> +	if (ret)
> +		return ret;
> +
> +	matrix = ap_matrix;

I'd just grab the (one) matrix device here...

> +	mutex_init(&mdev_devices_lock);
> +	INIT_LIST_HEAD(&mdev_devices);
> +	available_instances = AP_MATRIX_MAX_AVAILABLE_INSTANCES;
> +
> +	return 0;
> +}
> +
> +void ap_matrix_mdev_unregister(struct ap_matrix *ap_matrix)
> +{
> +	struct device *dev;
> +
> +	if (ap_matrix == matrix) {

...and lose this extra if.

> +		dev = &matrix->device;
> +		available_instances--;
> +		mdev_unregister_device(dev);
> +	}
> +}

Re: [RFC 07/19] KVM: s390: introduce AP matrix configuration interface

[Cornelia Huck]

On Fri, 13 Oct 2017 13:38:52 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> This patch introduces a new compilation unit that will contain
> all of the data structures and logic for configuring AP adapters,
> usage domains and control domains for a KVM guest.
> 
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>  MAINTAINERS                       |    2 ++
>  arch/s390/include/asm/ap-config.h |   12 ++++++++++++
>  arch/s390/kvm/Makefile            |    2 +-
>  arch/s390/kvm/ap-config.c         |    9 +++++++++
>  4 files changed, 24 insertions(+), 1 deletions(-)
>  create mode 100644 arch/s390/include/asm/ap-config.h
>  create mode 100644 arch/s390/kvm/ap-config.c

This is a bit useless on its own... merge this with the actual
implementation?

Re: [RFC 03/19] s390/zcrypt: new AP matrix bus

[Tony Krowiak]

On 11/14/2017 06:58 AM, Cornelia Huck wrote:
> On Fri, 13 Oct 2017 13:38:48 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> Introduces an AP matrix bus. The sysfs location of the
>> AP matrix bus is:
>>
>> /sys/bus/ap_matrix
>>
>> The AP matrix bus will create an AP matrix device that will
>> hold the AP queues reserved for use by KVM guests.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>>   MAINTAINERS                         |    8 +++++
>>   drivers/s390/crypto/Makefile        |    4 +-
>>   drivers/s390/crypto/ap_matrix_bus.c |   52 +++++++++++++++++++++++++++++++++++
>>   drivers/s390/crypto/ap_matrix_bus.h |   15 ++++++++++
>>   4 files changed, 77 insertions(+), 2 deletions(-)
>>   create mode 100644 drivers/s390/crypto/ap_matrix_bus.c
>>   create mode 100644 drivers/s390/crypto/ap_matrix_bus.h
>>
>> diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile
>> index be36f10..87646ca 100644
>> --- a/drivers/s390/crypto/Makefile
>> +++ b/drivers/s390/crypto/Makefile
>> @@ -2,7 +2,7 @@
>>   # S/390 crypto devices
>>   #
>>   
>> -ap-objs := ap_bus.o ap_card.o ap_queue.o
>> +ap-objs := ap_bus.o ap_card.o ap_queue.o ap_matrix_bus.o
>>   obj-$(subst m,y,$(CONFIG_ZCRYPT)) += ap.o
>>   # zcrypt_api.o and zcrypt_msgtype*.o depend on ap.o
>>   zcrypt-objs := zcrypt_api.o zcrypt_card.o zcrypt_queue.o
>> @@ -13,4 +13,4 @@ obj-$(CONFIG_ZCRYPT) += zcrypt_pcixcc.o zcrypt_cex2a.o zcrypt_cex4.o
>>   
>>   # pkey kernel module
>>   pkey-objs := pkey_api.o
>> -obj-$(CONFIG_PKEY) += pkey.o
>> +obj-$(CONFIG_PKEY) += pkey.o
> Unrelated change.
Will fix
>
>> \ No newline at end of file

Re: [RFC 08/19] s390/zcrypt: support for assigning adapters to matrix mdev

[Cornelia Huck]

On Fri, 13 Oct 2017 13:38:53 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> Provides the sysfs interfaces for assigning an adapter to
> and unassigning an AP adapter from a mediated matrix device.
> 
> The IDs of the AP adapters assigned to the mediated matrix
> device are stored in a bit mask. The bits in the mask, from
> left to right, correspond to AP adapter numbers 0 to 255. The
> bit corresponding to the ID of the adapter being assigned will
> be set in the bit mask.
> 
> The relevant sysfs structures are:
> 
> /sys/devices/ap_matrix
> ... [matrix]
> ...... [mdev_supported_types]
> ......... [ap_matrix-passthrough]
> ............ [devices]
> ...............[$uuid]
> .................. adapters
> .................. assign_adapter
> .................. unassign_adapter
> 
> To assign an adapter to the $uuid mediated matrix device, write
> ID of the adapter (hex value) to the assign_adapter file. To
> unassign an adapter, write the ID of the adapter (hex value)
> to the unassign_adapter file. The list of adapters that have
> been assigned can be viewed by displaying the contents of the
> adapters file.
> 
> For example, to assign adapter 0xad to mediated matrix device
> $uuid:
> 
> 	echo ad > assign_adapter
> 
> To unassign adapter 0xad:
> 
> 	echo ad > unassign_adapter
> 
> To see the list of adapters assigned:
> 
> 	cat adapters
> 
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>  arch/s390/include/asm/ap-config.h        |   13 +++
>  drivers/s390/crypto/vfio_ap_matrix_ops.c |  147 ++++++++++++++++++++++++++++++
>  2 files changed, 160 insertions(+), 0 deletions(-)
> 

> diff --git a/drivers/s390/crypto/vfio_ap_matrix_ops.c b/drivers/s390/crypto/vfio_ap_matrix_ops.c
> index 7d01f18..e4b1236 100644
> --- a/drivers/s390/crypto/vfio_ap_matrix_ops.c
> +++ b/drivers/s390/crypto/vfio_ap_matrix_ops.c
> @@ -23,6 +23,7 @@
>  struct ap_matrix_mdev {
>  	struct mdev_device *mdev;
>  	struct list_head node;
> +	struct ap_config_masks masks;
>  };
>  
>  struct ap_matrix	*matrix;
> @@ -136,9 +137,155 @@ static ssize_t device_api_show(struct kobject *kobj, struct device *dev,
>  	NULL,
>  };
>  
> +static int ap_matrix_id_is_xnum(char *id)
> +{
> +	size_t i;
> +
> +	for (i = 0; i < strlen(id); i++) {
> +		if (!isxdigit(id[i]))
> +			return 0;
> +	}
> +
> +	return 1;
> +}

This feels like something for which an utility function should already
exist...

> +
> +static int ap_matrix_parse_id(const char *buf, unsigned int *id)
> +{
> +	int ret;
> +	char *bufcpy;
> +	char *id_str;
> +
> +	if ((strlen(buf) == 0) || (*buf == '\n')) {
> +		pr_err("%s: input buffer '%s' contains no valid hex values",
> +		       VFIO_AP_MATRIX_MODULE_NAME, buf);
> +		return -EINVAL;
> +	}
> +
> +	bufcpy = kzalloc(strlen(buf) + 1, GFP_KERNEL);
> +	if (!bufcpy)
> +		return -ENOMEM;
> +
> +	strcpy(bufcpy, buf);
> +	id_str = strim(bufcpy);
> +
> +	if (!ap_matrix_id_is_xnum(id_str)) {
> +		pr_err("%s: input id '%s' contains an invalid hex value '%s'",
> +		       VFIO_AP_MATRIX_MODULE_NAME, buf, id_str);
> +		ret = -EINVAL;
> +		goto done;
> +	}
> +
> +	ret = kstrtouint (id_str, 16, id);
> +	if (ret || (*id >= AP_MATRIX_MAX_MASK_BITS)) {
> +		pr_err("%s: input id '%s' is not a value from 0 to %x",
> +		       VFIO_AP_MATRIX_MODULE_NAME, buf,
> +		       AP_MATRIX_MAX_MASK_BITS);
> +		ret = -EINVAL;
> +		goto done;
> +	}
> +
> +	ret = 0;
> +
> +done:
> +	kfree(bufcpy);
> +	return ret;
> +}

Similarly, I suspect you are not the first person with similar parsing
needs.

> +
> +static ssize_t ap_matrix_adapters_assign(struct device *dev,
> +					 struct device_attribute *attr,
> +					 const char *buf, size_t count)
> +{
> +	int ret;
> +	unsigned int apid;
> +	struct mdev_device *mdev = mdev_from_dev(dev);
> +	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
> +
> +	ret = ap_matrix_parse_id(buf, &apid);
> +	if (ret)
> +		return ret;
> +
> +	set_bit_inv((unsigned long)apid,
> +		    (unsigned long *)matrix_mdev->masks.apm);

Probably needs a comment regarding byte order of the masks somewhere.

> +
> +	return count;
> +}
> +static DEVICE_ATTR(assign_adapter, 0644, NULL, ap_matrix_adapters_assign);

Re: [RFC 16/19] KVM: s390: interface to configure KVM guest's AP matrix

[Cornelia Huck]

On Fri, 13 Oct 2017 13:39:01 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> Provides an interface to assign AP adapters, usage domains
> and control domains to a KVM guest.
> 
> A KVM guest is started by executing the Start Interpretive Execution (SIE)
> instruction. The SIE state description is a control block that contains the
> state information for a KVM guest and is supplied as input to the SIE
> instruction. The SIE state description contains a field that references
> a Crypto Control Block (CRYCB). The CRYCB contains three bitmask fields
> identifying the adapters, usage domains and control domains assigned to the
> KVM guest:
> 
> * The AP Adapter Matrix (APM) field identifies the AP adapters assigned to
>   the KVM guest
> 
> * The AP Queue Matrix (AQM) field identifies the usage domains assigned to
>   the KVM guest
> 
> * The AP Domain matrix (ADM) field identifies the control domains
>   assigned to the KVM guest.
> 
> Each adapter, usage domain and control domain are identified by a number
> from 0 to 255. The bits in each mask, from left to right, correspond to
> the numbers 0-255. When a bit is set, the corresponding adapter,
> usage domain or control domain will be assigned to the KVM guest.
> 
> This patch will set the bits in the APM, AQM and ADM fields of the
> CRYCB referenced by the KVM guest's SIE state description. The process
> used is:
> 
> 1. Perform a logical AND of the AP matrix masks configured for the
>    mediated AP matrix device via its sysfs attributes files with
>    the matrix masks assigned to the LPAR in which the host linux
>    system is running to create the effective masks, EAPM, EAQM and
>    EADM.
> 
> 2. Set the APM, AQM and ADM in the CRYCB from the EAPM, EAQM and
>    EADM calculated in step 1.
> 
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>  arch/s390/include/asm/ap-config.h |    7 ++
>  arch/s390/kvm/ap-config.c         |  144 +++++++++++++++++++++++++++++++++++++
>  2 files changed, 151 insertions(+), 0 deletions(-)
> 

> +static int ap_config_get_emasks(struct ap_config_masks *masks)
> +{
> +	int ret;
> +	struct ap_config_info config;
> +
> +	ret = ap_query_configuration(&config);
> +	if (ret) {
> +		if (ret == -EOPNOTSUPP) {
> +			pr_err("%s: Query AP configuration not supported",
> +			       __func__);
> +
> +			return ret;

I'm wondering whether we should check this at init time for vfio-ap? If
PQAP(QCI) is not supported, it probably does not make any sense to
pretend that we could do device assignment...

> +		}
> +
> +		pr_err("%s: Query AP configuration failed with rc=%d",
> +		       __func__, ret);
> +
> +		return ret;
> +	}
> +
> +	ret = ap_config_set_emask(AP_MATRIX_MASK_TYPE_ADAPTER,
> +				  (unsigned long *)masks->apm,
> +				  (unsigned long *)config.apm);
> +	if (ret)
> +		return ret;
> +
> +	ret = ap_config_set_emask(AP_MATRIX_MASK_TYPE_DOMAIN,
> +				  (unsigned long *)masks->aqm,
> +				  (unsigned long *)config.aqm);
> +	if (ret)
> +		return ret;
> +
> +	ret = ap_config_set_emask(AP_MATRIX_MASK_TYPE_CONTROL,
> +				  (unsigned long *)masks->adm,
> +				  (unsigned long *)config.adm);
> +	if (ret)
> +		return ret;
> +
> +	return 0;
> +}
> +
> +int ap_config_matrix(struct kvm *kvm, struct ap_config_masks *masks)
> +{
> +	int ret;
> +
> +	ret = ap_config_get_emasks(masks);
> +	if (ret)
> +		return ret;
> +
> +	ap_config_set_crycb_masks(kvm, masks);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL(ap_config_matrix);

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Cornelia Huck]

On Tue, 31 Oct 2017 15:39:09 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> On 10/13/2017 01:38 PM, Tony Krowiak wrote:
> Ping

> > Tony Krowiak (19):
> >    KVM: s390: SIE considerations for AP Queue virtualization
> >    KVM: s390: refactor crypto initialization
> >    s390/zcrypt: new AP matrix bus
> >    s390/zcrypt: create an AP matrix device on the AP matrix bus
> >    s390/zcrypt: base implementation of AP matrix device driver
> >    s390/zcrypt: register matrix device with VFIO mediated device
> >      framework
> >    KVM: s390: introduce AP matrix configuration interface
> >    s390/zcrypt: support for assigning adapters to matrix mdev
> >    s390/zcrypt: validate adapter assignment
> >    s390/zcrypt: sysfs interfaces supporting AP domain assignment
> >    s390/zcrypt: validate domain assignment
> >    s390/zcrypt: sysfs support for control domain assignment
> >    s390/zcrypt: validate control domain assignment
> >    KVM: s390: Connect the AP mediated matrix device to KVM
> >    s390/zcrypt: introduce ioctl access to VFIO AP Matrix driver
> >    KVM: s390: interface to configure KVM guest's AP matrix
> >    KVM: s390: validate input to AP matrix config interface
> >    KVM: s390: New ioctl to configure KVM guest's AP matrix
> >    s390/facilities: enable AP facilities needed by guest

I think the approach is fine, and the code also looks fine for the most
part. Some comments:

- various patches can be squashed together to give a better
  understanding at a glance
- this needs documentation (as I already said)
- some of the driver/device modelling feels a bit awkward (commented in
  patches) -- I'm not sure that my proposal is better, but I think we
  should make sure the interdependencies are modeled correctly
- some minor stuff

Re: [RFC 02/19] KVM: s390: refactor crypto initialization

[Tony Krowiak]

On 11/14/2017 06:50 AM, Cornelia Huck wrote:
> On Thu, 2 Nov 2017 13:41:18 +0100
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
>
>> On 10/13/2017 07:38 PM, Tony Krowiak wrote:
>>> This patch introduces the following changes to crypto initialization.
>>>
>>> * For key management operations support, the crypto control block
>>>    (CRYCB) referenced by the KVM guest's SIE state description is
>>>    formatted only if the Message-Security-Assist (MSA) extension 3
>>>    facility is installed (STFLE.76 is set). Virtualization of AP
>>>    facilities, however; requires that a CRYCB of the appropriate
>>>    format be made available to SIE regardless of the value of STFLE.76.
>>>
>>> * The Execution Controls A (ECA) field bit 28 in the SIE block needs
>>>    to be set to enable interpretive execution mode of adjunct processor (AP)
>>>    instructions.
>> We should fence setting ECA to cases where we have virtualization capability
>> for crypto. In addition we need to bind this somehow to the CPU model, so
>> I guess we need to add some CRYPTO feature e.g. add KVM_S390_VM_CPU_FEAT_AP to the
>> list of know features
>> (see arch/s390/include/uapi/asm/kvm.h)
>> ---snip---
>> #define KVM_S390_VM_CPU_FEAT_ESOP       0
>> #define KVM_S390_VM_CPU_FEAT_SIEF2      1
>> #define KVM_S390_VM_CPU_FEAT_64BSCAO    2
>> #define KVM_S390_VM_CPU_FEAT_SIIF       3
>> #define KVM_S390_VM_CPU_FEAT_GPERE      4
>> #define KVM_S390_VM_CPU_FEAT_GSLS       5
>> #define KVM_S390_VM_CPU_FEAT_IB         6
>> #define KVM_S390_VM_CPU_FEAT_CEI        7
>> #define KVM_S390_VM_CPU_FEAT_IBS        8
>> #define KVM_S390_VM_CPU_FEAT_SKEY       9
>> #define KVM_S390_VM_CPU_FEAT_CMMA       10
>> #define KVM_S390_VM_CPU_FEAT_PFMFI      11
>> #define KVM_S390_VM_CPU_FEAT_SIGPIF     12
>> #define KVM_S390_VM_CPU_FEAT_KSS        13
>> ---snip---
> So, we need this so userspace can add the appropriate flags, right?
>
>>
>> I will try to find out a way to properly detect that.
> Did you manage to find out
The AP bus (drivers/s390/crypto/ap_bus.c) calls 
ap_instructions_available() function in drivers/s390/crypto/ap_asm.c
to determine whether AP instructions are installed. This function 
executes the PQAP(TAPQ) function and incorporates
exception table to catch the operation exception if the AP instructions 
are not installed.  I propose externalizing
that function so it can be called from KVM.
>
>>
>>
>>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>>> ---
>>>   arch/s390/include/asm/kvm_host.h |    1 +
>>>   arch/s390/kvm/kvm-s390.c         |   17 +++++++++++++----
>>>   2 files changed, 14 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>>> index 50a6b25..5683f18 100644
>>> --- a/arch/s390/include/asm/kvm_host.h
>>> +++ b/arch/s390/include/asm/kvm_host.h
>>> @@ -188,6 +188,7 @@ struct kvm_s390_sie_block {
>>>   #define ECA_MVPGI	0x01000000
>>>   #define ECA_VX		0x00020000
>>>   #define ECA_PROTEXCI	0x00002000
>>> +#define ECA_AP		0x00000008
>>>   #define ECA_SII		0x00000001
>>>   	__u32	eca;			/* 0x004c */
>>>   #define ICPT_INST	0x04
>>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>>> index 40d0a1a..e57fc9b 100644
>>> --- a/arch/s390/kvm/kvm-s390.c
>>> +++ b/arch/s390/kvm/kvm-s390.c
>>> @@ -1819,7 +1819,9 @@ static void kvm_s390_set_crycb_format(struct kvm *kvm)
>>>   {
>>>   	kvm->arch.crypto.crycbd = (__u32)(unsigned long) kvm->arch.crypto.crycb;
>>>
>>> -	if (kvm_s390_apxa_installed())
>>> +	if (!test_kvm_facility(kvm, 76))
>>> +		kvm->arch.crypto.crycbd &= ~(CRYCB_FORMAT2); /* format 0 */
>>> +	else if (kvm_s390_apxa_installed())
>>>   		kvm->arch.crypto.crycbd |= CRYCB_FORMAT2;
>>>   	else
>>>   		kvm->arch.crypto.crycbd |= CRYCB_FORMAT1;
>>> @@ -1836,12 +1838,12 @@ static u64 kvm_s390_get_initial_cpuid(void)
>>>
>>>   static void kvm_s390_crypto_init(struct kvm *kvm)
>>>   {
>>> -	if (!test_kvm_facility(kvm, 76))
>>> -		return;
>>> -
>>>   	kvm->arch.crypto.crycb = &kvm->arch.sie_page2->crycb;
>>>   	kvm_s390_set_crycb_format(kvm);
>>>
>>> +	if (!test_kvm_facility(kvm, 76))
>>> +		return;
>>> +
>>>   	/* Enable AES/DEA protected key functions by default */
>>>   	kvm->arch.crypto.aes_kw = 1;
>>>   	kvm->arch.crypto.dea_kw = 1;
>>> @@ -2366,8 +2368,15 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)
>>>   	vcpu->arch.enabled_gmap = vcpu->arch.gmap;
>>>   }
>>>
>>> +static void kvm_s390_vcpu_set_crypto_exec_mode(struct kvm_vcpu *vcpu)
>>> +{
>>> +	vcpu->arch.sie_block->eca |= ECA_AP;
>>> +}
>>> +
>>>   static void kvm_s390_vcpu_crypto_setup(struct kvm_vcpu *vcpu)
>>>   {
>>> +	kvm_s390_vcpu_set_crypto_exec_mode(vcpu);
>>> +
>>>   	if (!test_kvm_facility(vcpu->kvm, 76))
>>>   		return;
>>>    

Re: [RFC 03/19] s390/zcrypt: new AP matrix bus

[Tony Krowiak]

On 11/14/2017 06:58 AM, Cornelia Huck wrote:
> On Fri, 13 Oct 2017 13:38:48 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> Introduces an AP matrix bus. The sysfs location of the
>> AP matrix bus is:
>>
>> /sys/bus/ap_matrix
>>
>> The AP matrix bus will create an AP matrix device that will
>> hold the AP queues reserved for use by KVM guests.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>>   MAINTAINERS                         |    8 +++++
>>   drivers/s390/crypto/Makefile        |    4 +-
>>   drivers/s390/crypto/ap_matrix_bus.c |   52 +++++++++++++++++++++++++++++++++++
>>   drivers/s390/crypto/ap_matrix_bus.h |   15 ++++++++++
>>   4 files changed, 77 insertions(+), 2 deletions(-)
>>   create mode 100644 drivers/s390/crypto/ap_matrix_bus.c
>>   create mode 100644 drivers/s390/crypto/ap_matrix_bus.h
>>
>> diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile
>> index be36f10..87646ca 100644
>> --- a/drivers/s390/crypto/Makefile
>> +++ b/drivers/s390/crypto/Makefile
>> @@ -2,7 +2,7 @@
>>   # S/390 crypto devices
>>   #
>>   
>> -ap-objs := ap_bus.o ap_card.o ap_queue.o
>> +ap-objs := ap_bus.o ap_card.o ap_queue.o ap_matrix_bus.o
>>   obj-$(subst m,y,$(CONFIG_ZCRYPT)) += ap.o
>>   # zcrypt_api.o and zcrypt_msgtype*.o depend on ap.o
>>   zcrypt-objs := zcrypt_api.o zcrypt_card.o zcrypt_queue.o
>> @@ -13,4 +13,4 @@ obj-$(CONFIG_ZCRYPT) += zcrypt_pcixcc.o zcrypt_cex2a.o zcrypt_cex4.o
>>   
>>   # pkey kernel module
>>   pkey-objs := pkey_api.o
>> -obj-$(CONFIG_PKEY) += pkey.o
>> +obj-$(CONFIG_PKEY) += pkey.o
> Unrelated change.
will fix
>
>> \ No newline at end of file

Re: [RFC 03/19] s390/zcrypt: new AP matrix bus

[Tony Krowiak]

On 11/14/2017 06:58 AM, Cornelia Huck wrote:
> On Fri, 13 Oct 2017 13:38:48 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> Introduces an AP matrix bus. The sysfs location of the
>> AP matrix bus is:
>>
>> /sys/bus/ap_matrix
>>
>> The AP matrix bus will create an AP matrix device that will
>> hold the AP queues reserved for use by KVM guests.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>>   MAINTAINERS                         |    8 +++++
>>   drivers/s390/crypto/Makefile        |    4 +-
>>   drivers/s390/crypto/ap_matrix_bus.c |   52 +++++++++++++++++++++++++++++++++++
>>   drivers/s390/crypto/ap_matrix_bus.h |   15 ++++++++++
>>   4 files changed, 77 insertions(+), 2 deletions(-)
>>   create mode 100644 drivers/s390/crypto/ap_matrix_bus.c
>>   create mode 100644 drivers/s390/crypto/ap_matrix_bus.h
>>
>> diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile
>> index be36f10..87646ca 100644
>> --- a/drivers/s390/crypto/Makefile
>> +++ b/drivers/s390/crypto/Makefile
>> @@ -2,7 +2,7 @@
>>   # S/390 crypto devices
>>   #
>>   
>> -ap-objs := ap_bus.o ap_card.o ap_queue.o
>> +ap-objs := ap_bus.o ap_card.o ap_queue.o ap_matrix_bus.o
>>   obj-$(subst m,y,$(CONFIG_ZCRYPT)) += ap.o
>>   # zcrypt_api.o and zcrypt_msgtype*.o depend on ap.o
>>   zcrypt-objs := zcrypt_api.o zcrypt_card.o zcrypt_queue.o
>> @@ -13,4 +13,4 @@ obj-$(CONFIG_ZCRYPT) += zcrypt_pcixcc.o zcrypt_cex2a.o zcrypt_cex4.o
>>   
>>   # pkey kernel module
>>   pkey-objs := pkey_api.o
>> -obj-$(CONFIG_PKEY) += pkey.o
>> +obj-$(CONFIG_PKEY) += pkey.o
> Unrelated change.
>
>> \ No newline at end of file
I will fix this

Re: [RFC 05/19] s390/zcrypt: base implementation of AP matrix device driver

[Tony Krowiak]

On 11/14/2017 07:40 AM, Cornelia Huck wrote:
> On Fri, 13 Oct 2017 13:38:50 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> Introduces a new AP matrix device driver. This device driver
>> will ultimately perform the following functions:
>>
>> * Register with the AP bus to let it know that the matrix
>>    driver can control AP queue devices. This will allow
>>    an administrator to unbind an AP queue device from its
>>    device driver and bind it to the matrix device driver.
>>    This is how AP queue devices will be reserved for use
>>    by guest machines.
>>
>> * Register the matrix device created by the AP matrix bus
>>    with the VFIO mediated device framework. This will create
>>    the sysfs entries needed to create mediated matrix devices.
>>    Each mediated matrix device can be configured with a matrix
>>    of adapters, usage domains and control domains that can be
>>    accessed by a guest machine.
>>
>> * Process requests via ioctl calls defined for the mediated
>>    matrix device. The guest can access the ioctl calls via
>>    the mediated device's file descriptor to:
>>
>>      * Grant access to the adapters, usage domains and
>>        control domains configured for the mediated matrix
>>        device.
>>
>> This device driver
>> is built on the VFIO mediated device framework. The VFIO mediated
>> device framework allows a mediated device to be dedicated exclusively
>> to a single guest VM.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>>   MAINTAINERS                                  |    2 +
>>   arch/s390/Kconfig                            |   13 +++
>>   arch/s390/configs/default_defconfig          |    1 +
>>   arch/s390/configs/gcov_defconfig             |    1 +
>>   arch/s390/configs/performance_defconfig      |    1 +
>>   arch/s390/defconfig                          |    1 +
>>   drivers/s390/crypto/Makefile                 |    6 +-
>>   drivers/s390/crypto/ap_matrix_bus.c          |    8 ++
>>   drivers/s390/crypto/ap_matrix_bus.h          |    2 +-
>>   drivers/s390/crypto/vfio_ap_matrix_drv.c     |  102 ++++++++++++++++++++++++++
>>   drivers/s390/crypto/vfio_ap_matrix_private.h |   47 ++++++++++++
>>   11 files changed, 182 insertions(+), 2 deletions(-)
>>   create mode 100644 drivers/s390/crypto/vfio_ap_matrix_drv.c
>>   create mode 100644 drivers/s390/crypto/vfio_ap_matrix_private.h
>>
>> diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
>> index 48af970..411c19a 100644
>> --- a/arch/s390/Kconfig
>> +++ b/arch/s390/Kconfig
>> @@ -722,6 +722,19 @@ config VFIO_CCW
>>   	  To compile this driver as a module, choose M here: the
>>   	  module will be called vfio_ccw.
>>   
>> +config VFIO_AP_MATRIX
>> +	def_tristate m
>> +	prompt "Support for Adjunct Processor Matrix device interface"
>> +	depends on ZCRYPT
>> +	select VFIO
>> +	select MDEV
>> +	select VFIO_MDEV
>> +	select VFIO_MDEV_DEVICE
>> +	select IOMMU_API
> I think the more common pattern is to depend on the VFIO configs
> instead of selecting them.
It's ironic because I originally changed from using 'depends on' and 
changed it based on review comments made
on our internal mailing list. I'll go with 'depends on'.
>
>> +	help
>> +		driver grants access to Adjunct Processor (AP) devices
>> +		via the VFIO mediated device interface.
>> +
>>   endmenu
>>   
>>   menu "Dump support"
>> diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile
>> index 87646ca..1983afa 100644
>> --- a/drivers/s390/crypto/Makefile
>> +++ b/drivers/s390/crypto/Makefile
>> @@ -13,4 +13,8 @@ obj-$(CONFIG_ZCRYPT) += zcrypt_pcixcc.o zcrypt_cex2a.o zcrypt_cex4.o
>>   
>>   # pkey kernel module
>>   pkey-objs := pkey_api.o
>> -obj-$(CONFIG_PKEY) += pkey.o
>> \ No newline at end of file
>> +obj-$(CONFIG_PKEY) += pkey.o
> Another change of that line.
Will fix this
>
>> +
>> +# adjunct processor matrix
>> +vfio_ap_matrix-objs := vfio_ap_matrix_drv.o
>> +obj-$(CONFIG_VFIO_AP_MATRIX) += vfio_ap_matrix.o
>> diff --git a/drivers/s390/crypto/ap_matrix_bus.c b/drivers/s390/crypto/ap_matrix_bus.c
>> index 4eb1e3c..66bfa54 100644
>> --- a/drivers/s390/crypto/ap_matrix_bus.c
>> +++ b/drivers/s390/crypto/ap_matrix_bus.c
>> @@ -75,10 +75,18 @@ static int ap_matrix_dev_create(void)
>>   	return 0;
>>   }
>>   
>> +struct ap_matrix *ap_matrix_get_device(void)
>> +{
>> +	return matrix;
> See the comments I had for the previous patch. In particular, I think
> it is better to retrieve a pointer to the matrix device via driver core
> methods.
I got some objections to creating a new bus and since there will only ever
be a single AP matrix device, I decided there really wasn't a need for an
AP matrix bus and got rid of it. I opted instead to create the matrix 
device
in the init function of the vfio_ap_matrix driver. Rather than passing 
around a
pointer, I put the following in vfio_ap_matrix_private.h:

     struct ap_matrix {
         struct device device;
         spinlock_t qlock;
         struct list_head queues;
     };

     extern struct ap_matrix ap_matrix;

... and declared the ap_matrix in  the driver (vfio_ap_matrix_drv.c) 
file as:

     struct ap_matrix ap_matrix;

Does this seem like a reasonable approach?


>
>> +}
>> +EXPORT_SYMBOL(ap_matrix_get_device);
>> +
>>   int __init ap_matrix_init(void)
>>   {
>>   	int ret;
>>   
>> +	matrix = NULL;
>> +
>>   	ap_matrix_root_device = root_device_register(AP_MATRIX_BUS_NAME);
>>   	ret = PTR_RET(ap_matrix_root_device);
>>   	if (ret)
>> diff --git a/drivers/s390/crypto/ap_matrix_bus.h b/drivers/s390/crypto/ap_matrix_bus.h
>> index 225db4f..c2aff23 100644
>> --- a/drivers/s390/crypto/ap_matrix_bus.h
>> +++ b/drivers/s390/crypto/ap_matrix_bus.h
>> @@ -16,6 +16,6 @@ struct ap_matrix {
>>   	struct device device;
>>   };
>>   
>> -int ap_matrix_init(void);
> So, was that not needed before?
Forgot to remove it when I refactored a previous patch in which was 
introduced.
>
>> +struct ap_matrix *ap_matrix_get_device(void);
>>   
>>   #endif /* _AP_MATRIX_BUS_H_ */
>> diff --git a/drivers/s390/crypto/vfio_ap_matrix_drv.c b/drivers/s390/crypto/vfio_ap_matrix_drv.c
>> new file mode 100644
>> index 0000000..760ed56
>> --- /dev/null
>> +++ b/drivers/s390/crypto/vfio_ap_matrix_drv.c
>> @@ -0,0 +1,102 @@
>> +/*
>> + * VFIO based AP Matrix device driver
>> + *
>> + * Copyright IBM Corp. 2017
>> + *
>> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> + */
>> +
>> +#include <linux/module.h>
>> +#include <linux/mod_devicetable.h>
>> +#include <linux/slab.h>
>> +
>> +#include "ap_bus.h"
>> +#include "ap_matrix_bus.h"
>> +#include "vfio_ap_matrix_private.h"
>> +
>> +#define VFIO_AP_MATRIX_DRV_NAME "vfio_ap_queue"
>> +
>> +MODULE_AUTHOR("IBM Corporation");
>> +MODULE_DESCRIPTION("AP Matrix device driver, Copyright IBM Corp. 2017");
>> +MODULE_LICENSE("GPL v2");
>> +
>> +static struct ap_device_id ap_queue_ids[] = {
>> +	{ .dev_type = AP_DEVICE_TYPE_CEX4,
>> +	  .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
>> +	{ .dev_type = AP_DEVICE_TYPE_CEX5,
>> +	  .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
>> +	{ .dev_type = AP_DEVICE_TYPE_CEX6,
>> +	  .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
> So, you explicitly don't match to all devices, but only to the newer
> ones? This needs an explaining comment.
Okay
>
>> +	{ /* end of list */ },
>> +};
>> +
>> +MODULE_DEVICE_TABLE(ap_matrix, ap_queue_ids);
>> +
>> +static struct ap_matrix_driver {
>> +	struct ap_driver ap_drv;
>> +	struct ap_matrix *ap_matrix;
> Do you actually need that pointer to the matrix device? One usage is to
> pass it as an parameter to the mdev registration in the next patch. As
> you only support one matrix device, would it be better to move getting
> that device into the mdev code?
>
> For the other usage, move getting it into find_vapq()?
See my comments above regarding getting rid of the AP matrix bus.
>
>> +} vfio_ap_matrix_drv;
>> +
>> +static int ap_matrix_queue_dev_probe(struct ap_device *apdev)
>> +{
>> +	struct vfio_ap_queue *vapq;
>> +	struct ap_queue *apq = to_ap_queue(&apdev->device);
>> +	struct ap_matrix *ap_matrix = vfio_ap_matrix_drv.ap_matrix;
>> +
>> +	vapq = kzalloc(sizeof(*vapq), GFP_KERNEL);
>> +	if (!vapq)
>> +		return -ENOMEM;
>> +
>> +	INIT_LIST_HEAD(&vapq->list);
>> +	vapq->queue = apq;
>> +	spin_lock_bh(&ap_matrix->qlock);
>> +	list_add_tail(&vapq->list, &ap_matrix->queues);
>> +	spin_unlock_bh(&ap_matrix->qlock);
>> +
>> +	return 0;
>> +}
>> +
>> +static void ap_matrix_queue_dev_remove(struct ap_device *apdev)
>> +{
>> +	struct vfio_ap_queue *vapq;
>> +	struct ap_queue *apq = to_ap_queue(&apdev->device);
>> +	struct ap_matrix *ap_matrix = vfio_ap_matrix_drv.ap_matrix;
>> +
>> +	vapq = find_vapq(ap_matrix, apq->qid);
>> +
>> +	if (vapq) {
>> +		spin_lock_bh(&ap_matrix->qlock);
>> +		list_del_init(&vapq->list);
>> +		spin_unlock_bh(&ap_matrix->qlock);
>> +		kfree(vapq);
>> +	}
>> +}
>> +
>> +int __init ap_matrix_init(void)
>> +{
>> +
>> +	int ret;
>> +
>> +	vfio_ap_matrix_drv.ap_matrix = ap_matrix_get_device();
>> +	if (!vfio_ap_matrix_drv.ap_matrix)
>> +		return -ENODEV;
>> +
>> +	vfio_ap_matrix_drv.ap_drv.probe = ap_matrix_queue_dev_probe;
>> +	vfio_ap_matrix_drv.ap_drv.remove = ap_matrix_queue_dev_remove;
>> +	vfio_ap_matrix_drv.ap_drv.ids = ap_queue_ids;
> Can you use an static initializer for that?
This is how its done for the AP bus drivers, for example; see zcrypt_cex4.c.
>
>> +
>> +	ret = ap_driver_register(&vfio_ap_matrix_drv.ap_drv,
>> +				 THIS_MODULE, VFIO_AP_MATRIX_DRV_NAME);
>> +	if (ret)
>> +		return ret;
>> +
>> +	return ret;
>> +}
>> +
>> +void __exit ap_matrix_exit(void)
>> +{
>> +	ap_driver_unregister(&vfio_ap_matrix_drv.ap_drv);
>> +}
>> +
>> +module_init(ap_matrix_init);
>> +module_exit(ap_matrix_exit);
>> diff --git a/drivers/s390/crypto/vfio_ap_matrix_private.h b/drivers/s390/crypto/vfio_ap_matrix_private.h
>> new file mode 100644
>> index 0000000..11c5e02
>> --- /dev/null
>> +++ b/drivers/s390/crypto/vfio_ap_matrix_private.h
>> @@ -0,0 +1,47 @@
>> +/*
>> + * Private data and functions for adjunct processor VFIO matrix driver.
>> + *
>> + * Copyright IBM Corp. 2016
>> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> + */
>> +
>> +#ifndef _VFIO_AP_PRIVATE_H_
>> +#define _VFIO_AP_PRIVATE_H_
>> +
>> +#include <linux/types.h>
>> +
>> +#include "ap_bus.h"
>> +#include "ap_matrix_bus.h"
>> +
>> +#define VFIO_AP_MATRIX_MODULE_NAME "vfio_ap_matrix"
>> +
>> +struct vfio_ap_queue {
>> +	struct ap_queue *queue;
>> +	struct list_head list;
>> +};
>> +
>> +static inline struct vfio_ap_queue *to_vapq(struct ap_device *ap_dev)
>> +{
>> +	struct ap_queue *ap_queue = to_ap_queue(&ap_dev->device);
>> +	struct vfio_ap_queue *vapq;
>> +
>> +	vapq = container_of(&ap_queue, struct vfio_ap_queue, queue);
> Why not just return container_of(...); ?
Will change it.
>
>> +
>> +	return vapq;
>> +}
>> +
>> +static inline struct vfio_ap_queue *find_vapq(struct ap_matrix *ap_matrix,
>> +					      ap_qid_t qid)
>> +{
>> +	struct vfio_ap_queue *vapq;
>> +
>> +	if (!list_empty(&ap_matrix->queues)) {
>> +		list_for_each_entry(vapq, &ap_matrix->queues, list)
>> +			if (vapq->queue->qid == qid)
>> +				return vapq;
>> +	}
>> +
>> +	return NULL;
>> +}
>> +
>> +#endif /* _VFIO_AP_PRIVATE_H_ */

Re: [RFC 05/19] s390/zcrypt: base implementation of AP matrix device driver

[Cornelia Huck]

On Tue, 14 Nov 2017 11:37:05 -0500
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> On 11/14/2017 07:40 AM, Cornelia Huck wrote:
> > On Fri, 13 Oct 2017 13:38:50 -0400
> > Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> >> diff --git a/drivers/s390/crypto/ap_matrix_bus.c b/drivers/s390/crypto/ap_matrix_bus.c
> >> index 4eb1e3c..66bfa54 100644
> >> --- a/drivers/s390/crypto/ap_matrix_bus.c
> >> +++ b/drivers/s390/crypto/ap_matrix_bus.c
> >> @@ -75,10 +75,18 @@ static int ap_matrix_dev_create(void)
> >>   	return 0;
> >>   }
> >>   
> >> +struct ap_matrix *ap_matrix_get_device(void)
> >> +{
> >> +	return matrix;  
> > See the comments I had for the previous patch. In particular, I think
> > it is better to retrieve a pointer to the matrix device via driver core
> > methods.  
> I got some objections to creating a new bus and since there will only ever
> be a single AP matrix device, I decided there really wasn't a need for an
> AP matrix bus and got rid of it. I opted instead to create the matrix 
> device
> in the init function of the vfio_ap_matrix driver. Rather than passing 
> around a
> pointer, I put the following in vfio_ap_matrix_private.h:
> 
>      struct ap_matrix {
>          struct device device;
>          spinlock_t qlock;
>          struct list_head queues;
>      };
> 
>      extern struct ap_matrix ap_matrix;
> 
> ... and declared the ap_matrix in  the driver (vfio_ap_matrix_drv.c) 
> file as:
> 
>      struct ap_matrix ap_matrix;
> 
> Does this seem like a reasonable approach?

Getting rid of the bus as overhead is not unreasonable.

I'm feeling a bit queasy about the extern, however. I'd prefer a getter
function (that also makes sure refcounting rules are followed).

We can't get around referencing this device from multiple files, can we?

Re: [RFC 05/19] s390/zcrypt: base implementation of AP matrix device driver

[Tony Krowiak]

On 11/14/2017 12:00 PM, Cornelia Huck wrote:
> On Tue, 14 Nov 2017 11:37:05 -0500
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> On 11/14/2017 07:40 AM, Cornelia Huck wrote:
>>> On Fri, 13 Oct 2017 13:38:50 -0400
>>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>>> diff --git a/drivers/s390/crypto/ap_matrix_bus.c b/drivers/s390/crypto/ap_matrix_bus.c
>>>> index 4eb1e3c..66bfa54 100644
>>>> --- a/drivers/s390/crypto/ap_matrix_bus.c
>>>> +++ b/drivers/s390/crypto/ap_matrix_bus.c
>>>> @@ -75,10 +75,18 @@ static int ap_matrix_dev_create(void)
>>>>    	return 0;
>>>>    }
>>>>    
>>>> +struct ap_matrix *ap_matrix_get_device(void)
>>>> +{
>>>> +	return matrix;
>>> See the comments I had for the previous patch. In particular, I think
>>> it is better to retrieve a pointer to the matrix device via driver core
>>> methods.
>> I got some objections to creating a new bus and since there will only ever
>> be a single AP matrix device, I decided there really wasn't a need for an
>> AP matrix bus and got rid of it. I opted instead to create the matrix
>> device
>> in the init function of the vfio_ap_matrix driver. Rather than passing
>> around a
>> pointer, I put the following in vfio_ap_matrix_private.h:
>>
>>       struct ap_matrix {
>>           struct device device;
>>           spinlock_t qlock;
>>           struct list_head queues;
>>       };
>>
>>       extern struct ap_matrix ap_matrix;
>>
>> ... and declared the ap_matrix in  the driver (vfio_ap_matrix_drv.c)
>> file as:
>>
>>       struct ap_matrix ap_matrix;
>>
>> Does this seem like a reasonable approach?
> Getting rid of the bus as overhead is not unreasonable.
>
> I'm feeling a bit queasy about the extern, however. I'd prefer a getter
> function (that also makes sure refcounting rules are followed).
I now think I can avoid having to reference the ap_matrix device from
multiple places. The reason the device is referenced in vfio_ap_matrix_ops.c
is because there is a need for information about the AP queues that have 
been
bound to the vfio_ap_matrix device driver. If interfaces are provided by 
the
vfio_ap_matrix device driver to access the needed information, it
won't be necessary to reference the ap_matrix device directly in
vfio_ap_matrix_ops. I think this would be a better solution, don't you?
>
> We can't get around referencing this device from multiple files, can we?
The only way would be to roll up vfio_ap_matrix_ops.c into 
vfio_ap_matrix_drv.c. I followed the pattern established
by the vfio_ccw
>

Re: [RFC 05/19] s390/zcrypt: base implementation of AP matrix device driver

[Cornelia Huck]

On Tue, 14 Nov 2017 13:15:27 -0500
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> On 11/14/2017 12:00 PM, Cornelia Huck wrote:
> > On Tue, 14 Nov 2017 11:37:05 -0500
> > Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> >  
> >> On 11/14/2017 07:40 AM, Cornelia Huck wrote:  
> >>> On Fri, 13 Oct 2017 13:38:50 -0400
> >>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:  
> >>>> diff --git a/drivers/s390/crypto/ap_matrix_bus.c b/drivers/s390/crypto/ap_matrix_bus.c
> >>>> index 4eb1e3c..66bfa54 100644
> >>>> --- a/drivers/s390/crypto/ap_matrix_bus.c
> >>>> +++ b/drivers/s390/crypto/ap_matrix_bus.c
> >>>> @@ -75,10 +75,18 @@ static int ap_matrix_dev_create(void)
> >>>>    	return 0;
> >>>>    }
> >>>>    
> >>>> +struct ap_matrix *ap_matrix_get_device(void)
> >>>> +{
> >>>> +	return matrix;  
> >>> See the comments I had for the previous patch. In particular, I think
> >>> it is better to retrieve a pointer to the matrix device via driver core
> >>> methods.  
> >> I got some objections to creating a new bus and since there will only ever
> >> be a single AP matrix device, I decided there really wasn't a need for an
> >> AP matrix bus and got rid of it. I opted instead to create the matrix
> >> device
> >> in the init function of the vfio_ap_matrix driver. Rather than passing
> >> around a
> >> pointer, I put the following in vfio_ap_matrix_private.h:
> >>
> >>       struct ap_matrix {
> >>           struct device device;
> >>           spinlock_t qlock;
> >>           struct list_head queues;
> >>       };
> >>
> >>       extern struct ap_matrix ap_matrix;
> >>
> >> ... and declared the ap_matrix in  the driver (vfio_ap_matrix_drv.c)
> >> file as:
> >>
> >>       struct ap_matrix ap_matrix;
> >>
> >> Does this seem like a reasonable approach?  
> > Getting rid of the bus as overhead is not unreasonable.
> >
> > I'm feeling a bit queasy about the extern, however. I'd prefer a getter
> > function (that also makes sure refcounting rules are followed).  
> I now think I can avoid having to reference the ap_matrix device from
> multiple places. The reason the device is referenced in vfio_ap_matrix_ops.c
> is because there is a need for information about the AP queues that have 
> been
> bound to the vfio_ap_matrix device driver. If interfaces are provided by 
> the
> vfio_ap_matrix device driver to access the needed information, it
> won't be necessary to reference the ap_matrix device directly in
> vfio_ap_matrix_ops. I think this would be a better solution, don't you?

That sounds good, agreed.

Re: [RFC 05/19] s390/zcrypt: base implementation of AP matrix device driver

[Pierre Morel]

On 14/11/2017 17:37, Tony Krowiak wrote:
> On 11/14/2017 07:40 AM, Cornelia Huck wrote:
>> On Fri, 13 Oct 2017 13:38:50 -0400
>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>
>>> Introduces a new AP matrix device driver. This device driver
>>> will ultimately perform the following functions:
>>>
>>> * Register with the AP bus to let it know that the matrix
>>>    driver can control AP queue devices. This will allow
>>>    an administrator to unbind an AP queue device from its
>>>    device driver and bind it to the matrix device driver.
>>>    This is how AP queue devices will be reserved for use
>>>    by guest machines.
>>>
>>> * Register the matrix device created by the AP matrix bus
>>>    with the VFIO mediated device framework. This will create
>>>    the sysfs entries needed to create mediated matrix devices.
>>>    Each mediated matrix device can be configured with a matrix
>>>    of adapters, usage domains and control domains that can be
>>>    accessed by a guest machine.
>>>
>>> * Process requests via ioctl calls defined for the mediated
>>>    matrix device. The guest can access the ioctl calls via
>>>    the mediated device's file descriptor to:
>>>
>>>      * Grant access to the adapters, usage domains and
>>>        control domains configured for the mediated matrix
>>>        device.
>>>
>>> This device driver
>>> is built on the VFIO mediated device framework. The VFIO mediated
>>> device framework allows a mediated device to be dedicated exclusively
>>> to a single guest VM.
>>>
>>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>>> ---
>>>   MAINTAINERS                                  |    2 +
>>>   arch/s390/Kconfig                            |   13 +++
>>>   arch/s390/configs/default_defconfig          |    1 +
>>>   arch/s390/configs/gcov_defconfig             |    1 +
>>>   arch/s390/configs/performance_defconfig      |    1 +
>>>   arch/s390/defconfig                          |    1 +
>>>   drivers/s390/crypto/Makefile                 |    6 +-
>>>   drivers/s390/crypto/ap_matrix_bus.c          |    8 ++
>>>   drivers/s390/crypto/ap_matrix_bus.h          |    2 +-
>>>   drivers/s390/crypto/vfio_ap_matrix_drv.c     |  102 
>>> ++++++++++++++++++++++++++
>>>   drivers/s390/crypto/vfio_ap_matrix_private.h |   47 ++++++++++++
>>>   11 files changed, 182 insertions(+), 2 deletions(-)
>>>   create mode 100644 drivers/s390/crypto/vfio_ap_matrix_drv.c
>>>   create mode 100644 drivers/s390/crypto/vfio_ap_matrix_private.h
>>>
>>> diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
>>> index 48af970..411c19a 100644
>>> --- a/arch/s390/Kconfig
>>> +++ b/arch/s390/Kconfig
>>> @@ -722,6 +722,19 @@ config VFIO_CCW
>>>         To compile this driver as a module, choose M here: the
>>>         module will be called vfio_ccw.
>>> +config VFIO_AP_MATRIX
>>> +    def_tristate m
>>> +    prompt "Support for Adjunct Processor Matrix device interface"
>>> +    depends on ZCRYPT
>>> +    select VFIO
>>> +    select MDEV
>>> +    select VFIO_MDEV
>>> +    select VFIO_MDEV_DEVICE
>>> +    select IOMMU_API
>> I think the more common pattern is to depend on the VFIO configs
>> instead of selecting them.
> It's ironic because I originally changed from using 'depends on' and 
> changed it based on review comments made
> on our internal mailing list. I'll go with 'depends on'.

Is doing like the others a sufficient good reason?
What if the first who did this did not really think about it?

When an administrator configure the kernel what does he think?

- I want to have AP through AP_VFIO in my guests
	and he get implicitly VFIO
or
- I want to have VFIO
	and he has to explicitly add AP_VFIO too

It seems to me that the first is much more user friendly.

Please tell me if I missed something. dependencies? collateral damages? 
my logic is wrong?

Regards,

Pierre

..snip...

-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [RFC 05/19] s390/zcrypt: base implementation of AP matrix device driver

[Cornelia Huck]

On Thu, 16 Nov 2017 13:02:26 +0100
Pierre Morel <pmorel@linux.vnet.ibm.com> wrote:

> On 14/11/2017 17:37, Tony Krowiak wrote:
> > On 11/14/2017 07:40 AM, Cornelia Huck wrote:  
> >> On Fri, 13 Oct 2017 13:38:50 -0400
> >> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> >>> diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
> >>> index 48af970..411c19a 100644
> >>> --- a/arch/s390/Kconfig
> >>> +++ b/arch/s390/Kconfig
> >>> @@ -722,6 +722,19 @@ config VFIO_CCW
> >>>         To compile this driver as a module, choose M here: the
> >>>         module will be called vfio_ccw.
> >>> +config VFIO_AP_MATRIX
> >>> +    def_tristate m
> >>> +    prompt "Support for Adjunct Processor Matrix device interface"
> >>> +    depends on ZCRYPT
> >>> +    select VFIO
> >>> +    select MDEV
> >>> +    select VFIO_MDEV
> >>> +    select VFIO_MDEV_DEVICE
> >>> +    select IOMMU_API  
> >> I think the more common pattern is to depend on the VFIO configs
> >> instead of selecting them.  
> > It's ironic because I originally changed from using 'depends on' and 
> > changed it based on review comments made
> > on our internal mailing list. I'll go with 'depends on'.  
> 
> Is doing like the others a sufficient good reason?
> What if the first who did this did not really think about it?
> 
> When an administrator configure the kernel what does he think?
> 
> - I want to have AP through AP_VFIO in my guests
> 	and he get implicitly VFIO
> or
> - I want to have VFIO
> 	and he has to explicitly add AP_VFIO too
> 
> It seems to me that the first is much more user friendly.
> 
> Please tell me if I missed something. dependencies? collateral damages? 
> my logic is wrong?

Using select for anything that's not a simple infrastructure dependency
may lead into trouble (we've had issues in the past where options tried
to enable other options but missed dependencies).

If a user wants to use vfio-ap, I think it is reasonable to expect them
to figure out that they need both ap and vfio for that.

[And config help has gotten much better than it was years ago; it's not
that hard to figure out what is actually needed.]

Re: [RFC 05/19] s390/zcrypt: base implementation of AP matrix device driver

[Tony Krowiak]

On 11/16/2017 07:35 AM, Cornelia Huck wrote:
> On Thu, 16 Nov 2017 13:02:26 +0100
> Pierre Morel <pmorel@linux.vnet.ibm.com> wrote:
>
>> On 14/11/2017 17:37, Tony Krowiak wrote:
>>> On 11/14/2017 07:40 AM, Cornelia Huck wrote:
>>>> On Fri, 13 Oct 2017 13:38:50 -0400
>>>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>>>> diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
>>>>> index 48af970..411c19a 100644
>>>>> --- a/arch/s390/Kconfig
>>>>> +++ b/arch/s390/Kconfig
>>>>> @@ -722,6 +722,19 @@ config VFIO_CCW
>>>>>          To compile this driver as a module, choose M here: the
>>>>>          module will be called vfio_ccw.
>>>>> +config VFIO_AP_MATRIX
>>>>> +    def_tristate m
>>>>> +    prompt "Support for Adjunct Processor Matrix device interface"
>>>>> +    depends on ZCRYPT
>>>>> +    select VFIO
>>>>> +    select MDEV
>>>>> +    select VFIO_MDEV
>>>>> +    select VFIO_MDEV_DEVICE
>>>>> +    select IOMMU_API
>>>> I think the more common pattern is to depend on the VFIO configs
>>>> instead of selecting them.
>>> It's ironic because I originally changed from using 'depends on' and
>>> changed it based on review comments made
>>> on our internal mailing list. I'll go with 'depends on'.
>> Is doing like the others a sufficient good reason?
>> What if the first who did this did not really think about it?
>>
>> When an administrator configure the kernel what does he think?
>>
>> - I want to have AP through AP_VFIO in my guests
>> 	and he get implicitly VFIO
>> or
>> - I want to have VFIO
>> 	and he has to explicitly add AP_VFIO too
>>
>> It seems to me that the first is much more user friendly.
>>
>> Please tell me if I missed something. dependencies? collateral damages?
>> my logic is wrong?
> Using select for anything that's not a simple infrastructure dependency
> may lead into trouble (we've had issues in the past where options tried
> to enable other options but missed dependencies).
>
> If a user wants to use vfio-ap, I think it is reasonable to expect them
> to figure out that they need both ap and vfio for that.
>
> [And config help has gotten much better than it was years ago; it's not
> that hard to figure out what is actually needed.]
Is it sufficient to specify 'depends on ZCRYPT && VFIO_MDEV_DEVICE'
since 'VFIO_MDEV_DEVICE depends on VFIO && VFIO_MDEV' and 'VFIO_MDEV
depends on VFIO' and 'VFIO depends on IOMMU_API'?
>

Re: [RFC 05/19] s390/zcrypt: base implementation of AP matrix device driver

[Pierre Morel]

On 16/11/2017 13:35, Cornelia Huck wrote:
> On Thu, 16 Nov 2017 13:02:26 +0100
> Pierre Morel <pmorel@linux.vnet.ibm.com> wrote:
> 
>> On 14/11/2017 17:37, Tony Krowiak wrote:
>>> On 11/14/2017 07:40 AM, Cornelia Huck wrote:
>>>> On Fri, 13 Oct 2017 13:38:50 -0400
>>>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> 
>>>>> diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
>>>>> index 48af970..411c19a 100644
>>>>> --- a/arch/s390/Kconfig
>>>>> +++ b/arch/s390/Kconfig
>>>>> @@ -722,6 +722,19 @@ config VFIO_CCW
>>>>>          To compile this driver as a module, choose M here: the
>>>>>          module will be called vfio_ccw.
>>>>> +config VFIO_AP_MATRIX
>>>>> +    def_tristate m
>>>>> +    prompt "Support for Adjunct Processor Matrix device interface"
>>>>> +    depends on ZCRYPT
>>>>> +    select VFIO
>>>>> +    select MDEV
>>>>> +    select VFIO_MDEV
>>>>> +    select VFIO_MDEV_DEVICE
>>>>> +    select IOMMU_API
>>>> I think the more common pattern is to depend on the VFIO configs
>>>> instead of selecting them.
>>> It's ironic because I originally changed from using 'depends on' and
>>> changed it based on review comments made
>>> on our internal mailing list. I'll go with 'depends on'.
>>
>> Is doing like the others a sufficient good reason?
>> What if the first who did this did not really think about it?
>>
>> When an administrator configure the kernel what does he think?
>>
>> - I want to have AP through AP_VFIO in my guests
>> 	and he get implicitly VFIO
>> or
>> - I want to have VFIO
>> 	and he has to explicitly add AP_VFIO too
>>
>> It seems to me that the first is much more user friendly.
>>
>> Please tell me if I missed something. dependencies? collateral damages?
>> my logic is wrong?
> 
> Using select for anything that's not a simple infrastructure dependency
> may lead into trouble (we've had issues in the past where options tried
> to enable other options but missed dependencies).

Understood, using dependencies is safer against a third party 
introducing a bug that would add a dependency to a member of the list 
but not update our list of selections.

> 
> If a user wants to use vfio-ap, I think it is reasonable to expect them
> to figure out that they need both ap and vfio for that.
> 
> [And config help has gotten much better than it was years ago; it's not
> that hard to figure out what is actually needed.]
> 

OK for Darwin selection for admins, (a gentle Darwin :) I acknowledge) 
and on our side we spare to us running after our disappeared AP VFIO.

Regards,

Pierre



-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Tony Krowiak]

On 11/14/2017 08:57 AM, Cornelia Huck wrote:
> On Tue, 31 Oct 2017 15:39:09 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> On 10/13/2017 01:38 PM, Tony Krowiak wrote:
>> Ping
>>> Tony Krowiak (19):
>>>     KVM: s390: SIE considerations for AP Queue virtualization
>>>     KVM: s390: refactor crypto initialization
>>>     s390/zcrypt: new AP matrix bus
>>>     s390/zcrypt: create an AP matrix device on the AP matrix bus
>>>     s390/zcrypt: base implementation of AP matrix device driver
>>>     s390/zcrypt: register matrix device with VFIO mediated device
>>>       framework
>>>     KVM: s390: introduce AP matrix configuration interface
>>>     s390/zcrypt: support for assigning adapters to matrix mdev
>>>     s390/zcrypt: validate adapter assignment
>>>     s390/zcrypt: sysfs interfaces supporting AP domain assignment
>>>     s390/zcrypt: validate domain assignment
>>>     s390/zcrypt: sysfs support for control domain assignment
>>>     s390/zcrypt: validate control domain assignment
>>>     KVM: s390: Connect the AP mediated matrix device to KVM
>>>     s390/zcrypt: introduce ioctl access to VFIO AP Matrix driver
>>>     KVM: s390: interface to configure KVM guest's AP matrix
>>>     KVM: s390: validate input to AP matrix config interface
>>>     KVM: s390: New ioctl to configure KVM guest's AP matrix
>>>     s390/facilities: enable AP facilities needed by guest
> I think the approach is fine, and the code also looks fine for the most
> part. Some comments:
>
> - various patches can be squashed together to give a better
>    understanding at a glance
Which patches would you squash?
> - this needs documentation (as I already said)
My plan is to take the cover letter patch and incorporate that into 
documentation,
then replace the cover letter patch with a more concise summary.
> - some of the driver/device modelling feels a bit awkward (commented in
>    patches) -- I'm not sure that my proposal is better, but I think we
>    should make sure the interdependencies are modeled correctly
I am responding to each patch review individually.
> - some minor stuff
>

Re: [RFC 06/19] s390/zcrypt: register matrix device with VFIO mediated device framework

[Tony Krowiak]

On 11/14/2017 08:14 AM, Cornelia Huck wrote:
> On Fri, 13 Oct 2017 13:38:51 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> Registers the matrix device created by the AP matrix bus with
>> the VFIO mediated device framework. Registering the matrix
>> device will create the sysfs structures needed to create
>> mediated matrix devices that can be configured with a matrix
>> of adapters, usage domains and control domains for a guest
>> machine.
>>
>> Registering the matrix device with the VFIO mediated device
>> framework will create the following sysfs structures:
>>
>> /sys/devices/ap_matrix
>> ... [matrix]
>> ...... [mdev_supported_types]
>> ......... [ap_matrix-passthrough]
>> ............ available_instances
>> ............ create
>> ............ device_api
>> ............ [devices]
>> ............ name
>>
>> To create a mediated device for the AP matrix device, write a UUID
>> to the create file:
>>
>>      uuidgen > create
>>
>> A symbolic link to the mediated device's directory will be created in the
>> devices subdirectory named after the generated $uuid:
>>
>> /sys/devices/ap_matrix
>> ... [matrix]
>> ...... [mdev_supported_types]
>> ......... [ap_matrix-passthrough]
>> ............ [devices]
>> ............... [$uuid]
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>>   MAINTAINERS                                  |    1 +
>>   drivers/s390/crypto/Makefile                 |    2 +-
>>   drivers/s390/crypto/ap_matrix_bus.c          |    1 +
>>   drivers/s390/crypto/ap_matrix_bus.h          |    4 +
>>   drivers/s390/crypto/vfio_ap_matrix_drv.c     |    5 +
>>   drivers/s390/crypto/vfio_ap_matrix_ops.c     |  172 ++++++++++++++++++++++++++
>>   drivers/s390/crypto/vfio_ap_matrix_private.h |    3 +
>>   include/uapi/linux/vfio.h                    |    1 +
>>   8 files changed, 188 insertions(+), 1 deletions(-)
>>   create mode 100644 drivers/s390/crypto/vfio_ap_matrix_ops.c
>>
>> diff --git a/drivers/s390/crypto/ap_matrix_bus.c b/drivers/s390/crypto/ap_matrix_bus.c
>> index 66bfa54..418c23b 100644
>> --- a/drivers/s390/crypto/ap_matrix_bus.c
>> +++ b/drivers/s390/crypto/ap_matrix_bus.c
>> @@ -61,6 +61,7 @@ static int ap_matrix_dev_create(void)
>>   	matrix->device.bus = &ap_matrix_bus_type;
>>   	matrix->device.parent = ap_matrix_root_device;
>>   	matrix->device.release = ap_matrix_dev_release;
>> +	INIT_LIST_HEAD(&matrix->queues);
> Don't you also need to init the spin lock?
Yes, but note that I am getting rid of the AP matrix bus - as we 
discussed earlier - so
this code will be moved to the AP matrix driver.
>
>>   
>>   	ret = device_register(&matrix->device);
>>   	if (ret) {
>> diff --git a/drivers/s390/crypto/ap_matrix_bus.h b/drivers/s390/crypto/ap_matrix_bus.h
>> index c2aff23..3eccc36 100644
>> --- a/drivers/s390/crypto/ap_matrix_bus.h
>> +++ b/drivers/s390/crypto/ap_matrix_bus.h
>> @@ -12,8 +12,12 @@
>>   
>>   #include <linux/device.h>
>>   
>> +#include "ap_bus.h"
> Why do you need this include now? (IOW, does that need to go into
> another patch?)
It is not needed here, but as discussed earlier, the AP matrix bus is
going away.
>
>> +
>>   struct ap_matrix {
>>   	struct device device;
>> +	spinlock_t qlock;
>> +	struct list_head queues;
>>   };
>>   
>>   struct ap_matrix *ap_matrix_get_device(void);
>> diff --git a/drivers/s390/crypto/vfio_ap_matrix_ops.c b/drivers/s390/crypto/vfio_ap_matrix_ops.c
>> new file mode 100644
>> index 0000000..7d01f18
>> --- /dev/null
>> +++ b/drivers/s390/crypto/vfio_ap_matrix_ops.c
>> @@ -0,0 +1,172 @@
>> +/*
>> + * Adjunct processor matrix VFIO device driver callbacks.
>> + *
>> + * Copyright IBM Corp. 2017
>> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> + *
>> + */
>> +#include <asm/ap-config.h>
>> +#include <linux/string.h>
>> +#include <linux/vfio.h>
>> +#include <linux/mdev.h>
>> +#include <linux/device.h>
>> +#include <linux/list.h>
>> +#include <linux/ctype.h>
>> +
>> +#include "vfio_ap_matrix_private.h"
>> +
>> +#define AP_MATRIX_MDEV_TYPE_HWVIRT "passthrough"
>> +#define AP_MATRIX_MDEV_NAME_HWVIRT "AP Matrix Passthrough Device"
>> +
>> +#define AP_MATRIX_MAX_AVAILABLE_INSTANCES	255
>> +
>> +struct ap_matrix_mdev {
>> +	struct mdev_device *mdev;
>> +	struct list_head node;
>> +};
>> +
>> +struct ap_matrix	*matrix;
>> +struct mutex		mdev_devices_lock;
>> +struct list_head	mdev_devices;
>> +int			available_instances;
> Having this as a global variable feels wrong. I see that vfio-ccw keeps
> things like this in driver data attached to its parent. Can you do
> something like that as well?
I am moving all of these variables into struct ap_matrix. The matrix device
will be stored as driver data for the device so it will be retrievable by
the mdev functions. This will eliminate the need for these globals and
also omit the need to store a reference to the matrix device in
multiple places.

>
>> +
>> +static struct ap_matrix_mdev *ap_matrix_mdev_find_by_uuid(uuid_le uuid)
>> +{
>> +	struct ap_matrix_mdev *matrix_mdev;
>> +
>> +	list_for_each_entry(matrix_mdev, &mdev_devices, node) {
>> +		if (uuid_le_cmp(mdev_uuid(matrix_mdev->mdev), uuid) == 0)
>> +			return matrix_mdev;
>> +	}
>> +
>> +	return NULL;
>> +}
> (...)
>
>> +int ap_matrix_mdev_register(struct ap_matrix *ap_matrix)
>> +{
>> +	int ret;
>> +	struct device *dev = &ap_matrix->device;
>> +
>> +	ret = mdev_register_device(dev, &vfio_ap_matrix_ops);
>> +	if (ret)
>> +		return ret;
>> +
>> +	matrix = ap_matrix;
> I'd just grab the (one) matrix device here...
I am changing this so the matrix device will be stored as
driver data with the device:

     dev_set_drvdata(dev)
>
>> +	mutex_init(&mdev_devices_lock);
>> +	INIT_LIST_HEAD(&mdev_devices);
>> +	available_instances = AP_MATRIX_MAX_AVAILABLE_INSTANCES;
>> +
>> +	return 0;
>> +}
>> +
>> +void ap_matrix_mdev_unregister(struct ap_matrix *ap_matrix)
>> +{
>> +	struct device *dev;
>> +
>> +	if (ap_matrix == matrix) {
> ...and lose this extra if.
It is going bye bye.
>
>> +		dev = &matrix->device;
>> +		available_instances--;
>> +		mdev_unregister_device(dev);
>> +	}
>> +}

Re: [RFC 07/19] KVM: s390: introduce AP matrix configuration interface

[Tony Krowiak]

On 11/14/2017 08:16 AM, Cornelia Huck wrote:
> On Fri, 13 Oct 2017 13:38:52 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> This patch introduces a new compilation unit that will contain
>> all of the data structures and logic for configuring AP adapters,
>> usage domains and control domains for a KVM guest.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>>   MAINTAINERS                       |    2 ++
>>   arch/s390/include/asm/ap-config.h |   12 ++++++++++++
>>   arch/s390/kvm/Makefile            |    2 +-
>>   arch/s390/kvm/ap-config.c         |    9 +++++++++
>>   4 files changed, 24 insertions(+), 1 deletions(-)
>>   create mode 100644 arch/s390/include/asm/ap-config.h
>>   create mode 100644 arch/s390/kvm/ap-config.c
> This is a bit useless on its own... merge this with the actual
> implementation?
I agree, I'm going to squash patches 7 and 8
>

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Pierre Morel]

On 16/11/2017 16:23, Tony Krowiak wrote:
> On 11/14/2017 08:57 AM, Cornelia Huck wrote:
>> On Tue, 31 Oct 2017 15:39:09 -0400
>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>
>>> On 10/13/2017 01:38 PM, Tony Krowiak wrote:
>>> Ping
>>>> Tony Krowiak (19):
>>>>     KVM: s390: SIE considerations for AP Queue virtualization
>>>>     KVM: s390: refactor crypto initialization
>>>>     s390/zcrypt: new AP matrix bus
>>>>     s390/zcrypt: create an AP matrix device on the AP matrix bus
>>>>     s390/zcrypt: base implementation of AP matrix device driver
>>>>     s390/zcrypt: register matrix device with VFIO mediated device
>>>>       framework
>>>>     KVM: s390: introduce AP matrix configuration interface
>>>>     s390/zcrypt: support for assigning adapters to matrix mdev
>>>>     s390/zcrypt: validate adapter assignment
>>>>     s390/zcrypt: sysfs interfaces supporting AP domain assignment
>>>>     s390/zcrypt: validate domain assignment
>>>>     s390/zcrypt: sysfs support for control domain assignment
>>>>     s390/zcrypt: validate control domain assignment
>>>>     KVM: s390: Connect the AP mediated matrix device to KVM
>>>>     s390/zcrypt: introduce ioctl access to VFIO AP Matrix driver
>>>>     KVM: s390: interface to configure KVM guest's AP matrix
>>>>     KVM: s390: validate input to AP matrix config interface
>>>>     KVM: s390: New ioctl to configure KVM guest's AP matrix
>>>>     s390/facilities: enable AP facilities needed by guest
>> I think the approach is fine, and the code also looks fine for the most
>> part. Some comments:
>>
>> - various patches can be squashed together to give a better
>>    understanding at a glance
> Which patches would you squash?
>> - this needs documentation (as I already said)
> My plan is to take the cover letter patch and incorporate that into 
> documentation,
> then replace the cover letter patch with a more concise summary.
>> - some of the driver/device modelling feels a bit awkward (commented in
>>    patches) -- I'm not sure that my proposal is better, but I think we
>>    should make sure the interdependencies are modeled correctly
> I am responding to each patch review individually.

I think that instead of responding to each patch individually we should 
have a discussion on the design because I think a lot could change and 
discussing about each patch as they may be completely redesigned for the 
next version may not be very useful.

So I totally agree with Conny on that we should stabilize the 
bus/device/driver modeling.

I think it would be here a good place to start the discussion on things 
like we started to discuss, Harald and I, off-line:
- why a matrix bus, in which case we can avoid it
- which kind of devices we need
- how to handle the repartition of queues on boot, reset and hotplug
- interaction with the host drivers
- validation of the matrix for guests and host views

or even features we need to add like
- interruptions
- PAPQ/TAPQ-t and APQI interception
- virtualization of the AP
- CPU model and KVM capabilities

In my understanding these points must be cleared before we really start 
to discuss the details of the implementation.

Best regards,

Pierre

>> - some minor stuff
>>
> 


-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [RFC 05/19] s390/zcrypt: base implementation of AP matrix device driver

[Cornelia Huck]

On Thu, 16 Nov 2017 09:25:27 -0500
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> On 11/16/2017 07:35 AM, Cornelia Huck wrote:
> > On Thu, 16 Nov 2017 13:02:26 +0100
> > Pierre Morel <pmorel@linux.vnet.ibm.com> wrote:
> >  
> >> On 14/11/2017 17:37, Tony Krowiak wrote:  
> >>> On 11/14/2017 07:40 AM, Cornelia Huck wrote:  
> >>>> On Fri, 13 Oct 2017 13:38:50 -0400
> >>>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:  
> >>>>> diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
> >>>>> index 48af970..411c19a 100644
> >>>>> --- a/arch/s390/Kconfig
> >>>>> +++ b/arch/s390/Kconfig
> >>>>> @@ -722,6 +722,19 @@ config VFIO_CCW
> >>>>>          To compile this driver as a module, choose M here: the
> >>>>>          module will be called vfio_ccw.
> >>>>> +config VFIO_AP_MATRIX
> >>>>> +    def_tristate m
> >>>>> +    prompt "Support for Adjunct Processor Matrix device interface"
> >>>>> +    depends on ZCRYPT
> >>>>> +    select VFIO
> >>>>> +    select MDEV
> >>>>> +    select VFIO_MDEV
> >>>>> +    select VFIO_MDEV_DEVICE
> >>>>> +    select IOMMU_API  
> >>>> I think the more common pattern is to depend on the VFIO configs
> >>>> instead of selecting them.  
> >>> It's ironic because I originally changed from using 'depends on' and
> >>> changed it based on review comments made
> >>> on our internal mailing list. I'll go with 'depends on'.  
> >> Is doing like the others a sufficient good reason?
> >> What if the first who did this did not really think about it?
> >>
> >> When an administrator configure the kernel what does he think?
> >>
> >> - I want to have AP through AP_VFIO in my guests
> >> 	and he get implicitly VFIO
> >> or
> >> - I want to have VFIO
> >> 	and he has to explicitly add AP_VFIO too
> >>
> >> It seems to me that the first is much more user friendly.
> >>
> >> Please tell me if I missed something. dependencies? collateral damages?
> >> my logic is wrong?  
> > Using select for anything that's not a simple infrastructure dependency
> > may lead into trouble (we've had issues in the past where options tried
> > to enable other options but missed dependencies).
> >
> > If a user wants to use vfio-ap, I think it is reasonable to expect them
> > to figure out that they need both ap and vfio for that.
> >
> > [And config help has gotten much better than it was years ago; it's not
> > that hard to figure out what is actually needed.]  
> Is it sufficient to specify 'depends on ZCRYPT && VFIO_MDEV_DEVICE'
> since 'VFIO_MDEV_DEVICE depends on VFIO && VFIO_MDEV' and 'VFIO_MDEV
> depends on VFIO' and 'VFIO depends on IOMMU_API'?

Perhaps ZCRYPT && VFIO_MDEV && VFIO_MDEV_DEVICE, to make it a bit more
obvious?

[Also, is IOMMU_API only needed to satisfy dependencies?]

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Cornelia Huck]

On Thu, 16 Nov 2017 10:23:25 -0500
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> On 11/14/2017 08:57 AM, Cornelia Huck wrote:
> > On Tue, 31 Oct 2017 15:39:09 -0400
> > Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> >  
> >> On 10/13/2017 01:38 PM, Tony Krowiak wrote:
> >> Ping  
> >>> Tony Krowiak (19):
> >>>     KVM: s390: SIE considerations for AP Queue virtualization
> >>>     KVM: s390: refactor crypto initialization
> >>>     s390/zcrypt: new AP matrix bus
> >>>     s390/zcrypt: create an AP matrix device on the AP matrix bus
> >>>     s390/zcrypt: base implementation of AP matrix device driver
> >>>     s390/zcrypt: register matrix device with VFIO mediated device
> >>>       framework
> >>>     KVM: s390: introduce AP matrix configuration interface
> >>>     s390/zcrypt: support for assigning adapters to matrix mdev
> >>>     s390/zcrypt: validate adapter assignment
> >>>     s390/zcrypt: sysfs interfaces supporting AP domain assignment
> >>>     s390/zcrypt: validate domain assignment
> >>>     s390/zcrypt: sysfs support for control domain assignment
> >>>     s390/zcrypt: validate control domain assignment
> >>>     KVM: s390: Connect the AP mediated matrix device to KVM
> >>>     s390/zcrypt: introduce ioctl access to VFIO AP Matrix driver
> >>>     KVM: s390: interface to configure KVM guest's AP matrix
> >>>     KVM: s390: validate input to AP matrix config interface
> >>>     KVM: s390: New ioctl to configure KVM guest's AP matrix
> >>>     s390/facilities: enable AP facilities needed by guest  
> > I think the approach is fine, and the code also looks fine for the most
> > part. Some comments:
> >
> > - various patches can be squashed together to give a better
> >    understanding at a glance  
> Which patches would you squash?

See the patches. As a rule, I find it more sensible to introduce
interface + implementation together rather than separate.

> > - this needs documentation (as I already said)  
> My plan is to take the cover letter patch and incorporate that into 
> documentation,
> then replace the cover letter patch with a more concise summary.

Sounds good.

> > - some of the driver/device modelling feels a bit awkward (commented in
> >    patches) -- I'm not sure that my proposal is better, but I think we
> >    should make sure the interdependencies are modeled correctly  
> I am responding to each patch review individually.
> > - some minor stuff
> >  
> 

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Cornelia Huck]

On Thu, 16 Nov 2017 17:06:58 +0100
Pierre Morel <pmorel@linux.vnet.ibm.com> wrote:

> On 16/11/2017 16:23, Tony Krowiak wrote:
> > On 11/14/2017 08:57 AM, Cornelia Huck wrote:  
> >> On Tue, 31 Oct 2017 15:39:09 -0400
> >> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> >>  
> >>> On 10/13/2017 01:38 PM, Tony Krowiak wrote:
> >>> Ping  
> >>>> Tony Krowiak (19):
> >>>>     KVM: s390: SIE considerations for AP Queue virtualization
> >>>>     KVM: s390: refactor crypto initialization
> >>>>     s390/zcrypt: new AP matrix bus
> >>>>     s390/zcrypt: create an AP matrix device on the AP matrix bus
> >>>>     s390/zcrypt: base implementation of AP matrix device driver
> >>>>     s390/zcrypt: register matrix device with VFIO mediated device
> >>>>       framework
> >>>>     KVM: s390: introduce AP matrix configuration interface
> >>>>     s390/zcrypt: support for assigning adapters to matrix mdev
> >>>>     s390/zcrypt: validate adapter assignment
> >>>>     s390/zcrypt: sysfs interfaces supporting AP domain assignment
> >>>>     s390/zcrypt: validate domain assignment
> >>>>     s390/zcrypt: sysfs support for control domain assignment
> >>>>     s390/zcrypt: validate control domain assignment
> >>>>     KVM: s390: Connect the AP mediated matrix device to KVM
> >>>>     s390/zcrypt: introduce ioctl access to VFIO AP Matrix driver
> >>>>     KVM: s390: interface to configure KVM guest's AP matrix
> >>>>     KVM: s390: validate input to AP matrix config interface
> >>>>     KVM: s390: New ioctl to configure KVM guest's AP matrix
> >>>>     s390/facilities: enable AP facilities needed by guest  
> >> I think the approach is fine, and the code also looks fine for the most
> >> part. Some comments:
> >>
> >> - various patches can be squashed together to give a better
> >>    understanding at a glance  
> > Which patches would you squash?  
> >> - this needs documentation (as I already said)  
> > My plan is to take the cover letter patch and incorporate that into 
> > documentation,
> > then replace the cover letter patch with a more concise summary.  
> >> - some of the driver/device modelling feels a bit awkward (commented in
> >>    patches) -- I'm not sure that my proposal is better, but I think we
> >>    should make sure the interdependencies are modeled correctly  
> > I am responding to each patch review individually.  
> 
> I think that instead of responding to each patch individually we should 
> have a discussion on the design because I think a lot could change and 
> discussing about each patch as they may be completely redesigned for the 
> next version may not be very useful.
> 
> So I totally agree with Conny on that we should stabilize the 
> bus/device/driver modeling.
> 
> I think it would be here a good place to start the discussion on things 
> like we started to discuss, Harald and I, off-line:
> - why a matrix bus, in which case we can avoid it

I thought it had been agreed that we should be able to ditch it?

> - which kind of devices we need

What is still unclear? Which card generations to support?

> - how to handle the repartition of queues on boot, reset and hotplug

That's something I'd like to see a writeup for.

> - interaction with the host drivers

The driver model should already handle that, no?

> - validation of the matrix for guests and host views

I saw validation code in the patches, although I have not reviewed it.

> 
> or even features we need to add like
> - interruptions

My understanding is that interrupts are optional so they can be left
out in the first shot. With the gisa (that has not yet been posted), it
should not be too difficult, no?

> - PAPQ/TAPQ-t and APQI interception

I can't say anything about that, as this is not documented :(

> - virtualization of the AP

Is this really needed? It would complicate everything a lot.

> - CPU model and KVM capabilities

That already has been discussed with the individual patches.

> 
> In my understanding these points must be cleared before we really start 
> to discuss the details of the implementation.

The general design already looks fine to me. Do you really expect that
a major redesign is needed?

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Pierre Morel]

On 16/11/2017 18:03, Cornelia Huck wrote:
> On Thu, 16 Nov 2017 17:06:58 +0100
> Pierre Morel <pmorel@linux.vnet.ibm.com> wrote:
> 
>> On 16/11/2017 16:23, Tony Krowiak wrote:
>>> On 11/14/2017 08:57 AM, Cornelia Huck wrote:
>>>> On Tue, 31 Oct 2017 15:39:09 -0400
>>>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>>>   
>>>>> On 10/13/2017 01:38 PM, Tony Krowiak wrote:
>>>>> Ping
>>>>>> Tony Krowiak (19):
>>>>>>      KVM: s390: SIE considerations for AP Queue virtualization
>>>>>>      KVM: s390: refactor crypto initialization
>>>>>>      s390/zcrypt: new AP matrix bus
>>>>>>      s390/zcrypt: create an AP matrix device on the AP matrix bus
>>>>>>      s390/zcrypt: base implementation of AP matrix device driver
>>>>>>      s390/zcrypt: register matrix device with VFIO mediated device
>>>>>>        framework
>>>>>>      KVM: s390: introduce AP matrix configuration interface
>>>>>>      s390/zcrypt: support for assigning adapters to matrix mdev
>>>>>>      s390/zcrypt: validate adapter assignment
>>>>>>      s390/zcrypt: sysfs interfaces supporting AP domain assignment
>>>>>>      s390/zcrypt: validate domain assignment
>>>>>>      s390/zcrypt: sysfs support for control domain assignment
>>>>>>      s390/zcrypt: validate control domain assignment
>>>>>>      KVM: s390: Connect the AP mediated matrix device to KVM
>>>>>>      s390/zcrypt: introduce ioctl access to VFIO AP Matrix driver
>>>>>>      KVM: s390: interface to configure KVM guest's AP matrix
>>>>>>      KVM: s390: validate input to AP matrix config interface
>>>>>>      KVM: s390: New ioctl to configure KVM guest's AP matrix
>>>>>>      s390/facilities: enable AP facilities needed by guest
>>>> I think the approach is fine, and the code also looks fine for the most
>>>> part. Some comments:
>>>>
>>>> - various patches can be squashed together to give a better
>>>>     understanding at a glance
>>> Which patches would you squash?
>>>> - this needs documentation (as I already said)
>>> My plan is to take the cover letter patch and incorporate that into
>>> documentation,
>>> then replace the cover letter patch with a more concise summary.
>>>> - some of the driver/device modelling feels a bit awkward (commented in
>>>>     patches) -- I'm not sure that my proposal is better, but I think we
>>>>     should make sure the interdependencies are modeled correctly
>>> I am responding to each patch review individually.
>>
>> I think that instead of responding to each patch individually we should
>> have a discussion on the design because I think a lot could change and
>> discussing about each patch as they may be completely redesigned for the
>> next version may not be very useful.
>>
>> So I totally agree with Conny on that we should stabilize the
>> bus/device/driver modeling.
>>
>> I think it would be here a good place to start the discussion on things
>> like we started to discuss, Harald and I, off-line:
>> - why a matrix bus, in which case we can avoid it
> 
> I thought it had been agreed that we should be able to ditch it?

I have not see any comment on the matrix bus.

> 
>> - which kind of devices we need
> 
> What is still unclear? Which card generations to support?

No, I mean the relation bus/device/driver/mdev...

> 
>> - how to handle the repartition of queues on boot, reset and hotplug
> 
> That's something I'd like to see a writeup for.

yes, and it may have an influence on the bus/device/driver/mdev design

> 
>> - interaction with the host drivers
> 
> The driver model should already handle that, no?

yes it should, but it is not clear for me.

> 
>> - validation of the matrix for guests and host views
> 
> I saw validation code in the patches, although I have not reviewed it.
> 
>>
>> or even features we need to add like
>> - interruptions
> 
> My understanding is that interrupts are optional so they can be left
> out in the first shot. With the gisa (that has not yet been posted), it
> should not be too difficult, no?

you are right I forgot that it is optional

> 
>> - PAPQ/TAPQ-t and APQI interception
> 
> I can't say anything about that, as this is not documented :(

Right we can live without these too.

> 
>> - virtualization of the AP
> 
> Is this really needed? It would complicate everything a lot.

Concern has no sens without interception.
> 
>> - CPU model and KVM capabilities
> 
> That already has been discussed with the individual patches.

Well, if there are no interceptions the individual patches discussions 
are enough.

> 
>>
>> In my understanding these points must be cleared before we really start
>> to discuss the details of the implementation.
> 
> The general design already looks fine to me. Do you really expect that
> a major redesign is needed?
> 

I am worry about the following:
- Will the matrix bus be accepted
- What happens on host reset and hot plug/unplug in host
- What happens with the queues on guest start/halt/restart

Regards,

Pierre

-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Tony Krowiak]

On 11/16/2017 03:25 PM, Pierre Morel wrote:
> On 16/11/2017 18:03, Cornelia Huck wrote:
>> On Thu, 16 Nov 2017 17:06:58 +0100
>> Pierre Morel <pmorel@linux.vnet.ibm.com> wrote:
>>
>>> On 16/11/2017 16:23, Tony Krowiak wrote:
>>>> On 11/14/2017 08:57 AM, Cornelia Huck wrote:
>>>>> On Tue, 31 Oct 2017 15:39:09 -0400
>>>>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>>>>> On 10/13/2017 01:38 PM, Tony Krowiak wrote:
>>>>>> Ping
>>>>>>> Tony Krowiak (19):
>>>>>>>      KVM: s390: SIE considerations for AP Queue virtualization
>>>>>>>      KVM: s390: refactor crypto initialization
>>>>>>>      s390/zcrypt: new AP matrix bus
>>>>>>>      s390/zcrypt: create an AP matrix device on the AP matrix bus
>>>>>>>      s390/zcrypt: base implementation of AP matrix device driver
>>>>>>>      s390/zcrypt: register matrix device with VFIO mediated device
>>>>>>>        framework
>>>>>>>      KVM: s390: introduce AP matrix configuration interface
>>>>>>>      s390/zcrypt: support for assigning adapters to matrix mdev
>>>>>>>      s390/zcrypt: validate adapter assignment
>>>>>>>      s390/zcrypt: sysfs interfaces supporting AP domain assignment
>>>>>>>      s390/zcrypt: validate domain assignment
>>>>>>>      s390/zcrypt: sysfs support for control domain assignment
>>>>>>>      s390/zcrypt: validate control domain assignment
>>>>>>>      KVM: s390: Connect the AP mediated matrix device to KVM
>>>>>>>      s390/zcrypt: introduce ioctl access to VFIO AP Matrix driver
>>>>>>>      KVM: s390: interface to configure KVM guest's AP matrix
>>>>>>>      KVM: s390: validate input to AP matrix config interface
>>>>>>>      KVM: s390: New ioctl to configure KVM guest's AP matrix
>>>>>>>      s390/facilities: enable AP facilities needed by guest
>>>>> I think the approach is fine, and the code also looks fine for the 
>>>>> most
>>>>> part. Some comments:
>>>>>
>>>>> - various patches can be squashed together to give a better
>>>>>     understanding at a glance
>>>> Which patches would you squash?
>>>>> - this needs documentation (as I already said)
>>>> My plan is to take the cover letter patch and incorporate that into
>>>> documentation,
>>>> then replace the cover letter patch with a more concise summary.
>>>>> - some of the driver/device modelling feels a bit awkward 
>>>>> (commented in
>>>>>     patches) -- I'm not sure that my proposal is better, but I 
>>>>> think we
>>>>>     should make sure the interdependencies are modeled correctly
>>>> I am responding to each patch review individually.
>>>
>>> I think that instead of responding to each patch individually we should
>>> have a discussion on the design because I think a lot could change and
>>> discussing about each patch as they may be completely redesigned for 
>>> the
>>> next version may not be very useful.
How do you suggest this discussion should be structured? Aren't the patches
themselves an ultimate expression of the design? A lot could change, but
can't those issues can be dealt with and discussed as we move forward?

>>>
>>>
>>> So I totally agree with Conny on that we should stabilize the
>>> bus/device/driver modeling.
>>>
>>> I think it would be here a good place to start the discussion on things
>>> like we started to discuss, Harald and I, off-line:
>>> - why a matrix bus, in which case we can avoid it
>>
>> I thought it had been agreed that we should be able to ditch it?
>
> I have not see any comment on the matrix bus.
As stated in a previous email responding to Connie, I decided to scrap the
AP matrix bus. There will only ever be one matrix device that serves two
purposes: To hold the APQNs of the queue devices bound to the VFIO AP matrix
device driver; to serve as a parent of the mediated devices created for
guests requiring access to the APQNs reserved for their use. So, instead
of an AP matrix bus creating the matrix device, it will be created by the
VFIO AP matrix driver in /sys/devices/ap_matrix/ during driver 
initialization.
>
>
>>
>>> - which kind of devices we need
>>
>> What is still unclear? Which card generations to support?
>
> No, I mean the relation bus/device/driver/mdev...
I think that is pretty well spelled out in the cover letter
patch and in the descriptions of the patches themselves. What is it
you don't understand?
>
>
>>
>>> - how to handle the repartition of queues on boot, reset and hotplug
What do you mean by repartition of queues on boot?
>>
>> That's something I'd like to see a writeup for.
>
> yes, and it may have an influence on the bus/device/driver/mdev design
I don't understand the need to avoid implementation details. If you recall,
the original design was modeled on AP queue devices. It was only after
implementing that design that the shortcomings were revealed which is
why we decided to base the model on the AP matrix. Keep in mind, this is
an RFC, not a final patch set. I would expect some change from the
implementation herein. In fact, I've already made many changes based on
Connie's and Christian's review comments, none of which resulted in an
overhaul of the design.
>
>
>>
>>> - interaction with the host drivers
>>
>> The driver model should already handle that, no?
>
> yes it should, but it is not clear for me.
What is it that is not clear? This cover letter seeks to describe the
patch set, so why not annotate those areas that are not clear? I'm don't
understand what it is you are expecting. I thought the purpose of
submitting these patches here was to generate discussion.
>
>
>>
>>> - validation of the matrix for guests and host views
>>
>> I saw validation code in the patches, although I have not reviewed it.
Patches 9, 11, and 13 validate the adapters, domains and control domains
configured for the mdev matrix device and patch 17 verifies that the
KVM guest's matrix does not define any APQNs in use by other guests.
>>
>>
>>>
>>> or even features we need to add like
>>> - interruptions
>>
>> My understanding is that interrupts are optional so they can be left
>> out in the first shot. With the gisa (that has not yet been posted), it
>> should not be too difficult, no?
>
> you are right I forgot that it is optional
If the facilities bit (STFLE.65) indicating interrupts are available is not
set for the guest, then the AP bus running on the guest will poll and
interrupts will not have to be handled. This patch set does not enable
interrupts, so it is not relevant at this time. We will not be able to
handle interrupts for the guest until the GISA for passthrough patches
are available. This will be addressed at that time.
>
>
>>
>>> - PAPQ/TAPQ-t and APQI interception
>>
>> I can't say anything about that, as this is not documented :(
>
> Right we can live without these too.
I have implemented interception of the PQAP(TAPQ) instruction and will
include it in the next set of patches. It was not documented here
because this patch set was submitted as an RFC for the purpose of
determining if we are on the right track with regard to the overall
AP matrix design.
>
>
>>
>>> - virtualization of the AP
>>
>> Is this really needed? It would complicate everything a lot.
>
> Concern has no sens without interception.
Virtualization of AP is not on the table right now.
>
>>
>>> - CPU model and KVM capabilities
>>
>> That already has been discussed with the individual patches.
>
> Well, if there are no interceptions the individual patches discussions 
> are enough.
As I stated above, these patches were submitted as an RFC for the 
purpose of
getting a stamp of approval for the general design. Additional functions 
such as
hot plug and interception will be introduced in phases in the near 
future. As
I stated above, I already have the implementation of PQAP(TAPQ) and will 
include
it in the next submission. It does not change the general design one iota.
>
>
>>
>>>
>>> In my understanding these points must be cleared before we really start
>>> to discuss the details of the implementation.
>>
>> The general design already looks fine to me. Do you really expect that
>> a major redesign is needed?
I thought the point of submitting this RFC was to get a sanity check of the
design concepts. According to Christian, he discussed the design with
several maintainers at the KVM forum and they agreed this design was sane.
>>
>
> I am worry about the following:
> - Will the matrix bus be accepted
I am eliminating the matrix bus - based on comments made by Connie for an
individual patch - so no need to worry;-)
>
> - What happens on host reset and hot plug/unplug in host
TBD, but I don't anticipate a major overhaul of the design to accommodate
these eventualities, particularly hot plug which I considered while
creating this design.
>
> - What happens with the queues on guest start/halt/restart
TBD
>
> Regards,
>
> Pierre
>

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Tony Krowiak]

On 11/16/2017 11:49 AM, Cornelia Huck wrote:
> On Thu, 16 Nov 2017 10:23:25 -0500
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> On 11/14/2017 08:57 AM, Cornelia Huck wrote:
>>> On Tue, 31 Oct 2017 15:39:09 -0400
>>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>>   
>>>> On 10/13/2017 01:38 PM, Tony Krowiak wrote:
>>>> Ping
>>>>> Tony Krowiak (19):
>>>>>      KVM: s390: SIE considerations for AP Queue virtualization
>>>>>      KVM: s390: refactor crypto initialization
>>>>>      s390/zcrypt: new AP matrix bus
>>>>>      s390/zcrypt: create an AP matrix device on the AP matrix bus
>>>>>      s390/zcrypt: base implementation of AP matrix device driver
>>>>>      s390/zcrypt: register matrix device with VFIO mediated device
>>>>>        framework
>>>>>      KVM: s390: introduce AP matrix configuration interface
>>>>>      s390/zcrypt: support for assigning adapters to matrix mdev
>>>>>      s390/zcrypt: validate adapter assignment
>>>>>      s390/zcrypt: sysfs interfaces supporting AP domain assignment
>>>>>      s390/zcrypt: validate domain assignment
>>>>>      s390/zcrypt: sysfs support for control domain assignment
>>>>>      s390/zcrypt: validate control domain assignment
>>>>>      KVM: s390: Connect the AP mediated matrix device to KVM
>>>>>      s390/zcrypt: introduce ioctl access to VFIO AP Matrix driver
>>>>>      KVM: s390: interface to configure KVM guest's AP matrix
>>>>>      KVM: s390: validate input to AP matrix config interface
>>>>>      KVM: s390: New ioctl to configure KVM guest's AP matrix
>>>>>      s390/facilities: enable AP facilities needed by guest
>>> I think the approach is fine, and the code also looks fine for the most
>>> part. Some comments:
>>>
>>> - various patches can be squashed together to give a better
>>>     understanding at a glance
>> Which patches would you squash?
> See the patches. As a rule, I find it more sensible to introduce
> interface + implementation together rather than separate.
The only patch that introduces an interface separate from the implementation
is patch7:

	KVM: s390: introduce AP matrix configuration interface

I've squashed that with patch8, s390/zcrypt: support for assigning adapters to matrix mdev

>
>>> - this needs documentation (as I already said)
>> My plan is to take the cover letter patch and incorporate that into
>> documentation,
>> then replace the cover letter patch with a more concise summary.
> Sounds good.
>
>>> - some of the driver/device modelling feels a bit awkward (commented in
>>>     patches) -- I'm not sure that my proposal is better, but I think we
>>>     should make sure the interdependencies are modeled correctly
>> I am responding to each patch review individually.
>>> - some minor stuff
>>>   

Re: [RFC 08/19] s390/zcrypt: support for assigning adapters to matrix mdev

[Tony Krowiak]

On 11/14/2017 08:22 AM, Cornelia Huck wrote:
> On Fri, 13 Oct 2017 13:38:53 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> Provides the sysfs interfaces for assigning an adapter to
>> and unassigning an AP adapter from a mediated matrix device.
>>
>> The IDs of the AP adapters assigned to the mediated matrix
>> device are stored in a bit mask. The bits in the mask, from
>> left to right, correspond to AP adapter numbers 0 to 255. The
>> bit corresponding to the ID of the adapter being assigned will
>> be set in the bit mask.
>>
>> The relevant sysfs structures are:
>>
>> /sys/devices/ap_matrix
>> ... [matrix]
>> ...... [mdev_supported_types]
>> ......... [ap_matrix-passthrough]
>> ............ [devices]
>> ...............[$uuid]
>> .................. adapters
>> .................. assign_adapter
>> .................. unassign_adapter
>>
>> To assign an adapter to the $uuid mediated matrix device, write
>> ID of the adapter (hex value) to the assign_adapter file. To
>> unassign an adapter, write the ID of the adapter (hex value)
>> to the unassign_adapter file. The list of adapters that have
>> been assigned can be viewed by displaying the contents of the
>> adapters file.
>>
>> For example, to assign adapter 0xad to mediated matrix device
>> $uuid:
>>
>> 	echo ad > assign_adapter
>>
>> To unassign adapter 0xad:
>>
>> 	echo ad > unassign_adapter
>>
>> To see the list of adapters assigned:
>>
>> 	cat adapters
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>>   arch/s390/include/asm/ap-config.h        |   13 +++
>>   drivers/s390/crypto/vfio_ap_matrix_ops.c |  147 ++++++++++++++++++++++++++++++
>>   2 files changed, 160 insertions(+), 0 deletions(-)
>>
>> diff --git a/drivers/s390/crypto/vfio_ap_matrix_ops.c b/drivers/s390/crypto/vfio_ap_matrix_ops.c
>> index 7d01f18..e4b1236 100644
>> --- a/drivers/s390/crypto/vfio_ap_matrix_ops.c
>> +++ b/drivers/s390/crypto/vfio_ap_matrix_ops.c
>> @@ -23,6 +23,7 @@
>>   struct ap_matrix_mdev {
>>   	struct mdev_device *mdev;
>>   	struct list_head node;
>> +	struct ap_config_masks masks;
>>   };
>>   
>>   struct ap_matrix	*matrix;
>> @@ -136,9 +137,155 @@ static ssize_t device_api_show(struct kobject *kobj, struct device *dev,
>>   	NULL,
>>   };
>>   
>> +static int ap_matrix_id_is_xnum(char *id)
>> +{
>> +	size_t i;
>> +
>> +	for (i = 0; i < strlen(id); i++) {
>> +		if (!isxdigit(id[i]))
>> +			return 0;
>> +	}
>> +
>> +	return 1;
>> +}
> This feels like something for which an utility function should already
> exist...
See comment below
>
>> +
>> +static int ap_matrix_parse_id(const char *buf, unsigned int *id)
>> +{
>> +	int ret;
>> +	char *bufcpy;
>> +	char *id_str;
>> +
>> +	if ((strlen(buf) == 0) || (*buf == '\n')) {
>> +		pr_err("%s: input buffer '%s' contains no valid hex values",
>> +		       VFIO_AP_MATRIX_MODULE_NAME, buf);
>> +		return -EINVAL;
>> +	}
>> +
>> +	bufcpy = kzalloc(strlen(buf) + 1, GFP_KERNEL);
>> +	if (!bufcpy)
>> +		return -ENOMEM;
>> +
>> +	strcpy(bufcpy, buf);
>> +	id_str = strim(bufcpy);
>> +
>> +	if (!ap_matrix_id_is_xnum(id_str)) {
>> +		pr_err("%s: input id '%s' contains an invalid hex value '%s'",
>> +		       VFIO_AP_MATRIX_MODULE_NAME, buf, id_str);
>> +		ret = -EINVAL;
>> +		goto done;
>> +	}
>> +
>> +	ret = kstrtouint (id_str, 16, id);
>> +	if (ret || (*id >= AP_MATRIX_MAX_MASK_BITS)) {
>> +		pr_err("%s: input id '%s' is not a value from 0 to %x",
>> +		       VFIO_AP_MATRIX_MODULE_NAME, buf,
>> +		       AP_MATRIX_MAX_MASK_BITS);
>> +		ret = -EINVAL;
>> +		goto done;
>> +	}
>> +
>> +	ret = 0;
>> +
>> +done:
>> +	kfree(bufcpy);
>> +	return ret;
>> +}
> Similarly, I suspect you are not the first person with similar parsing
> needs.
I am going to remove this function
>
>> +
>> +static ssize_t ap_matrix_adapters_assign(struct device *dev,
>> +					 struct device_attribute *attr,
>> +					 const char *buf, size_t count)
>> +{
>> +	int ret;
>> +	unsigned int apid;
>> +	struct mdev_device *mdev = mdev_from_dev(dev);
>> +	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
>> +
>> +	ret = ap_matrix_parse_id(buf, &apid);
I'm going to replace this with a call to kstrtouint (buf, 0, &apid). This
will allow the user to enter the adapter ID string using conventional
semantics - e.g., 71 or  0x47 - and the function will determine if the
value is valid.
>> +	if (ret)
>> +		return ret;
>> +
>> +	set_bit_inv((unsigned long)apid,
>> +		    (unsigned long *)matrix_mdev->masks.apm);
> Probably needs a comment regarding byte order of the masks somewhere.
Okay
>
>> +
>> +	return count;
>> +}
>> +static DEVICE_ATTR(assign_adapter, 0644, NULL, ap_matrix_adapters_assign);

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Pierre Morel]

On 17/11/2017 00:35, Tony Krowiak wrote:
> On 11/16/2017 03:25 PM, Pierre Morel wrote:
>> On 16/11/2017 18:03, Cornelia Huck wrote:
>>> On Thu, 16 Nov 2017 17:06:58 +0100
>>> Pierre Morel <pmorel@linux.vnet.ibm.com> wrote:
>>>
>>>> On 16/11/2017 16:23, Tony Krowiak wrote:
>>>>> On 11/14/2017 08:57 AM, Cornelia Huck wrote:
>>>>>> On Tue, 31 Oct 2017 15:39:09 -0400
>>>>>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>>>>>> On 10/13/2017 01:38 PM, Tony Krowiak wrote:
>>>>>>> Ping
>>>>>>>> Tony Krowiak (19):
>>>>>>>>      KVM: s390: SIE considerations for AP Queue virtualization
>>>>>>>>      KVM: s390: refactor crypto initialization
>>>>>>>>      s390/zcrypt: new AP matrix bus
>>>>>>>>      s390/zcrypt: create an AP matrix device on the AP matrix bus
>>>>>>>>      s390/zcrypt: base implementation of AP matrix device driver
>>>>>>>>      s390/zcrypt: register matrix device with VFIO mediated device
>>>>>>>>        framework
>>>>>>>>      KVM: s390: introduce AP matrix configuration interface
>>>>>>>>      s390/zcrypt: support for assigning adapters to matrix mdev
>>>>>>>>      s390/zcrypt: validate adapter assignment
>>>>>>>>      s390/zcrypt: sysfs interfaces supporting AP domain assignment
>>>>>>>>      s390/zcrypt: validate domain assignment
>>>>>>>>      s390/zcrypt: sysfs support for control domain assignment
>>>>>>>>      s390/zcrypt: validate control domain assignment
>>>>>>>>      KVM: s390: Connect the AP mediated matrix device to KVM
>>>>>>>>      s390/zcrypt: introduce ioctl access to VFIO AP Matrix driver
>>>>>>>>      KVM: s390: interface to configure KVM guest's AP matrix
>>>>>>>>      KVM: s390: validate input to AP matrix config interface
>>>>>>>>      KVM: s390: New ioctl to configure KVM guest's AP matrix
>>>>>>>>      s390/facilities: enable AP facilities needed by guest
>>>>>> I think the approach is fine, and the code also looks fine for the 
>>>>>> most
>>>>>> part. Some comments:
>>>>>>
>>>>>> - various patches can be squashed together to give a better
>>>>>>     understanding at a glance
>>>>> Which patches would you squash?
>>>>>> - this needs documentation (as I already said)
>>>>> My plan is to take the cover letter patch and incorporate that into
>>>>> documentation,
>>>>> then replace the cover letter patch with a more concise summary.
>>>>>> - some of the driver/device modelling feels a bit awkward 
>>>>>> (commented in
>>>>>>     patches) -- I'm not sure that my proposal is better, but I 
>>>>>> think we
>>>>>>     should make sure the interdependencies are modeled correctly
>>>>> I am responding to each patch review individually.
>>>>
>>>> I think that instead of responding to each patch individually we should
>>>> have a discussion on the design because I think a lot could change and
>>>> discussing about each patch as they may be completely redesigned for 
>>>> the
>>>> next version may not be very useful.
> How do you suggest this discussion should be structured? Aren't the patches
> themselves an ultimate expression of the design? A lot could change, but
> can't those issues can be dealt with and discussed as we move forward?
> 
>>>>
>>>>
>>>> So I totally agree with Conny on that we should stabilize the
>>>> bus/device/driver modeling.
>>>>
>>>> I think it would be here a good place to start the discussion on things
>>>> like we started to discuss, Harald and I, off-line:
>>>> - why a matrix bus, in which case we can avoid it
>>>
>>> I thought it had been agreed that we should be able to ditch it?
>>
>> I have not see any comment on the matrix bus.
> As stated in a previous email responding to Connie, I decided to scrap the
> AP matrix bus. There will only ever be one matrix device that serves two
> purposes: To hold the APQNs of the queue devices bound to the VFIO AP 
> matrix
> device driver; to serve as a parent of the mediated devices created for
> guests requiring access to the APQNs reserved for their use. So, instead
> of an AP matrix bus creating the matrix device, it will be created by the
> VFIO AP matrix driver in /sys/devices/ap_matrix/ during driver 
> initialization.

Sorry, I did not see the mail, this of course change a lot of things...

>>
>>
>>>
>>>> - which kind of devices we need
>>>
>>> What is still unclear? Which card generations to support?
>>
>> No, I mean the relation bus/device/driver/mdev...
> I think that is pretty well spelled out in the cover letter
> patch and in the descriptions of the patches themselves. What is it
> you don't understand?

If we have no matrix bus anymore I prefer to wait for the cover letter 
of V2 to discuss this.

>>
>>
>>>
>>>> - how to handle the repartition of queues on boot, reset and hotplug
> What do you mean by repartition of queues on boot?
>>>
>>> That's something I'd like to see a writeup for.
>>
>> yes, and it may have an influence on the bus/device/driver/mdev design
> I don't understand the need to avoid implementation details. If you recall,
> the original design was modeled on AP queue devices. It was only after
> implementing that design that the shortcomings were revealed which is
> why we decided to base the model on the AP matrix. Keep in mind, this is
> an RFC, not a final patch set. I would expect some change from the
> implementation herein. In fact, I've already made many changes based on
> Connie's and Christian's review comments, none of which resulted in an
> overhaul of the design.
>>
>>
>>>
>>>> - interaction with the host drivers
>>>
>>> The driver model should already handle that, no?
>>
>> yes it should, but it is not clear for me.
> What is it that is not clear? This cover letter seeks to describe the
> patch set, so why not annotate those areas that are not clear? I'm don't
> understand what it is you are expecting. I thought the purpose of
> submitting these patches here was to generate discussion.
>>
>>
>>>
>>>> - validation of the matrix for guests and host views
>>>
>>> I saw validation code in the patches, although I have not reviewed it.
> Patches 9, 11, and 13 validate the adapters, domains and control domains
> configured for the mdev matrix device and patch 17 verifies that the
> KVM guest's matrix does not define any APQNs in use by other guests.
>>>
>>>
>>>>
>>>> or even features we need to add like
>>>> - interruptions
>>>
>>> My understanding is that interrupts are optional so they can be left
>>> out in the first shot. With the gisa (that has not yet been posted), it
>>> should not be too difficult, no?
>>
>> you are right I forgot that it is optional
> If the facilities bit (STFLE.65) indicating interrupts are available is not
> set for the guest, then the AP bus running on the guest will poll and
> interrupts will not have to be handled. This patch set does not enable
> interrupts, so it is not relevant at this time. We will not be able to
> handle interrupts for the guest until the GISA for passthrough patches
> are available. This will be addressed at that time.
>>
>>
>>>
>>>> - PAPQ/TAPQ-t and APQI interception
>>>
>>> I can't say anything about that, as this is not documented :(
>>
>> Right we can live without these too.
> I have implemented interception of the PQAP(TAPQ) instruction and will
> include it in the next set of patches. It was not documented here
> because this patch set was submitted as an RFC for the purpose of
> determining if we are on the right track with regard to the overall
> AP matrix design. >>
>>
>>>
>>>> - virtualization of the AP
>>>
>>> Is this really needed? It would complicate everything a lot.
>>
>> Concern has no sens without interception.
> Virtualization of AP is not on the table right now.

If we implement interception, we must speak about this, even to say how 
we do not implement virtualization.

>>
>>>
>>>> - CPU model and KVM capabilities
>>>
>>> That already has been discussed with the individual patches.
>>
>> Well, if there are no interceptions the individual patches discussions 
>> are enough.
> As I stated above, these patches were submitted as an RFC for the 
> purpose of
> getting a stamp of approval for the general design. Additional functions 
> such as
> hot plug and interception will be introduced in phases in the near 
> future. As
> I stated above, I already have the implementation of PQAP(TAPQ) and will 
> include
> it in the next submission. It does not change the general design one iota.
>>
>>
>>>
>>>>
>>>> In my understanding these points must be cleared before we really start
>>>> to discuss the details of the implementation.
>>>
>>> The general design already looks fine to me. Do you really expect that
>>> a major redesign is needed?
> I thought the point of submitting this RFC was to get a sanity check of the
> design concepts. According to Christian, he discussed the design with
> several maintainers at the KVM forum and they agreed this design was sane.
>>>
>>
>> I am worry about the following:
>> - Will the matrix bus be accepted
> I am eliminating the matrix bus - based on comments made by Connie for an
> individual patch - so no need to worry;-)
>>
>> - What happens on host reset and hot plug/unplug in host
> TBD, but I don't anticipate a major overhaul of the design to accommodate
> these eventualities, particularly hot plug which I considered while
> creating this design.
>>
>> - What happens with the queues on guest start/halt/restart
> TBD

AFAIU These two points are crucials for device driver design.

Pierre

>>
>> Regards,
>>
>> Pierre
>>
> 


-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Cornelia Huck]

On Thu, 16 Nov 2017 18:41:29 -0500
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> On 11/16/2017 11:49 AM, Cornelia Huck wrote:
> > On Thu, 16 Nov 2017 10:23:25 -0500
> > Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> >  
> >> On 11/14/2017 08:57 AM, Cornelia Huck wrote:  
> >>> On Tue, 31 Oct 2017 15:39:09 -0400
> >>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> >>>     
> >>>> On 10/13/2017 01:38 PM, Tony Krowiak wrote:
> >>>> Ping  
> >>>>> Tony Krowiak (19):
> >>>>>      KVM: s390: SIE considerations for AP Queue virtualization
> >>>>>      KVM: s390: refactor crypto initialization
> >>>>>      s390/zcrypt: new AP matrix bus
> >>>>>      s390/zcrypt: create an AP matrix device on the AP matrix bus
> >>>>>      s390/zcrypt: base implementation of AP matrix device driver
> >>>>>      s390/zcrypt: register matrix device with VFIO mediated device
> >>>>>        framework
> >>>>>      KVM: s390: introduce AP matrix configuration interface
> >>>>>      s390/zcrypt: support for assigning adapters to matrix mdev
> >>>>>      s390/zcrypt: validate adapter assignment
> >>>>>      s390/zcrypt: sysfs interfaces supporting AP domain assignment
> >>>>>      s390/zcrypt: validate domain assignment
> >>>>>      s390/zcrypt: sysfs support for control domain assignment
> >>>>>      s390/zcrypt: validate control domain assignment
> >>>>>      KVM: s390: Connect the AP mediated matrix device to KVM
> >>>>>      s390/zcrypt: introduce ioctl access to VFIO AP Matrix driver
> >>>>>      KVM: s390: interface to configure KVM guest's AP matrix
> >>>>>      KVM: s390: validate input to AP matrix config interface
> >>>>>      KVM: s390: New ioctl to configure KVM guest's AP matrix
> >>>>>      s390/facilities: enable AP facilities needed by guest  
> >>> I think the approach is fine, and the code also looks fine for the most
> >>> part. Some comments:
> >>>
> >>> - various patches can be squashed together to give a better
> >>>     understanding at a glance  
> >> Which patches would you squash?  
> > See the patches. As a rule, I find it more sensible to introduce
> > interface + implementation together rather than separate.  
> The only patch that introduces an interface separate from the implementation
> is patch7:
> 
> 	KVM: s390: introduce AP matrix configuration interface
> 
> I've squashed that with patch8, s390/zcrypt: support for assigning adapters to matrix mdev

Sounds good!

Re: [RFC 08/19] s390/zcrypt: support for assigning adapters to matrix mdev

[Cornelia Huck]

On Thu, 16 Nov 2017 18:53:07 -0500
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> On 11/14/2017 08:22 AM, Cornelia Huck wrote:
> > On Fri, 13 Oct 2017 13:38:53 -0400
> > Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> >> +static ssize_t ap_matrix_adapters_assign(struct device *dev,
> >> +					 struct device_attribute *attr,
> >> +					 const char *buf, size_t count)
> >> +{
> >> +	int ret;
> >> +	unsigned int apid;
> >> +	struct mdev_device *mdev = mdev_from_dev(dev);
> >> +	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
> >> +
> >> +	ret = ap_matrix_parse_id(buf, &apid);  
> I'm going to replace this with a call to kstrtouint (buf, 0, &apid). This
> will allow the user to enter the adapter ID string using conventional
> semantics - e.g., 71 or  0x47 - and the function will determine if the
> value is valid.

Using an existing interface sounds good.

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Cornelia Huck]

On Fri, 17 Nov 2017 08:07:15 +0100
Pierre Morel <pmorel@linux.vnet.ibm.com> wrote:

> On 17/11/2017 00:35, Tony Krowiak wrote:
> > On 11/16/2017 03:25 PM, Pierre Morel wrote:  
> >> On 16/11/2017 18:03, Cornelia Huck wrote:  
> >>> On Thu, 16 Nov 2017 17:06:58 +0100
> >>> Pierre Morel <pmorel@linux.vnet.ibm.com> wrote:
> >>>  
> >>>> On 16/11/2017 16:23, Tony Krowiak wrote:  
> >>>>> On 11/14/2017 08:57 AM, Cornelia Huck wrote:  
> >>>>>> On Tue, 31 Oct 2017 15:39:09 -0400
> >>>>>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:  
> >>>>>>> On 10/13/2017 01:38 PM, Tony Krowiak wrote:
> >>>>>>> Ping  
> >>>>>>>> Tony Krowiak (19):
> >>>>>>>>      KVM: s390: SIE considerations for AP Queue virtualization
> >>>>>>>>      KVM: s390: refactor crypto initialization
> >>>>>>>>      s390/zcrypt: new AP matrix bus
> >>>>>>>>      s390/zcrypt: create an AP matrix device on the AP matrix bus
> >>>>>>>>      s390/zcrypt: base implementation of AP matrix device driver
> >>>>>>>>      s390/zcrypt: register matrix device with VFIO mediated device
> >>>>>>>>        framework
> >>>>>>>>      KVM: s390: introduce AP matrix configuration interface
> >>>>>>>>      s390/zcrypt: support for assigning adapters to matrix mdev
> >>>>>>>>      s390/zcrypt: validate adapter assignment
> >>>>>>>>      s390/zcrypt: sysfs interfaces supporting AP domain assignment
> >>>>>>>>      s390/zcrypt: validate domain assignment
> >>>>>>>>      s390/zcrypt: sysfs support for control domain assignment
> >>>>>>>>      s390/zcrypt: validate control domain assignment
> >>>>>>>>      KVM: s390: Connect the AP mediated matrix device to KVM
> >>>>>>>>      s390/zcrypt: introduce ioctl access to VFIO AP Matrix driver
> >>>>>>>>      KVM: s390: interface to configure KVM guest's AP matrix
> >>>>>>>>      KVM: s390: validate input to AP matrix config interface
> >>>>>>>>      KVM: s390: New ioctl to configure KVM guest's AP matrix
> >>>>>>>>      s390/facilities: enable AP facilities needed by guest  
> >>>>>> I think the approach is fine, and the code also looks fine for the 
> >>>>>> most
> >>>>>> part. Some comments:
> >>>>>>
> >>>>>> - various patches can be squashed together to give a better
> >>>>>>     understanding at a glance  
> >>>>> Which patches would you squash?  
> >>>>>> - this needs documentation (as I already said)  
> >>>>> My plan is to take the cover letter patch and incorporate that into
> >>>>> documentation,
> >>>>> then replace the cover letter patch with a more concise summary.  
> >>>>>> - some of the driver/device modelling feels a bit awkward 
> >>>>>> (commented in
> >>>>>>     patches) -- I'm not sure that my proposal is better, but I 
> >>>>>> think we
> >>>>>>     should make sure the interdependencies are modeled correctly  
> >>>>> I am responding to each patch review individually.  
> >>>>
> >>>> I think that instead of responding to each patch individually we should
> >>>> have a discussion on the design because I think a lot could change and
> >>>> discussing about each patch as they may be completely redesigned for 
> >>>> the
> >>>> next version may not be very useful.  
> > How do you suggest this discussion should be structured? Aren't the patches
> > themselves an ultimate expression of the design? A lot could change, but
> > can't those issues can be dealt with and discussed as we move forward?

FWIW, I think the basic design is already fine, and I don't think many
of the points raised would become useless. For one, if something looks
a bit strange in detail, it might be a pointer to something bigger that
should be reshuffled (like where to anchor information).

> >   
> >>>>
> >>>>
> >>>> So I totally agree with Conny on that we should stabilize the
> >>>> bus/device/driver modeling.
> >>>>
> >>>> I think it would be here a good place to start the discussion on things
> >>>> like we started to discuss, Harald and I, off-line:
> >>>> - why a matrix bus, in which case we can avoid it  
> >>>
> >>> I thought it had been agreed that we should be able to ditch it?  
> >>
> >> I have not see any comment on the matrix bus.  
> > As stated in a previous email responding to Connie, I decided to scrap the
> > AP matrix bus. There will only ever be one matrix device that serves two
> > purposes: To hold the APQNs of the queue devices bound to the VFIO AP 
> > matrix
> > device driver; to serve as a parent of the mediated devices created for
> > guests requiring access to the APQNs reserved for their use. So, instead
> > of an AP matrix bus creating the matrix device, it will be created by the
> > VFIO AP matrix driver in /sys/devices/ap_matrix/ during driver 
> > initialization.  
> 
> Sorry, I did not see the mail, this of course change a lot of things...

One thing that would be useful for the next iteration is some ascii-art
representation that shows how the different parts (matrix, ap driver,
mdev, ...) tie together. That also would be useful to have in the
documentation.

> 
> >>
> >>  
> >>>  
> >>>> - which kind of devices we need  
> >>>
> >>> What is still unclear? Which card generations to support?  
> >>
> >> No, I mean the relation bus/device/driver/mdev...  
> > I think that is pretty well spelled out in the cover letter
> > patch and in the descriptions of the patches themselves. What is it
> > you don't understand?  
> 
> If we have no matrix bus anymore I prefer to wait for the cover letter 
> of V2 to discuss this.
> 
> >>
> >>  
> >>>  
> >>>> - how to handle the repartition of queues on boot, reset and hotplug  
> > What do you mean by repartition of queues on boot?  
> >>>
> >>> That's something I'd like to see a writeup for.  
> >>
> >> yes, and it may have an influence on the bus/device/driver/mdev design  
> > I don't understand the need to avoid implementation details. If you recall,
> > the original design was modeled on AP queue devices. It was only after
> > implementing that design that the shortcomings were revealed which is
> > why we decided to base the model on the AP matrix. Keep in mind, this is
> > an RFC, not a final patch set. I would expect some change from the
> > implementation herein. In fact, I've already made many changes based on
> > Connie's and Christian's review comments, none of which resulted in an
> > overhaul of the design.  

I expect that any of the above can be accommodated by the design. A
short writeup of what we may want to do for that would certainly help
to validate that, though.

> >>
> >>  
> >>>  
> >>>> - interaction with the host drivers  
> >>>
> >>> The driver model should already handle that, no?  
> >>
> >> yes it should, but it is not clear for me.  
> > What is it that is not clear? This cover letter seeks to describe the
> > patch set, so why not annotate those areas that are not clear? I'm don't
> > understand what it is you are expecting. I thought the purpose of
> > submitting these patches here was to generate discussion.  

The simple fact of unbinding from the normal drivers and rebinding to
the specialized ones should already take care of interactions,
shouldn't it? Combined with the validation we should be all set.

> >>
> >>  
> >>>  
> >>>> - validation of the matrix for guests and host views  
> >>>
> >>> I saw validation code in the patches, although I have not reviewed it.  
> > Patches 9, 11, and 13 validate the adapters, domains and control domains
> > configured for the mdev matrix device and patch 17 verifies that the
> > KVM guest's matrix does not define any APQNs in use by other guests.  
> >>>
> >>>  
> >>>>
> >>>> or even features we need to add like
> >>>> - interruptions  
> >>>
> >>> My understanding is that interrupts are optional so they can be left
> >>> out in the first shot. With the gisa (that has not yet been posted), it
> >>> should not be too difficult, no?  
> >>
> >> you are right I forgot that it is optional  
> > If the facilities bit (STFLE.65) indicating interrupts are available is not
> > set for the guest, then the AP bus running on the guest will poll and
> > interrupts will not have to be handled. This patch set does not enable
> > interrupts, so it is not relevant at this time. We will not be able to
> > handle interrupts for the guest until the GISA for passthrough patches
> > are available. This will be addressed at that time.  

If you think it can be easily added later on, that would be fine for
me. (I cannot comment on gisa details until it has been posted,
obviously.)

> >>
> >>  
> >>>  
> >>>> - PAPQ/TAPQ-t and APQI interception  
> >>>
> >>> I can't say anything about that, as this is not documented :(  
> >>
> >> Right we can live without these too.  
> > I have implemented interception of the PQAP(TAPQ) instruction and will
> > include it in the next set of patches. It was not documented here
> > because this patch set was submitted as an RFC for the purpose of
> > determining if we are on the right track with regard to the overall
> > AP matrix design. >>  
> >>  
> >>>  
> >>>> - virtualization of the AP  
> >>>
> >>> Is this really needed? It would complicate everything a lot.  
> >>
> >> Concern has no sens without interception.  
> > Virtualization of AP is not on the table right now.  
> 
> If we implement interception, we must speak about this, even to say how 
> we do not implement virtualization.

A note that we do not plan to virtualize it right now would be
sensible, yes.

>From what I remember, this would mean opening a huge can of worms for
something that might be only of limited use. I'd prefer a simplistic
but usable approach first. If virtualization should really become a
requirement in the future, it might be better served by a different
mechanism anyway.

> 
> >>  
> >>>  
> >>>> - CPU model and KVM capabilities  
> >>>
> >>> That already has been discussed with the individual patches.  
> >>
> >> Well, if there are no interceptions the individual patches discussions 
> >> are enough.  
> > As I stated above, these patches were submitted as an RFC for the 
> > purpose of
> > getting a stamp of approval for the general design. Additional functions 
> > such as
> > hot plug and interception will be introduced in phases in the near 
> > future. As
> > I stated above, I already have the implementation of PQAP(TAPQ) and will 
> > include
> > it in the next submission. It does not change the general design one iota.  

The intersection was mainly with things like the cpu model. A different
part of the interface that does not really have an impact on the rest
of the design.

(That does not mean that we don't need to figure it out, of course.)

> >>
> >>  
> >>>  
> >>>>
> >>>> In my understanding these points must be cleared before we really start
> >>>> to discuss the details of the implementation.  
> >>>
> >>> The general design already looks fine to me. Do you really expect that
> >>> a major redesign is needed?  
> > I thought the point of submitting this RFC was to get a sanity check of the
> > design concepts. According to Christian, he discussed the design with
> > several maintainers at the KVM forum and they agreed this design was sane.  
> >>>  
> >>
> >> I am worry about the following:
> >> - Will the matrix bus be accepted  
> > I am eliminating the matrix bus - based on comments made by Connie for an
> > individual patch - so no need to worry;-)  
> >>
> >> - What happens on host reset and hot plug/unplug in host  
> > TBD, but I don't anticipate a major overhaul of the design to accommodate
> > these eventualities, particularly hot plug which I considered while
> > creating this design.  
> >>
> >> - What happens with the queues on guest start/halt/restart  
> > TBD  
> 
> AFAIU These two points are crucials for device driver design.

A writeup for these two points would be most welcome. I think the base
design is sane; it does not hurt to validate, though.

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Tony Krowiak]

On 11/17/2017 05:07 AM, Cornelia Huck wrote:
> On Fri, 17 Nov 2017 08:07:15 +0100
> Pierre Morel <pmorel@linux.vnet.ibm.com> wrote:
>
>> On 17/11/2017 00:35, Tony Krowiak wrote:
>>> On 11/16/2017 03:25 PM, Pierre Morel wrote:
>>>> On 16/11/2017 18:03, Cornelia Huck wrote:
>>>>> On Thu, 16 Nov 2017 17:06:58 +0100
>>>>> Pierre Morel <pmorel@linux.vnet.ibm.com> wrote:
>>>>>   
>>>>>> On 16/11/2017 16:23, Tony Krowiak wrote:
>>>>>>> On 11/14/2017 08:57 AM, Cornelia Huck wrote:
>>>>>>>> On Tue, 31 Oct 2017 15:39:09 -0400
>>>>>>>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>>>>>>>> On 10/13/2017 01:38 PM, Tony Krowiak wrote:
>>>>>>>>> Ping
>>>>>>>>>> Tony Krowiak (19):
>>>>>>>>>>       KVM: s390: SIE considerations for AP Queue virtualization
>>>>>>>>>>       KVM: s390: refactor crypto initialization
>>>>>>>>>>       s390/zcrypt: new AP matrix bus
>>>>>>>>>>       s390/zcrypt: create an AP matrix device on the AP matrix bus
>>>>>>>>>>       s390/zcrypt: base implementation of AP matrix device driver
>>>>>>>>>>       s390/zcrypt: register matrix device with VFIO mediated device
>>>>>>>>>>         framework
>>>>>>>>>>       KVM: s390: introduce AP matrix configuration interface
>>>>>>>>>>       s390/zcrypt: support for assigning adapters to matrix mdev
>>>>>>>>>>       s390/zcrypt: validate adapter assignment
>>>>>>>>>>       s390/zcrypt: sysfs interfaces supporting AP domain assignment
>>>>>>>>>>       s390/zcrypt: validate domain assignment
>>>>>>>>>>       s390/zcrypt: sysfs support for control domain assignment
>>>>>>>>>>       s390/zcrypt: validate control domain assignment
>>>>>>>>>>       KVM: s390: Connect the AP mediated matrix device to KVM
>>>>>>>>>>       s390/zcrypt: introduce ioctl access to VFIO AP Matrix driver
>>>>>>>>>>       KVM: s390: interface to configure KVM guest's AP matrix
>>>>>>>>>>       KVM: s390: validate input to AP matrix config interface
>>>>>>>>>>       KVM: s390: New ioctl to configure KVM guest's AP matrix
>>>>>>>>>>       s390/facilities: enable AP facilities needed by guest
>>>>>>>> I think the approach is fine, and the code also looks fine for the
>>>>>>>> most
>>>>>>>> part. Some comments:
>>>>>>>>
>>>>>>>> - various patches can be squashed together to give a better
>>>>>>>>      understanding at a glance
>>>>>>> Which patches would you squash?
>>>>>>>> - this needs documentation (as I already said)
>>>>>>> My plan is to take the cover letter patch and incorporate that into
>>>>>>> documentation,
>>>>>>> then replace the cover letter patch with a more concise summary.
>>>>>>>> - some of the driver/device modelling feels a bit awkward
>>>>>>>> (commented in
>>>>>>>>      patches) -- I'm not sure that my proposal is better, but I
>>>>>>>> think we
>>>>>>>>      should make sure the interdependencies are modeled correctly
>>>>>>> I am responding to each patch review individually.
>>>>>> I think that instead of responding to each patch individually we should
>>>>>> have a discussion on the design because I think a lot could change and
>>>>>> discussing about each patch as they may be completely redesigned for
>>>>>> the
>>>>>> next version may not be very useful.
>>> How do you suggest this discussion should be structured? Aren't the patches
>>> themselves an ultimate expression of the design? A lot could change, but
>>> can't those issues can be dealt with and discussed as we move forward?
> FWIW, I think the basic design is already fine, and I don't think many
> of the points raised would become useless. For one, if something looks
> a bit strange in detail, it might be a pointer to something bigger that
> should be reshuffled (like where to anchor information).
I agree.
>
>>>    
>>>>>>
>>>>>> So I totally agree with Conny on that we should stabilize the
>>>>>> bus/device/driver modeling.
>>>>>>
>>>>>> I think it would be here a good place to start the discussion on things
>>>>>> like we started to discuss, Harald and I, off-line:
>>>>>> - why a matrix bus, in which case we can avoid it
>>>>> I thought it had been agreed that we should be able to ditch it?
>>>> I have not see any comment on the matrix bus.
>>> As stated in a previous email responding to Connie, I decided to scrap the
>>> AP matrix bus. There will only ever be one matrix device that serves two
>>> purposes: To hold the APQNs of the queue devices bound to the VFIO AP
>>> matrix
>>> device driver; to serve as a parent of the mediated devices created for
>>> guests requiring access to the APQNs reserved for their use. So, instead
>>> of an AP matrix bus creating the matrix device, it will be created by the
>>> VFIO AP matrix driver in /sys/devices/ap_matrix/ during driver
>>> initialization.
>> Sorry, I did not see the mail, this of course change a lot of things...
> One thing that would be useful for the next iteration is some ascii-art
> representation that shows how the different parts (matrix, ap driver,
> mdev, ...) tie together. That also would be useful to have in the
> documentation.
I plan on including some drawings with the documentation and will include it
in the cover letter as well.
>
>>>>   
>>>>>   
>>>>>> - which kind of devices we need
>>>>> What is still unclear? Which card generations to support?
>>>> No, I mean the relation bus/device/driver/mdev...
>>> I think that is pretty well spelled out in the cover letter
>>> patch and in the descriptions of the patches themselves. What is it
>>> you don't understand?
>> If we have no matrix bus anymore I prefer to wait for the cover letter
>> of V2 to discuss this.
>>
>>>>   
>>>>>   
>>>>>> - how to handle the repartition of queues on boot, reset and hotplug
>>> What do you mean by repartition of queues on boot?
>>>>> That's something I'd like to see a writeup for.
>>>> yes, and it may have an influence on the bus/device/driver/mdev design
>>> I don't understand the need to avoid implementation details. If you recall,
>>> the original design was modeled on AP queue devices. It was only after
>>> implementing that design that the shortcomings were revealed which is
>>> why we decided to base the model on the AP matrix. Keep in mind, this is
>>> an RFC, not a final patch set. I would expect some change from the
>>> implementation herein. In fact, I've already made many changes based on
>>> Connie's and Christian's review comments, none of which resulted in an
>>> overhaul of the design.
> I expect that any of the above can be accommodated by the design. A
> short writeup of what we may want to do for that would certainly help
> to validate that, though.
I have spent some time thinking about hotplug implementation and I
believe it can be accommodated within this design. I haven't looked
at the implications for reset yet and I don't really know what is
meant by "repartition of queues". I will include a write-up in the
next submission.
>
>>>>   
>>>>>   
>>>>>> - interaction with the host drivers
>>>>> The driver model should already handle that, no?
>>>> yes it should, but it is not clear for me.
>>> What is it that is not clear? This cover letter seeks to describe the
>>> patch set, so why not annotate those areas that are not clear? I'm don't
>>> understand what it is you are expecting. I thought the purpose of
>>> submitting these patches here was to generate discussion.
> The simple fact of unbinding from the normal drivers and rebinding to
> the specialized ones should already take care of interactions,
> shouldn't it? Combined with the validation we should be all set.
Unbinding from the host drivers and binding to the VFIO AP matrix driver
reserves the APQNs (adapter ID and queue index tuple) for use by guests.
The APQNs reserved are stored with the AP matrix device. Mediated matrix
devices can be created for the matrix device, one for each guest that
will be using the reserved APQNs. When a mediated matrix device
is created, sysfs attribute files are created to assign adapters, domains
and control domains to the mediated matrix device to configure the AP
matrix for a guest. Validation is done at the time of assignment to insure
that APQNs that can be derived from the matrix have in fact been reserved.

The mediated device file descriptor provides a communication pathway 
between
QEMU and the VFIO AP matrix device driver. Communication between QEMU
and the device driver is established by opening the file descriptor.
The device driver provides mediated device ioctl to configure the
the APM, AQM and ADM of the KVM guest's CRYCB from the AP matrix
configured via the mediated matrix device's attribute files. Validation
is performed by the ioctl to ensure no other guest is using any APQN
that can be derived from matrix.

The interception of AP instructions will be handled via an ioctl call
on the same communication pathway.

I believe that about covers it.
>
>>>>   
>>>>>   
>>>>>> - validation of the matrix for guests and host views
>>>>> I saw validation code in the patches, although I have not reviewed it.
>>> Patches 9, 11, and 13 validate the adapters, domains and control domains
>>> configured for the mdev matrix device and patch 17 verifies that the
>>> KVM guest's matrix does not define any APQNs in use by other guests.
>>>>>   
>>>>>> or even features we need to add like
>>>>>> - interruptions
>>>>> My understanding is that interrupts are optional so they can be left
>>>>> out in the first shot. With the gisa (that has not yet been posted), it
>>>>> should not be too difficult, no?
>>>> you are right I forgot that it is optional
>>> If the facilities bit (STFLE.65) indicating interrupts are available is not
>>> set for the guest, then the AP bus running on the guest will poll and
>>> interrupts will not have to be handled. This patch set does not enable
>>> interrupts, so it is not relevant at this time. We will not be able to
>>> handle interrupts for the guest until the GISA for passthrough patches
>>> are available. This will be addressed at that time.
> If you think it can be easily added later on, that would be fine for
> me. (I cannot comment on gisa details until it has been posted,
> obviously.)
Enabling AP interrupts is accomplished using the PQAP(AQIC) instruction
which is a mandatory interception. The instruction will be forwarded to
the VFIO AP device driver via an ioctl call on the mediated matrix
device file descriptor. There will be some GISA set up needed and code
to feed the interrupt back to user space, but I believe that will be
provided by the forthcoming GISA passthrough patches. The bottom line is,
I don't anticipate any major design change to handle interrupt processing.
>
>>>>   
>>>>>   
>>>>>> - PAPQ/TAPQ-t and APQI interception
>>>>> I can't say anything about that, as this is not documented :(
>>>> Right we can live without these too.
>>> I have implemented interception of the PQAP(TAPQ) instruction and will
>>> include it in the next set of patches. It was not documented here
>>> because this patch set was submitted as an RFC for the purpose of
>>> determining if we are on the right track with regard to the overall
>>> AP matrix design. >>
>>>>   
>>>>>   
>>>>>> - virtualization of the AP
>>>>> Is this really needed? It would complicate everything a lot.
>>>> Concern has no sens without interception.
>>> Virtualization of AP is not on the table right now.
>> If we implement interception, we must speak about this, even to say how
>> we do not implement virtualization.
> A note that we do not plan to virtualize it right now would be
> sensible, yes.
Will do.
>
>  From what I remember, this would mean opening a huge can of worms for
> something that might be only of limited use. I'd prefer a simplistic
> but usable approach first. If virtualization should really become a
> requirement in the future, it might be better served by a different
> mechanism anyway.
I have done a little proof of concept code to get an idea if the AP matrix
design will be extensible to handle virtualization. I modeled the
proof of concept on the AP matrix model by creating a second mediated
matrix device type for virtualization. Of course, virtual and passthrough
matrix device types would have to be mutually exclusive; the admin would 
have
to choose one or the other. The sysfs model looked like this:

/sys/devices/ap_matrix
... [matrix]
...... [mdev_supported_types]
......... [vfio_ap_matrix-virtual]
............ create
............... [devices]
.................. [$uuid]
..................... assign_adapter
..................... assign_domain

Using the a assign_adapter file, one can assign a virtual adapter
ID to one or more real adapter IDs. For example, to assign virtual adapter
4 to real adapters 3, 22 and 254:

echo 4:3,22,254 > assign_adapter

Using the a assign_domain file, one can assign a virtual domain
ID to one or more real domain IDs. For example, to assign virtual domain
0 to real domains 8 and 71:

echo 0:8,0x47 > assign_domain

All AP instructions would be intercepted for a virtual matrix. The 
intercepted
instructions would be forwarded to the VFIO AP matrix device driver by QEMU
using an ioctl implemented by the VFIO AP matrix driver. If the mediated 
matrix
device is vfio_ap_matrix-passthrough type, things would work as they do now.
If the type is vfio_ap_matrix-virtual, the the driver would:

1. Calculate all of the real APQNs that can be used by:
    * Retrieving the adapter IDs mapped to the APID specified in the APQN
      contained in the AP instruction
    * Retrieving the domain IDs mapped to the APQI specified in the APQN
      contained in the AP instruction
    * Combining all of the permutations of APID/APQI
2. Determine which APQN would be best to use.
3. Execute the instruction
4. Return the result to the caller

In other words, I think the current design is extensible; but even if not,
I see no reason we can't design a completely different mechanism for
virtualization.




>
>>>>   
>>>>>   
>>>>>> - CPU model and KVM capabilities
>>>>> That already has been discussed with the individual patches.
>>>> Well, if there are no interceptions the individual patches discussions
>>>> are enough.
>>> As I stated above, these patches were submitted as an RFC for the
>>> purpose of
>>> getting a stamp of approval for the general design. Additional functions
>>> such as
>>> hot plug and interception will be introduced in phases in the near
>>> future. As
>>> I stated above, I already have the implementation of PQAP(TAPQ) and will
>>> include
>>> it in the next submission. It does not change the general design one iota.
> The intersection was mainly with things like the cpu model. A different
> part of the interface that does not really have an impact on the rest
> of the design.
>
> (That does not mean that we don't need to figure it out, of course.)
Of course.
>
>>>>   
>>>>>   
>>>>>> In my understanding these points must be cleared before we really start
>>>>>> to discuss the details of the implementation.
>>>>> The general design already looks fine to me. Do you really expect that
>>>>> a major redesign is needed?
>>> I thought the point of submitting this RFC was to get a sanity check of the
>>> design concepts. According to Christian, he discussed the design with
>>> several maintainers at the KVM forum and they agreed this design was sane.
>>>>>   
>>>> I am worry about the following:
>>>> - Will the matrix bus be accepted
>>> I am eliminating the matrix bus - based on comments made by Connie for an
>>> individual patch - so no need to worry;-)
>>>> - What happens on host reset and hot plug/unplug in host
>>> TBD, but I don't anticipate a major overhaul of the design to accommodate
>>> these eventualities, particularly hot plug which I considered while
>>> creating this design.
>>>> - What happens with the queues on guest start/halt/restart
>>> TBD
>> AFAIU These two points are crucials for device driver design.
> A writeup for these two points would be most welcome. I think the base
> design is sane; it does not hurt to validate, though.
Will do.
>

Re: [RFC 05/19] s390/zcrypt: base implementation of AP matrix device driver

[Tony Krowiak]

On 11/16/2017 11:47 AM, Cornelia Huck wrote:
> On Thu, 16 Nov 2017 09:25:27 -0500
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> On 11/16/2017 07:35 AM, Cornelia Huck wrote:
>>> On Thu, 16 Nov 2017 13:02:26 +0100
>>> Pierre Morel <pmorel@linux.vnet.ibm.com> wrote:
>>>   
>>>> On 14/11/2017 17:37, Tony Krowiak wrote:
>>>>> On 11/14/2017 07:40 AM, Cornelia Huck wrote:
>>>>>> On Fri, 13 Oct 2017 13:38:50 -0400
>>>>>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>>>>>> diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
>>>>>>> index 48af970..411c19a 100644
>>>>>>> --- a/arch/s390/Kconfig
>>>>>>> +++ b/arch/s390/Kconfig
>>>>>>> @@ -722,6 +722,19 @@ config VFIO_CCW
>>>>>>>           To compile this driver as a module, choose M here: the
>>>>>>>           module will be called vfio_ccw.
>>>>>>> +config VFIO_AP_MATRIX
>>>>>>> +    def_tristate m
>>>>>>> +    prompt "Support for Adjunct Processor Matrix device interface"
>>>>>>> +    depends on ZCRYPT
>>>>>>> +    select VFIO
>>>>>>> +    select MDEV
>>>>>>> +    select VFIO_MDEV
>>>>>>> +    select VFIO_MDEV_DEVICE
>>>>>>> +    select IOMMU_API
>>>>>> I think the more common pattern is to depend on the VFIO configs
>>>>>> instead of selecting them.
>>>>> It's ironic because I originally changed from using 'depends on' and
>>>>> changed it based on review comments made
>>>>> on our internal mailing list. I'll go with 'depends on'.
>>>> Is doing like the others a sufficient good reason?
>>>> What if the first who did this did not really think about it?
>>>>
>>>> When an administrator configure the kernel what does he think?
>>>>
>>>> - I want to have AP through AP_VFIO in my guests
>>>> 	and he get implicitly VFIO
>>>> or
>>>> - I want to have VFIO
>>>> 	and he has to explicitly add AP_VFIO too
>>>>
>>>> It seems to me that the first is much more user friendly.
>>>>
>>>> Please tell me if I missed something. dependencies? collateral damages?
>>>> my logic is wrong?
>>> Using select for anything that's not a simple infrastructure dependency
>>> may lead into trouble (we've had issues in the past where options tried
>>> to enable other options but missed dependencies).
>>>
>>> If a user wants to use vfio-ap, I think it is reasonable to expect them
>>> to figure out that they need both ap and vfio for that.
>>>
>>> [And config help has gotten much better than it was years ago; it's not
>>> that hard to figure out what is actually needed.]
>> Is it sufficient to specify 'depends on ZCRYPT && VFIO_MDEV_DEVICE'
>> since 'VFIO_MDEV_DEVICE depends on VFIO && VFIO_MDEV' and 'VFIO_MDEV
>> depends on VFIO' and 'VFIO depends on IOMMU_API'?
> Perhaps ZCRYPT && VFIO_MDEV && VFIO_MDEV_DEVICE, to make it a bit more
> obvious?
Sure, why not.
>
> [Also, is IOMMU_API only needed to satisfy dependencies?]
Yes, VFIO is dependent upon it.
>

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Cornelia Huck]

On Fri, 17 Nov 2017 15:28:16 -0500
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> On 11/17/2017 05:07 AM, Cornelia Huck wrote:
> > On Fri, 17 Nov 2017 08:07:15 +0100
> > Pierre Morel <pmorel@linux.vnet.ibm.com> wrote:
> >  
> >> On 17/11/2017 00:35, Tony Krowiak wrote:  
> >>> On 11/16/2017 03:25 PM, Pierre Morel wrote:  
> >>>> On 16/11/2017 18:03, Cornelia Huck wrote:  
> >>>>> On Thu, 16 Nov 2017 17:06:58 +0100
> >>>>> Pierre Morel <pmorel@linux.vnet.ibm.com> wrote:

> >>>>>> So I totally agree with Conny on that we should stabilize the
> >>>>>> bus/device/driver modeling.
> >>>>>>
> >>>>>> I think it would be here a good place to start the discussion on things
> >>>>>> like we started to discuss, Harald and I, off-line:
> >>>>>> - why a matrix bus, in which case we can avoid it  
> >>>>> I thought it had been agreed that we should be able to ditch it?  
> >>>> I have not see any comment on the matrix bus.  
> >>> As stated in a previous email responding to Connie, I decided to scrap the
> >>> AP matrix bus. There will only ever be one matrix device that serves two
> >>> purposes: To hold the APQNs of the queue devices bound to the VFIO AP
> >>> matrix
> >>> device driver; to serve as a parent of the mediated devices created for
> >>> guests requiring access to the APQNs reserved for their use. So, instead
> >>> of an AP matrix bus creating the matrix device, it will be created by the
> >>> VFIO AP matrix driver in /sys/devices/ap_matrix/ during driver
> >>> initialization.  
> >> Sorry, I did not see the mail, this of course change a lot of things...  
> > One thing that would be useful for the next iteration is some ascii-art
> > representation that shows how the different parts (matrix, ap driver,
> > mdev, ...) tie together. That also would be useful to have in the
> > documentation.  
> I plan on including some drawings with the documentation and will include it
> in the cover letter as well.

Sounds good.

> >>>>>> - how to handle the repartition of queues on boot, reset and hotplug  
> >>> What do you mean by repartition of queues on boot?  
> >>>>> That's something I'd like to see a writeup for.  
> >>>> yes, and it may have an influence on the bus/device/driver/mdev design  
> >>> I don't understand the need to avoid implementation details. If you recall,
> >>> the original design was modeled on AP queue devices. It was only after
> >>> implementing that design that the shortcomings were revealed which is
> >>> why we decided to base the model on the AP matrix. Keep in mind, this is
> >>> an RFC, not a final patch set. I would expect some change from the
> >>> implementation herein. In fact, I've already made many changes based on
> >>> Connie's and Christian's review comments, none of which resulted in an
> >>> overhaul of the design.  
> > I expect that any of the above can be accommodated by the design. A
> > short writeup of what we may want to do for that would certainly help
> > to validate that, though.  
> I have spent some time thinking about hotplug implementation and I
> believe it can be accommodated within this design. I haven't looked
> at the implications for reset yet and I don't really know what is
> meant by "repartition of queues". I will include a write-up in the
> next submission.

FWIW, "repartition of queues" is also unclear to me.

> >>>>>> - interruptions  
> >>>>> My understanding is that interrupts are optional so they can be left
> >>>>> out in the first shot. With the gisa (that has not yet been posted), it
> >>>>> should not be too difficult, no?  
> >>>> you are right I forgot that it is optional  
> >>> If the facilities bit (STFLE.65) indicating interrupts are available is not
> >>> set for the guest, then the AP bus running on the guest will poll and
> >>> interrupts will not have to be handled. This patch set does not enable
> >>> interrupts, so it is not relevant at this time. We will not be able to
> >>> handle interrupts for the guest until the GISA for passthrough patches
> >>> are available. This will be addressed at that time.  
> > If you think it can be easily added later on, that would be fine for
> > me. (I cannot comment on gisa details until it has been posted,
> > obviously.)  
> Enabling AP interrupts is accomplished using the PQAP(AQIC) instruction
> which is a mandatory interception. The instruction will be forwarded to
> the VFIO AP device driver via an ioctl call on the mediated matrix
> device file descriptor. There will be some GISA set up needed and code
> to feed the interrupt back to user space, but I believe that will be
> provided by the forthcoming GISA passthrough patches. The bottom line is,
> I don't anticipate any major design change to handle interrupt processing.

Cool, that's what I wanted to hear.

> >>>>>> - virtualization of the AP  
> >>>>> Is this really needed? It would complicate everything a lot.  
> >>>> Concern has no sens without interception.  
> >>> Virtualization of AP is not on the table right now.  
> >> If we implement interception, we must speak about this, even to say how
> >> we do not implement virtualization.  
> > A note that we do not plan to virtualize it right now would be
> > sensible, yes.  
> Will do.
> >
> >  From what I remember, this would mean opening a huge can of worms for
> > something that might be only of limited use. I'd prefer a simplistic
> > but usable approach first. If virtualization should really become a
> > requirement in the future, it might be better served by a different
> > mechanism anyway.  
> I have done a little proof of concept code to get an idea if the AP matrix
> design will be extensible to handle virtualization. I modeled the
> proof of concept on the AP matrix model by creating a second mediated
> matrix device type for virtualization. Of course, virtual and passthrough
> matrix device types would have to be mutually exclusive; the admin would 
> have
> to choose one or the other. The sysfs model looked like this:
> 
> /sys/devices/ap_matrix
> ... [matrix]
> ...... [mdev_supported_types]
> ......... [vfio_ap_matrix-virtual]
> ............ create
> ............... [devices]
> .................. [$uuid]
> ..................... assign_adapter
> ..................... assign_domain
> 
> Using the a assign_adapter file, one can assign a virtual adapter
> ID to one or more real adapter IDs. For example, to assign virtual adapter
> 4 to real adapters 3, 22 and 254:
> 
> echo 4:3,22,254 > assign_adapter
> 
> Using the a assign_domain file, one can assign a virtual domain
> ID to one or more real domain IDs. For example, to assign virtual domain
> 0 to real domains 8 and 71:
> 
> echo 0:8,0x47 > assign_domain
> 
> All AP instructions would be intercepted for a virtual matrix. The 
> intercepted
> instructions would be forwarded to the VFIO AP matrix device driver by QEMU
> using an ioctl implemented by the VFIO AP matrix driver. If the mediated 
> matrix
> device is vfio_ap_matrix-passthrough type, things would work as they do now.
> If the type is vfio_ap_matrix-virtual, the the driver would:
> 
> 1. Calculate all of the real APQNs that can be used by:
>     * Retrieving the adapter IDs mapped to the APID specified in the APQN
>       contained in the AP instruction
>     * Retrieving the domain IDs mapped to the APQI specified in the APQN
>       contained in the AP instruction
>     * Combining all of the permutations of APID/APQI
> 2. Determine which APQN would be best to use.
> 3. Execute the instruction
> 4. Return the result to the caller
> 
> In other words, I think the current design is extensible; but even if not,
> I see no reason we can't design a completely different mechanism for
> virtualization.

So it's basically a one-time effort at (re)configuration, and the
virtualization facility will basically take care of the rest?

Re: [RFC 05/19] s390/zcrypt: base implementation of AP matrix device driver

[Cornelia Huck]

On Fri, 17 Nov 2017 16:13:45 -0500
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> On 11/16/2017 11:47 AM, Cornelia Huck wrote:

> > [Also, is IOMMU_API only needed to satisfy dependencies?]  
> Yes, VFIO is dependent upon it.

Ah, my question was more whether we just need to make sure the api is
there, but do not need to implement anything.

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Tony Krowiak]

On 11/20/2017 12:13 PM, Cornelia Huck wrote:
> On Fri, 17 Nov 2017 15:28:16 -0500
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> On 11/17/2017 05:07 AM, Cornelia Huck wrote:
>>> On Fri, 17 Nov 2017 08:07:15 +0100
>>> Pierre Morel <pmorel@linux.vnet.ibm.com> wrote:
>>>   
>>>> On 17/11/2017 00:35, Tony Krowiak wrote:
>>>>> On 11/16/2017 03:25 PM, Pierre Morel wrote:
>>>>>> On 16/11/2017 18:03, Cornelia Huck wrote:
>>>>>>> On Thu, 16 Nov 2017 17:06:58 +0100
>>>>>>> Pierre Morel <pmorel@linux.vnet.ibm.com> wrote:
>>>>>>>> So I totally agree with Conny on that we should stabilize the
>>>>>>>> bus/device/driver modeling.
>>>>>>>>
>>>>>>>> I think it would be here a good place to start the discussion on things
>>>>>>>> like we started to discuss, Harald and I, off-line:
>>>>>>>> - why a matrix bus, in which case we can avoid it
>>>>>>> I thought it had been agreed that we should be able to ditch it?
>>>>>> I have not see any comment on the matrix bus.
>>>>> As stated in a previous email responding to Connie, I decided to scrap the
>>>>> AP matrix bus. There will only ever be one matrix device that serves two
>>>>> purposes: To hold the APQNs of the queue devices bound to the VFIO AP
>>>>> matrix
>>>>> device driver; to serve as a parent of the mediated devices created for
>>>>> guests requiring access to the APQNs reserved for their use. So, instead
>>>>> of an AP matrix bus creating the matrix device, it will be created by the
>>>>> VFIO AP matrix driver in /sys/devices/ap_matrix/ during driver
>>>>> initialization.
>>>> Sorry, I did not see the mail, this of course change a lot of things...
>>> One thing that would be useful for the next iteration is some ascii-art
>>> representation that shows how the different parts (matrix, ap driver,
>>> mdev, ...) tie together. That also would be useful to have in the
>>> documentation.
>> I plan on including some drawings with the documentation and will include it
>> in the cover letter as well.
> Sounds good.
>
>>>>>>>> - how to handle the repartition of queues on boot, reset and hotplug
>>>>> What do you mean by repartition of queues on boot?
>>>>>>> That's something I'd like to see a writeup for.
>>>>>> yes, and it may have an influence on the bus/device/driver/mdev design
>>>>> I don't understand the need to avoid implementation details. If you recall,
>>>>> the original design was modeled on AP queue devices. It was only after
>>>>> implementing that design that the shortcomings were revealed which is
>>>>> why we decided to base the model on the AP matrix. Keep in mind, this is
>>>>> an RFC, not a final patch set. I would expect some change from the
>>>>> implementation herein. In fact, I've already made many changes based on
>>>>> Connie's and Christian's review comments, none of which resulted in an
>>>>> overhaul of the design.
>>> I expect that any of the above can be accommodated by the design. A
>>> short writeup of what we may want to do for that would certainly help
>>> to validate that, though.
>> I have spent some time thinking about hotplug implementation and I
>> believe it can be accommodated within this design. I haven't looked
>> at the implications for reset yet and I don't really know what is
>> meant by "repartition of queues". I will include a write-up in the
>> next submission.
> FWIW, "repartition of queues" is also unclear to me.
>
>>>>>>>> - interruptions
>>>>>>> My understanding is that interrupts are optional so they can be left
>>>>>>> out in the first shot. With the gisa (that has not yet been posted), it
>>>>>>> should not be too difficult, no?
>>>>>> you are right I forgot that it is optional
>>>>> If the facilities bit (STFLE.65) indicating interrupts are available is not
>>>>> set for the guest, then the AP bus running on the guest will poll and
>>>>> interrupts will not have to be handled. This patch set does not enable
>>>>> interrupts, so it is not relevant at this time. We will not be able to
>>>>> handle interrupts for the guest until the GISA for passthrough patches
>>>>> are available. This will be addressed at that time.
>>> If you think it can be easily added later on, that would be fine for
>>> me. (I cannot comment on gisa details until it has been posted,
>>> obviously.)
>> Enabling AP interrupts is accomplished using the PQAP(AQIC) instruction
>> which is a mandatory interception. The instruction will be forwarded to
>> the VFIO AP device driver via an ioctl call on the mediated matrix
>> device file descriptor. There will be some GISA set up needed and code
>> to feed the interrupt back to user space, but I believe that will be
>> provided by the forthcoming GISA passthrough patches. The bottom line is,
>> I don't anticipate any major design change to handle interrupt processing.
> Cool, that's what I wanted to hear.
>
>>>>>>>> - virtualization of the AP
>>>>>>> Is this really needed? It would complicate everything a lot.
>>>>>> Concern has no sens without interception.
>>>>> Virtualization of AP is not on the table right now.
>>>> If we implement interception, we must speak about this, even to say how
>>>> we do not implement virtualization.
>>> A note that we do not plan to virtualize it right now would be
>>> sensible, yes.
>> Will do.
>>>   From what I remember, this would mean opening a huge can of worms for
>>> something that might be only of limited use. I'd prefer a simplistic
>>> but usable approach first. If virtualization should really become a
>>> requirement in the future, it might be better served by a different
>>> mechanism anyway.
>> I have done a little proof of concept code to get an idea if the AP matrix
>> design will be extensible to handle virtualization. I modeled the
>> proof of concept on the AP matrix model by creating a second mediated
>> matrix device type for virtualization. Of course, virtual and passthrough
>> matrix device types would have to be mutually exclusive; the admin would
>> have
>> to choose one or the other. The sysfs model looked like this:
>>
>> /sys/devices/ap_matrix
>> ... [matrix]
>> ...... [mdev_supported_types]
>> ......... [vfio_ap_matrix-virtual]
>> ............ create
>> ............... [devices]
>> .................. [$uuid]
>> ..................... assign_adapter
>> ..................... assign_domain
>>
>> Using the a assign_adapter file, one can assign a virtual adapter
>> ID to one or more real adapter IDs. For example, to assign virtual adapter
>> 4 to real adapters 3, 22 and 254:
>>
>> echo 4:3,22,254 > assign_adapter
>>
>> Using the a assign_domain file, one can assign a virtual domain
>> ID to one or more real domain IDs. For example, to assign virtual domain
>> 0 to real domains 8 and 71:
>>
>> echo 0:8,0x47 > assign_domain
>>
>> All AP instructions would be intercepted for a virtual matrix. The
>> intercepted
>> instructions would be forwarded to the VFIO AP matrix device driver by QEMU
>> using an ioctl implemented by the VFIO AP matrix driver. If the mediated
>> matrix
>> device is vfio_ap_matrix-passthrough type, things would work as they do now.
>> If the type is vfio_ap_matrix-virtual, the the driver would:
>>
>> 1. Calculate all of the real APQNs that can be used by:
>>      * Retrieving the adapter IDs mapped to the APID specified in the APQN
>>        contained in the AP instruction
>>      * Retrieving the domain IDs mapped to the APQI specified in the APQN
>>        contained in the AP instruction
>>      * Combining all of the permutations of APID/APQI
>> 2. Determine which APQN would be best to use.
>> 3. Execute the instruction
>> 4. Return the result to the caller
>>
>> In other words, I think the current design is extensible; but even if not,
>> I see no reason we can't design a completely different mechanism for
>> virtualization.
> So it's basically a one-time effort at (re)configuration, and the
> virtualization facility will basically take care of the rest?
I am not quite sure what you are asking, but I'll attempt to answer
what I think you're asking. A new type of mediated matrix device
will be introduced to configure a virtual matrix for a guest that
provides the interfaces to map a virtual adapter/domain ID to one
or more real adapter/domain IDs. If by virtualization facility,
you are talking about the VFIO AP matrix driver, then yes,
the driver will handle ioctl requests based on the type of the
mediated matrix device through which the request was submitted:

If the request is to configure the KVM guest's matrix:

* If the mediated matrix device type is passthrough:
   * Do validation of matrix
   * Configure the APM, AQM and ADM in the KVM guest's CRYCB
     according to the configuration specified via the mediated
     device's sysfs attribute files.
* If the mediated matrix device type is virtual:
   * Do validation of matrix
   * No need to configure CRYCB since all instructions will be
     intercepted

If the request is to execute an intercepted AP instruction:

* If the mediated matrix device type is passthrough:
   * Forward the instruction to the AP device and return the
     result to QEMU.

* If the mediated matrix device type is virtual:

   * Retrieve all of the real APQNs mapped to the virtual
     adapter and domain IDs configured in the mediated matrix
     device's sysfs attribute files
   * If there is more than one APQN mapping, then determine
     which would be best to use - algorithm TBD
   * Forward the instruction to the AP device and return the
     result.

Of course, these are just preliminary ideas at this time.
I've only prototyped the sysfs configuration interfaces. No
back end prototyping has been undertaken yet. If the ideas do
not pan out, however; I think virtualization can be introduced
as an independent design.

>

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Cornelia Huck]

On Tue, 21 Nov 2017 11:08:01 -0500
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:


> I am not quite sure what you are asking, but I'll attempt to answer
> what I think you're asking. A new type of mediated matrix device
> will be introduced to configure a virtual matrix for a guest that
> provides the interfaces to map a virtual adapter/domain ID to one
> or more real adapter/domain IDs. If by virtualization facility,
> you are talking about the VFIO AP matrix driver, then yes,
> the driver will handle ioctl requests based on the type of the
> mediated matrix device through which the request was submitted:
> 
> If the request is to configure the KVM guest's matrix:
> 
> * If the mediated matrix device type is passthrough:
>    * Do validation of matrix
>    * Configure the APM, AQM and ADM in the KVM guest's CRYCB
>      according to the configuration specified via the mediated
>      device's sysfs attribute files.
> * If the mediated matrix device type is virtual:
>    * Do validation of matrix
>    * No need to configure CRYCB since all instructions will be
>      intercepted

Ok, so we would have two distinct paths here...

> 
> If the request is to execute an intercepted AP instruction:
> 
> * If the mediated matrix device type is passthrough:
>    * Forward the instruction to the AP device and return the
>      result to QEMU.
> 
> * If the mediated matrix device type is virtual:
> 
>    * Retrieve all of the real APQNs mapped to the virtual
>      adapter and domain IDs configured in the mediated matrix
>      device's sysfs attribute files
>    * If there is more than one APQN mapping, then determine
>      which would be best to use - algorithm TBD
>    * Forward the instruction to the AP device and return the
>      result.

...and two distinct paths for most instructions here as well.

> 
> Of course, these are just preliminary ideas at this time.
> I've only prototyped the sysfs configuration interfaces. No
> back end prototyping has been undertaken yet. If the ideas do
> not pan out, however; I think virtualization can be introduced
> as an independent design.

Yes, let's cross that bridge when we get to it.

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Tony Krowiak]

On 11/22/2017 08:47 AM, Cornelia Huck wrote:
> On Tue, 21 Nov 2017 11:08:01 -0500
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>
>> I am not quite sure what you are asking, but I'll attempt to answer
>> what I think you're asking. A new type of mediated matrix device
>> will be introduced to configure a virtual matrix for a guest that
>> provides the interfaces to map a virtual adapter/domain ID to one
>> or more real adapter/domain IDs. If by virtualization facility,
>> you are talking about the VFIO AP matrix driver, then yes,
>> the driver will handle ioctl requests based on the type of the
>> mediated matrix device through which the request was submitted:
>>
>> If the request is to configure the KVM guest's matrix:
>>
>> * If the mediated matrix device type is passthrough:
>>     * Do validation of matrix
>>     * Configure the APM, AQM and ADM in the KVM guest's CRYCB
>>       according to the configuration specified via the mediated
>>       device's sysfs attribute files.
>> * If the mediated matrix device type is virtual:
>>     * Do validation of matrix
>>     * No need to configure CRYCB since all instructions will be
>>       intercepted
> Ok, so we would have two distinct paths here...
It depends upon what you mean by two distinct paths. Configuring the
mediated device would require a new mediated device type for virtualized
AP matrices. The ioctl for configuring the KVM guest's CRYCB would
require an additional check to determine whether the CRYCB need be
configured or not.
>
>> If the request is to execute an intercepted AP instruction:
>>
>> * If the mediated matrix device type is passthrough:
>>     * Forward the instruction to the AP device and return the
>>       result to QEMU.
>>
>> * If the mediated matrix device type is virtual:
>>
>>     * Retrieve all of the real APQNs mapped to the virtual
>>       adapter and domain IDs configured in the mediated matrix
>>       device's sysfs attribute files
>>     * If there is more than one APQN mapping, then determine
>>       which would be best to use - algorithm TBD
>>     * Forward the instruction to the AP device and return the
>>       result.
> ...and two distinct paths for most instructions here as well.
The driver would require additional ioctls to handle
interception of all AP instructions for virtual matrices and additional
code to remap virtual APQNs to real APQNs and determine which real APQN
to which intercepted instructions should be forwarded.
>
>> Of course, these are just preliminary ideas at this time.
>> I've only prototyped the sysfs configuration interfaces. No
>> back end prototyping has been undertaken yet. If the ideas do
>> not pan out, however; I think virtualization can be introduced
>> as an independent design.
> Yes, let's cross that bridge when we get to it.
That is the plan. Given Pierre's objections, I thought it might help
to touch on this.
>

Re: [RFC 19/19] s390/facilities: enable AP facilities needed by guest

[Tony Krowiak]

On 11/03/2017 04:47 AM, Christian Borntraeger wrote:
>
> On 11/02/2017 07:49 PM, Tony Krowiak wrote:
>> On 11/02/2017 11:53 AM, Christian Borntraeger wrote:
>>> On 11/02/2017 04:36 PM, Tony Krowiak wrote:
>>>> On 11/02/2017 08:08 AM, Christian Borntraeger wrote:
>>>>> On 10/16/2017 11:25 AM, Martin Schwidefsky wrote:
>>>>>> On Fri, 13 Oct 2017 13:39:04 -0400
>>>>>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>>>>>
>>>>>>> Sets up the following facilities bits to enable the specified AP
>>>>>>> facilities for the guest VM:
>>>>>>>       * STFLE.12: Enables the AP Query Configuration Information
>>>>>>>                   facility. The AP bus running in the guest uses
>>>>>>>                   the information returned from this instruction
>>>>>>>                   to configure AP adapters and domains for the
>>>>>>>                   guest machine.
>>>>>>>       * STFLE.15: Indicates the AP facilities test is available.
>>>>>>>                   The AP bus running in the guest uses the
>>>>>>>                   information.
>>>>>>>
>>>>>>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>>>>>>> ---
>>>>>>>    arch/s390/tools/gen_facilities.c |    2 ++
>>>>>>>    1 files changed, 2 insertions(+), 0 deletions(-)
>>>>>>>
>>>>>>> diff --git a/arch/s390/tools/gen_facilities.c b/arch/s390/tools/gen_facilities.c
>>>>>>> index 70dd8f1..eeaa7db 100644
>>>>>>> --- a/arch/s390/tools/gen_facilities.c
>>>>>>> +++ b/arch/s390/tools/gen_facilities.c
>>>>>>> @@ -74,8 +74,10 @@ struct facility_def {
>>>>>>>                8,  /* enhanced-DAT 1 */
>>>>>>>                9,  /* sense-running-status */
>>>>>>>                10, /* conditional sske */
>>>>>>> +            12, /* AP query configuration */
>>>>>>>                13, /* ipte-range */
>>>>>>>                14, /* nonquiescing key-setting */
>>>>>>> +            15, /* AP special-command facility */
>>>>>>>                73, /* transactional execution */
>>>>>>>                75, /* access-exception-fetch/store indication */
>>>>>>>                76, /* msa extension 3 */
>>>>>> With this all KVM guests will always have the AP instructions available, no?
>>>>>> In principles I like this approach, but it differs from the way z/VM does things,
>>>>>> there the guest will get an exception if it tries to execute an AP instruction
>>>>>> if there are no AP devices assigned to the guest. I wonder if there is a reason
>>>>>> why z/VM does it the way it does.
>>>>> A good question. For LPAR it seems that you have AP instructions even if you have
>>>>> no crypto cards.
>>>>>
>>>> I don't believe these facilities control whether or not AP instructions will be available
>>>>
>>>> to the guest.
>>> This is actually handled by your patch2 enabling the ECA bit.
>>> I think we must decide if we want to be able to disable these instructions
>>> via the cpu model. If yes we must then couple the facilities with the enablement.
>> The ECA.28 bit controls whether instructions are intercepted or interpreted - i.e., handled via hardware
>> virtualization. If set, as is done in patch2, then instructions will be interpreted. I don't see how
>> that affects enabling or disabling AP instructions, unless we don't set ECA.28, intercept every instruction
>> and program check. Am I missing something here?
> If we do not set ECA.28 these instructions intercept and we (the hypervisor) can then
> decide what to do. For example we can give an PIC01 operation exception (illegal
> instruction) - thats what we do today.
>
> Now: if we want to be able to migrate a guest from a new kernel back to an old kernel,
> there must be a way to disable the new behaviour so that the user can configure a guest
> that does NOT have these 3 instructions. That means, I want to bind the ap instruction
> to a cpu model feature, so that we only enable ECA.28 and the facility bits, if the
> feature is enabled in the CPU model. Otherwise we have no control on what happens
> when the guest issues these instructions.
>
> Imagine what happens if we not do this and you migrate from an identical hw with an
> identical libvirt/qemu but from a new kernel to an old kernel:
>
> The guest boots starts up on the new kernel
> guest kernel: drivers/s390/crypto/ap_bus.c  ap_module_init -> ap_instructions_available
> checks if the instructions work. They do and now the guest driver assumes that all
> instructions will continue to work.
>
> Now the guest is migrated back to an old kernel
> sooner or later the ap_scan_bus kthread will run to scan the bus (or some crypto operation
> is started) and the instruction will be rejected with a PIC01. kernel oops.
There are several scenarios that have to be accounted for, such as:
* Migrating from a linux host where both the KVM/kernel and QEMU support 
AP matrix
   devices to a guest host where neither the KVM/kernel nor QEMU support 
AP matrix
   devices;
* Migrating from a linux host where both the KVM/kernel and QEMU support 
AP matrix
   devices to a guest host where the KVM/kernel does not support AP 
matrix devices
   but QEMU does;
* Starting a guest on a linux host where QEMU supports AP matrix devices 
and
   the KVM/kernel does not;
* etc.

I agree with your suggestion that defining a new CPU model feature is 
probably
the best way to resolve this issue. The question is, should we define a 
single
feature indicating whether AP instructions are installed and set 
features bits
for the guest based on whether or not they are set in the linux host, or 
should
we define additional CPU model features for turning features bits on and 
off?
I guess it boils down to what behavior is expected for the AP bus 
running on
the linux guest. Here is a rundown of the facilities bits associated 
with AP
and how they affect the behavior of the AP bus:

* STFLE.12 indicates whether the AP query function is available. If this 
bit
   is not set, then the AP bus scan will only test domains 0-15. For 
example,
   if adapters 4, 5, and 6 and domains 12 and 71 (0x47) are installed, 
then AP
   queues 04.0047, 05.0047 and 06.0047 will not be made available.

* STFLE.15 indicates whether the AP facilities test function is 
available. If
   this bit is not set, then the CEX4, CEX5 and CEX6 device drivers 
discovered
   by the AP bus scan will not get bound to any AP device drivers. Since 
the
   AP matrix model supports only CEX4 and greater, no devices will be bound
   to any driver for a guest.

* STFLE.65 indicates whether AP interrupts are available. If this bit is not
   set, then the AP bus will use polling instead of using interrupt handlers
   to process AP events.

If the AP bus running on the guest is expected to mimic the behavior of an
AP bus running on the host, then I think we need a CPU model feature for 
each
facility. Otherwise, I think we can group them within a CPU model feature
that indicates AP matrix devices are supported. What say you?
>
>

Re: [RFC 19/19] s390/facilities: enable AP facilities needed by guest

[Harald Freudenberger]

On 12/02/2017 02:30 AM, Tony Krowiak wrote:
> On 11/03/2017 04:47 AM, Christian Borntraeger wrote:
>>
>> On 11/02/2017 07:49 PM, Tony Krowiak wrote:
>>> On 11/02/2017 11:53 AM, Christian Borntraeger wrote:
>>>> On 11/02/2017 04:36 PM, Tony Krowiak wrote:
>>>>> On 11/02/2017 08:08 AM, Christian Borntraeger wrote:
>>>>>> On 10/16/2017 11:25 AM, Martin Schwidefsky wrote:
>>>>>>> On Fri, 13 Oct 2017 13:39:04 -0400
>>>>>>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>>>>>>
>>>>>>>> Sets up the following facilities bits to enable the specified AP
>>>>>>>> facilities for the guest VM:
>>>>>>>>       * STFLE.12: Enables the AP Query Configuration Information
>>>>>>>>                   facility. The AP bus running in the guest uses
>>>>>>>>                   the information returned from this instruction
>>>>>>>>                   to configure AP adapters and domains for the
>>>>>>>>                   guest machine.
>>>>>>>>       * STFLE.15: Indicates the AP facilities test is available.
>>>>>>>>                   The AP bus running in the guest uses the
>>>>>>>>                   information.
>>>>>>>>
>>>>>>>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>>>>>>>> ---
>>>>>>>>    arch/s390/tools/gen_facilities.c |    2 ++
>>>>>>>>    1 files changed, 2 insertions(+), 0 deletions(-)
>>>>>>>>
>>>>>>>> diff --git a/arch/s390/tools/gen_facilities.c b/arch/s390/tools/gen_facilities.c
>>>>>>>> index 70dd8f1..eeaa7db 100644
>>>>>>>> --- a/arch/s390/tools/gen_facilities.c
>>>>>>>> +++ b/arch/s390/tools/gen_facilities.c
>>>>>>>> @@ -74,8 +74,10 @@ struct facility_def {
>>>>>>>>                8,  /* enhanced-DAT 1 */
>>>>>>>>                9,  /* sense-running-status */
>>>>>>>>                10, /* conditional sske */
>>>>>>>> +            12, /* AP query configuration */
>>>>>>>>                13, /* ipte-range */
>>>>>>>>                14, /* nonquiescing key-setting */
>>>>>>>> +            15, /* AP special-command facility */
>>>>>>>>                73, /* transactional execution */
>>>>>>>>                75, /* access-exception-fetch/store indication */
>>>>>>>>                76, /* msa extension 3 */
>>>>>>> With this all KVM guests will always have the AP instructions available, no?
>>>>>>> In principles I like this approach, but it differs from the way z/VM does things,
>>>>>>> there the guest will get an exception if it tries to execute an AP instruction
>>>>>>> if there are no AP devices assigned to the guest. I wonder if there is a reason
>>>>>>> why z/VM does it the way it does.
>>>>>> A good question. For LPAR it seems that you have AP instructions even if you have
>>>>>> no crypto cards.
>>>>>>
>>>>> I don't believe these facilities control whether or not AP instructions will be available
>>>>>
>>>>> to the guest.
>>>> This is actually handled by your patch2 enabling the ECA bit.
>>>> I think we must decide if we want to be able to disable these instructions
>>>> via the cpu model. If yes we must then couple the facilities with the enablement.
>>> The ECA.28 bit controls whether instructions are intercepted or interpreted - i.e., handled via hardware
>>> virtualization. If set, as is done in patch2, then instructions will be interpreted. I don't see how
>>> that affects enabling or disabling AP instructions, unless we don't set ECA.28, intercept every instruction
>>> and program check. Am I missing something here?
>> If we do not set ECA.28 these instructions intercept and we (the hypervisor) can then
>> decide what to do. For example we can give an PIC01 operation exception (illegal
>> instruction) - thats what we do today.
>>
>> Now: if we want to be able to migrate a guest from a new kernel back to an old kernel,
>> there must be a way to disable the new behaviour so that the user can configure a guest
>> that does NOT have these 3 instructions. That means, I want to bind the ap instruction
>> to a cpu model feature, so that we only enable ECA.28 and the facility bits, if the
>> feature is enabled in the CPU model. Otherwise we have no control on what happens
>> when the guest issues these instructions.
>>
>> Imagine what happens if we not do this and you migrate from an identical hw with an
>> identical libvirt/qemu but from a new kernel to an old kernel:
>>
>> The guest boots starts up on the new kernel
>> guest kernel: drivers/s390/crypto/ap_bus.c  ap_module_init -> ap_instructions_available
>> checks if the instructions work. They do and now the guest driver assumes that all
>> instructions will continue to work.
>>
>> Now the guest is migrated back to an old kernel
>> sooner or later the ap_scan_bus kthread will run to scan the bus (or some crypto operation
>> is started) and the instruction will be rejected with a PIC01. kernel oops.
> There are several scenarios that have to be accounted for, such as:
> * Migrating from a linux host where both the KVM/kernel and QEMU support AP matrix
>   devices to a guest host where neither the KVM/kernel nor QEMU support AP matrix
>   devices;
> * Migrating from a linux host where both the KVM/kernel and QEMU support AP matrix
>   devices to a guest host where the KVM/kernel does not support AP matrix devices
>   but QEMU does;
> * Starting a guest on a linux host where QEMU supports AP matrix devices and
>   the KVM/kernel does not;
> * etc.
>
> I agree with your suggestion that defining a new CPU model feature is probably
> the best way to resolve this issue. The question is, should we define a single
> feature indicating whether AP instructions are installed and set features bits
> for the guest based on whether or not they are set in the linux host, or should
> we define additional CPU model features for turning features bits on and off?
> I guess it boils down to what behavior is expected for the AP bus running on
> the linux guest. Here is a rundown of the facilities bits associated with AP
> and how they affect the behavior of the AP bus:
>
> * STFLE.12 indicates whether the AP query function is available. If this bit
>   is not set, then the AP bus scan will only test domains 0-15. For example,
>   if adapters 4, 5, and 6 and domains 12 and 71 (0x47) are installed, then AP
>   queues 04.0047, 05.0047 and 06.0047 will not be made available.
STFLE 12 is the indication for Query AP Configuration Information (QCI) available.
> * STFLE.15 indicates whether the AP facilities test function is available. If
>   this bit is not set, then the CEX4, CEX5 and CEX6 device drivers discovered
>   by the AP bus scan will not get bound to any AP device drivers. Since the
>   AP matrix model supports only CEX4 and greater, no devices will be bound
>   to any driver for a guest.
This T-Bit extension to the TAPQ subfunction is a must have. When kvm only
supports CEX4 and upper then this bit could also act as the indicator for
AP instructions available. Of course if you want to implement pure virtual
full simulated AP without any real AP hardware on the host this bit can't
be the indicator.
> * STFLE.65 indicates whether AP interrupts are available. If this bit is not
>   set, then the AP bus will use polling instead of using interrupt handlers
>   to process AP events.
>
> If the AP bus running on the guest is expected to mimic the behavior of an
> AP bus running on the host, then I think we need a CPU model feature for each
> facility. Otherwise, I think we can group them within a CPU model feature
> that indicates AP matrix devices are supported. What say you?
>>
>>
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-s390" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [RFC 19/19] s390/facilities: enable AP facilities needed by guest

[Cornelia Huck]

On Tue, 5 Dec 2017 08:52:57 +0100
Harald Freudenberger <freude@linux.vnet.ibm.com> wrote:

> On 12/02/2017 02:30 AM, Tony Krowiak wrote:

> > I agree with your suggestion that defining a new CPU model feature is probably
> > the best way to resolve this issue. The question is, should we define a single
> > feature indicating whether AP instructions are installed and set features bits
> > for the guest based on whether or not they are set in the linux host, or should
> > we define additional CPU model features for turning features bits on and off?
> > I guess it boils down to what behavior is expected for the AP bus running on
> > the linux guest. Here is a rundown of the facilities bits associated with AP
> > and how they affect the behavior of the AP bus:
> >
> > * STFLE.12 indicates whether the AP query function is available. If this bit
> >   is not set, then the AP bus scan will only test domains 0-15. For example,
> >   if adapters 4, 5, and 6 and domains 12 and 71 (0x47) are installed, then AP
> >   queues 04.0047, 05.0047 and 06.0047 will not be made available.  
> STFLE 12 is the indication for Query AP Configuration Information (QCI) available.
> > * STFLE.15 indicates whether the AP facilities test function is available. If
> >   this bit is not set, then the CEX4, CEX5 and CEX6 device drivers discovered
> >   by the AP bus scan will not get bound to any AP device drivers. Since the
> >   AP matrix model supports only CEX4 and greater, no devices will be bound
> >   to any driver for a guest.  
> This T-Bit extension to the TAPQ subfunction is a must have. When kvm only
> supports CEX4 and upper then this bit could also act as the indicator for
> AP instructions available. Of course if you want to implement pure virtual
> full simulated AP without any real AP hardware on the host this bit can't
> be the indicator.

It would probably make sense to group these two together. Or is there
any advantage in supporting only a part of it?

> > * STFLE.65 indicates whether AP interrupts are available. If this bit is not
> >   set, then the AP bus will use polling instead of using interrupt handlers
> >   to process AP events.

So, does this indicate "adapter interrupts for AP" only? If so, we
should keep this separate and only enable it when we have the gisa etc.
ready.

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Cornelia Huck]

On Mon, 27 Nov 2017 19:39:32 -0500
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> On 11/22/2017 08:47 AM, Cornelia Huck wrote:
> > On Tue, 21 Nov 2017 11:08:01 -0500
> > Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> >
> >  
> >> I am not quite sure what you are asking, but I'll attempt to answer
> >> what I think you're asking. A new type of mediated matrix device
> >> will be introduced to configure a virtual matrix for a guest that
> >> provides the interfaces to map a virtual adapter/domain ID to one
> >> or more real adapter/domain IDs. If by virtualization facility,
> >> you are talking about the VFIO AP matrix driver, then yes,
> >> the driver will handle ioctl requests based on the type of the
> >> mediated matrix device through which the request was submitted:
> >>
> >> If the request is to configure the KVM guest's matrix:
> >>
> >> * If the mediated matrix device type is passthrough:
> >>     * Do validation of matrix
> >>     * Configure the APM, AQM and ADM in the KVM guest's CRYCB
> >>       according to the configuration specified via the mediated
> >>       device's sysfs attribute files.
> >> * If the mediated matrix device type is virtual:
> >>     * Do validation of matrix
> >>     * No need to configure CRYCB since all instructions will be
> >>       intercepted  
> > Ok, so we would have two distinct paths here...  
> It depends upon what you mean by two distinct paths. Configuring the
> mediated device would require a new mediated device type for virtualized
> AP matrices. The ioctl for configuring the KVM guest's CRYCB would
> require an additional check to determine whether the CRYCB need be
> configured or not.

Yes, the new type makes this distinct enough so that we can think about
this later.

> >  
> >> If the request is to execute an intercepted AP instruction:
> >>
> >> * If the mediated matrix device type is passthrough:
> >>     * Forward the instruction to the AP device and return the
> >>       result to QEMU.
> >>
> >> * If the mediated matrix device type is virtual:
> >>
> >>     * Retrieve all of the real APQNs mapped to the virtual
> >>       adapter and domain IDs configured in the mediated matrix
> >>       device's sysfs attribute files
> >>     * If there is more than one APQN mapping, then determine
> >>       which would be best to use - algorithm TBD
> >>     * Forward the instruction to the AP device and return the
> >>       result.  
> > ...and two distinct paths for most instructions here as well.  
> The driver would require additional ioctls to handle
> interception of all AP instructions for virtual matrices and additional
> code to remap virtual APQNs to real APQNs and determine which real APQN
> to which intercepted instructions should be forwarded.
> >  
> >> Of course, these are just preliminary ideas at this time.
> >> I've only prototyped the sysfs configuration interfaces. No
> >> back end prototyping has been undertaken yet. If the ideas do
> >> not pan out, however; I think virtualization can be introduced
> >> as an independent design.  
> > Yes, let's cross that bridge when we get to it.  
> That is the plan. Given Pierre's objections, I thought it might help
> to touch on this.

OK, so I admit that I lost track a bit. Are there any remaining issues
beyond the feature handling? Would it make sense to post a v2?

Re: [RFC 19/19] s390/facilities: enable AP facilities needed by guest

[Tony Krowiak]

On 12/05/2017 02:52 AM, Harald Freudenberger wrote:
> On 12/02/2017 02:30 AM, Tony Krowiak wrote:
>> On 11/03/2017 04:47 AM, Christian Borntraeger wrote:
>>> On 11/02/2017 07:49 PM, Tony Krowiak wrote:
>>>> On 11/02/2017 11:53 AM, Christian Borntraeger wrote:
>>>>> On 11/02/2017 04:36 PM, Tony Krowiak wrote:
>>>>>> On 11/02/2017 08:08 AM, Christian Borntraeger wrote:
>>>>>>> On 10/16/2017 11:25 AM, Martin Schwidefsky wrote:
>>>>>>>> On Fri, 13 Oct 2017 13:39:04 -0400
>>>>>>>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>>>>>>>
>>>>>>>>> Sets up the following facilities bits to enable the specified AP
>>>>>>>>> facilities for the guest VM:
>>>>>>>>>        * STFLE.12: Enables the AP Query Configuration Information
>>>>>>>>>                    facility. The AP bus running in the guest uses
>>>>>>>>>                    the information returned from this instruction
>>>>>>>>>                    to configure AP adapters and domains for the
>>>>>>>>>                    guest machine.
>>>>>>>>>        * STFLE.15: Indicates the AP facilities test is available.
>>>>>>>>>                    The AP bus running in the guest uses the
>>>>>>>>>                    information.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>>>>>>>>> ---
>>>>>>>>>     arch/s390/tools/gen_facilities.c |    2 ++
>>>>>>>>>     1 files changed, 2 insertions(+), 0 deletions(-)
>>>>>>>>>
>>>>>>>>> diff --git a/arch/s390/tools/gen_facilities.c b/arch/s390/tools/gen_facilities.c
>>>>>>>>> index 70dd8f1..eeaa7db 100644
>>>>>>>>> --- a/arch/s390/tools/gen_facilities.c
>>>>>>>>> +++ b/arch/s390/tools/gen_facilities.c
>>>>>>>>> @@ -74,8 +74,10 @@ struct facility_def {
>>>>>>>>>                 8,  /* enhanced-DAT 1 */
>>>>>>>>>                 9,  /* sense-running-status */
>>>>>>>>>                 10, /* conditional sske */
>>>>>>>>> +            12, /* AP query configuration */
>>>>>>>>>                 13, /* ipte-range */
>>>>>>>>>                 14, /* nonquiescing key-setting */
>>>>>>>>> +            15, /* AP special-command facility */
>>>>>>>>>                 73, /* transactional execution */
>>>>>>>>>                 75, /* access-exception-fetch/store indication */
>>>>>>>>>                 76, /* msa extension 3 */
>>>>>>>> With this all KVM guests will always have the AP instructions available, no?
>>>>>>>> In principles I like this approach, but it differs from the way z/VM does things,
>>>>>>>> there the guest will get an exception if it tries to execute an AP instruction
>>>>>>>> if there are no AP devices assigned to the guest. I wonder if there is a reason
>>>>>>>> why z/VM does it the way it does.
>>>>>>> A good question. For LPAR it seems that you have AP instructions even if you have
>>>>>>> no crypto cards.
>>>>>>>
>>>>>> I don't believe these facilities control whether or not AP instructions will be available
>>>>>>
>>>>>> to the guest.
>>>>> This is actually handled by your patch2 enabling the ECA bit.
>>>>> I think we must decide if we want to be able to disable these instructions
>>>>> via the cpu model. If yes we must then couple the facilities with the enablement.
>>>> The ECA.28 bit controls whether instructions are intercepted or interpreted - i.e., handled via hardware
>>>> virtualization. If set, as is done in patch2, then instructions will be interpreted. I don't see how
>>>> that affects enabling or disabling AP instructions, unless we don't set ECA.28, intercept every instruction
>>>> and program check. Am I missing something here?
>>> If we do not set ECA.28 these instructions intercept and we (the hypervisor) can then
>>> decide what to do. For example we can give an PIC01 operation exception (illegal
>>> instruction) - thats what we do today.
>>>
>>> Now: if we want to be able to migrate a guest from a new kernel back to an old kernel,
>>> there must be a way to disable the new behaviour so that the user can configure a guest
>>> that does NOT have these 3 instructions. That means, I want to bind the ap instruction
>>> to a cpu model feature, so that we only enable ECA.28 and the facility bits, if the
>>> feature is enabled in the CPU model. Otherwise we have no control on what happens
>>> when the guest issues these instructions.
>>>
>>> Imagine what happens if we not do this and you migrate from an identical hw with an
>>> identical libvirt/qemu but from a new kernel to an old kernel:
>>>
>>> The guest boots starts up on the new kernel
>>> guest kernel: drivers/s390/crypto/ap_bus.c  ap_module_init -> ap_instructions_available
>>> checks if the instructions work. They do and now the guest driver assumes that all
>>> instructions will continue to work.
>>>
>>> Now the guest is migrated back to an old kernel
>>> sooner or later the ap_scan_bus kthread will run to scan the bus (or some crypto operation
>>> is started) and the instruction will be rejected with a PIC01. kernel oops.
>> There are several scenarios that have to be accounted for, such as:
>> * Migrating from a linux host where both the KVM/kernel and QEMU support AP matrix
>>    devices to a guest host where neither the KVM/kernel nor QEMU support AP matrix
>>    devices;
>> * Migrating from a linux host where both the KVM/kernel and QEMU support AP matrix
>>    devices to a guest host where the KVM/kernel does not support AP matrix devices
>>    but QEMU does;
>> * Starting a guest on a linux host where QEMU supports AP matrix devices and
>>    the KVM/kernel does not;
>> * etc.
>>
>> I agree with your suggestion that defining a new CPU model feature is probably
>> the best way to resolve this issue. The question is, should we define a single
>> feature indicating whether AP instructions are installed and set features bits
>> for the guest based on whether or not they are set in the linux host, or should
>> we define additional CPU model features for turning features bits on and off?
>> I guess it boils down to what behavior is expected for the AP bus running on
>> the linux guest. Here is a rundown of the facilities bits associated with AP
>> and how they affect the behavior of the AP bus:
>>
>> * STFLE.12 indicates whether the AP query function is available. If this bit
>>    is not set, then the AP bus scan will only test domains 0-15. For example,
>>    if adapters 4, 5, and 6 and domains 12 and 71 (0x47) are installed, then AP
>>    queues 04.0047, 05.0047 and 06.0047 will not be made available.
> STFLE 12 is the indication for Query AP Configuration Information (QCI) available.
>> * STFLE.15 indicates whether the AP facilities test function is available. If
>>    this bit is not set, then the CEX4, CEX5 and CEX6 device drivers discovered
>>    by the AP bus scan will not get bound to any AP device drivers. Since the
>>    AP matrix model supports only CEX4 and greater, no devices will be bound
>>    to any driver for a guest.
> This T-Bit extension to the TAPQ subfunction is a must have. When kvm only
> supports CEX4 and upper then this bit could also act as the indicator for
> AP instructions available. Of course if you want to implement pure virtual
> full simulated AP without any real AP hardware on the host this bit can't
> be the indicator.
The context for this discussion is whether we should model the STFLE bits as
individual CPU model features or include them with the CPU model feature
indicating AP instructions are installed. In either case, the setting of
these bits for the guest is dependent upon whether they are set for the
host or not. In other words, we can not set a bit for the guest if it is
not set for the host for dedicated crypto.
>> * STFLE.65 indicates whether AP interrupts are available. If this bit is not
>>    set, then the AP bus will use polling instead of using interrupt handlers
>>    to process AP events.
>>
>> If the AP bus running on the guest is expected to mimic the behavior of an
>> AP bus running on the host, then I think we need a CPU model feature for each
>> facility. Otherwise, I think we can group them within a CPU model feature
>> that indicates AP matrix devices are supported. What say you?
>>>
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe linux-s390" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>

Re: [RFC 19/19] s390/facilities: enable AP facilities needed by guest

[Pierre Morel]

On 05/12/2017 15:04, Cornelia Huck wrote:
> On Tue, 5 Dec 2017 08:52:57 +0100
> Harald Freudenberger <freude@linux.vnet.ibm.com> wrote:
> 
>> On 12/02/2017 02:30 AM, Tony Krowiak wrote:
> 
>>> I agree with your suggestion that defining a new CPU model feature is probably
>>> the best way to resolve this issue. The question is, should we define a single
>>> feature indicating whether AP instructions are installed and set features bits
>>> for the guest based on whether or not they are set in the linux host, or should
>>> we define additional CPU model features for turning features bits on and off?
>>> I guess it boils down to what behavior is expected for the AP bus running on
>>> the linux guest. Here is a rundown of the facilities bits associated with AP
>>> and how they affect the behavior of the AP bus:
>>>
>>> * STFLE.12 indicates whether the AP query function is available. If this bit
>>>    is not set, then the AP bus scan will only test domains 0-15. For example,
>>>    if adapters 4, 5, and 6 and domains 12 and 71 (0x47) are installed, then AP
>>>    queues 04.0047, 05.0047 and 06.0047 will not be made available.
>> STFLE 12 is the indication for Query AP Configuration Information (QCI) available.
>>> * STFLE.15 indicates whether the AP facilities test function is available. If
>>>    this bit is not set, then the CEX4, CEX5 and CEX6 device drivers discovered
>>>    by the AP bus scan will not get bound to any AP device drivers. Since the
>>>    AP matrix model supports only CEX4 and greater, no devices will be bound
>>>    to any driver for a guest.
>> This T-Bit extension to the TAPQ subfunction is a must have. When kvm only
>> supports CEX4 and upper then this bit could also act as the indicator for
>> AP instructions available. Of course if you want to implement pure virtual
>> full simulated AP without any real AP hardware on the host this bit can't
>> be the indicator.
> 
> It would probably make sense to group these two together. Or is there
> any advantage in supporting only a part of it?
> 
>>> * STFLE.65 indicates whether AP interrupts are available. If this bit is not
>>>    set, then the AP bus will use polling instead of using interrupt handlers
>>>    to process AP events.
> 
> So, does this indicate "adapter interrupts for AP" only? If so, we
> should keep this separate and only enable it when we have the gisa etc.
> ready.
> 

Yes, STFLE 65, it is for AP only.

QCI, STFLE 12, is no present on older systems, in this case AP uses TAPQ 
to retrieve information for each AP

So for my point of view, it make sense to separate the three facilities 
to enable migration on older systems.

Pierre


-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [RFC 19/19] s390/facilities: enable AP facilities needed by guest

[Cornelia Huck]

On Tue, 5 Dec 2017 15:23:50 +0100
Pierre Morel <pmorel@linux.vnet.ibm.com> wrote:

> On 05/12/2017 15:04, Cornelia Huck wrote:
> > On Tue, 5 Dec 2017 08:52:57 +0100
> > Harald Freudenberger <freude@linux.vnet.ibm.com> wrote:
> >   
> >> On 12/02/2017 02:30 AM, Tony Krowiak wrote:  
> >   
> >>> I agree with your suggestion that defining a new CPU model feature is probably
> >>> the best way to resolve this issue. The question is, should we define a single
> >>> feature indicating whether AP instructions are installed and set features bits
> >>> for the guest based on whether or not they are set in the linux host, or should
> >>> we define additional CPU model features for turning features bits on and off?
> >>> I guess it boils down to what behavior is expected for the AP bus running on
> >>> the linux guest. Here is a rundown of the facilities bits associated with AP
> >>> and how they affect the behavior of the AP bus:
> >>>
> >>> * STFLE.12 indicates whether the AP query function is available. If this bit
> >>>    is not set, then the AP bus scan will only test domains 0-15. For example,
> >>>    if adapters 4, 5, and 6 and domains 12 and 71 (0x47) are installed, then AP
> >>>    queues 04.0047, 05.0047 and 06.0047 will not be made available.  
> >> STFLE 12 is the indication for Query AP Configuration Information (QCI) available.  
> >>> * STFLE.15 indicates whether the AP facilities test function is available. If
> >>>    this bit is not set, then the CEX4, CEX5 and CEX6 device drivers discovered
> >>>    by the AP bus scan will not get bound to any AP device drivers. Since the
> >>>    AP matrix model supports only CEX4 and greater, no devices will be bound
> >>>    to any driver for a guest.  
> >> This T-Bit extension to the TAPQ subfunction is a must have. When kvm only
> >> supports CEX4 and upper then this bit could also act as the indicator for
> >> AP instructions available. Of course if you want to implement pure virtual
> >> full simulated AP without any real AP hardware on the host this bit can't
> >> be the indicator.  
> > 
> > It would probably make sense to group these two together. Or is there
> > any advantage in supporting only a part of it?
> >   
> >>> * STFLE.65 indicates whether AP interrupts are available. If this bit is not
> >>>    set, then the AP bus will use polling instead of using interrupt handlers
> >>>    to process AP events.  
> > 
> > So, does this indicate "adapter interrupts for AP" only? If so, we
> > should keep this separate and only enable it when we have the gisa etc.
> > ready.
> >   
> 
> Yes, STFLE 65, it is for AP only.
> 
> QCI, STFLE 12, is no present on older systems, in this case AP uses TAPQ 
> to retrieve information for each AP

Dumb question: How old? Machines that are still supported?

> 
> So for my point of view, it make sense to separate the three facilities 
> to enable migration on older systems.

OK, if STFLE 12 might not be present (pending my question above), but
STFLE 15 is indeed a must-have, we should split this up.

Re: [RFC 19/19] s390/facilities: enable AP facilities needed by guest

[Pierre Morel]

On 05/12/2017 15:30, Cornelia Huck wrote:
> On Tue, 5 Dec 2017 15:23:50 +0100
> Pierre Morel <pmorel@linux.vnet.ibm.com> wrote:
> 
>> On 05/12/2017 15:04, Cornelia Huck wrote:
>>> On Tue, 5 Dec 2017 08:52:57 +0100
>>> Harald Freudenberger <freude@linux.vnet.ibm.com> wrote:
>>>    
>>>> On 12/02/2017 02:30 AM, Tony Krowiak wrote:
>>>    
>>>>> I agree with your suggestion that defining a new CPU model feature is probably
>>>>> the best way to resolve this issue. The question is, should we define a single
>>>>> feature indicating whether AP instructions are installed and set features bits
>>>>> for the guest based on whether or not they are set in the linux host, or should
>>>>> we define additional CPU model features for turning features bits on and off?
>>>>> I guess it boils down to what behavior is expected for the AP bus running on
>>>>> the linux guest. Here is a rundown of the facilities bits associated with AP
>>>>> and how they affect the behavior of the AP bus:
>>>>>
>>>>> * STFLE.12 indicates whether the AP query function is available. If this bit
>>>>>     is not set, then the AP bus scan will only test domains 0-15. For example,
>>>>>     if adapters 4, 5, and 6 and domains 12 and 71 (0x47) are installed, then AP
>>>>>     queues 04.0047, 05.0047 and 06.0047 will not be made available.
>>>> STFLE 12 is the indication for Query AP Configuration Information (QCI) available.
>>>>> * STFLE.15 indicates whether the AP facilities test function is available. If
>>>>>     this bit is not set, then the CEX4, CEX5 and CEX6 device drivers discovered
>>>>>     by the AP bus scan will not get bound to any AP device drivers. Since the
>>>>>     AP matrix model supports only CEX4 and greater, no devices will be bound
>>>>>     to any driver for a guest.
>>>> This T-Bit extension to the TAPQ subfunction is a must have. When kvm only
>>>> supports CEX4 and upper then this bit could also act as the indicator for
>>>> AP instructions available. Of course if you want to implement pure virtual
>>>> full simulated AP without any real AP hardware on the host this bit can't
>>>> be the indicator.
>>>
>>> It would probably make sense to group these two together. Or is there
>>> any advantage in supporting only a part of it?
>>>    
>>>>> * STFLE.65 indicates whether AP interrupts are available. If this bit is not
>>>>>     set, then the AP bus will use polling instead of using interrupt handlers
>>>>>     to process AP events.
>>>
>>> So, does this indicate "adapter interrupts for AP" only? If so, we
>>> should keep this separate and only enable it when we have the gisa etc.
>>> ready.
>>>    
>>
>> Yes, STFLE 65, it is for AP only.
>>
>> QCI, STFLE 12, is no present on older systems, in this case AP uses TAPQ
>> to retrieve information for each AP
> 
> Dumb question: How old? Machines that are still supported?

No idea which machine are supported or not, will ask.

What I can say is that I have here a Lpar which does not support QCI.
It seems to be a zEC12.2.
z13 support it.

> 
>>
>> So for my point of view, it make sense to separate the three facilities
>> to enable migration on older systems.
> 
> OK, if STFLE 12 might not be present (pending my question above), but
> STFLE 15 is indeed a must-have, we should split this up.
> 



-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [RFC 19/19] s390/facilities: enable AP facilities needed by guest

[Tony Krowiak]

On 12/05/2017 09:04 AM, Cornelia Huck wrote:
> On Tue, 5 Dec 2017 08:52:57 +0100
> Harald Freudenberger <freude@linux.vnet.ibm.com> wrote:
>
>> On 12/02/2017 02:30 AM, Tony Krowiak wrote:
>>> I agree with your suggestion that defining a new CPU model feature is probably
>>> the best way to resolve this issue. The question is, should we define a single
>>> feature indicating whether AP instructions are installed and set features bits
>>> for the guest based on whether or not they are set in the linux host, or should
>>> we define additional CPU model features for turning features bits on and off?
>>> I guess it boils down to what behavior is expected for the AP bus running on
>>> the linux guest. Here is a rundown of the facilities bits associated with AP
>>> and how they affect the behavior of the AP bus:
>>>
>>> * STFLE.12 indicates whether the AP query function is available. If this bit
>>>    is not set, then the AP bus scan will only test domains 0-15. For example,
>>>    if adapters 4, 5, and 6 and domains 12 and 71 (0x47) are installed, then AP
>>>    queues 04.0047, 05.0047 and 06.0047 will not be made available.
>> STFLE 12 is the indication for Query AP Configuration Information (QCI) available.
>>> * STFLE.15 indicates whether the AP facilities test function is available. If
>>>    this bit is not set, then the CEX4, CEX5 and CEX6 device drivers discovered
>>>    by the AP bus scan will not get bound to any AP device drivers. Since the
>>>    AP matrix model supports only CEX4 and greater, no devices will be bound
>>>    to any driver for a guest.
>> This T-Bit extension to the TAPQ subfunction is a must have. When kvm only
>> supports CEX4 and upper then this bit could also act as the indicator for
>> AP instructions available. Of course if you want to implement pure virtual
>> full simulated AP without any real AP hardware on the host this bit can't
>> be the indicator.
> It would probably make sense to group these two together. Or is there
> any advantage in supporting only a part of it?
After thinking about this a little more, I've come to the conclusion that
all of this might be moot for the following reasons:

* If STFLE.12 is not set for the linux host, then AP bus scan running on
   the host will not detect any domains with a domain number higher than 15,
   so no AP queues with a queue index higher than 15 will be available to
   bind to the vfio_ap_matrix driver. Consequently, no domain higher than
   15 can be assigned to any guest. In this case, the AP bus scan 
running on
   the guest will never detect a domain higher than 15, regardless of the
   setting of STFLE.12 for the guest.

* If STFLE.15 is not set for the linux host, then then there will be no
   CEX4, CEX5 or CEX6 queues available to bind to the vfio_ap_matrix
   driver, so no AP adapters or domains can be assigned to any KVM guest.

The bottom line is the STFLE bit settings for the linux host will control
what APs are available to the KVM guest. Since STFLE.15 controls whether
any CEX4,5 or 6 devices are even available, I think this bit can be
combined into the feature that indicates whether AP is available. As long
as AP instructions are available on the linux host, I'm not sure whether
STFLE.12 needs a feature at all.
>
>>> * STFLE.65 indicates whether AP interrupts are available. If this bit is not
>>>    set, then the AP bus will use polling instead of using interrupt handlers
>>>    to process AP events.
> So, does this indicate "adapter interrupts for AP" only? If so, we
> should keep this separate and only enable it when we have the gisa etc.
> ready.
Yes, this indicates AP interrupts only. The plan is to enable this when
GISA is available and we can implement interrupt processing.
>

Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Tony Krowiak]

On 12/05/2017 09:06 AM, Cornelia Huck wrote:
> On Mon, 27 Nov 2017 19:39:32 -0500
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>
>> On 11/22/2017 08:47 AM, Cornelia Huck wrote:
>>> On Tue, 21 Nov 2017 11:08:01 -0500
>>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
>>>
>>>   
>>>> I am not quite sure what you are asking, but I'll attempt to answer
>>>> what I think you're asking. A new type of mediated matrix device
>>>> will be introduced to configure a virtual matrix for a guest that
>>>> provides the interfaces to map a virtual adapter/domain ID to one
>>>> or more real adapter/domain IDs. If by virtualization facility,
>>>> you are talking about the VFIO AP matrix driver, then yes,
>>>> the driver will handle ioctl requests based on the type of the
>>>> mediated matrix device through which the request was submitted:
>>>>
>>>> If the request is to configure the KVM guest's matrix:
>>>>
>>>> * If the mediated matrix device type is passthrough:
>>>>      * Do validation of matrix
>>>>      * Configure the APM, AQM and ADM in the KVM guest's CRYCB
>>>>        according to the configuration specified via the mediated
>>>>        device's sysfs attribute files.
>>>> * If the mediated matrix device type is virtual:
>>>>      * Do validation of matrix
>>>>      * No need to configure CRYCB since all instructions will be
>>>>        intercepted
>>> Ok, so we would have two distinct paths here...
>> It depends upon what you mean by two distinct paths. Configuring the
>> mediated device would require a new mediated device type for virtualized
>> AP matrices. The ioctl for configuring the KVM guest's CRYCB would
>> require an additional check to determine whether the CRYCB need be
>> configured or not.
> Yes, the new type makes this distinct enough so that we can think about
> this later.
>
>>>   
>>>> If the request is to execute an intercepted AP instruction:
>>>>
>>>> * If the mediated matrix device type is passthrough:
>>>>      * Forward the instruction to the AP device and return the
>>>>        result to QEMU.
>>>>
>>>> * If the mediated matrix device type is virtual:
>>>>
>>>>      * Retrieve all of the real APQNs mapped to the virtual
>>>>        adapter and domain IDs configured in the mediated matrix
>>>>        device's sysfs attribute files
>>>>      * If there is more than one APQN mapping, then determine
>>>>        which would be best to use - algorithm TBD
>>>>      * Forward the instruction to the AP device and return the
>>>>        result.
>>> ...and two distinct paths for most instructions here as well.
>> The driver would require additional ioctls to handle
>> interception of all AP instructions for virtual matrices and additional
>> code to remap virtual APQNs to real APQNs and determine which real APQN
>> to which intercepted instructions should be forwarded.
>>>   
>>>> Of course, these are just preliminary ideas at this time.
>>>> I've only prototyped the sysfs configuration interfaces. No
>>>> back end prototyping has been undertaken yet. If the ideas do
>>>> not pan out, however; I think virtualization can be introduced
>>>> as an independent design.
>>> Yes, let's cross that bridge when we get to it.
>> That is the plan. Given Pierre's objections, I thought it might help
>> to touch on this.
> OK, so I admit that I lost track a bit. Are there any remaining issues
> beyond the feature handling? Would it make sense to post a v2?
I was planning on posting a V2 once the features issue is settled.
>

Re: [RFC 19/19] s390/facilities: enable AP facilities needed by guest

[Tony Krowiak]

On 12/05/2017 09:23 AM, Pierre Morel wrote:
> On 05/12/2017 15:04, Cornelia Huck wrote:
>> On Tue, 5 Dec 2017 08:52:57 +0100
>> Harald Freudenberger <freude@linux.vnet.ibm.com> wrote:
>>
>>> On 12/02/2017 02:30 AM, Tony Krowiak wrote:
>>
>>>> I agree with your suggestion that defining a new CPU model feature 
>>>> is probably
>>>> the best way to resolve this issue. The question is, should we 
>>>> define a single
>>>> feature indicating whether AP instructions are installed and set 
>>>> features bits
>>>> for the guest based on whether or not they are set in the linux 
>>>> host, or should
>>>> we define additional CPU model features for turning features bits 
>>>> on and off?
>>>> I guess it boils down to what behavior is expected for the AP bus 
>>>> running on
>>>> the linux guest. Here is a rundown of the facilities bits 
>>>> associated with AP
>>>> and how they affect the behavior of the AP bus:
>>>>
>>>> * STFLE.12 indicates whether the AP query function is available. If 
>>>> this bit
>>>>    is not set, then the AP bus scan will only test domains 0-15. 
>>>> For example,
>>>>    if adapters 4, 5, and 6 and domains 12 and 71 (0x47) are 
>>>> installed, then AP
>>>>    queues 04.0047, 05.0047 and 06.0047 will not be made available.
>>> STFLE 12 is the indication for Query AP Configuration Information 
>>> (QCI) available.
>>>> * STFLE.15 indicates whether the AP facilities test function is 
>>>> available. If
>>>>    this bit is not set, then the CEX4, CEX5 and CEX6 device drivers 
>>>> discovered
>>>>    by the AP bus scan will not get bound to any AP device drivers. 
>>>> Since theI think
>>>> STFLE.12
>>>>    AP matrix model supports only CEX4 and greater, no devices will 
>>>> be bound
>>>>    to any driver for a guest.
>>> This T-Bit extension to the TAPQ subfunction is a must have. When 
>>> kvm only
>>> supports CEX4 and upper then this bit could also act as the 
>>> indicator for
>>> AP instructions available. Of course if you want to implement pure 
>>> virtual
>>> full simulated AP without any real AP hardware on the host this bit 
>>> can't
>>> be the indicator.
>>
>> It would probably make sense to group these two together. Or is there
>> any advantage in supporting only a part of it?
>>
>>>> * STFLE.65 indicates whether AP interrupts are available. If this 
>>>> bit is not
>>>>    set, then the AP bus will use polling instead of using interrupt 
>>>> handlers
>>>>    to process AP events.
>>
>> So, does this indicate "adapter interrupts for AP" only? If so, we
>> should keep this separate and only enable it when we have the gisa etc.
>> ready.
>>
>
> Yes, STFLE 65, it is for AP only.
>
> QCI, STFLE 12, is no present on older systems, in this case AP uses 
> TAPQ to retrieve information for each AP
>
> So for my point of view, it make sense to separate the three 
> facilities to enable migration on older syste
In the interest of keeping things simple, this makes sense.
>
>
> Pierre
>
>

Re: [RFC 19/19] s390/facilities: enable AP facilities needed by guest

[Pierre Morel]

On 05/12/2017 16:01, Tony Krowiak wrote:
> On 12/05/2017 09:04 AM, Cornelia Huck wrote:
>> On Tue, 5 Dec 2017 08:52:57 +0100
>> Harald Freudenberger <freude@linux.vnet.ibm.com> wrote:
>>
>>> On 12/02/2017 02:30 AM, Tony Krowiak wrote:
>>>> I agree with your suggestion that defining a new CPU model feature 
>>>> is probably
>>>> the best way to resolve this issue. The question is, should we 
>>>> define a single
>>>> feature indicating whether AP instructions are installed and set 
>>>> features bits
>>>> for the guest based on whether or not they are set in the linux 
>>>> host, or should
>>>> we define additional CPU model features for turning features bits on 
>>>> and off?
>>>> I guess it boils down to what behavior is expected for the AP bus 
>>>> running on
>>>> the linux guest. Here is a rundown of the facilities bits associated 
>>>> with AP
>>>> and how they affect the behavior of the AP bus:
>>>>
>>>> * STFLE.12 indicates whether the AP query function is available. If 
>>>> this bit
>>>>    is not set, then the AP bus scan will only test domains 0-15. For 
>>>> example,
>>>>    if adapters 4, 5, and 6 and domains 12 and 71 (0x47) are 
>>>> installed, then AP
>>>>    queues 04.0047, 05.0047 and 06.0047 will not be made available.
>>> STFLE 12 is the indication for Query AP Configuration Information 
>>> (QCI) available.
>>>> * STFLE.15 indicates whether the AP facilities test function is 
>>>> available. If
>>>>    this bit is not set, then the CEX4, CEX5 and CEX6 device drivers 
>>>> discovered
>>>>    by the AP bus scan will not get bound to any AP device drivers. 
>>>> Since the
>>>>    AP matrix model supports only CEX4 and greater, no devices will 
>>>> be bound
>>>>    to any driver for a guest.
>>> This T-Bit extension to the TAPQ subfunction is a must have. When kvm 
>>> only
>>> supports CEX4 and upper then this bit could also act as the indicator 
>>> for
>>> AP instructions available. Of course if you want to implement pure 
>>> virtual
>>> full simulated AP without any real AP hardware on the host this bit 
>>> can't
>>> be the indicator.
>> It would probably make sense to group these two together. Or is there
>> any advantage in supporting only a part of it?
> After thinking about this a little more, I've come to the conclusion that
> all of this might be moot for the following reasons:
> 
> * If STFLE.12 is not set for the linux host, then AP bus scan running on
>    the host will not detect any domains with a domain number higher than 
> 15,
>    so no AP queues with a queue index higher than 15 will be available to
>    bind to the vfio_ap_matrix driver. Consequently, no domain higher than
>    15 can be assigned to any guest. In this case, the AP bus scan 
> running on
>    the guest will never detect a domain higher than 15, regardless of the
>    setting of STFLE.12 for the guest.
> 
> * If STFLE.15 is not set for the linux host, then then there will be no
>    CEX4, CEX5 or CEX6 queues available to bind to the vfio_ap_matrix
>    driver, so no AP adapters or domains can be assigned to any KVM guest.
> 
> The bottom line is the STFLE bit settings for the linux host will control
> what APs are available to the KVM guest. Since STFLE.15 controls whether
> any CEX4,5 or 6 devices are even available, I think this bit can be
> combined into the feature that indicates whether AP is available. As long
> as AP instructions are available on the linux host, I'm not sure whether
> STFLE.12 needs a feature at all.

We are implementing VFIO with SIE interpretation.

1) Providing more:
The simple way is to provide to the guest only features existing on the 
host.
If we do provide features not existing on the host we need to be able to 
emulate them.
Even it is possible, it could be done in a future enhancement, but AFAIK 
it is not the goal of the current development.

2) Providing less:
On the other hand we can mask to the guest some of the features provided 
by the host if we can intercept the scanning of the features.


What I understand from this is that we need all these features being 
separately toggled to be able to be compatible with an older system even 
if we have a 1:1 host:guest features match in a first version.

If several features where introduced together in a new architecture and 
are available on all systems issued from this architecture we can then 
gather them in a set. (But I will wonder why we have several features then)


>>
>>>> * STFLE.65 indicates whether AP interrupts are available. If this 
>>>> bit is not
>>>>    set, then the AP bus will use polling instead of using interrupt 
>>>> handlers
>>>>    to process AP events.
>> So, does this indicate "adapter interrupts for AP" only? If so, we
>> should keep this separate and only enable it when we have the gisa etc.
>> ready.
> Yes, this indicates AP interrupts only. The plan is to enable this when
> GISA is available and we can implement interrupt processing.

If we want to be able to work on system where STFLE.65 is not available, 
even if GISA is available I think it would be interesting to have a 
Matrix implementation with only polling.

Regards,

Pierre


-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany

Re: [RFC 19/19] s390/facilities: enable AP facilities needed by guest

[Cornelia Huck]

On Wed, 6 Dec 2017 10:15:51 +0100
Pierre Morel <pmorel@linux.vnet.ibm.com> wrote:

> On 05/12/2017 16:01, Tony Krowiak wrote:
> > On 12/05/2017 09:04 AM, Cornelia Huck wrote:  
> >> On Tue, 5 Dec 2017 08:52:57 +0100
> >> Harald Freudenberger <freude@linux.vnet.ibm.com> wrote:
> >>  
> >>> On 12/02/2017 02:30 AM, Tony Krowiak wrote:  
> >>>> I agree with your suggestion that defining a new CPU model feature 
> >>>> is probably
> >>>> the best way to resolve this issue. The question is, should we 
> >>>> define a single
> >>>> feature indicating whether AP instructions are installed and set 
> >>>> features bits
> >>>> for the guest based on whether or not they are set in the linux 
> >>>> host, or should
> >>>> we define additional CPU model features for turning features bits on 
> >>>> and off?
> >>>> I guess it boils down to what behavior is expected for the AP bus 
> >>>> running on
> >>>> the linux guest. Here is a rundown of the facilities bits associated 
> >>>> with AP
> >>>> and how they affect the behavior of the AP bus:
> >>>>
> >>>> * STFLE.12 indicates whether the AP query function is available. If 
> >>>> this bit
> >>>>    is not set, then the AP bus scan will only test domains 0-15. For 
> >>>> example,
> >>>>    if adapters 4, 5, and 6 and domains 12 and 71 (0x47) are 
> >>>> installed, then AP
> >>>>    queues 04.0047, 05.0047 and 06.0047 will not be made available.  
> >>> STFLE 12 is the indication for Query AP Configuration Information 
> >>> (QCI) available.  
> >>>> * STFLE.15 indicates whether the AP facilities test function is 
> >>>> available. If
> >>>>    this bit is not set, then the CEX4, CEX5 and CEX6 device drivers 
> >>>> discovered
> >>>>    by the AP bus scan will not get bound to any AP device drivers. 
> >>>> Since the
> >>>>    AP matrix model supports only CEX4 and greater, no devices will 
> >>>> be bound
> >>>>    to any driver for a guest.  
> >>> This T-Bit extension to the TAPQ subfunction is a must have. When kvm 
> >>> only
> >>> supports CEX4 and upper then this bit could also act as the indicator 
> >>> for
> >>> AP instructions available. Of course if you want to implement pure 
> >>> virtual
> >>> full simulated AP without any real AP hardware on the host this bit 
> >>> can't
> >>> be the indicator.  
> >> It would probably make sense to group these two together. Or is there
> >> any advantage in supporting only a part of it?  
> > After thinking about this a little more, I've come to the conclusion that
> > all of this might be moot for the following reasons:
> > 
> > * If STFLE.12 is not set for the linux host, then AP bus scan running on
> >    the host will not detect any domains with a domain number higher than 
> > 15,
> >    so no AP queues with a queue index higher than 15 will be available to
> >    bind to the vfio_ap_matrix driver. Consequently, no domain higher than
> >    15 can be assigned to any guest. In this case, the AP bus scan 
> > running on
> >    the guest will never detect a domain higher than 15, regardless of the
> >    setting of STFLE.12 for the guest.
> > 
> > * If STFLE.15 is not set for the linux host, then then there will be no
> >    CEX4, CEX5 or CEX6 queues available to bind to the vfio_ap_matrix
> >    driver, so no AP adapters or domains can be assigned to any KVM guest.
> > 
> > The bottom line is the STFLE bit settings for the linux host will control
> > what APs are available to the KVM guest. Since STFLE.15 controls whether
> > any CEX4,5 or 6 devices are even available, I think this bit can be
> > combined into the feature that indicates whether AP is available. As long
> > as AP instructions are available on the linux host, I'm not sure whether
> > STFLE.12 needs a feature at all.  
> 
> We are implementing VFIO with SIE interpretation.
> 
> 1) Providing more:
> The simple way is to provide to the guest only features existing on the 
> host.
> If we do provide features not existing on the host we need to be able to 
> emulate them.
> Even it is possible, it could be done in a future enhancement, but AFAIK 
> it is not the goal of the current development.

Yes. I think we currently want to provide a subset of what the SIE can
do. Any emulation would be icing on top.

> 
> 2) Providing less:
> On the other hand we can mask to the guest some of the features provided 
> by the host if we can intercept the scanning of the features.

Yes. I think that applies to bit 65 (interrupts) for now.

> 
> 
> What I understand from this is that we need all these features being 
> separately toggled to be able to be compatible with an older system even 
> if we have a 1:1 host:guest features match in a first version.

This seems to be the case for the mentioned bits if I followed the
discussion correctly.

> 
> If several features where introduced together in a new architecture and 
> are available on all systems issued from this architecture we can then 
> gather them in a set. (But I will wonder why we have several features then)

You're expecting that all architecture is making sense ;)

> 
> 
> >>  
> >>>> * STFLE.65 indicates whether AP interrupts are available. If this 
> >>>> bit is not
> >>>>    set, then the AP bus will use polling instead of using interrupt 
> >>>> handlers
> >>>>    to process AP events.  
> >> So, does this indicate "adapter interrupts for AP" only? If so, we
> >> should keep this separate and only enable it when we have the gisa etc.
> >> ready.  
> > Yes, this indicates AP interrupts only. The plan is to enable this when
> > GISA is available and we can implement interrupt processing.  
> 
> If we want to be able to work on system where STFLE.65 is not available, 
> even if GISA is available I think it would be interesting to have a 
> Matrix implementation with only polling.

Agreed.

So, it seems what we want is:

- A feature for STFLE 12. This seems to be a z13 or later facility, so
  it probably makes sense to indicate it in the guest for z13 or newer
  if the host supports it and for older machines if it is explicitly
  specified (and the host supports it).
- A feature for STFLE 15. Similar to the one above, but starting with a
  different generation (when was this introduced?)
- A feature for STFLE 65. Can be deferred to when GISA is implemented
  (and exploited by vfio-ap). Same as above (I think this has existed
  for a long time, probably for any of the machines we support?)

Thoughts?