* [PATCH v9 00/15] s390/vfio-ap: dynamic configuration support
@ 2020-07-20 15:03 Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 01/15] s390/vfio-ap: add version vfio_ap module Tony Krowiak
` (16 more replies)
0 siblings, 17 replies; 21+ messages in thread
From: Tony Krowiak @ 2020-07-20 15:03 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, borntraeger, cohuck, mjrosato, pasic, alex.williamson,
kwankhede, fiuczy, Tony Krowiak
The current design for AP pass-through does not support making dynamic
changes to the AP matrix of a running guest resulting in a few
deficiencies this patch series is intended to mitigate:
1. Adapters, domains and control domains can not be added to or removed
from a running guest. In order to modify a guest's AP configuration,
the guest must be terminated; only then can AP resources be assigned
to or unassigned from the guest's matrix mdev. The new AP
configuration becomes available to the guest when it is subsequently
restarted.
2. The AP bus's /sys/bus/ap/apmask and /sys/bus/ap/aqmask interfaces can
be modified by a root user without any restrictions. A change to
either mask can result in AP queue devices being unbound from the
vfio_ap device driver and bound to a zcrypt device driver even if a
guest is using the queues, thus giving the host access to the guest's
private crypto data and vice versa.
3. The APQNs derived from the Cartesian product of the APIDs of the
adapters and APQIs of the domains assigned to a matrix mdev must
reference an AP queue device bound to the vfio_ap device driver. The
AP architecture allows assignment of AP resources that are not
available to the system, so this artificial restriction is not
compliant with the architecture.
4. The AP configuration profile can be dynamically changed for the linux
host after a KVM guest is started. For example, a new domain can be
dynamically added to the configuration profile via the SE or an HMC
connected to a DPM enabled lpar. Likewise, AP adapters can be
dynamically configured (online state) and deconfigured (standby state)
using the SE, an SCLP command or an HMC connected to a DPM enabled
lpar. This can result in inadvertent sharing of AP queues between the
guest and host.
5. A root user can manually unbind an AP queue device representing a
queue in use by a KVM guest via the vfio_ap device driver's sysfs
unbind attribute. In this case, the guest will be using a queue that
is not bound to the driver which violates the device model.
This patch series introduces the following changes to the current design
to alleviate the shortcomings described above as well as to implement
more of the AP architecture:
1. A root user will be prevented from making changes to the AP bus's
/sys/bus/ap/apmask or /sys/bus/ap/aqmask if the ownership of an APQN
changes from the vfio_ap device driver to a zcrypt driver when the
APQN is assigned to a matrix mdev.
2. Allow a root user to hot plug/unplug AP adapters, domains and control
domains using the matrix mdev's assign/unassign attributes.
4. Allow assignment of an AP adapter or domain to a matrix mdev even if
it results in assignment of an APQN that does not reference an AP
queue device bound to the vfio_ap device driver, as long as the APQN
is not reserved for use by the default zcrypt drivers (also known as
over-provisioning of AP resources). Allowing over-provisioning of AP
resources better models the architecture which does not preclude
assigning AP resources that are not yet available in the system. Such
APQNs, however, will not be assigned to the guest using the matrix
mdev; only APQNs referencing AP queue devices bound to the vfio_ap
device driver will actually get assigned to the guest.
5. Handle dynamic changes to the AP device model.
1. Rationale for changes to AP bus's apmask/aqmask interfaces:
----------------------------------------------------------
Due to the extremely sensitive nature of cryptographic data, it is
imperative that great care be taken to ensure that such data is secured.
Allowing a root user, either inadvertently or maliciously, to configure
these masks such that a queue is shared between the host and a guest is
not only avoidable, it is advisable. It was suggested that this scenario
is better handled in user space with management software, but that does
not preclude a malicious administrator from using the sysfs interfaces
to gain access to a guest's crypto data. It was also suggested that this
scenario could be avoided by taking access to the adapter away from the
guest and zeroing out the queues prior to the vfio_ap driver releasing the
device; however, stealing an adapter in use from a guest as a by-product
of an operation is bad and will likely cause problems for the guest
unnecessarily. It was decided that the most effective solution with the
least number of negative side effects is to prevent the situation at the
source.
2. Rationale for hot plug/unplug using matrix mdev sysfs interfaces:
----------------------------------------------------------------
Allowing a user to hot plug/unplug AP resources using the matrix mdev
sysfs interfaces circumvents the need to terminate the guest in order to
modify its AP configuration. Allowing dynamic configuration makes
reconfiguring a guest's AP matrix much less disruptive.
3. Rationale for allowing over-provisioning of AP resources:
-----------------------------------------------------------
Allowing assignment of AP resources to a matrix mdev and ultimately to a
guest better models the AP architecture. The architecture does not
preclude assignment of unavailable AP resources. If a queue subsequently
becomes available while a guest using the matrix mdev to which its APQN
is assigned, the guest will be given access to it. If an APQN
is dynamically unassigned from the underlying host system, it will
automatically become unavailable to the guest.
Change log v8-v9:
----------------
* Fixed errors flagged by the kernel test robot
* Fixed issue with guest losing queues when a new queue is probed due to
manual bind operation.
Change log v7-v8:
----------------
* Now logging a message when an attempt to reserve APQNs for the zcrypt
drivers will result in taking a queue away from a KVM guest to provide
the sysadmin a way to ascertain why the sysfs operation failed.
* Created locked and unlocked versions of the ap_parse_mask_str() function.
* Now using new interface provided by an AP bus patch -
s390/ap: introduce new ap function ap_get_qdev() - to retrieve
struct ap_queue representing an AP queue device. This patch is not a
part of this series but is a prerequisite for this series.
Change log v6-v7:
----------------
* Added callbacks to AP bus:
- on_config_changed: Notifies implementing drivers that
the AP configuration has changed since last AP device scan.
- on_scan_complete: Notifies implementing drivers that the device scan
has completed.
- implemented on_config_changed and on_scan_complete callbacks for
vfio_ap device driver.
- updated vfio_ap device driver's probe and remove callbacks to handle
dynamic changes to the AP device model.
* Added code to filter APQNs when assigning AP resources to a KVM guest's
CRYCB
Change log v5-v6:
----------------
* Fixed a bug in ap_bus.c introduced with patch 2/7 of the v5
series. Harald Freudenberer pointed out that the mutex lock
for ap_perms_mutex in the apmask_store and aqmask_store functions
was not being freed.
* Removed patch 6/7 which added logging to the vfio_ap driver
to expedite acceptance of this series. The logging will be introduced
with a separate patch series to allow more time to explore options
such as DBF logging vs. tracepoints.
* Added 3 patches related to ensuring that APQNs that do not reference
AP queue devices bound to the vfio_ap device driver are not assigned
to the guest CRYCB:
Patch 4: Filter CRYCB bits for unavailable queue devices
Patch 5: sysfs attribute to display the guest CRYCB
Patch 6: update guest CRYCB in vfio_ap probe and remove callbacks
* Added a patch (Patch 9) to version the vfio_ap module.
* Reshuffled patches to allow the in_use callback implementation to
invoke the vfio_ap_mdev_verify_no_sharing() function introduced in
patch 2.
Change log v4-v5:
----------------
* Added a patch to provide kernel s390dbf debug logs for VFIO AP
Change log v3->v4:
-----------------
* Restored patches preventing root user from changing ownership of
APQNs from zcrypt drivers to the vfio_ap driver if the APQN is
assigned to an mdev.
* No longer enforcing requirement restricting guest access to
queues represented by a queue device bound to the vfio_ap
device driver.
* Removed shadow CRYCB and now directly updating the guest CRYCB
from the matrix mdev's matrix.
* Rebased the patch series on top of 'vfio: ap: AP Queue Interrupt
Control' patches.
* Disabled bind/unbind sysfs interfaces for vfio_ap driver
Change log v2->v3:
-----------------
* Allow guest access to an AP queue only if the queue is bound to
the vfio_ap device driver.
* Removed the patch to test CRYCB masks before taking the vCPUs
out of SIE. Now checking the shadow CRYCB in the vfio_ap driver.
Change log v1->v2:
-----------------
* Removed patches preventing root user from unbinding AP queues from
the vfio_ap device driver
* Introduced a shadow CRYCB in the vfio_ap driver to manage dynamic
changes to the AP guest configuration due to root user interventions
or hardware anomalies.
Harald Freudenberger (1):
s390/zcrypt: Notify driver on config changed and scan complete
callbacks
Tony Krowiak (14):
s390/vfio-ap: add version vfio_ap module
s390/vfio-ap: use new AP bus interface to search for queue devices
s390/vfio-ap: manage link between queue struct and matrix mdev
s390/zcrypt: driver callback to indicate resource in use
s390/vfio-ap: implement in-use callback for vfio_ap driver
s390/vfio-ap: introduce shadow APCB
s390/vfio-ap: sysfs attribute to display the guest's matrix
s390/vfio-ap: filter matrix for unavailable queue devices
s390/vfio-ap: allow assignment of unavailable AP queues to mdev device
s390/vfio-ap: allow configuration of matrix mdev in use by a KVM guest
s390/vfio-ap: allow hot plug/unplug of AP resources using mdev device
s390/vfio-ap: handle host AP config change notification
s390/vfio-ap: handle AP bus scan completed notification
s390/vfio-ap: handle probe/remove not due to host AP config changes
drivers/s390/crypto/ap_bus.c | 323 +++++--
drivers/s390/crypto/ap_bus.h | 16 +
drivers/s390/crypto/vfio_ap_drv.c | 36 +-
drivers/s390/crypto/vfio_ap_ops.c | 1216 ++++++++++++++++++++-----
drivers/s390/crypto/vfio_ap_private.h | 23 +-
5 files changed, 1294 insertions(+), 320 deletions(-)
--
2.21.1
^ permalink raw reply [flat|nested] 21+ messages in thread
* [PATCH v9 01/15] s390/vfio-ap: add version vfio_ap module
2020-07-20 15:03 [PATCH v9 00/15] s390/vfio-ap: dynamic configuration support Tony Krowiak
@ 2020-07-20 15:03 ` Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 02/15] s390/vfio-ap: use new AP bus interface to search for queue devices Tony Krowiak
` (15 subsequent siblings)
16 siblings, 0 replies; 21+ messages in thread
From: Tony Krowiak @ 2020-07-20 15:03 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, borntraeger, cohuck, mjrosato, pasic, alex.williamson,
kwankhede, fiuczy, Tony Krowiak
Let's set a version for the vfio_ap module so that automated regression
tests can determine whether dynamic configuration tests can be run or
not.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
---
drivers/s390/crypto/vfio_ap_drv.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c
index be2520cc010b..f4ceb380dd61 100644
--- a/drivers/s390/crypto/vfio_ap_drv.c
+++ b/drivers/s390/crypto/vfio_ap_drv.c
@@ -17,10 +17,12 @@
#define VFIO_AP_ROOT_NAME "vfio_ap"
#define VFIO_AP_DEV_NAME "matrix"
+#define VFIO_AP_MODULE_VERSION "1.2.0"
MODULE_AUTHOR("IBM Corporation");
MODULE_DESCRIPTION("VFIO AP device driver, Copyright IBM Corp. 2018");
MODULE_LICENSE("GPL v2");
+MODULE_VERSION(VFIO_AP_MODULE_VERSION);
static struct ap_driver vfio_ap_drv;
--
2.21.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v9 02/15] s390/vfio-ap: use new AP bus interface to search for queue devices
2020-07-20 15:03 [PATCH v9 00/15] s390/vfio-ap: dynamic configuration support Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 01/15] s390/vfio-ap: add version vfio_ap module Tony Krowiak
@ 2020-07-20 15:03 ` Tony Krowiak
2020-07-24 8:38 ` Pierre Morel
2020-07-20 15:03 ` [PATCH v9 03/15] s390/vfio-ap: manage link between queue struct and matrix mdev Tony Krowiak
` (14 subsequent siblings)
16 siblings, 1 reply; 21+ messages in thread
From: Tony Krowiak @ 2020-07-20 15:03 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, borntraeger, cohuck, mjrosato, pasic, alex.williamson,
kwankhede, fiuczy, Tony Krowiak, kernel test robot
This patch refactor's the vfio_ap device driver to use the AP bus's
ap_get_qdev() function to retrieve the vfio_ap_queue struct containing
information about a queue that is bound to the vfio_ap device driver.
The bus's ap_get_qdev() function retrieves the queue device from a
hashtable keyed by APQN. This is much more efficient than looping over
the list of devices attached to the AP bus by several orders of
magnitude.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
Reported-by: kernel test robot <lkp@intel.com>
---
drivers/s390/crypto/vfio_ap_drv.c | 27 ++-------
drivers/s390/crypto/vfio_ap_ops.c | 86 +++++++++++++++------------
drivers/s390/crypto/vfio_ap_private.h | 8 ++-
3 files changed, 59 insertions(+), 62 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c
index f4ceb380dd61..24cdef60039a 100644
--- a/drivers/s390/crypto/vfio_ap_drv.c
+++ b/drivers/s390/crypto/vfio_ap_drv.c
@@ -53,15 +53,9 @@ MODULE_DEVICE_TABLE(vfio_ap, ap_queue_ids);
*/
static int vfio_ap_queue_dev_probe(struct ap_device *apdev)
{
- struct vfio_ap_queue *q;
-
- q = kzalloc(sizeof(*q), GFP_KERNEL);
- if (!q)
- return -ENOMEM;
- dev_set_drvdata(&apdev->device, q);
- q->apqn = to_ap_queue(&apdev->device)->qid;
- q->saved_isc = VFIO_AP_ISC_INVALID;
- return 0;
+ struct ap_queue *queue = to_ap_queue(&apdev->device);
+
+ return vfio_ap_mdev_probe_queue(queue);
}
/**
@@ -72,18 +66,9 @@ static int vfio_ap_queue_dev_probe(struct ap_device *apdev)
*/
static void vfio_ap_queue_dev_remove(struct ap_device *apdev)
{
- struct vfio_ap_queue *q;
- int apid, apqi;
-
- mutex_lock(&matrix_dev->lock);
- q = dev_get_drvdata(&apdev->device);
- dev_set_drvdata(&apdev->device, NULL);
- apid = AP_QID_CARD(q->apqn);
- apqi = AP_QID_QUEUE(q->apqn);
- vfio_ap_mdev_reset_queue(apid, apqi, 1);
- vfio_ap_irq_disable(q);
- kfree(q);
- mutex_unlock(&matrix_dev->lock);
+ struct ap_queue *queue = to_ap_queue(&apdev->device);
+
+ vfio_ap_mdev_remove_queue(queue);
}
static void vfio_ap_matrix_dev_release(struct device *dev)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index e0bde8518745..ad3925f04f61 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -26,43 +26,26 @@
static int vfio_ap_mdev_reset_queues(struct mdev_device *mdev);
-static int match_apqn(struct device *dev, const void *data)
-{
- struct vfio_ap_queue *q = dev_get_drvdata(dev);
-
- return (q->apqn == *(int *)(data)) ? 1 : 0;
-}
-
/**
- * vfio_ap_get_queue: Retrieve a queue with a specific APQN from a list
- * @matrix_mdev: the associated mediated matrix
+ * vfio_ap_get_queue: Retrieve a queue with a specific APQN.
* @apqn: The queue APQN
*
- * Retrieve a queue with a specific APQN from the list of the
- * devices of the vfio_ap_drv.
- * Verify that the APID and the APQI are set in the matrix.
+ * Retrieve a queue with a specific APQN from the AP queue devices attached to
+ * the AP bus.
*
- * Returns the pointer to the associated vfio_ap_queue
+ * Returns the pointer to the vfio_ap_queue with the specified APQN, or NULL.
*/
-static struct vfio_ap_queue *vfio_ap_get_queue(
- struct ap_matrix_mdev *matrix_mdev,
- int apqn)
+static struct vfio_ap_queue *vfio_ap_get_queue(unsigned long apqn)
{
+ struct ap_queue *queue;
struct vfio_ap_queue *q;
- struct device *dev;
- if (!test_bit_inv(AP_QID_CARD(apqn), matrix_mdev->matrix.apm))
- return NULL;
- if (!test_bit_inv(AP_QID_QUEUE(apqn), matrix_mdev->matrix.aqm))
+ queue = ap_get_qdev(apqn);
+ if (!queue)
return NULL;
- dev = driver_find_device(&matrix_dev->vfio_ap_drv->driver, NULL,
- &apqn, match_apqn);
- if (!dev)
- return NULL;
- q = dev_get_drvdata(dev);
- q->matrix_mdev = matrix_mdev;
- put_device(dev);
+ q = dev_get_drvdata(&queue->ap_dev.device);
+ put_device(&queue->ap_dev.device);
return q;
}
@@ -144,7 +127,7 @@ static void vfio_ap_free_aqic_resources(struct vfio_ap_queue *q)
* Returns if ap_aqic function failed with invalid, deconfigured or
* checkstopped AP.
*/
-struct ap_queue_status vfio_ap_irq_disable(struct vfio_ap_queue *q)
+static struct ap_queue_status vfio_ap_irq_disable(struct vfio_ap_queue *q)
{
struct ap_qirq_ctrl aqic_gisa = {};
struct ap_queue_status status;
@@ -293,10 +276,11 @@ static int handle_pqap(struct kvm_vcpu *vcpu)
matrix_mdev = container_of(vcpu->kvm->arch.crypto.pqap_hook,
struct ap_matrix_mdev, pqap_hook);
- q = vfio_ap_get_queue(matrix_mdev, apqn);
+ q = vfio_ap_get_queue(apqn);
if (!q)
goto out_unlock;
+ q->matrix_mdev = matrix_mdev;
status = vcpu->run->s.regs.gprs[1];
/* If IR bit(16) is set we enable the interrupt */
@@ -1116,20 +1100,15 @@ static int vfio_ap_mdev_group_notifier(struct notifier_block *nb,
static void vfio_ap_irq_disable_apqn(int apqn)
{
- struct device *dev;
struct vfio_ap_queue *q;
- dev = driver_find_device(&matrix_dev->vfio_ap_drv->driver, NULL,
- &apqn, match_apqn);
- if (dev) {
- q = dev_get_drvdata(dev);
+ q = vfio_ap_get_queue(apqn);
+ if (q)
vfio_ap_irq_disable(q);
- put_device(dev);
- }
}
-int vfio_ap_mdev_reset_queue(unsigned int apid, unsigned int apqi,
- unsigned int retry)
+static int vfio_ap_mdev_reset_queue(unsigned int apid, unsigned int apqi,
+ unsigned int retry)
{
struct ap_queue_status status;
int retry2 = 2;
@@ -1302,3 +1281,34 @@ void vfio_ap_mdev_unregister(void)
{
mdev_unregister_device(&matrix_dev->device);
}
+
+int vfio_ap_mdev_probe_queue(struct ap_queue *queue)
+{
+ struct vfio_ap_queue *q;
+
+ q = kzalloc(sizeof(*q), GFP_KERNEL);
+ if (!q)
+ return -ENOMEM;
+
+ dev_set_drvdata(&queue->ap_dev.device, q);
+ q->apqn = queue->qid;
+ q->saved_isc = VFIO_AP_ISC_INVALID;
+
+ return 0;
+}
+
+void vfio_ap_mdev_remove_queue(struct ap_queue *queue)
+{
+ struct vfio_ap_queue *q;
+ int apid, apqi;
+
+ mutex_lock(&matrix_dev->lock);
+ q = dev_get_drvdata(&queue->ap_dev.device);
+ dev_set_drvdata(&queue->ap_dev.device, NULL);
+ apid = AP_QID_CARD(q->apqn);
+ apqi = AP_QID_QUEUE(q->apqn);
+ vfio_ap_mdev_reset_queue(apid, apqi, 1);
+ vfio_ap_irq_disable(q);
+ kfree(q);
+ mutex_unlock(&matrix_dev->lock);
+}
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index f46dde56b464..a2aa05bec718 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -18,6 +18,7 @@
#include <linux/delay.h>
#include <linux/mutex.h>
#include <linux/kvm_host.h>
+#include <linux/hashtable.h>
#include "ap_bus.h"
@@ -90,8 +91,6 @@ struct ap_matrix_mdev {
extern int vfio_ap_mdev_register(void);
extern void vfio_ap_mdev_unregister(void);
-int vfio_ap_mdev_reset_queue(unsigned int apid, unsigned int apqi,
- unsigned int retry);
struct vfio_ap_queue {
struct ap_matrix_mdev *matrix_mdev;
@@ -100,5 +99,8 @@ struct vfio_ap_queue {
#define VFIO_AP_ISC_INVALID 0xff
unsigned char saved_isc;
};
-struct ap_queue_status vfio_ap_irq_disable(struct vfio_ap_queue *q);
+
+int vfio_ap_mdev_probe_queue(struct ap_queue *queue);
+void vfio_ap_mdev_remove_queue(struct ap_queue *queue);
+
#endif /* _VFIO_AP_PRIVATE_H_ */
--
2.21.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v9 03/15] s390/vfio-ap: manage link between queue struct and matrix mdev
2020-07-20 15:03 [PATCH v9 00/15] s390/vfio-ap: dynamic configuration support Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 01/15] s390/vfio-ap: add version vfio_ap module Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 02/15] s390/vfio-ap: use new AP bus interface to search for queue devices Tony Krowiak
@ 2020-07-20 15:03 ` Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 04/15] s390/zcrypt: driver callback to indicate resource in use Tony Krowiak
` (13 subsequent siblings)
16 siblings, 0 replies; 21+ messages in thread
From: Tony Krowiak @ 2020-07-20 15:03 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, borntraeger, cohuck, mjrosato, pasic, alex.williamson,
kwankhede, fiuczy, Tony Krowiak
Let's create links between each queue device bound to the vfio_ap device
driver and the matrix mdev to which the queue is assigned. The idea is to
facilitate efficient retrieval of the objects representing the queue
devices and matrix mdevs as well as to verify that a queue assigned to
a matrix mdev is bound to the driver.
The links will be created as follows:
* When the queue device is probed, if its APQN is assigned to a matrix
mdev, the structures representing the queue device and the matrix mdev
will be linked.
* When an adapter or domain is assigned to a matrix mdev, for each new
APQN assigned that references a queue device bound to the vfio_ap
device driver, the structures representing the queue device and the
matrix mdev will be linked.
The links will be removed as follows:
* When the queue device is removed, if its APQN is assigned to a matrix
mdev, the structures representing the queue device and the matrix mdev
will be unlinked.
* When an adapter or domain is unassigned from a matrix mdev, for each
APQN unassigned that references a queue device bound to the vfio_ap
device driver, the structures representing the queue device and the
matrix mdev will be unlinked.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
---
drivers/s390/crypto/vfio_ap_ops.c | 132 +++++++++++++++++++++++++-
drivers/s390/crypto/vfio_ap_private.h | 2 +
2 files changed, 129 insertions(+), 5 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index ad3925f04f61..2e37ee82e422 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -50,6 +50,19 @@ static struct vfio_ap_queue *vfio_ap_get_queue(unsigned long apqn)
return q;
}
+static struct vfio_ap_queue *vfio_ap_get_mdev_queue(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long apqn)
+{
+ struct vfio_ap_queue *q;
+
+ hash_for_each_possible(matrix_mdev->qtable, q, mdev_qnode, apqn) {
+ if (q && (q->apqn == apqn))
+ return q;
+ }
+
+ return NULL;
+}
+
/**
* vfio_ap_wait_for_irqclear
* @apqn: The AP Queue number
@@ -160,7 +173,6 @@ static struct ap_queue_status vfio_ap_irq_disable(struct vfio_ap_queue *q)
status.response_code);
end_free:
vfio_ap_free_aqic_resources(q);
- q->matrix_mdev = NULL;
return status;
}
@@ -262,7 +274,6 @@ static int handle_pqap(struct kvm_vcpu *vcpu)
struct vfio_ap_queue *q;
struct ap_queue_status qstatus = {
.response_code = AP_RESPONSE_Q_NOT_AVAIL, };
- struct ap_matrix_mdev *matrix_mdev;
/* If we do not use the AIV facility just go to userland */
if (!(vcpu->arch.sie_block->eca & ECA_AIV))
@@ -273,14 +284,11 @@ static int handle_pqap(struct kvm_vcpu *vcpu)
if (!vcpu->kvm->arch.crypto.pqap_hook)
goto out_unlock;
- matrix_mdev = container_of(vcpu->kvm->arch.crypto.pqap_hook,
- struct ap_matrix_mdev, pqap_hook);
q = vfio_ap_get_queue(apqn);
if (!q)
goto out_unlock;
- q->matrix_mdev = matrix_mdev;
status = vcpu->run->s.regs.gprs[1];
/* If IR bit(16) is set we enable the interrupt */
@@ -320,6 +328,7 @@ static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev)
matrix_mdev->mdev = mdev;
vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->matrix);
+ hash_init(matrix_mdev->qtable);
mdev_set_drvdata(mdev, matrix_mdev);
matrix_mdev->pqap_hook.hook = handle_pqap;
matrix_mdev->pqap_hook.owner = THIS_MODULE;
@@ -548,6 +557,87 @@ static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev)
return 0;
}
+enum qlink_type {
+ LINK_APID,
+ LINK_APQI,
+ UNLINK_APID,
+ UNLINK_APQI,
+};
+
+static void vfio_ap_mdev_link_queue(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long apid, unsigned long apqi)
+{
+ struct vfio_ap_queue *q;
+
+ q = vfio_ap_get_queue(AP_MKQID(apid, apqi));
+ if (q) {
+ q->matrix_mdev = matrix_mdev;
+ hash_add(matrix_mdev->qtable,
+ &q->mdev_qnode, q->apqn);
+ }
+}
+
+static void vfio_ap_mdev_unlink_queue(unsigned long apid, unsigned long apqi)
+{
+ struct vfio_ap_queue *q;
+
+ q = vfio_ap_get_queue(AP_MKQID(apid, apqi));
+ if (q) {
+ q->matrix_mdev = NULL;
+ hash_del(&q->mdev_qnode);
+ }
+}
+
+/**
+ * vfio_ap_mdev_link_queues
+ *
+ * @matrix_mdev: The matrix mdev to link.
+ * @type: The type of @qlink_id.
+ * @qlink_id: The APID or APQI of the queues to link.
+ *
+ * Sets or clears the links between the queues with the specified @qlink_id
+ * and the @matrix_mdev:
+ * @type == LINK_APID: Set the links between the @matrix_mdev and the
+ * queues with the specified @qlink_id (APID)
+ * @type == LINK_APQI: Set the links between the @matrix_mdev and the
+ * queues with the specified @qlink_id (APQI)
+ * @type == UNLINK_APID: Clear the links between the @matrix_mdev and the
+ * queues with the specified @qlink_id (APID)
+ * @type == UNLINK_APQI: Clear the links between the @matrix_mdev and the
+ * queues with the specified @qlink_id (APQI)
+ */
+static void vfio_ap_mdev_link_queues(struct ap_matrix_mdev *matrix_mdev,
+ enum qlink_type type,
+ unsigned long qlink_id)
+{
+ unsigned long id;
+
+ switch (type) {
+ case LINK_APID:
+ for_each_set_bit_inv(id, matrix_mdev->matrix.aqm,
+ matrix_mdev->matrix.aqm_max + 1)
+ vfio_ap_mdev_link_queue(matrix_mdev, qlink_id, id);
+ break;
+ case UNLINK_APID:
+ for_each_set_bit_inv(id, matrix_mdev->matrix.aqm,
+ matrix_mdev->matrix.aqm_max + 1)
+ vfio_ap_mdev_unlink_queue(qlink_id, id);
+ break;
+ case LINK_APQI:
+ for_each_set_bit_inv(id, matrix_mdev->matrix.apm,
+ matrix_mdev->matrix.apm_max + 1)
+ vfio_ap_mdev_link_queue(matrix_mdev, id, qlink_id);
+ break;
+ case UNLINK_APQI:
+ for_each_set_bit_inv(id, matrix_mdev->matrix.apm,
+ matrix_mdev->matrix.apm_max + 1)
+ vfio_ap_mdev_link_queue(matrix_mdev, id, qlink_id);
+ break;
+ default:
+ WARN_ON_ONCE(1);
+ }
+}
+
/**
* assign_adapter_store
*
@@ -617,6 +707,7 @@ static ssize_t assign_adapter_store(struct device *dev,
if (ret)
goto share_err;
+ vfio_ap_mdev_link_queues(matrix_mdev, LINK_APID, apid);
ret = count;
goto done;
@@ -668,6 +759,7 @@ static ssize_t unassign_adapter_store(struct device *dev,
mutex_lock(&matrix_dev->lock);
clear_bit_inv((unsigned long)apid, matrix_mdev->matrix.apm);
+ vfio_ap_mdev_link_queues(matrix_mdev, UNLINK_APID, apid);
mutex_unlock(&matrix_dev->lock);
return count;
@@ -758,6 +850,7 @@ static ssize_t assign_domain_store(struct device *dev,
if (ret)
goto share_err;
+ vfio_ap_mdev_link_queues(matrix_mdev, LINK_APQI, apqi);
ret = count;
goto done;
@@ -810,6 +903,7 @@ static ssize_t unassign_domain_store(struct device *dev,
mutex_lock(&matrix_dev->lock);
clear_bit_inv((unsigned long)apqi, matrix_mdev->matrix.aqm);
+ vfio_ap_mdev_link_queues(matrix_mdev, UNLINK_APQI, apqi);
mutex_unlock(&matrix_dev->lock);
return count;
@@ -1282,6 +1376,29 @@ void vfio_ap_mdev_unregister(void)
mdev_unregister_device(&matrix_dev->device);
}
+/**
+ * vfio_ap_queue_link_mdev
+ *
+ * @q: The queue to link with the matrix mdev.
+ *
+ * Links @q with the matrix mdev to which the queue's APQN is assigned.
+ */
+static void vfio_ap_queue_link_mdev(struct vfio_ap_queue *q)
+{
+ unsigned long apid = AP_QID_CARD(q->apqn);
+ unsigned long apqi = AP_QID_QUEUE(q->apqn);
+ struct ap_matrix_mdev *matrix_mdev;
+
+ list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) {
+ if (test_bit_inv(apid, matrix_mdev->matrix.apm) &&
+ test_bit_inv(apqi, matrix_mdev->matrix.aqm)) {
+ q->matrix_mdev = matrix_mdev;
+ hash_add(matrix_mdev->qtable, &q->mdev_qnode, q->apqn);
+ break;
+ }
+ }
+}
+
int vfio_ap_mdev_probe_queue(struct ap_queue *queue)
{
struct vfio_ap_queue *q;
@@ -1290,9 +1407,12 @@ int vfio_ap_mdev_probe_queue(struct ap_queue *queue)
if (!q)
return -ENOMEM;
+ mutex_lock(&matrix_dev->lock);
dev_set_drvdata(&queue->ap_dev.device, q);
q->apqn = queue->qid;
q->saved_isc = VFIO_AP_ISC_INVALID;
+ vfio_ap_queue_link_mdev(q);
+ mutex_unlock(&matrix_dev->lock);
return 0;
}
@@ -1309,6 +1429,8 @@ void vfio_ap_mdev_remove_queue(struct ap_queue *queue)
apqi = AP_QID_QUEUE(q->apqn);
vfio_ap_mdev_reset_queue(apid, apqi, 1);
vfio_ap_irq_disable(q);
+ if (q->matrix_mdev)
+ hash_del(&q->mdev_qnode);
kfree(q);
mutex_unlock(&matrix_dev->lock);
}
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index a2aa05bec718..57da703b549a 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -87,6 +87,7 @@ struct ap_matrix_mdev {
struct kvm *kvm;
struct kvm_s390_module_hook pqap_hook;
struct mdev_device *mdev;
+ DECLARE_HASHTABLE(qtable, 8);
};
extern int vfio_ap_mdev_register(void);
@@ -98,6 +99,7 @@ struct vfio_ap_queue {
int apqn;
#define VFIO_AP_ISC_INVALID 0xff
unsigned char saved_isc;
+ struct hlist_node mdev_qnode;
};
int vfio_ap_mdev_probe_queue(struct ap_queue *queue);
--
2.21.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v9 04/15] s390/zcrypt: driver callback to indicate resource in use
2020-07-20 15:03 [PATCH v9 00/15] s390/vfio-ap: dynamic configuration support Tony Krowiak
` (2 preceding siblings ...)
2020-07-20 15:03 ` [PATCH v9 03/15] s390/vfio-ap: manage link between queue struct and matrix mdev Tony Krowiak
@ 2020-07-20 15:03 ` Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 05/15] s390/vfio-ap: implement in-use callback for vfio_ap driver Tony Krowiak
` (12 subsequent siblings)
16 siblings, 0 replies; 21+ messages in thread
From: Tony Krowiak @ 2020-07-20 15:03 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, borntraeger, cohuck, mjrosato, pasic, alex.williamson,
kwankhede, fiuczy, Tony Krowiak, kernel test robot
Introduces a new driver callback to prevent a root user from unbinding
an AP queue from its device driver if the queue is in use. The intent of
this callback is to provide a driver with the means to prevent a root user
from inadvertently taking a queue away from a matrix mdev and giving it to
the host while it is assigned to the matrix mdev. The callback will
be invoked whenever a change to the AP bus's sysfs apmask or aqmask
attributes would result in one or more AP queues being removed from its
driver. If the callback responds in the affirmative for any driver
queried, the change to the apmask or aqmask will be rejected with a device
in use error.
For this patch, only non-default drivers will be queried. Currently,
there is only one non-default driver, the vfio_ap device driver. The
vfio_ap device driver facilitates pass-through of an AP queue to a
guest. The idea here is that a guest may be administered by a different
sysadmin than the host and we don't want AP resources to unexpectedly
disappear from a guest's AP configuration (i.e., adapters, domains and
control domains assigned to the matrix mdev). This will enforce the proper
procedure for removing AP resources intended for guest usage which is to
first unassign them from the matrix mdev, then unbind them from the
vfio_ap device driver.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
Reported-by: kernel test robot <lkp@intel.com>
---
drivers/s390/crypto/ap_bus.c | 148 ++++++++++++++++++++++++++++++++---
drivers/s390/crypto/ap_bus.h | 4 +
2 files changed, 142 insertions(+), 10 deletions(-)
diff --git a/drivers/s390/crypto/ap_bus.c b/drivers/s390/crypto/ap_bus.c
index e71ca4a719a5..d6a732c8d90a 100644
--- a/drivers/s390/crypto/ap_bus.c
+++ b/drivers/s390/crypto/ap_bus.c
@@ -35,6 +35,7 @@
#include <linux/mod_devicetable.h>
#include <linux/debugfs.h>
#include <linux/ctype.h>
+#include <linux/module.h>
#include "ap_bus.h"
#include "ap_debug.h"
@@ -876,6 +877,23 @@ static int modify_bitmap(const char *str, unsigned long *bitmap, int bits)
return 0;
}
+static int ap_parse_bitmap_str(const char *str, unsigned long *bitmap, int bits,
+ unsigned long *newmap)
+{
+ unsigned long size;
+ int rc;
+
+ size = BITS_TO_LONGS(bits)*sizeof(unsigned long);
+ if (*str == '+' || *str == '-') {
+ memcpy(newmap, bitmap, size);
+ rc = modify_bitmap(str, newmap, bits);
+ } else {
+ memset(newmap, 0, size);
+ rc = hex2bitmap(str, newmap, bits);
+ }
+ return rc;
+}
+
int ap_parse_mask_str(const char *str,
unsigned long *bitmap, int bits,
struct mutex *lock)
@@ -895,14 +913,7 @@ int ap_parse_mask_str(const char *str,
kfree(newmap);
return -ERESTARTSYS;
}
-
- if (*str == '+' || *str == '-') {
- memcpy(newmap, bitmap, size);
- rc = modify_bitmap(str, newmap, bits);
- } else {
- memset(newmap, 0, size);
- rc = hex2bitmap(str, newmap, bits);
- }
+ rc = ap_parse_bitmap_str(str, bitmap, bits, newmap);
if (rc == 0)
memcpy(bitmap, newmap, size);
mutex_unlock(lock);
@@ -1092,12 +1103,70 @@ static ssize_t apmask_show(struct bus_type *bus, char *buf)
return rc;
}
+static int __verify_card_reservations(struct device_driver *drv, void *data)
+{
+ int rc = 0;
+ struct ap_driver *ap_drv = to_ap_drv(drv);
+ unsigned long *newapm = (unsigned long *)data;
+
+ /*
+ * No need to verify whether the driver is using the queues if it is the
+ * default driver.
+ */
+ if (ap_drv->flags & AP_DRIVER_FLAG_DEFAULT)
+ return 0;
+
+ /* The non-default driver's module must be loaded */
+ if (!try_module_get(drv->owner))
+ return 0;
+
+ if (ap_drv->in_use)
+ if (ap_drv->in_use(newapm, ap_perms.aqm))
+ rc = -EADDRINUSE;
+
+ module_put(drv->owner);
+
+ return rc;
+}
+
+static int apmask_commit(unsigned long *newapm)
+{
+ int rc;
+ unsigned long reserved[BITS_TO_LONGS(AP_DEVICES)];
+
+ /*
+ * Check if any bits in the apmask have been set which will
+ * result in queues being removed from non-default drivers
+ */
+ if (bitmap_andnot(reserved, newapm, ap_perms.apm, AP_DEVICES)) {
+ rc = bus_for_each_drv(&ap_bus_type, NULL, reserved,
+ __verify_card_reservations);
+ if (rc)
+ return rc;
+ }
+
+ memcpy(ap_perms.apm, newapm, APMASKSIZE);
+
+ return 0;
+}
+
static ssize_t apmask_store(struct bus_type *bus, const char *buf,
size_t count)
{
int rc;
+ DECLARE_BITMAP(newapm, AP_DEVICES);
+
+ if (mutex_lock_interruptible(&ap_perms_mutex))
+ return -ERESTARTSYS;
+
+ rc = ap_parse_bitmap_str(buf, ap_perms.apm, AP_DEVICES, newapm);
+ if (rc)
+ goto done;
- rc = ap_parse_mask_str(buf, ap_perms.apm, AP_DEVICES, &ap_perms_mutex);
+ rc = apmask_commit(newapm);
+
+done:
+ mutex_unlock(&ap_perms_mutex);
if (rc)
return rc;
@@ -1123,12 +1192,71 @@ static ssize_t aqmask_show(struct bus_type *bus, char *buf)
return rc;
}
+static int __verify_queue_reservations(struct device_driver *drv, void *data)
+{
+ int rc = 0;
+ struct ap_driver *ap_drv = to_ap_drv(drv);
+ unsigned long *newaqm = (unsigned long *)data;
+
+ /*
+ * If the reserved bits do not identify queues reserved for use by the
+ * non-default driver, there is no need to verify the driver is using
+ * the queues.
+ */
+ if (ap_drv->flags & AP_DRIVER_FLAG_DEFAULT)
+ return 0;
+
+ /* The non-default driver's module must be loaded */
+ if (!try_module_get(drv->owner))
+ return 0;
+
+ if (ap_drv->in_use)
+ if (ap_drv->in_use(ap_perms.apm, newaqm))
+ rc = -EADDRINUSE;
+
+ module_put(drv->owner);
+
+ return rc;
+}
+
+static int aqmask_commit(unsigned long *newaqm)
+{
+ int rc;
+ unsigned long reserved[BITS_TO_LONGS(AP_DOMAINS)];
+
+ /*
+ * Check if any bits in the aqmask have been set which will
+ * result in queues being removed from non-default drivers
+ */
+ if (bitmap_andnot(reserved, newaqm, ap_perms.aqm, AP_DOMAINS)) {
+ rc = bus_for_each_drv(&ap_bus_type, NULL, reserved,
+ __verify_queue_reservations);
+ if (rc)
+ return rc;
+ }
+
+ memcpy(ap_perms.aqm, newaqm, AQMASKSIZE);
+
+ return 0;
+}
+
static ssize_t aqmask_store(struct bus_type *bus, const char *buf,
size_t count)
{
int rc;
+ DECLARE_BITMAP(newaqm, AP_DOMAINS);
- rc = ap_parse_mask_str(buf, ap_perms.aqm, AP_DOMAINS, &ap_perms_mutex);
+ if (mutex_lock_interruptible(&ap_perms_mutex))
+ return -ERESTARTSYS;
+
+ rc = ap_parse_bitmap_str(buf, ap_perms.aqm, AP_DOMAINS, newaqm);
+ if (rc)
+ goto done;
+
+ rc = aqmask_commit(newaqm);
+
+done:
+ mutex_unlock(&ap_perms_mutex);
if (rc)
return rc;
diff --git a/drivers/s390/crypto/ap_bus.h b/drivers/s390/crypto/ap_bus.h
index 053cc34d2ca2..7d9646251bfd 100644
--- a/drivers/s390/crypto/ap_bus.h
+++ b/drivers/s390/crypto/ap_bus.h
@@ -136,6 +136,7 @@ struct ap_driver {
int (*probe)(struct ap_device *);
void (*remove)(struct ap_device *);
+ bool (*in_use)(unsigned long *apm, unsigned long *aqm);
};
#define to_ap_drv(x) container_of((x), struct ap_driver, driver)
@@ -254,6 +255,9 @@ void ap_queue_init_state(struct ap_queue *aq);
struct ap_card *ap_card_create(int id, int queue_depth, int raw_device_type,
int comp_device_type, unsigned int functions);
+#define APMASKSIZE (BITS_TO_LONGS(AP_DEVICES) * sizeof(unsigned long))
+#define AQMASKSIZE (BITS_TO_LONGS(AP_DOMAINS) * sizeof(unsigned long))
+
struct ap_perms {
unsigned long ioctlm[BITS_TO_LONGS(AP_IOCTLS)];
unsigned long apm[BITS_TO_LONGS(AP_DEVICES)];
--
2.21.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v9 05/15] s390/vfio-ap: implement in-use callback for vfio_ap driver
2020-07-20 15:03 [PATCH v9 00/15] s390/vfio-ap: dynamic configuration support Tony Krowiak
` (3 preceding siblings ...)
2020-07-20 15:03 ` [PATCH v9 04/15] s390/zcrypt: driver callback to indicate resource in use Tony Krowiak
@ 2020-07-20 15:03 ` Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 06/15] s390/vfio-ap: introduce shadow APCB Tony Krowiak
` (11 subsequent siblings)
16 siblings, 0 replies; 21+ messages in thread
From: Tony Krowiak @ 2020-07-20 15:03 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, borntraeger, cohuck, mjrosato, pasic, alex.williamson,
kwankhede, fiuczy, Tony Krowiak
Let's implement the callback to indicate when an APQN
is in use by the vfio_ap device driver. The callback is
invoked whenever a change to the apmask or aqmask would
result in one or more queue devices being removed from the driver. The
vfio_ap device driver will indicate a resource is in use
if the APQN of any of the queue devices to be removed are assigned to
any of the matrix mdevs under the driver's control.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
---
drivers/s390/crypto/vfio_ap_drv.c | 1 +
drivers/s390/crypto/vfio_ap_ops.c | 68 ++++++++++++++++++++-------
drivers/s390/crypto/vfio_ap_private.h | 2 +
3 files changed, 53 insertions(+), 18 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c
index 24cdef60039a..aae5b3d8e3fa 100644
--- a/drivers/s390/crypto/vfio_ap_drv.c
+++ b/drivers/s390/crypto/vfio_ap_drv.c
@@ -175,6 +175,7 @@ static int __init vfio_ap_init(void)
memset(&vfio_ap_drv, 0, sizeof(vfio_ap_drv));
vfio_ap_drv.probe = vfio_ap_queue_dev_probe;
vfio_ap_drv.remove = vfio_ap_queue_dev_remove;
+ vfio_ap_drv.in_use = vfio_ap_mdev_resource_in_use;
vfio_ap_drv.ids = ap_queue_ids;
ret = ap_driver_register(&vfio_ap_drv, THIS_MODULE, VFIO_AP_DRV_NAME);
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 2e37ee82e422..fc1aa6f947eb 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -515,18 +515,36 @@ vfio_ap_mdev_verify_queues_reserved_for_apid(struct ap_matrix_mdev *matrix_mdev,
return 0;
}
+#define MDEV_SHARING_ERR "Userspace may not re-assign queue %02lx.%04lx " \
+ "already assigned to %s"
+
+static void vfio_ap_mdev_log_sharing_err(const char *mdev_name,
+ unsigned long *apm,
+ unsigned long *aqm)
+{
+ unsigned long apid, apqi;
+
+ for_each_set_bit_inv(apid, apm, AP_DEVICES)
+ for_each_set_bit_inv(apqi, aqm, AP_DOMAINS)
+ pr_err(MDEV_SHARING_ERR, apid, apqi, mdev_name);
+}
+
/**
* vfio_ap_mdev_verify_no_sharing
*
* Verifies that the APQNs derived from the cross product of the AP adapter IDs
- * and AP queue indexes comprising the AP matrix are not configured for another
+ * and AP queue indexes comprising an AP matrix are not assigned to another
* mediated device. AP queue sharing is not allowed.
*
* @matrix_mdev: the mediated matrix device
+ * @mdev_apm: mask indicating the APIDs of the APQNs to be verified
+ * @mdev_aqm: mask indicating the APQIs of the APQNs to be verified
*
* Returns 0 if the APQNs are not shared, otherwise; returns -EADDRINUSE.
*/
-static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev)
+static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long *mdev_apm,
+ unsigned long *mdev_aqm)
{
struct ap_matrix_mdev *lstdev;
DECLARE_BITMAP(apm, AP_DEVICES);
@@ -543,14 +561,15 @@ static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev)
* We work on full longs, as we can only exclude the leftover
* bits in non-inverse order. The leftover is all zeros.
*/
- if (!bitmap_and(apm, matrix_mdev->matrix.apm,
- lstdev->matrix.apm, AP_DEVICES))
+ if (!bitmap_and(apm, mdev_apm, lstdev->matrix.apm, AP_DEVICES))
continue;
- if (!bitmap_and(aqm, matrix_mdev->matrix.aqm,
- lstdev->matrix.aqm, AP_DOMAINS))
+ if (!bitmap_and(aqm, mdev_aqm, lstdev->matrix.aqm, AP_DOMAINS))
continue;
+ vfio_ap_mdev_log_sharing_err(dev_name(mdev_dev(lstdev->mdev)),
+ apm, aqm);
+
return -EADDRINUSE;
}
@@ -676,6 +695,7 @@ static ssize_t assign_adapter_store(struct device *dev,
{
int ret;
unsigned long apid;
+ DECLARE_BITMAP(apm, AP_DEVICES);
struct mdev_device *mdev = mdev_from_dev(dev);
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
@@ -701,18 +721,18 @@ static ssize_t assign_adapter_store(struct device *dev,
if (ret)
goto done;
- set_bit_inv(apid, matrix_mdev->matrix.apm);
+ memset(apm, 0, sizeof(apm));
+ set_bit_inv(apid, apm);
- ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev);
+ ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev, apm,
+ matrix_mdev->matrix.aqm);
if (ret)
- goto share_err;
+ goto done;
+ set_bit_inv(apid, matrix_mdev->matrix.apm);
vfio_ap_mdev_link_queues(matrix_mdev, LINK_APID, apid);
ret = count;
- goto done;
-share_err:
- clear_bit_inv(apid, matrix_mdev->matrix.apm);
done:
mutex_unlock(&matrix_dev->lock);
@@ -824,6 +844,7 @@ static ssize_t assign_domain_store(struct device *dev,
{
int ret;
unsigned long apqi;
+ DECLARE_BITMAP(aqm, AP_DOMAINS);
struct mdev_device *mdev = mdev_from_dev(dev);
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
unsigned long max_apqi = matrix_mdev->matrix.aqm_max;
@@ -844,18 +865,18 @@ static ssize_t assign_domain_store(struct device *dev,
if (ret)
goto done;
- set_bit_inv(apqi, matrix_mdev->matrix.aqm);
+ memset(aqm, 0, sizeof(aqm));
+ set_bit_inv(apqi, aqm);
- ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev);
+ ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev,
+ matrix_mdev->matrix.apm, aqm);
if (ret)
- goto share_err;
+ goto done;
+ set_bit_inv(apqi, matrix_mdev->matrix.aqm);
vfio_ap_mdev_link_queues(matrix_mdev, LINK_APQI, apqi);
ret = count;
- goto done;
-share_err:
- clear_bit_inv(apqi, matrix_mdev->matrix.aqm);
done:
mutex_unlock(&matrix_dev->lock);
@@ -1434,3 +1455,14 @@ void vfio_ap_mdev_remove_queue(struct ap_queue *queue)
kfree(q);
mutex_unlock(&matrix_dev->lock);
}
+
+bool vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm)
+{
+ bool in_use;
+
+ mutex_lock(&matrix_dev->lock);
+ in_use = !!vfio_ap_mdev_verify_no_sharing(NULL, apm, aqm);
+ mutex_unlock(&matrix_dev->lock);
+
+ return in_use;
+}
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index 57da703b549a..0c796ef11426 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -105,4 +105,6 @@ struct vfio_ap_queue {
int vfio_ap_mdev_probe_queue(struct ap_queue *queue);
void vfio_ap_mdev_remove_queue(struct ap_queue *queue);
+bool vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm);
+
#endif /* _VFIO_AP_PRIVATE_H_ */
--
2.21.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v9 06/15] s390/vfio-ap: introduce shadow APCB
2020-07-20 15:03 [PATCH v9 00/15] s390/vfio-ap: dynamic configuration support Tony Krowiak
` (4 preceding siblings ...)
2020-07-20 15:03 ` [PATCH v9 05/15] s390/vfio-ap: implement in-use callback for vfio_ap driver Tony Krowiak
@ 2020-07-20 15:03 ` Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 07/15] s390/vfio-ap: sysfs attribute to display the guest's matrix Tony Krowiak
` (10 subsequent siblings)
16 siblings, 0 replies; 21+ messages in thread
From: Tony Krowiak @ 2020-07-20 15:03 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, borntraeger, cohuck, mjrosato, pasic, alex.williamson,
kwankhede, fiuczy, Tony Krowiak
The APCB is a field within the CRYCB that provides the AP configuration
to a KVM guest. Let's introduce a shadow copy of the KVM guest's APCB and
maintain it for the lifespan of the guest.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
---
drivers/s390/crypto/vfio_ap_ops.c | 32 ++++++++++++++++++++++-----
drivers/s390/crypto/vfio_ap_private.h | 2 ++
2 files changed, 29 insertions(+), 5 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index fc1aa6f947eb..efb229033f9e 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -305,14 +305,35 @@ static int handle_pqap(struct kvm_vcpu *vcpu)
return 0;
}
+static void vfio_ap_matrix_clear_masks(struct ap_matrix *matrix)
+{
+ bitmap_clear(matrix->apm, 0, AP_DEVICES);
+ bitmap_clear(matrix->aqm, 0, AP_DOMAINS);
+ bitmap_clear(matrix->adm, 0, AP_DOMAINS);
+}
+
static void vfio_ap_matrix_init(struct ap_config_info *info,
struct ap_matrix *matrix)
{
+ vfio_ap_matrix_clear_masks(matrix);
matrix->apm_max = info->apxa ? info->Na : 63;
matrix->aqm_max = info->apxa ? info->Nd : 15;
matrix->adm_max = info->apxa ? info->Nd : 15;
}
+static bool vfio_ap_mdev_has_crycb(struct ap_matrix_mdev *matrix_mdev)
+{
+ return (matrix_mdev->kvm && matrix_mdev->kvm->arch.crypto.crycbd);
+}
+
+static void vfio_ap_mdev_commit_crycb(struct ap_matrix_mdev *matrix_mdev)
+{
+ kvm_arch_crypto_set_masks(matrix_mdev->kvm,
+ matrix_mdev->shadow_apcb.apm,
+ matrix_mdev->shadow_apcb.aqm,
+ matrix_mdev->shadow_apcb.adm);
+}
+
static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev)
{
struct ap_matrix_mdev *matrix_mdev;
@@ -1202,13 +1223,12 @@ static int vfio_ap_mdev_group_notifier(struct notifier_block *nb,
if (ret)
return NOTIFY_DONE;
- /* If there is no CRYCB pointer, then we can't copy the masks */
- if (!matrix_mdev->kvm->arch.crypto.crycbd)
+ if (!vfio_ap_mdev_has_crycb(matrix_mdev))
return NOTIFY_DONE;
- kvm_arch_crypto_set_masks(matrix_mdev->kvm, matrix_mdev->matrix.apm,
- matrix_mdev->matrix.aqm,
- matrix_mdev->matrix.adm);
+ memcpy(&matrix_mdev->shadow_apcb, &matrix_mdev->matrix,
+ sizeof(matrix_mdev->shadow_apcb));
+ vfio_ap_mdev_commit_crycb(matrix_mdev);
return NOTIFY_OK;
}
@@ -1323,6 +1343,8 @@ static void vfio_ap_mdev_release(struct mdev_device *mdev)
kvm_put_kvm(matrix_mdev->kvm);
matrix_mdev->kvm = NULL;
}
+
+ vfio_ap_matrix_clear_masks(&matrix_mdev->shadow_apcb);
mutex_unlock(&matrix_dev->lock);
vfio_unregister_notifier(mdev_dev(mdev), VFIO_IOMMU_NOTIFY,
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index 0c796ef11426..055bce6d45db 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -75,6 +75,7 @@ struct ap_matrix {
* @list: allows the ap_matrix_mdev struct to be added to a list
* @matrix: the adapters, usage domains and control domains assigned to the
* mediated matrix device.
+ * @shadow_apcb: the shadow copy of the APCB field of the KVM guest's CRYCB
* @group_notifier: notifier block used for specifying callback function for
* handling the VFIO_GROUP_NOTIFY_SET_KVM event
* @kvm: the struct holding guest's state
@@ -82,6 +83,7 @@ struct ap_matrix {
struct ap_matrix_mdev {
struct list_head node;
struct ap_matrix matrix;
+ struct ap_matrix shadow_apcb;
struct notifier_block group_notifier;
struct notifier_block iommu_notifier;
struct kvm *kvm;
--
2.21.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v9 07/15] s390/vfio-ap: sysfs attribute to display the guest's matrix
2020-07-20 15:03 [PATCH v9 00/15] s390/vfio-ap: dynamic configuration support Tony Krowiak
` (5 preceding siblings ...)
2020-07-20 15:03 ` [PATCH v9 06/15] s390/vfio-ap: introduce shadow APCB Tony Krowiak
@ 2020-07-20 15:03 ` Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 08/15] s390/vfio-ap: filter matrix for unavailable queue devices Tony Krowiak
` (9 subsequent siblings)
16 siblings, 0 replies; 21+ messages in thread
From: Tony Krowiak @ 2020-07-20 15:03 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, borntraeger, cohuck, mjrosato, pasic, alex.williamson,
kwankhede, fiuczy, Tony Krowiak
The matrix of adapters and domains configured in a guest's CRYCB may
differ from the matrix of adapters and domains assigned to the matrix mdev,
so this patch introduces a sysfs attribute to display the matrix of a guest
using the matrix mdev. For a matrix mdev denoted by $uuid, the crycb for a
guest using the matrix mdev can be displayed as follows:
cat /sys/devices/vfio_ap/matrix/$uuid/guest_matrix
If a guest is not using the matrix mdev at the time the crycb is displayed,
an error (ENODEV) will be returned.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
---
drivers/s390/crypto/vfio_ap_ops.c | 58 +++++++++++++++++++++++++++++++
1 file changed, 58 insertions(+)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index efb229033f9e..30bf23734af6 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -1119,6 +1119,63 @@ static ssize_t matrix_show(struct device *dev, struct device_attribute *attr,
}
static DEVICE_ATTR_RO(matrix);
+static ssize_t guest_matrix_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ char *bufpos = buf;
+ unsigned long apid;
+ unsigned long apqi;
+ unsigned long apid1;
+ unsigned long apqi1;
+ unsigned long napm_bits = matrix_mdev->shadow_apcb.apm_max + 1;
+ unsigned long naqm_bits = matrix_mdev->shadow_apcb.aqm_max + 1;
+ int nchars = 0;
+ int n;
+
+ if (!vfio_ap_mdev_has_crycb(matrix_mdev))
+ return -ENODEV;
+
+ apid1 = find_first_bit_inv(matrix_mdev->shadow_apcb.apm, napm_bits);
+ apqi1 = find_first_bit_inv(matrix_mdev->shadow_apcb.aqm, naqm_bits);
+
+ mutex_lock(&matrix_dev->lock);
+
+ if ((apid1 < napm_bits) && (apqi1 < naqm_bits)) {
+ for_each_set_bit_inv(apid, matrix_mdev->shadow_apcb.apm,
+ napm_bits) {
+ for_each_set_bit_inv(apqi,
+ matrix_mdev->shadow_apcb.aqm,
+ naqm_bits) {
+ n = sprintf(bufpos, "%02lx.%04lx\n", apid,
+ apqi);
+ bufpos += n;
+ nchars += n;
+ }
+ }
+ } else if (apid1 < napm_bits) {
+ for_each_set_bit_inv(apid, matrix_mdev->shadow_apcb.apm,
+ napm_bits) {
+ n = sprintf(bufpos, "%02lx.\n", apid);
+ bufpos += n;
+ nchars += n;
+ }
+ } else if (apqi1 < naqm_bits) {
+ for_each_set_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm,
+ naqm_bits) {
+ n = sprintf(bufpos, ".%04lx\n", apqi);
+ bufpos += n;
+ nchars += n;
+ }
+ }
+
+ mutex_unlock(&matrix_dev->lock);
+
+ return nchars;
+}
+static DEVICE_ATTR_RO(guest_matrix);
+
static struct attribute *vfio_ap_mdev_attrs[] = {
&dev_attr_assign_adapter.attr,
&dev_attr_unassign_adapter.attr,
@@ -1128,6 +1185,7 @@ static struct attribute *vfio_ap_mdev_attrs[] = {
&dev_attr_unassign_control_domain.attr,
&dev_attr_control_domains.attr,
&dev_attr_matrix.attr,
+ &dev_attr_guest_matrix.attr,
NULL,
};
--
2.21.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v9 08/15] s390/vfio-ap: filter matrix for unavailable queue devices
2020-07-20 15:03 [PATCH v9 00/15] s390/vfio-ap: dynamic configuration support Tony Krowiak
` (6 preceding siblings ...)
2020-07-20 15:03 ` [PATCH v9 07/15] s390/vfio-ap: sysfs attribute to display the guest's matrix Tony Krowiak
@ 2020-07-20 15:03 ` Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 09/15] s390/vfio-ap: allow assignment of unavailable AP queues to mdev device Tony Krowiak
` (8 subsequent siblings)
16 siblings, 0 replies; 21+ messages in thread
From: Tony Krowiak @ 2020-07-20 15:03 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, borntraeger, cohuck, mjrosato, pasic, alex.williamson,
kwankhede, fiuczy, Tony Krowiak
Even though APQNs for queues that are not in the host's AP configuration
may be assigned to a matrix mdev, we do not want to set bits in the guest's
APCB for APQNs that do not reference AP queue devices bound to the vfio_ap
device driver. Ideally, it would be great if such APQNs could be filtered
out before setting the bits in the guest's APCB; however, the architecture
precludes filtering individual APQNs. Consequently, either the APID or APQI
must be filtered.
This patch introduces code to filter the APIDs or APQIs assigned to the
matrix mdev's AP configuration before assigning them to the guest's AP
configuration (i.e., APCB). We'll start by filtering the APIDs:
If an APQN assigned to the matrix mdev's AP configuration does not
reference a queue device bound to the vfio_ap device driver, the APID
will be filtered out (i.e., not assigned to the guest's APCB).
If every APID assigned to the matrix mdev is filtered out, then we'll try
filtering the APQI's:
If an APQN assigned to the matrix mdev's AP configuration does not
reference a queue device bound to the vfio_ap device driver, the APQI
will be filtered out (i.e., not assigned to the guest's APCB).
In any case, if after filtering either the APIDs or APQIs there are any
APQNs that can be assigned to the guest's APCB, they will be assigned and
the CRYCB will be hot plugged into the guest.
Example
=======
APQNs bound to vfio_ap device driver:
04.0004
04.0047
04.0054
05.0005
05.0047
05.0054
Assignments to matrix mdev:
APIDs APQIs -> APQNs
04 0004 04.0004
05 0005 04.0005
0047 04.0047
0054 04.0054
05.0004
05.0005
05.0047
04.0054
Filter APIDs:
APID 04 will be filtered because APQN 04.0005 is not bound.
APID 05 will be filtered because APQN 05.0004 is not bound.
APQNs remaining: None
Filter APQIs:
APQI 04 will be filtered because APQN 05.0004 is not bound.
APQI 05 will be filtered because APQN 04.0005 is not bound.
APQNs remaining: 04.0047, 04.0054, 05.0047, 05.0054
APQNs 04.0047, 04.0054, 05.0047, 05.0054 will be assigned to the CRYCB and
hot plugged into the KVM guest.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
---
drivers/s390/crypto/vfio_ap_ops.c | 159 +++++++++++++++++++++++++++++-
1 file changed, 155 insertions(+), 4 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 30bf23734af6..eaf4e9eab6cb 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -326,7 +326,7 @@ static bool vfio_ap_mdev_has_crycb(struct ap_matrix_mdev *matrix_mdev)
return (matrix_mdev->kvm && matrix_mdev->kvm->arch.crypto.crycbd);
}
-static void vfio_ap_mdev_commit_crycb(struct ap_matrix_mdev *matrix_mdev)
+static void vfio_ap_mdev_commit_shadow_apcb(struct ap_matrix_mdev *matrix_mdev)
{
kvm_arch_crypto_set_masks(matrix_mdev->kvm,
matrix_mdev->shadow_apcb.apm,
@@ -597,6 +597,157 @@ static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev,
return 0;
}
+/**
+ * vfio_ap_mdev_filter_matrix
+ *
+ * Filter APQNs assigned to the matrix mdev that do not reference an AP queue
+ * device bound to the vfio_ap device driver.
+ *
+ * @matrix_mdev: the matrix mdev whose AP configuration is to be filtered
+ * @shadow_apcb: the shadow of the KVM guest's APCB (contains AP configuration
+ * for guest)
+ * @filter_apids: boolean value indicating whether the APQNs shall be filtered
+ * by APID (true) or by APQI (false).
+ *
+ * Returns the number of APQNs remaining after filtering is complete.
+ */
+static int vfio_ap_mdev_filter_matrix(struct ap_matrix_mdev *matrix_mdev,
+ struct ap_matrix *shadow_apcb,
+ bool filter_apids)
+{
+ unsigned long apid, apqi, apqn;
+
+ memcpy(shadow_apcb, &matrix_mdev->matrix, sizeof(*shadow_apcb));
+
+ for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, AP_DEVICES) {
+ /*
+ * If the APID is not assigned to the host AP configuration,
+ * we can not assign it to the guest's AP configuration
+ */
+ if (!test_bit_inv(apid,
+ (unsigned long *)matrix_dev->info.apm)) {
+ clear_bit_inv(apid, shadow_apcb->apm);
+ continue;
+ }
+
+ for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm,
+ AP_DOMAINS) {
+ /*
+ * If the APQI is not assigned to the host AP
+ * configuration, then it can not be assigned to the
+ * guest's AP configuration
+ */
+ if (!test_bit_inv(apqi, (unsigned long *)
+ matrix_dev->info.aqm)) {
+ clear_bit_inv(apqi, shadow_apcb->aqm);
+ continue;
+ }
+
+ /*
+ * If the APQN is not bound to the vfio_ap device
+ * driver, then we can't assign it to the guest's
+ * AP configuration. The AP architecture won't
+ * allow filtering of a single APQN, so if we're
+ * filtering APIDs, then filter the APID; otherwise,
+ * filter the APQI.
+ */
+ apqn = AP_MKQID(apid, apqi);
+ if (!vfio_ap_get_queue(apqn)) {
+ if (filter_apids)
+ clear_bit_inv(apid, shadow_apcb->apm);
+ else
+ clear_bit_inv(apqi, shadow_apcb->aqm);
+ break;
+ }
+ }
+
+ /*
+ * If we're filtering APQIs and all of them have been filtered,
+ * there's no need to continue filtering.
+ */
+ if (!filter_apids)
+ if (bitmap_empty(shadow_apcb->aqm, AP_DOMAINS))
+ break;
+ }
+
+ return bitmap_weight(shadow_apcb->apm, AP_DEVICES) *
+ bitmap_weight(shadow_apcb->aqm, AP_DOMAINS);
+}
+
+/**
+ * vfio_ap_mdev_config_shadow_apcb
+ *
+ * Configure the shadow of a KVM guest's APCB specifying the adapters, domains
+ * and control domains to be assigned to the guest. The shadow APCB will be
+ * configured after filtering the APQNs assigned to the matrix mdev that do not
+ * reference a queue device bound to the vfio_ap device driver.
+ *
+ * @matrix_mdev: the matrix mdev whose shadow APCB is to be configured.
+ *
+ * Returns true if the shadow APCB contents have been changed; otherwise,
+ * returns false.
+ */
+static bool vfio_ap_mdev_config_shadow_apcb(struct ap_matrix_mdev *matrix_mdev)
+{
+ int napm, naqm;
+ struct ap_matrix shadow_apcb;
+
+ vfio_ap_matrix_init(&matrix_dev->info, &shadow_apcb);
+ napm = bitmap_weight(matrix_mdev->matrix.apm, AP_DEVICES);
+ naqm = bitmap_weight(matrix_mdev->matrix.aqm, AP_DOMAINS);
+
+ /*
+ * If there are no APIDs or no APQIs assigned to the matrix mdev,
+ * then no APQNs shall be assigned to the guest CRYCB.
+ */
+ if ((napm != 0) || (naqm != 0)) {
+ /*
+ * Filter the APIDs assigned to the matrix mdev for APQNs that
+ * do not reference an AP queue device bound to the driver.
+ */
+ napm = vfio_ap_mdev_filter_matrix(matrix_mdev, &shadow_apcb,
+ true);
+ /*
+ * If there are no APQNs that can be assigned to the guest's
+ * CRYCB after filtering, then try filtering the APQIs.
+ */
+ if (napm == 0) {
+ naqm = vfio_ap_mdev_filter_matrix(matrix_mdev,
+ &shadow_apcb, false);
+
+ /*
+ * If there are no APQNs that can be assigned to the
+ * matrix mdev after filtering the APQIs, then no APQNs
+ * shall be assigned to the guest's CRYCB.
+ */
+ if (naqm == 0) {
+ bitmap_clear(shadow_apcb.apm, 0, AP_DEVICES);
+ bitmap_clear(shadow_apcb.aqm, 0, AP_DOMAINS);
+ }
+ }
+ }
+
+ /*
+ * If the guest's AP configuration has not changed, then return
+ * indicating such.
+ */
+ if (bitmap_equal(matrix_mdev->shadow_apcb.apm, shadow_apcb.apm,
+ AP_DEVICES) &&
+ bitmap_equal(matrix_mdev->shadow_apcb.aqm, shadow_apcb.aqm,
+ AP_DOMAINS) &&
+ bitmap_equal(matrix_mdev->shadow_apcb.adm, shadow_apcb.adm,
+ AP_DOMAINS))
+ return false;
+
+ /*
+ * Copy the changes to the guest's CRYCB, then return indicating that
+ * the guest's AP configuration has changed.
+ */
+ memcpy(&matrix_mdev->shadow_apcb, &shadow_apcb, sizeof(shadow_apcb));
+
+ return true;
+}
+
enum qlink_type {
LINK_APID,
LINK_APQI,
@@ -1284,9 +1435,8 @@ static int vfio_ap_mdev_group_notifier(struct notifier_block *nb,
if (!vfio_ap_mdev_has_crycb(matrix_mdev))
return NOTIFY_DONE;
- memcpy(&matrix_mdev->shadow_apcb, &matrix_mdev->matrix,
- sizeof(matrix_mdev->shadow_apcb));
- vfio_ap_mdev_commit_crycb(matrix_mdev);
+ if (vfio_ap_mdev_config_shadow_apcb(matrix_mdev))
+ vfio_ap_mdev_commit_shadow_apcb(matrix_mdev);
return NOTIFY_OK;
}
@@ -1396,6 +1546,7 @@ static void vfio_ap_mdev_release(struct mdev_device *mdev)
mutex_lock(&matrix_dev->lock);
if (matrix_mdev->kvm) {
kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
+ vfio_ap_matrix_clear_masks(&matrix_mdev->shadow_apcb);
matrix_mdev->kvm->arch.crypto.pqap_hook = NULL;
vfio_ap_mdev_reset_queues(mdev);
kvm_put_kvm(matrix_mdev->kvm);
--
2.21.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v9 09/15] s390/vfio-ap: allow assignment of unavailable AP queues to mdev device
2020-07-20 15:03 [PATCH v9 00/15] s390/vfio-ap: dynamic configuration support Tony Krowiak
` (7 preceding siblings ...)
2020-07-20 15:03 ` [PATCH v9 08/15] s390/vfio-ap: filter matrix for unavailable queue devices Tony Krowiak
@ 2020-07-20 15:03 ` Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 10/15] s390/vfio-ap: allow configuration of matrix mdev in use by a KVM guest Tony Krowiak
` (7 subsequent siblings)
16 siblings, 0 replies; 21+ messages in thread
From: Tony Krowiak @ 2020-07-20 15:03 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, borntraeger, cohuck, mjrosato, pasic, alex.williamson,
kwankhede, fiuczy, Tony Krowiak
The current implementation does not allow assignment of an AP adapter or
domain to an mdev device if the APQNs resulting from the assignment
do not reference AP queue devices that are bound to the vfio_ap device
driver. This patch allows assignment of AP resources to the matrix mdev as
long as the APQNs resulting from the assignment:
1. Are not reserved by the AP BUS for use by the zcrypt device drivers.
2. Are not assigned to another matrix mdev.
The rationale behind this is twofold:
1. The AP architecture does not preclude assignment of APQNs to an AP
configuration that are not available to the system.
2. APQNs that do not reference a queue device bound to the vfio_ap
device driver will not be assigned to the guest's CRYCB, so the
guest will not get access to queues not bound to the vfio_ap driver.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
---
drivers/s390/crypto/vfio_ap_ops.c | 212 +++++-------------------------
1 file changed, 35 insertions(+), 177 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index eaf4e9eab6cb..24fd47e43b80 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -1,4 +1,3 @@
-// SPDX-License-Identifier: GPL-2.0+
/*
* Adjunct processor matrix VFIO device driver callbacks.
*
@@ -420,122 +419,6 @@ static struct attribute_group *vfio_ap_mdev_type_groups[] = {
NULL,
};
-struct vfio_ap_queue_reserved {
- unsigned long *apid;
- unsigned long *apqi;
- bool reserved;
-};
-
-/**
- * vfio_ap_has_queue
- *
- * @dev: an AP queue device
- * @data: a struct vfio_ap_queue_reserved reference
- *
- * Flags whether the AP queue device (@dev) has a queue ID containing the APQN,
- * apid or apqi specified in @data:
- *
- * - If @data contains both an apid and apqi value, then @data will be flagged
- * as reserved if the APID and APQI fields for the AP queue device matches
- *
- * - If @data contains only an apid value, @data will be flagged as
- * reserved if the APID field in the AP queue device matches
- *
- * - If @data contains only an apqi value, @data will be flagged as
- * reserved if the APQI field in the AP queue device matches
- *
- * Returns 0 to indicate the input to function succeeded. Returns -EINVAL if
- * @data does not contain either an apid or apqi.
- */
-static int vfio_ap_has_queue(struct device *dev, void *data)
-{
- struct vfio_ap_queue_reserved *qres = data;
- struct ap_queue *ap_queue = to_ap_queue(dev);
- ap_qid_t qid;
- unsigned long id;
-
- if (qres->apid && qres->apqi) {
- qid = AP_MKQID(*qres->apid, *qres->apqi);
- if (qid == ap_queue->qid)
- qres->reserved = true;
- } else if (qres->apid && !qres->apqi) {
- id = AP_QID_CARD(ap_queue->qid);
- if (id == *qres->apid)
- qres->reserved = true;
- } else if (!qres->apid && qres->apqi) {
- id = AP_QID_QUEUE(ap_queue->qid);
- if (id == *qres->apqi)
- qres->reserved = true;
- } else {
- return -EINVAL;
- }
-
- return 0;
-}
-
-/**
- * vfio_ap_verify_queue_reserved
- *
- * @matrix_dev: a mediated matrix device
- * @apid: an AP adapter ID
- * @apqi: an AP queue index
- *
- * Verifies that the AP queue with @apid/@apqi is reserved by the VFIO AP device
- * driver according to the following rules:
- *
- * - If both @apid and @apqi are not NULL, then there must be an AP queue
- * device bound to the vfio_ap driver with the APQN identified by @apid and
- * @apqi
- *
- * - If only @apid is not NULL, then there must be an AP queue device bound
- * to the vfio_ap driver with an APQN containing @apid
- *
- * - If only @apqi is not NULL, then there must be an AP queue device bound
- * to the vfio_ap driver with an APQN containing @apqi
- *
- * Returns 0 if the AP queue is reserved; otherwise, returns -EADDRNOTAVAIL.
- */
-static int vfio_ap_verify_queue_reserved(unsigned long *apid,
- unsigned long *apqi)
-{
- int ret;
- struct vfio_ap_queue_reserved qres;
-
- qres.apid = apid;
- qres.apqi = apqi;
- qres.reserved = false;
-
- ret = driver_for_each_device(&matrix_dev->vfio_ap_drv->driver, NULL,
- &qres, vfio_ap_has_queue);
- if (ret)
- return ret;
-
- if (qres.reserved)
- return 0;
-
- return -EADDRNOTAVAIL;
-}
-
-static int
-vfio_ap_mdev_verify_queues_reserved_for_apid(struct ap_matrix_mdev *matrix_mdev,
- unsigned long apid)
-{
- int ret;
- unsigned long apqi;
- unsigned long nbits = matrix_mdev->matrix.aqm_max + 1;
-
- if (find_first_bit_inv(matrix_mdev->matrix.aqm, nbits) >= nbits)
- return vfio_ap_verify_queue_reserved(&apid, NULL);
-
- for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, nbits) {
- ret = vfio_ap_verify_queue_reserved(&apid, &apqi);
- if (ret)
- return ret;
- }
-
- return 0;
-}
-
#define MDEV_SHARING_ERR "Userspace may not re-assign queue %02lx.%04lx " \
"already assigned to %s"
@@ -572,6 +455,11 @@ static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev,
DECLARE_BITMAP(aqm, AP_DOMAINS);
list_for_each_entry(lstdev, &matrix_dev->mdev_list, node) {
+ /*
+ * If either of the input masks belongs to the mdev to which an
+ * AP resource is being assigned, then we don't need to verify
+ * that mdev's masks.
+ */
if (matrix_mdev == lstdev)
continue;
@@ -597,6 +485,20 @@ static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev,
return 0;
}
+static int vfio_ap_mdev_validate_masks(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long *mdev_apm,
+ unsigned long *mdev_aqm)
+{
+ DECLARE_BITMAP(apm, AP_DEVICES);
+ DECLARE_BITMAP(aqm, AP_DOMAINS);
+
+ if (bitmap_and(apm, mdev_apm, ap_perms.apm, AP_DEVICES) &&
+ bitmap_and(aqm, mdev_aqm, ap_perms.aqm, AP_DOMAINS))
+ return -EADDRNOTAVAIL;
+
+ return vfio_ap_mdev_verify_no_sharing(matrix_mdev, mdev_apm, mdev_aqm);
+}
+
/**
* vfio_ap_mdev_filter_matrix
*
@@ -882,33 +784,21 @@ static ssize_t assign_adapter_store(struct device *dev,
if (apid > matrix_mdev->matrix.apm_max)
return -ENODEV;
- /*
- * Set the bit in the AP mask (APM) corresponding to the AP adapter
- * number (APID). The bits in the mask, from most significant to least
- * significant bit, correspond to APIDs 0-255.
- */
- mutex_lock(&matrix_dev->lock);
-
- ret = vfio_ap_mdev_verify_queues_reserved_for_apid(matrix_mdev, apid);
- if (ret)
- goto done;
-
memset(apm, 0, sizeof(apm));
set_bit_inv(apid, apm);
- ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev, apm,
- matrix_mdev->matrix.aqm);
- if (ret)
- goto done;
-
+ mutex_lock(&matrix_dev->lock);
+ ret = vfio_ap_mdev_validate_masks(matrix_mdev, apm,
+ matrix_mdev->matrix.aqm);
+ if (ret) {
+ mutex_unlock(&matrix_dev->lock);
+ return ret;
+ }
set_bit_inv(apid, matrix_mdev->matrix.apm);
vfio_ap_mdev_link_queues(matrix_mdev, LINK_APID, apid);
- ret = count;
-
-done:
mutex_unlock(&matrix_dev->lock);
- return ret;
+ return count;
}
static DEVICE_ATTR_WO(assign_adapter);
@@ -958,26 +848,6 @@ static ssize_t unassign_adapter_store(struct device *dev,
}
static DEVICE_ATTR_WO(unassign_adapter);
-static int
-vfio_ap_mdev_verify_queues_reserved_for_apqi(struct ap_matrix_mdev *matrix_mdev,
- unsigned long apqi)
-{
- int ret;
- unsigned long apid;
- unsigned long nbits = matrix_mdev->matrix.apm_max + 1;
-
- if (find_first_bit_inv(matrix_mdev->matrix.apm, nbits) >= nbits)
- return vfio_ap_verify_queue_reserved(NULL, &apqi);
-
- for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, nbits) {
- ret = vfio_ap_verify_queue_reserved(&apid, &apqi);
- if (ret)
- return ret;
- }
-
- return 0;
-}
-
/**
* assign_domain_store
*
@@ -1031,28 +901,21 @@ static ssize_t assign_domain_store(struct device *dev,
if (apqi > max_apqi)
return -ENODEV;
- mutex_lock(&matrix_dev->lock);
-
- ret = vfio_ap_mdev_verify_queues_reserved_for_apqi(matrix_mdev, apqi);
- if (ret)
- goto done;
-
memset(aqm, 0, sizeof(aqm));
set_bit_inv(apqi, aqm);
- ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev,
- matrix_mdev->matrix.apm, aqm);
- if (ret)
- goto done;
-
+ mutex_lock(&matrix_dev->lock);
+ ret = vfio_ap_mdev_validate_masks(matrix_mdev, matrix_mdev->matrix.apm,
+ aqm);
+ if (ret) {
+ mutex_unlock(&matrix_dev->lock);
+ return ret;
+ }
set_bit_inv(apqi, matrix_mdev->matrix.aqm);
vfio_ap_mdev_link_queues(matrix_mdev, LINK_APQI, apqi);
- ret = count;
-
-done:
mutex_unlock(&matrix_dev->lock);
- return ret;
+ return count;
}
static DEVICE_ATTR_WO(assign_domain);
@@ -1139,11 +1002,6 @@ static ssize_t assign_control_domain_store(struct device *dev,
if (id > matrix_mdev->matrix.adm_max)
return -ENODEV;
- /* Set the bit in the ADM (bitmask) corresponding to the AP control
- * domain number (id). The bits in the mask, from most significant to
- * least significant, correspond to IDs 0 up to the one less than the
- * number of control domains that can be assigned.
- */
mutex_lock(&matrix_dev->lock);
set_bit_inv(id, matrix_mdev->matrix.adm);
mutex_unlock(&matrix_dev->lock);
--
2.21.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v9 10/15] s390/vfio-ap: allow configuration of matrix mdev in use by a KVM guest
2020-07-20 15:03 [PATCH v9 00/15] s390/vfio-ap: dynamic configuration support Tony Krowiak
` (8 preceding siblings ...)
2020-07-20 15:03 ` [PATCH v9 09/15] s390/vfio-ap: allow assignment of unavailable AP queues to mdev device Tony Krowiak
@ 2020-07-20 15:03 ` Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 11/15] s390/vfio-ap: allow hot plug/unplug of AP resources using mdev device Tony Krowiak
` (6 subsequent siblings)
16 siblings, 0 replies; 21+ messages in thread
From: Tony Krowiak @ 2020-07-20 15:03 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, borntraeger, cohuck, mjrosato, pasic, alex.williamson,
kwankhede, fiuczy, Tony Krowiak
The current support for pass-through crypto adapters does not allow
configuration of a matrix mdev when it is in use by a KVM guest. Let's
allow AP resources - i.e., adapters, domains and control domains - to be
assigned to or unassigned from a matrix mdev while it is in use by a guest.
This is in preparation for the introduction of support for dynamic
configuration of the AP matrix for a running KVM guest.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
---
drivers/s390/crypto/vfio_ap_ops.c | 24 ------------------------
1 file changed, 24 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 24fd47e43b80..cf3321eb239b 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -773,10 +773,6 @@ static ssize_t assign_adapter_store(struct device *dev,
struct mdev_device *mdev = mdev_from_dev(dev);
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
- /* If the guest is running, disallow assignment of adapter */
- if (matrix_mdev->kvm)
- return -EBUSY;
-
ret = kstrtoul(buf, 0, &apid);
if (ret)
return ret;
@@ -828,10 +824,6 @@ static ssize_t unassign_adapter_store(struct device *dev,
struct mdev_device *mdev = mdev_from_dev(dev);
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
- /* If the guest is running, disallow un-assignment of adapter */
- if (matrix_mdev->kvm)
- return -EBUSY;
-
ret = kstrtoul(buf, 0, &apid);
if (ret)
return ret;
@@ -891,10 +883,6 @@ static ssize_t assign_domain_store(struct device *dev,
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
unsigned long max_apqi = matrix_mdev->matrix.aqm_max;
- /* If the guest is running, disallow assignment of domain */
- if (matrix_mdev->kvm)
- return -EBUSY;
-
ret = kstrtoul(buf, 0, &apqi);
if (ret)
return ret;
@@ -946,10 +934,6 @@ static ssize_t unassign_domain_store(struct device *dev,
struct mdev_device *mdev = mdev_from_dev(dev);
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
- /* If the guest is running, disallow un-assignment of domain */
- if (matrix_mdev->kvm)
- return -EBUSY;
-
ret = kstrtoul(buf, 0, &apqi);
if (ret)
return ret;
@@ -991,10 +975,6 @@ static ssize_t assign_control_domain_store(struct device *dev,
struct mdev_device *mdev = mdev_from_dev(dev);
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
- /* If the guest is running, disallow assignment of control domain */
- if (matrix_mdev->kvm)
- return -EBUSY;
-
ret = kstrtoul(buf, 0, &id);
if (ret)
return ret;
@@ -1036,10 +1016,6 @@ static ssize_t unassign_control_domain_store(struct device *dev,
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
unsigned long max_domid = matrix_mdev->matrix.adm_max;
- /* If the guest is running, disallow un-assignment of control domain */
- if (matrix_mdev->kvm)
- return -EBUSY;
-
ret = kstrtoul(buf, 0, &domid);
if (ret)
return ret;
--
2.21.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v9 11/15] s390/vfio-ap: allow hot plug/unplug of AP resources using mdev device
2020-07-20 15:03 [PATCH v9 00/15] s390/vfio-ap: dynamic configuration support Tony Krowiak
` (9 preceding siblings ...)
2020-07-20 15:03 ` [PATCH v9 10/15] s390/vfio-ap: allow configuration of matrix mdev in use by a KVM guest Tony Krowiak
@ 2020-07-20 15:03 ` Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 12/15] s390/zcrypt: Notify driver on config changed and scan complete callbacks Tony Krowiak
` (5 subsequent siblings)
16 siblings, 0 replies; 21+ messages in thread
From: Tony Krowiak @ 2020-07-20 15:03 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, borntraeger, cohuck, mjrosato, pasic, alex.williamson,
kwankhede, fiuczy, Tony Krowiak
Let's hot plug/unplug adapters, domains and control domains assigned to or
unassigned from an AP matrix mdev device while it is in use by a guest per
the following:
* When the APID of an adapter is assigned to a matrix mdev in use by a KVM
guest, the adapter will be hot plugged into the KVM guest as long as each
APQN derived from the Cartesian product of the APID being assigned and
the APQIs already assigned to the guest's CRYCB references a queue device
bound to the vfio_ap device driver.
* When the APID of an adapter is unassigned from a matrix mdev in use by a
KVM guest, the adapter will be hot unplugged from the KVM guest.
* When the APQI of a domain is assigned to a matrix mdev in use by a KVM
guest, the domain will be hot plugged into the KVM guest as long as each
APQN derived from the Cartesian product of the APQI being assigned and
the APIDs already assigned to the guest's CRYCB references a queue device
bound to the vfio_ap device driver.
* When the APQI of a domain is unassigned from a matrix mdev in use by a
KVM guest, the domain will be hot unplugged from the KVM guest
* When the domain number of a control domain is assigned to a matrix mdev
in use by a KVM guest, the control domain will be hot plugged into the
KVM guest.
* When the domain number of a control domain is unassigned from a matrix
mdev in use by a KVM guest, the control domain will be hot unplugged
from the KVM guest.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
---
drivers/s390/crypto/vfio_ap_ops.c | 196 ++++++++++++++++++++++++++++++
1 file changed, 196 insertions(+)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index cf3321eb239b..2b01a8eb6ee7 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -731,6 +731,56 @@ static void vfio_ap_mdev_link_queues(struct ap_matrix_mdev *matrix_mdev,
}
}
+static bool vfio_ap_mdev_assign_apqis_4_apid(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long apid)
+{
+ DECLARE_BITMAP(aqm, AP_DOMAINS);
+ unsigned long apqi, apqn;
+
+ bitmap_copy(aqm, matrix_mdev->matrix.aqm, AP_DOMAINS);
+
+ for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, AP_DOMAINS) {
+ if (!test_bit_inv(apqi,
+ (unsigned long *) matrix_dev->info.aqm))
+ clear_bit_inv(apqi, aqm);
+
+ apqn = AP_MKQID(apid, apqi);
+ if (!vfio_ap_get_mdev_queue(matrix_mdev, apqn))
+ clear_bit_inv(apqi, aqm);
+ }
+
+ if (bitmap_empty(aqm, AP_DOMAINS))
+ return false;
+
+ set_bit_inv(apid, matrix_mdev->shadow_apcb.apm);
+ bitmap_copy(matrix_mdev->shadow_apcb.aqm, aqm, AP_DOMAINS);
+
+ return true;
+}
+
+static bool vfio_ap_mdev_assign_guest_apid(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long apid)
+{
+ unsigned long apqi, apqn;
+
+ if (!vfio_ap_mdev_has_crycb(matrix_mdev) ||
+ !test_bit_inv(apid, (unsigned long *)matrix_dev->info.apm))
+ return false;
+
+ if (bitmap_empty(matrix_mdev->shadow_apcb.aqm, AP_DOMAINS))
+ return vfio_ap_mdev_assign_apqis_4_apid(matrix_mdev, apid);
+
+ for_each_set_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm, AP_DOMAINS) {
+ apqn = AP_MKQID(apid, apqi);
+ if (!vfio_ap_get_mdev_queue(matrix_mdev, apqn))
+ return false;
+ }
+
+ set_bit_inv(apid, matrix_mdev->shadow_apcb.apm);
+
+ return true;
+}
+
/**
* assign_adapter_store
*
@@ -792,12 +842,42 @@ static ssize_t assign_adapter_store(struct device *dev,
}
set_bit_inv(apid, matrix_mdev->matrix.apm);
vfio_ap_mdev_link_queues(matrix_mdev, LINK_APID, apid);
+ if (vfio_ap_mdev_assign_guest_apid(matrix_mdev, apid))
+ vfio_ap_mdev_commit_shadow_apcb(matrix_mdev);
mutex_unlock(&matrix_dev->lock);
return count;
}
static DEVICE_ATTR_WO(assign_adapter);
+static bool vfio_ap_mdev_unassign_guest_apid(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long apid)
+{
+ if (vfio_ap_mdev_has_crycb(matrix_mdev)) {
+ if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm)) {
+ clear_bit_inv(apid, matrix_mdev->shadow_apcb.apm);
+
+ /*
+ * If there are no APIDs assigned to the guest, then
+ * the guest will not have access to any queues, so
+ * let's also go ahead and unassign the APQIs. Keeping
+ * them around may yield unpredictable results during
+ * a probe that is not related to a host AP
+ * configuration change (i.e., an AP adapter is
+ * configured online).
+ */
+ if (bitmap_empty(matrix_mdev->shadow_apcb.apm,
+ AP_DEVICES))
+ bitmap_clear(matrix_mdev->shadow_apcb.aqm, 0,
+ AP_DOMAINS);
+
+ return true;
+ }
+ }
+
+ return false;
+}
+
/**
* unassign_adapter_store
*
@@ -834,12 +914,64 @@ static ssize_t unassign_adapter_store(struct device *dev,
mutex_lock(&matrix_dev->lock);
clear_bit_inv((unsigned long)apid, matrix_mdev->matrix.apm);
vfio_ap_mdev_link_queues(matrix_mdev, UNLINK_APID, apid);
+ if (vfio_ap_mdev_unassign_guest_apid(matrix_mdev, apid))
+ vfio_ap_mdev_commit_shadow_apcb(matrix_mdev);
mutex_unlock(&matrix_dev->lock);
return count;
}
static DEVICE_ATTR_WO(unassign_adapter);
+static bool vfio_ap_mdev_assign_apids_4_apqi(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long apqi)
+{
+ DECLARE_BITMAP(apm, AP_DEVICES);
+ unsigned long apid, apqn;
+
+ bitmap_copy(apm, matrix_mdev->matrix.apm, AP_DEVICES);
+
+ for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, AP_DEVICES) {
+ if (!test_bit_inv(apid,
+ (unsigned long *) matrix_dev->info.apm))
+ clear_bit_inv(apqi, apm);
+
+ apqn = AP_MKQID(apid, apqi);
+ if (!vfio_ap_get_mdev_queue(matrix_mdev, apqn))
+ clear_bit_inv(apid, apm);
+ }
+
+ if (bitmap_empty(apm, AP_DEVICES))
+ return false;
+
+ set_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm);
+ bitmap_copy(matrix_mdev->shadow_apcb.apm, apm, AP_DEVICES);
+
+ return true;
+}
+
+static bool vfio_ap_mdev_assign_guest_apqi(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long apqi)
+{
+ unsigned long apid, apqn;
+
+ if (!vfio_ap_mdev_has_crycb(matrix_mdev) ||
+ !test_bit_inv(apqi, (unsigned long *)matrix_dev->info.aqm))
+ return false;
+
+ if (bitmap_empty(matrix_mdev->shadow_apcb.apm, AP_DEVICES))
+ return vfio_ap_mdev_assign_apids_4_apqi(matrix_mdev, apqi);
+
+ for_each_set_bit_inv(apid, matrix_mdev->shadow_apcb.apm, AP_DEVICES) {
+ apqn = AP_MKQID(apid, apqi);
+ if (!vfio_ap_get_mdev_queue(matrix_mdev, apqn))
+ return false;
+ }
+
+ set_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm);
+
+ return true;
+}
+
/**
* assign_domain_store
*
@@ -901,12 +1033,41 @@ static ssize_t assign_domain_store(struct device *dev,
}
set_bit_inv(apqi, matrix_mdev->matrix.aqm);
vfio_ap_mdev_link_queues(matrix_mdev, LINK_APQI, apqi);
+ if (vfio_ap_mdev_assign_guest_apqi(matrix_mdev, apqi))
+ vfio_ap_mdev_commit_shadow_apcb(matrix_mdev);
mutex_unlock(&matrix_dev->lock);
return count;
}
static DEVICE_ATTR_WO(assign_domain);
+static bool vfio_ap_mdev_unassign_guest_apqi(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long apqi)
+{
+ if (vfio_ap_mdev_has_crycb(matrix_mdev)) {
+ if (test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm)) {
+ clear_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm);
+
+ /*
+ * If there are no APQIs assigned to the guest, then
+ * the guest will not have access to any queues, so
+ * let's also go ahead and unassign the APIDs. Keeping
+ * them around may yield unpredictable results during
+ * a probe that is not related to a host AP
+ * configuration change (i.e., an AP adapter is
+ * configured online).
+ */
+ if (bitmap_empty(matrix_mdev->shadow_apcb.aqm,
+ AP_DOMAINS))
+ bitmap_clear(matrix_mdev->shadow_apcb.apm, 0,
+ AP_DEVICES);
+
+ return true;
+ }
+ }
+
+ return false;
+}
/**
* unassign_domain_store
@@ -944,12 +1105,28 @@ static ssize_t unassign_domain_store(struct device *dev,
mutex_lock(&matrix_dev->lock);
clear_bit_inv((unsigned long)apqi, matrix_mdev->matrix.aqm);
vfio_ap_mdev_link_queues(matrix_mdev, UNLINK_APQI, apqi);
+ if (vfio_ap_mdev_unassign_guest_apqi(matrix_mdev, apqi))
+ vfio_ap_mdev_commit_shadow_apcb(matrix_mdev);
mutex_unlock(&matrix_dev->lock);
return count;
}
static DEVICE_ATTR_WO(unassign_domain);
+static bool vfio_ap_mdev_assign_guest_cdom(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long domid)
+{
+ if (vfio_ap_mdev_has_crycb(matrix_mdev)) {
+ if (!test_bit_inv(domid, matrix_mdev->shadow_apcb.adm)) {
+ set_bit_inv(domid, matrix_mdev->shadow_apcb.adm);
+
+ return true;
+ }
+ }
+
+ return false;
+}
+
/**
* assign_control_domain_store
*
@@ -984,12 +1161,29 @@ static ssize_t assign_control_domain_store(struct device *dev,
mutex_lock(&matrix_dev->lock);
set_bit_inv(id, matrix_mdev->matrix.adm);
+ if (vfio_ap_mdev_assign_guest_cdom(matrix_mdev, id))
+ vfio_ap_mdev_commit_shadow_apcb(matrix_mdev);
mutex_unlock(&matrix_dev->lock);
return count;
}
static DEVICE_ATTR_WO(assign_control_domain);
+static bool
+vfio_ap_mdev_unassign_guest_cdom(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long domid)
+{
+ if (vfio_ap_mdev_has_crycb(matrix_mdev)) {
+ if (test_bit_inv(domid, matrix_mdev->shadow_apcb.adm)) {
+ clear_bit_inv(domid, matrix_mdev->shadow_apcb.adm);
+
+ return true;
+ }
+ }
+
+ return false;
+}
+
/**
* unassign_control_domain_store
*
@@ -1024,6 +1218,8 @@ static ssize_t unassign_control_domain_store(struct device *dev,
mutex_lock(&matrix_dev->lock);
clear_bit_inv(domid, matrix_mdev->matrix.adm);
+ if (vfio_ap_mdev_unassign_guest_cdom(matrix_mdev, domid))
+ vfio_ap_mdev_commit_shadow_apcb(matrix_mdev);
mutex_unlock(&matrix_dev->lock);
return count;
--
2.21.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v9 12/15] s390/zcrypt: Notify driver on config changed and scan complete callbacks
2020-07-20 15:03 [PATCH v9 00/15] s390/vfio-ap: dynamic configuration support Tony Krowiak
` (10 preceding siblings ...)
2020-07-20 15:03 ` [PATCH v9 11/15] s390/vfio-ap: allow hot plug/unplug of AP resources using mdev device Tony Krowiak
@ 2020-07-20 15:03 ` Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 13/15] s390/vfio-ap: handle host AP config change notification Tony Krowiak
` (4 subsequent siblings)
16 siblings, 0 replies; 21+ messages in thread
From: Tony Krowiak @ 2020-07-20 15:03 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, borntraeger, cohuck, mjrosato, pasic, alex.williamson,
kwankhede, fiuczy, Tony Krowiak
From: Harald Freudenberger <freude@linux.ibm.com>
This patch intruduces an extension to the ap bus to notify drivers
on crypto config changed and bus scan complete events.
Two new callbacks are introduced for ap_drivers:
void (*on_config_changed)(struct ap_config_info *new_config_info,
struct ap_config_info *old_config_info);
void (*on_scan_complete)(struct ap_config_info *new_config_info,
struct ap_config_info *old_config_info);
Both callbacks are optional. Both callbacks are only triggered
when QCI information is available (facility bit 12):
* The on_config_changed callback is invoked at the start of the AP bus scan
function when it determines that the host AP configuration information
has changed since the previous scan. This is done by storing
an old and current QCI info struct and comparing them. If there is any
difference, the callback is invoked.
Note that when the AP bus scan detects that AP adapters or domains have
been removed from the host's AP configuration, it will remove the
associated devices from the AP bus subsystem's device model. This
callback gives the device driver a chance to respond to the removal
of the AP devices in bulk rather than one at a time as its remove
callback is invoked. It will also allow the device driver to do any
any cleanup prior to giving control back to the bus piecemeal. This is
particularly important for the vfio_ap driver because there may be
guests using the queues at the time.
* The on_scan_complete callback is invoked after the ap bus scan is
complete if the host AP configuration data has changed.
Note that when the AP bus scan detects that adapters or domains have
been added to the host's configuration, it will create new devices in
the AP bus subsystem's device model. This callback also allows the driver
to process all of the new devices in bulk.
Please note that changes to the apmask and aqmask do not trigger
these two callbacks since the bus scan function is not invoked by changes
to those masks.
Signed-off-by: Harald Freudenberger <freude@linux.ibm.com>
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
---
drivers/s390/crypto/ap_bus.c | 175 ++++++++++++++++++++++++++---------
drivers/s390/crypto/ap_bus.h | 12 +++
2 files changed, 142 insertions(+), 45 deletions(-)
diff --git a/drivers/s390/crypto/ap_bus.c b/drivers/s390/crypto/ap_bus.c
index d6a732c8d90a..282002d39389 100644
--- a/drivers/s390/crypto/ap_bus.c
+++ b/drivers/s390/crypto/ap_bus.c
@@ -73,8 +73,12 @@ struct ap_perms ap_perms;
EXPORT_SYMBOL(ap_perms);
DEFINE_MUTEX(ap_perms_mutex);
EXPORT_SYMBOL(ap_perms_mutex);
+DEFINE_MUTEX(ap_config_lock);
+
+/* current and old qci info structs */
+static struct ap_config_info *ap_config_info;
+static struct ap_config_info *ap_old_config_info;
-static struct ap_config_info *ap_configuration;
static bool initialised;
/*
@@ -183,8 +187,8 @@ static int ap_apft_available(void)
*/
static inline int ap_qact_available(void)
{
- if (ap_configuration)
- return ap_configuration->qact;
+ if (ap_config_info)
+ return ap_config_info->qact;
return 0;
}
@@ -213,13 +217,15 @@ static void ap_init_configuration(void)
if (!ap_configuration_available())
return;
- ap_configuration = kzalloc(sizeof(*ap_configuration), GFP_KERNEL);
- if (!ap_configuration)
- return;
- if (ap_query_configuration(ap_configuration) != 0) {
- kfree(ap_configuration);
- ap_configuration = NULL;
+ /* allocate current qci info struct */
+ ap_config_info = kzalloc(sizeof(*ap_config_info), GFP_KERNEL);
+ if (!ap_config_info)
return;
+
+ /* fetch qci info into the current qci info struct */
+ if (ap_query_configuration(ap_config_info)) {
+ kfree(ap_config_info);
+ ap_config_info = NULL;
}
}
@@ -242,10 +248,10 @@ static inline int ap_test_config(unsigned int *field, unsigned int nr)
*/
static inline int ap_test_config_card_id(unsigned int id)
{
- if (!ap_configuration) /* QCI not supported */
- /* only ids 0...3F may be probed */
+ if (!ap_config_info)
+ /* QCI not available, only ids 0...3F may be probed */
return id < 0x40 ? 1 : 0;
- return ap_test_config(ap_configuration->apm, id);
+ return ap_test_config(ap_config_info->apm, id);
}
/*
@@ -259,9 +265,9 @@ static inline int ap_test_config_card_id(unsigned int id)
*/
int ap_test_config_usage_domain(unsigned int domain)
{
- if (!ap_configuration) /* QCI not supported */
+ if (!ap_config_info) /* QCI not supported */
return domain < 16;
- return ap_test_config(ap_configuration->aqm, domain);
+ return ap_test_config(ap_config_info->aqm, domain);
}
EXPORT_SYMBOL(ap_test_config_usage_domain);
@@ -275,9 +281,9 @@ EXPORT_SYMBOL(ap_test_config_usage_domain);
*/
int ap_test_config_ctrl_domain(unsigned int domain)
{
- if (!ap_configuration) /* QCI not supported */
+ if (!ap_config_info) /* QCI not supported */
return 0;
- return ap_test_config(ap_configuration->adm, domain);
+ return ap_test_config(ap_config_info->adm, domain);
}
EXPORT_SYMBOL(ap_test_config_ctrl_domain);
@@ -953,45 +959,45 @@ static BUS_ATTR_RW(ap_domain);
static ssize_t ap_control_domain_mask_show(struct bus_type *bus, char *buf)
{
- if (!ap_configuration) /* QCI not supported */
- return scnprintf(buf, PAGE_SIZE, "not supported\n");
+ if (!ap_config_info) /* QCI not supported */
+ return snprintf(buf, PAGE_SIZE, "not supported\n");
- return scnprintf(buf, PAGE_SIZE,
- "0x%08x%08x%08x%08x%08x%08x%08x%08x\n",
- ap_configuration->adm[0], ap_configuration->adm[1],
- ap_configuration->adm[2], ap_configuration->adm[3],
- ap_configuration->adm[4], ap_configuration->adm[5],
- ap_configuration->adm[6], ap_configuration->adm[7]);
+ return snprintf(buf, PAGE_SIZE,
+ "0x%08x%08x%08x%08x%08x%08x%08x%08x\n",
+ ap_config_info->adm[0], ap_config_info->adm[1],
+ ap_config_info->adm[2], ap_config_info->adm[3],
+ ap_config_info->adm[4], ap_config_info->adm[5],
+ ap_config_info->adm[6], ap_config_info->adm[7]);
}
static BUS_ATTR_RO(ap_control_domain_mask);
static ssize_t ap_usage_domain_mask_show(struct bus_type *bus, char *buf)
{
- if (!ap_configuration) /* QCI not supported */
- return scnprintf(buf, PAGE_SIZE, "not supported\n");
+ if (!ap_config_info) /* QCI not supported */
+ return snprintf(buf, PAGE_SIZE, "not supported\n");
- return scnprintf(buf, PAGE_SIZE,
- "0x%08x%08x%08x%08x%08x%08x%08x%08x\n",
- ap_configuration->aqm[0], ap_configuration->aqm[1],
- ap_configuration->aqm[2], ap_configuration->aqm[3],
- ap_configuration->aqm[4], ap_configuration->aqm[5],
- ap_configuration->aqm[6], ap_configuration->aqm[7]);
+ return snprintf(buf, PAGE_SIZE,
+ "0x%08x%08x%08x%08x%08x%08x%08x%08x\n",
+ ap_config_info->aqm[0], ap_config_info->aqm[1],
+ ap_config_info->aqm[2], ap_config_info->aqm[3],
+ ap_config_info->aqm[4], ap_config_info->aqm[5],
+ ap_config_info->aqm[6], ap_config_info->aqm[7]);
}
static BUS_ATTR_RO(ap_usage_domain_mask);
static ssize_t ap_adapter_mask_show(struct bus_type *bus, char *buf)
{
- if (!ap_configuration) /* QCI not supported */
- return scnprintf(buf, PAGE_SIZE, "not supported\n");
+ if (!ap_config_info) /* QCI not supported */
+ return snprintf(buf, PAGE_SIZE, "not supported\n");
- return scnprintf(buf, PAGE_SIZE,
- "0x%08x%08x%08x%08x%08x%08x%08x%08x\n",
- ap_configuration->apm[0], ap_configuration->apm[1],
- ap_configuration->apm[2], ap_configuration->apm[3],
- ap_configuration->apm[4], ap_configuration->apm[5],
- ap_configuration->apm[6], ap_configuration->apm[7]);
+ return snprintf(buf, PAGE_SIZE,
+ "0x%08x%08x%08x%08x%08x%08x%08x%08x\n",
+ ap_config_info->apm[0], ap_config_info->apm[1],
+ ap_config_info->apm[2], ap_config_info->apm[3],
+ ap_config_info->apm[4], ap_config_info->apm[5],
+ ap_config_info->apm[6], ap_config_info->apm[7]);
}
static BUS_ATTR_RO(ap_adapter_mask);
@@ -1079,7 +1085,7 @@ static ssize_t ap_max_domain_id_show(struct bus_type *bus, char *buf)
{
int max_domain_id;
- if (ap_configuration)
+ if (ap_config_info)
max_domain_id = ap_max_domain_id ? : -1;
else
max_domain_id = 15;
@@ -1373,6 +1379,50 @@ static int ap_get_compatible_type(ap_qid_t qid, int rawtype, unsigned int func)
return comp_type;
}
+/* Helper function for notify_config_changed */
+static int __drv_notify_config_changed(struct device_driver *drv, void *data)
+{
+ struct ap_driver *ap_drv = to_ap_drv(drv);
+
+ if (try_module_get(drv->owner)) {
+ if (ap_drv->on_config_changed)
+ ap_drv->on_config_changed(ap_config_info,
+ ap_old_config_info);
+ module_put(drv->owner);
+ }
+
+ return 0;
+}
+
+/* Notify all drivers about an qci config change */
+static inline void notify_config_changed(void)
+{
+ bus_for_each_drv(&ap_bus_type, NULL, NULL,
+ __drv_notify_config_changed);
+}
+
+/* Helper function for notify_scan_complete */
+static int __drv_notify_scan_complete(struct device_driver *drv, void *data)
+{
+ struct ap_driver *ap_drv = to_ap_drv(drv);
+
+ if (try_module_get(drv->owner)) {
+ if (ap_drv->on_scan_complete)
+ ap_drv->on_scan_complete(ap_config_info,
+ ap_old_config_info);
+ module_put(drv->owner);
+ }
+
+ return 0;
+}
+
+/* Notify all drivers about bus scan complete */
+static inline void notify_scan_complete(void)
+{
+ bus_for_each_drv(&ap_bus_type, NULL, NULL,
+ __drv_notify_scan_complete);
+}
+
/*
* Helper function to be used with bus_find_dev
* matches for the card device with the given id
@@ -1555,23 +1605,57 @@ static void _ap_scan_bus_adapter(int id)
put_device(&ac->ap_dev.device);
}
+static int ap_config_changed(void)
+{
+ int cfg_chg = 0;
+
+ if (ap_config_info) {
+ if (!ap_old_config_info) {
+ ap_old_config_info = kzalloc(
+ sizeof(*ap_old_config_info), GFP_KERNEL);
+ if (!ap_old_config_info)
+ return 0;
+ } else {
+ memcpy(ap_old_config_info, ap_config_info,
+ sizeof(struct ap_config_info));
+ }
+ ap_query_configuration(ap_config_info);
+ cfg_chg = memcmp(ap_config_info,
+ ap_old_config_info,
+ sizeof(struct ap_config_info)) != 0;
+ }
+
+ return cfg_chg;
+}
+
/**
* ap_scan_bus(): Scan the AP bus for new devices
* Runs periodically, workqueue timer (ap_config_time)
*/
static void ap_scan_bus(struct work_struct *unused)
{
- int id;
+ int id, config_changed = 0;
AP_DBF(DBF_DEBUG, "%s running\n", __func__);
- ap_query_configuration(ap_configuration);
+ mutex_lock(&ap_config_lock);
+
+ /* config change notify */
+ config_changed = ap_config_changed();
+ if (config_changed)
+ notify_config_changed();
ap_select_domain();
/* loop over all possible adapters */
for (id = 0; id < AP_DEVICES; id++)
_ap_scan_bus_adapter(id);
+ /* scan complete notify */
+ if (config_changed)
+ notify_scan_complete();
+
+ mutex_unlock(&ap_config_lock);
+
/* check if there is at least one queue available with default domain */
if (ap_domain_index >= 0) {
struct device *dev =
@@ -1654,7 +1738,7 @@ static int __init ap_module_init(void)
/* Get AP configuration data if available */
ap_init_configuration();
- if (ap_configuration)
+ if (ap_config_info)
max_domain_id =
ap_max_domain_id ? ap_max_domain_id : AP_DOMAINS - 1;
else
@@ -1723,7 +1807,8 @@ static int __init ap_module_init(void)
out:
if (ap_using_interrupts())
unregister_adapter_interrupt(&ap_airq);
- kfree(ap_configuration);
+ kfree(ap_config_info);
+ kfree(ap_old_config_info);
return rc;
}
device_initcall(ap_module_init);
diff --git a/drivers/s390/crypto/ap_bus.h b/drivers/s390/crypto/ap_bus.h
index 7d9646251bfd..491ca8543398 100644
--- a/drivers/s390/crypto/ap_bus.h
+++ b/drivers/s390/crypto/ap_bus.h
@@ -137,6 +137,18 @@ struct ap_driver {
int (*probe)(struct ap_device *);
void (*remove)(struct ap_device *);
bool (*in_use)(unsigned long *apm, unsigned long *aqm);
+ /*
+ * Called at the start of the ap bus scan function when
+ * the crypto config information (qci) has changed.
+ */
+ void (*on_config_changed)(struct ap_config_info *new_config_info,
+ struct ap_config_info *old_config_info);
+ /*
+ * Called at the end of the ap bus scan function when
+ * the crypto config information (qci) has changed.
+ */
+ void (*on_scan_complete)(struct ap_config_info *new_config_info,
+ struct ap_config_info *old_config_info);
};
#define to_ap_drv(x) container_of((x), struct ap_driver, driver)
--
2.21.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v9 13/15] s390/vfio-ap: handle host AP config change notification
2020-07-20 15:03 [PATCH v9 00/15] s390/vfio-ap: dynamic configuration support Tony Krowiak
` (11 preceding siblings ...)
2020-07-20 15:03 ` [PATCH v9 12/15] s390/zcrypt: Notify driver on config changed and scan complete callbacks Tony Krowiak
@ 2020-07-20 15:03 ` Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 14/15] s390/vfio-ap: handle AP bus scan completed notification Tony Krowiak
` (3 subsequent siblings)
16 siblings, 0 replies; 21+ messages in thread
From: Tony Krowiak @ 2020-07-20 15:03 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, borntraeger, cohuck, mjrosato, pasic, alex.williamson,
kwankhede, fiuczy, Tony Krowiak, kernel test robot
Implements the driver callback invoked by the AP bus when the host
AP configuration has changed. Since this callback is invoked prior to
unbinding a device from its device driver, the vfio_ap driver will
respond by unplugging the AP adapters, domains and control domains
removed from the host's AP configuration from the guests using them.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
Reported-by: kernel test robot <lkp@intel.com>
---
drivers/s390/crypto/vfio_ap_drv.c | 5 +-
drivers/s390/crypto/vfio_ap_ops.c | 147 ++++++++++++++++++++++++--
drivers/s390/crypto/vfio_ap_private.h | 7 +-
3 files changed, 146 insertions(+), 13 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c
index aae5b3d8e3fa..ea0a7603e886 100644
--- a/drivers/s390/crypto/vfio_ap_drv.c
+++ b/drivers/s390/crypto/vfio_ap_drv.c
@@ -115,9 +115,11 @@ static int vfio_ap_matrix_dev_create(void)
/* Fill in config info via PQAP(QCI), if available */
if (test_facility(12)) {
- ret = ap_qci(&matrix_dev->info);
+ ret = ap_qci(&matrix_dev->config_info);
if (ret)
goto matrix_alloc_err;
+ memcpy(&matrix_dev->config_info_prev, &matrix_dev->config_info,
+ sizeof(struct ap_config_info));
}
mutex_init(&matrix_dev->lock);
@@ -177,6 +179,7 @@ static int __init vfio_ap_init(void)
vfio_ap_drv.remove = vfio_ap_queue_dev_remove;
vfio_ap_drv.in_use = vfio_ap_mdev_resource_in_use;
vfio_ap_drv.ids = ap_queue_ids;
+ vfio_ap_drv.on_config_changed = vfio_ap_on_cfg_changed;
ret = ap_driver_register(&vfio_ap_drv, THIS_MODULE, VFIO_AP_DRV_NAME);
if (ret) {
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 2b01a8eb6ee7..e002d556abab 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -347,7 +347,9 @@ static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev)
}
matrix_mdev->mdev = mdev;
- vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->matrix);
+ vfio_ap_matrix_init(&matrix_dev->config_info, &matrix_mdev->matrix);
+ vfio_ap_matrix_init(&matrix_dev->config_info,
+ &matrix_mdev->shadow_apcb);
hash_init(matrix_mdev->qtable);
mdev_set_drvdata(mdev, matrix_mdev);
matrix_mdev->pqap_hook.hook = handle_pqap;
@@ -526,8 +528,8 @@ static int vfio_ap_mdev_filter_matrix(struct ap_matrix_mdev *matrix_mdev,
* If the APID is not assigned to the host AP configuration,
* we can not assign it to the guest's AP configuration
*/
- if (!test_bit_inv(apid,
- (unsigned long *)matrix_dev->info.apm)) {
+ if (!test_bit_inv(apid, (unsigned long *)
+ matrix_dev->config_info.apm)) {
clear_bit_inv(apid, shadow_apcb->apm);
continue;
}
@@ -540,7 +542,7 @@ static int vfio_ap_mdev_filter_matrix(struct ap_matrix_mdev *matrix_mdev,
* guest's AP configuration
*/
if (!test_bit_inv(apqi, (unsigned long *)
- matrix_dev->info.aqm)) {
+ matrix_dev->config_info.aqm)) {
clear_bit_inv(apqi, shadow_apcb->aqm);
continue;
}
@@ -594,7 +596,7 @@ static bool vfio_ap_mdev_config_shadow_apcb(struct ap_matrix_mdev *matrix_mdev)
int napm, naqm;
struct ap_matrix shadow_apcb;
- vfio_ap_matrix_init(&matrix_dev->info, &shadow_apcb);
+ vfio_ap_matrix_init(&matrix_dev->config_info, &shadow_apcb);
napm = bitmap_weight(matrix_mdev->matrix.apm, AP_DEVICES);
naqm = bitmap_weight(matrix_mdev->matrix.aqm, AP_DOMAINS);
@@ -741,7 +743,7 @@ static bool vfio_ap_mdev_assign_apqis_4_apid(struct ap_matrix_mdev *matrix_mdev,
for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, AP_DOMAINS) {
if (!test_bit_inv(apqi,
- (unsigned long *) matrix_dev->info.aqm))
+ (unsigned long *)matrix_dev->config_info.aqm))
clear_bit_inv(apqi, aqm);
apqn = AP_MKQID(apid, apqi);
@@ -764,7 +766,7 @@ static bool vfio_ap_mdev_assign_guest_apid(struct ap_matrix_mdev *matrix_mdev,
unsigned long apqi, apqn;
if (!vfio_ap_mdev_has_crycb(matrix_mdev) ||
- !test_bit_inv(apid, (unsigned long *)matrix_dev->info.apm))
+ !test_bit_inv(apid, (unsigned long *)matrix_dev->config_info.apm))
return false;
if (bitmap_empty(matrix_mdev->shadow_apcb.aqm, AP_DOMAINS))
@@ -931,8 +933,8 @@ static bool vfio_ap_mdev_assign_apids_4_apqi(struct ap_matrix_mdev *matrix_mdev,
bitmap_copy(apm, matrix_mdev->matrix.apm, AP_DEVICES);
for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, AP_DEVICES) {
- if (!test_bit_inv(apid,
- (unsigned long *) matrix_dev->info.apm))
+ if (!test_bit_inv(apid, (unsigned long *)
+ matrix_dev->config_info.apm))
clear_bit_inv(apqi, apm);
apqn = AP_MKQID(apid, apqi);
@@ -955,7 +957,7 @@ static bool vfio_ap_mdev_assign_guest_apqi(struct ap_matrix_mdev *matrix_mdev,
unsigned long apid, apqn;
if (!vfio_ap_mdev_has_crycb(matrix_mdev) ||
- !test_bit_inv(apqi, (unsigned long *)matrix_dev->info.aqm))
+ !test_bit_inv(apqi, (unsigned long *)matrix_dev->config_info.aqm))
return false;
if (bitmap_empty(matrix_mdev->shadow_apcb.apm, AP_DEVICES))
@@ -1702,7 +1704,7 @@ int vfio_ap_mdev_probe_queue(struct ap_queue *queue)
void vfio_ap_mdev_remove_queue(struct ap_queue *queue)
{
struct vfio_ap_queue *q;
- int apid, apqi;
+ unsigned long apid, apqi;
mutex_lock(&matrix_dev->lock);
q = dev_get_drvdata(&queue->ap_dev.device);
@@ -1727,3 +1729,126 @@ bool vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm)
return in_use;
}
+
+/**
+ * vfio_ap_mdev_unassign_apids
+ *
+ * @matrix_mdev: The matrix mediated device
+ *
+ * @aqm: A bitmap with 256 bits. Each bit in the map represents an APID from 0
+ * to 255 (with the leftmost bit corresponding to APID 0).
+ *
+ * Unassigns each APID specified in @aqm that is assigned to the shadow CRYCB
+ * of @matrix_mdev. Returns true if at least one APID is unassigned; otherwise,
+ * returns false.
+ */
+static bool vfio_ap_mdev_unassign_apids(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long *apm_unassign)
+{
+ unsigned long apid;
+ bool unassigned = false;
+
+ /*
+ * If the matrix mdev is not in use by a KVM guest, return indicating
+ * that no APIDs have been unassigned.
+ */
+ if (!vfio_ap_mdev_has_crycb(matrix_mdev))
+ return false;
+
+ for_each_set_bit_inv(apid, apm_unassign, AP_DEVICES) {
+ unassigned |= vfio_ap_mdev_unassign_guest_apid(matrix_mdev,
+ apid);
+ }
+
+ return unassigned;
+}
+
+/**
+ * vfio_ap_mdev_unassign_apqis
+ *
+ * @matrix_mdev: The matrix mediated device
+ *
+ * @aqm: A bitmap with 256 bits. Each bit in the map represents an APQI from 0
+ * to 255 (with the leftmost bit corresponding to APQI 0).
+ *
+ * Unassigns each APQI specified in @aqm that is assigned to the shadow CRYCB
+ * of @matrix_mdev. Returns true if at least one APQI is unassigned; otherwise,
+ * returns false.
+ */
+static bool vfio_ap_mdev_unassign_apqis(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long *aqm_unassign)
+{
+ unsigned long apqi;
+ bool unassigned = false;
+
+ /*
+ * If the matrix mdev is not in use by a KVM guest, return indicating
+ * that no APQIs have been unassigned.
+ */
+ if (!vfio_ap_mdev_has_crycb(matrix_mdev))
+ return false;
+
+ for_each_set_bit_inv(apqi, aqm_unassign, AP_DOMAINS) {
+ unassigned |= vfio_ap_mdev_unassign_guest_apqi(matrix_mdev,
+ apqi);
+ }
+
+ return unassigned;
+}
+
+void vfio_ap_on_cfg_changed(struct ap_config_info *new_config_info,
+ struct ap_config_info *old_config_info)
+{
+ bool unassigned;
+ int ap_remove, aq_remove;
+ struct ap_matrix_mdev *matrix_mdev;
+ DECLARE_BITMAP(apm_unassign, AP_DEVICES);
+ DECLARE_BITMAP(aqm_unassign, AP_DOMAINS);
+
+ unsigned long *cur_apm, *cur_aqm, *prev_apm, *prev_aqm;
+
+ if (matrix_dev->flags & AP_MATRIX_CFG_CHG) {
+ WARN_ONCE(1, "AP host configuration change already reported");
+ return;
+ }
+
+ memcpy(&matrix_dev->config_info, new_config_info,
+ sizeof(struct ap_config_info));
+ memcpy(&matrix_dev->config_info_prev, old_config_info,
+ sizeof(struct ap_config_info));
+
+ cur_apm = (unsigned long *)matrix_dev->config_info.apm;
+ cur_aqm = (unsigned long *)matrix_dev->config_info.aqm;
+ prev_apm = (unsigned long *)matrix_dev->config_info_prev.apm;
+ prev_aqm = (unsigned long *)matrix_dev->config_info_prev.aqm;
+
+ ap_remove = bitmap_andnot(apm_unassign, prev_apm, cur_apm, AP_DEVICES);
+ aq_remove = bitmap_andnot(aqm_unassign, prev_aqm, cur_aqm, AP_DOMAINS);
+
+ mutex_lock(&matrix_dev->lock);
+ matrix_dev->flags |= AP_MATRIX_CFG_CHG;
+
+ list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) {
+ if (!vfio_ap_mdev_has_crycb(matrix_mdev))
+ continue;
+
+ unassigned = false;
+
+ if (ap_remove)
+ if (bitmap_intersects(matrix_mdev->shadow_apcb.apm,
+ apm_unassign, AP_DEVICES))
+ if (vfio_ap_mdev_unassign_apids(matrix_mdev,
+ apm_unassign))
+ unassigned = true;
+ if (aq_remove)
+ if (bitmap_intersects(matrix_mdev->shadow_apcb.aqm,
+ aqm_unassign, AP_DOMAINS))
+ if (vfio_ap_mdev_unassign_apqis(matrix_mdev,
+ aqm_unassign))
+ unassigned = true;
+
+ if (unassigned)
+ vfio_ap_mdev_commit_shadow_apcb(matrix_mdev);
+ }
+ mutex_unlock(&matrix_dev->lock);
+}
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index 055bce6d45db..fc8629e28ad3 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -40,10 +40,13 @@
struct ap_matrix_dev {
struct device device;
atomic_t available_instances;
- struct ap_config_info info;
+ struct ap_config_info config_info;
+ struct ap_config_info config_info_prev;
struct list_head mdev_list;
struct mutex lock;
struct ap_driver *vfio_ap_drv;
+ #define AP_MATRIX_CFG_CHG (1UL << 0)
+ unsigned long flags;
};
extern struct ap_matrix_dev *matrix_dev;
@@ -108,5 +111,7 @@ int vfio_ap_mdev_probe_queue(struct ap_queue *queue);
void vfio_ap_mdev_remove_queue(struct ap_queue *queue);
bool vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm);
+void vfio_ap_on_cfg_changed(struct ap_config_info *new_config_info,
+ struct ap_config_info *old_config_info);
#endif /* _VFIO_AP_PRIVATE_H_ */
--
2.21.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v9 14/15] s390/vfio-ap: handle AP bus scan completed notification
2020-07-20 15:03 [PATCH v9 00/15] s390/vfio-ap: dynamic configuration support Tony Krowiak
` (12 preceding siblings ...)
2020-07-20 15:03 ` [PATCH v9 13/15] s390/vfio-ap: handle host AP config change notification Tony Krowiak
@ 2020-07-20 15:03 ` Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 15/15] s390/vfio-ap: handle probe/remove not due to host AP config changes Tony Krowiak
` (2 subsequent siblings)
16 siblings, 0 replies; 21+ messages in thread
From: Tony Krowiak @ 2020-07-20 15:03 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, borntraeger, cohuck, mjrosato, pasic, alex.williamson,
kwankhede, fiuczy, Tony Krowiak
Implements the driver callback invoked by the AP bus when the AP bus
scan has completed. Since this callback is invoked after binding the newly
added devices to their respective device drivers, the vfio_ap driver will
attempt to plug the adapters, domains and control domains into each guest
using a matrix mdev to which they are assigned. Keep in mind that an
adapter or domain can be plugged in only if each APQN with the APID of the
adapter or the APQI of the domain references a queue device bound to the
vfio_ap device driver. Consequently, not all newly added adapters and
domains will necessarily get hot plugged.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
---
drivers/s390/crypto/vfio_ap_drv.c | 1 +
drivers/s390/crypto/vfio_ap_ops.c | 110 +++++++++++++++++++++++++-
drivers/s390/crypto/vfio_ap_private.h | 2 +
3 files changed, 110 insertions(+), 3 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c
index ea0a7603e886..21bfae928be5 100644
--- a/drivers/s390/crypto/vfio_ap_drv.c
+++ b/drivers/s390/crypto/vfio_ap_drv.c
@@ -180,6 +180,7 @@ static int __init vfio_ap_init(void)
vfio_ap_drv.in_use = vfio_ap_mdev_resource_in_use;
vfio_ap_drv.ids = ap_queue_ids;
vfio_ap_drv.on_config_changed = vfio_ap_on_cfg_changed;
+ vfio_ap_drv.on_scan_complete = vfio_ap_on_scan_complete;
ret = ap_driver_register(&vfio_ap_drv, THIS_MODULE, VFIO_AP_DRV_NAME);
if (ret) {
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index e002d556abab..e6480f31a42b 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -616,14 +616,13 @@ static bool vfio_ap_mdev_config_shadow_apcb(struct ap_matrix_mdev *matrix_mdev)
* CRYCB after filtering, then try filtering the APQIs.
*/
if (napm == 0) {
- naqm = vfio_ap_mdev_filter_matrix(matrix_mdev,
- &shadow_apcb, false);
-
/*
* If there are no APQNs that can be assigned to the
* matrix mdev after filtering the APQIs, then no APQNs
* shall be assigned to the guest's CRYCB.
*/
+ naqm = vfio_ap_mdev_filter_matrix(matrix_mdev,
+ &shadow_apcb, false);
if (naqm == 0) {
bitmap_clear(shadow_apcb.apm, 0, AP_DEVICES);
bitmap_clear(shadow_apcb.aqm, 0, AP_DOMAINS);
@@ -1758,6 +1757,16 @@ static bool vfio_ap_mdev_unassign_apids(struct ap_matrix_mdev *matrix_mdev,
for_each_set_bit_inv(apid, apm_unassign, AP_DEVICES) {
unassigned |= vfio_ap_mdev_unassign_guest_apid(matrix_mdev,
apid);
+ /*
+ * If the APID is not assigned to the matrix mdev's shadow
+ * CRYCB, continue with the next APID.
+ */
+ if (!test_bit_inv(apid, matrix_mdev->shadow_apcb.apm))
+ continue;
+
+ /* Unassign the APID from the matrix mdev's shadow CRYCB */
+ clear_bit_inv(apid, matrix_mdev->shadow_apcb.apm);
+ unassigned = true;
}
return unassigned;
@@ -1791,6 +1800,17 @@ static bool vfio_ap_mdev_unassign_apqis(struct ap_matrix_mdev *matrix_mdev,
for_each_set_bit_inv(apqi, aqm_unassign, AP_DOMAINS) {
unassigned |= vfio_ap_mdev_unassign_guest_apqi(matrix_mdev,
apqi);
+
+ /*
+ * If the APQI is not assigned to the matrix mdev's shadow
+ * CRYCB, continue with the next APQI
+ */
+ if (!test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm))
+ continue;
+
+ /* Unassign the APQI from the matrix mdev's shadow CRYCB */
+ clear_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm);
+ unassigned = true;
}
return unassigned;
@@ -1852,3 +1872,87 @@ void vfio_ap_on_cfg_changed(struct ap_config_info *new_config_info,
}
mutex_unlock(&matrix_dev->lock);
}
+
+bool vfio_ap_mdev_assign_apids(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long *apm_assign)
+{
+ unsigned long apid;
+ bool assigned = false;
+
+ for_each_set_bit_inv(apid, apm_assign, AP_DEVICES)
+ if (test_bit_inv(apid, matrix_mdev->matrix.apm))
+ if (vfio_ap_mdev_assign_guest_apid(matrix_mdev, apid))
+ assigned = true;
+
+ return assigned;
+}
+
+bool vfio_ap_mdev_assign_apqis(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long *aqm_assign)
+{
+ unsigned long apqi;
+ bool assigned = false;
+
+ for_each_set_bit_inv(apqi, aqm_assign, AP_DOMAINS)
+ if (test_bit_inv(apqi, matrix_mdev->matrix.aqm))
+ if (vfio_ap_mdev_assign_guest_apqi(matrix_mdev, apqi))
+ assigned = true;
+
+ return assigned;
+}
+
+void vfio_ap_on_scan_complete(struct ap_config_info *new_config_info,
+ struct ap_config_info *old_config_info)
+{
+ struct ap_matrix_mdev *matrix_mdev;
+ DECLARE_BITMAP(apm_assign, AP_DEVICES);
+ DECLARE_BITMAP(aqm_assign, AP_DOMAINS);
+ int ap_add, aq_add;
+ bool assign;
+ unsigned long *cur_apm, *cur_aqm, *prev_apm, *prev_aqm;
+
+ /*
+ * If we are not in the middle of a host configuration change scan it is
+ * likely that the vfio_ap driver was loaded mid-scan, so let's handle
+ * this scenario by calling the vfio_ap_on_cfg_changed function which
+ * gets called at the start of an AP bus scan when the host AP
+ * configuration has changed.
+ */
+ if (!(matrix_dev->flags & AP_MATRIX_CFG_CHG))
+ vfio_ap_on_cfg_changed(new_config_info, old_config_info);
+
+ cur_apm = (unsigned long *)matrix_dev->config_info.apm;
+ cur_aqm = (unsigned long *)matrix_dev->config_info.aqm;
+
+ prev_apm = (unsigned long *)matrix_dev->config_info_prev.apm;
+ prev_aqm = (unsigned long *)matrix_dev->config_info_prev.aqm;
+
+ ap_add = bitmap_andnot(apm_assign, cur_apm, prev_apm, AP_DEVICES);
+ aq_add = bitmap_andnot(aqm_assign, cur_aqm, prev_aqm, AP_DOMAINS);
+
+ mutex_lock(&matrix_dev->lock);
+ list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) {
+ if (!vfio_ap_mdev_has_crycb(matrix_mdev))
+ continue;
+
+ assign = false;
+
+ if (ap_add)
+ if (bitmap_intersects(matrix_mdev->matrix.apm,
+ apm_assign, AP_DEVICES))
+ assign |= vfio_ap_mdev_assign_apids(matrix_mdev,
+ apm_assign);
+
+ if (aq_add)
+ if (bitmap_intersects(matrix_mdev->matrix.aqm,
+ aqm_assign, AP_DOMAINS))
+ assign |= vfio_ap_mdev_assign_apqis(matrix_mdev,
+ aqm_assign);
+
+ if (assign)
+ vfio_ap_mdev_commit_shadow_apcb(matrix_mdev);
+ }
+
+ matrix_dev->flags &= ~AP_MATRIX_CFG_CHG;
+ mutex_unlock(&matrix_dev->lock);
+}
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index fc8629e28ad3..da1754fd4f66 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -113,5 +113,7 @@ void vfio_ap_mdev_remove_queue(struct ap_queue *queue);
bool vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm);
void vfio_ap_on_cfg_changed(struct ap_config_info *new_config_info,
struct ap_config_info *old_config_info);
+void vfio_ap_on_scan_complete(struct ap_config_info *new_config_info,
+ struct ap_config_info *old_config_info);
#endif /* _VFIO_AP_PRIVATE_H_ */
--
2.21.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v9 15/15] s390/vfio-ap: handle probe/remove not due to host AP config changes
2020-07-20 15:03 [PATCH v9 00/15] s390/vfio-ap: dynamic configuration support Tony Krowiak
` (13 preceding siblings ...)
2020-07-20 15:03 ` [PATCH v9 14/15] s390/vfio-ap: handle AP bus scan completed notification Tony Krowiak
@ 2020-07-20 15:03 ` Tony Krowiak
2020-08-04 14:28 ` [PATCH v9 00/15] s390/vfio-ap: dynamic configuration support Tony Krowiak
2020-08-10 15:39 ` Tony Krowiak
16 siblings, 0 replies; 21+ messages in thread
From: Tony Krowiak @ 2020-07-20 15:03 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, borntraeger, cohuck, mjrosato, pasic, alex.williamson,
kwankhede, fiuczy, Tony Krowiak
AP queue devices are probed or removed for reasons other than changes
to the host AP configuration. For example:
* The state of an AP adapter can be dynamically changed from standby to
online via the SE or by execution of the SCLP Configure AP command. When
the state changes, each queue device associated with the card device
representing the adapter will get created and probed.
* The state of an AP adapter can be dynamically changed from online to
standby via the SE or by execution of the SCLP Deconfigure AP command.
When the state changes, each queue device associated with the card device
representing the adapter will get removed.
* Each queue device associated with a card device will get removed
when the type of the AP adapter represented by the card device
dynamically changes.
* Each queue device associated with a card device will get removed
when the status of the queue represented by the queue device changes
from operating to check stop.
* AP queue devices can be manually bound to or unbound from the vfio_ap
device driver by a root user via the sysfs bind/unbind attributes of the
driver.
In response to a queue device probe or remove that is not the result of a
change to the host's AP configuration, if a KVM guest is using the matrix
mdev to which the APQN of the queue device is assigned, the vfio_ap device
driver must respond accordingly. In an ideal world, the queue device being
probed would be hot plugged into the guest. Likewise, the queue
corresponding to the queue device being removed would
be hot unplugged from the guest. Unfortunately, the AP architecture
precludes plugging or unplugging individual queues, so let's handle
the probe or remove of an AP queue device as follows:
Handling Probe
--------------
There are two requirements that must be met in order to give a
guest access to the queue corresponding to the queue device being probed:
* Each APQN derived from the APID of the queue device and the APQIs of the
domains already assigned to the guest's AP configuration must reference
a queue device bound to the vfio_ap device driver.
* Each APQN derived from the APQI of the queue device and the APIDs of the
adapters assigned to the guest's AP configuration must reference a queue
device bound to the vfio_ap device driver.
If the above conditions are met, the APQN will be assigned to the guest's
AP configuration and the guest will be given access to the queue.
Handling Remove
---------------
Since the AP architecture precludes us from taking access to an individual
queue from a guest, we are left with the choice of taking access away from
either the adapter or the domain to which the queue is connected. Access to
the adapter will be taken away because it is likely that most of the time,
the remove callback will be invoked because the adapter state has
transitioned from online to standby. In such a case, no queue connected
to the adapter will be available to access.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
---
drivers/s390/crypto/vfio_ap_ops.c | 84 +++++++++++++++++++++++++++++++
1 file changed, 84 insertions(+)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index e6480f31a42b..b6a1e280991d 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -1682,6 +1682,61 @@ static void vfio_ap_queue_link_mdev(struct vfio_ap_queue *q)
}
}
+static bool vfio_ap_mdev_assign_shadow_apid(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long apid)
+{
+ unsigned long apqi;
+
+ for_each_set_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm,
+ matrix_mdev->shadow_apcb.aqm_max + 1) {
+ if (!vfio_ap_get_queue(AP_MKQID(apid, apqi)))
+ return false;
+ }
+
+ set_bit_inv(apid, matrix_mdev->shadow_apcb.apm);
+
+ return true;
+}
+
+static bool vfio_ap_mdev_assign_shadow_apqi(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long apqi)
+{
+ unsigned long apid;
+
+ for_each_set_bit_inv(apid, matrix_mdev->shadow_apcb.apm,
+ matrix_mdev->shadow_apcb.apm_max + 1) {
+ if (!vfio_ap_get_queue(AP_MKQID(apid, apqi)))
+ return false;
+ }
+
+ set_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm);
+
+ return true;
+}
+
+static void vfio_ap_mdev_hot_plug_queue(struct vfio_ap_queue *q)
+{
+ bool commit = false;
+ unsigned long apid = AP_QID_CARD(q->apqn);
+ unsigned long apqi = AP_QID_QUEUE(q->apqn);
+
+ if ((q->matrix_mdev == NULL) || !vfio_ap_mdev_has_crycb(q->matrix_mdev))
+ return;
+
+ if (!test_bit_inv(apid, q->matrix_mdev->matrix.apm) ||
+ !test_bit_inv(apqi, q->matrix_mdev->matrix.aqm))
+ return;
+
+ if (!test_bit_inv(apid, q->matrix_mdev->shadow_apcb.apm))
+ commit |= vfio_ap_mdev_assign_shadow_apid(q->matrix_mdev, apid);
+
+ if (!test_bit_inv(apqi, q->matrix_mdev->shadow_apcb.aqm))
+ commit |= vfio_ap_mdev_assign_shadow_apqi(q->matrix_mdev, apqi);
+
+ if (commit)
+ vfio_ap_mdev_commit_shadow_apcb(q->matrix_mdev);
+}
+
int vfio_ap_mdev_probe_queue(struct ap_queue *queue)
{
struct vfio_ap_queue *q;
@@ -1695,11 +1750,35 @@ int vfio_ap_mdev_probe_queue(struct ap_queue *queue)
q->apqn = queue->qid;
q->saved_isc = VFIO_AP_ISC_INVALID;
vfio_ap_queue_link_mdev(q);
+ /* Make sure we're not in the middle of an AP configuration change. */
+ if (!(matrix_dev->flags & AP_MATRIX_CFG_CHG))
+ vfio_ap_mdev_hot_plug_queue(q);
mutex_unlock(&matrix_dev->lock);
return 0;
}
+void vfio_ap_mdev_hot_unplug_queue(struct vfio_ap_queue *q)
+{
+ unsigned long apid = AP_QID_CARD(q->apqn);
+ unsigned long apqi = AP_QID_QUEUE(q->apqn);
+
+ if ((q->matrix_mdev == NULL) || !vfio_ap_mdev_has_crycb(q->matrix_mdev))
+ return;
+
+ /*
+ * If the APQN is assigned to the guest, then let's
+ * go ahead and unplug the adapter since the
+ * architecture does not provide a means to unplug
+ * an individual queue.
+ */
+ if (test_bit_inv(apid, q->matrix_mdev->shadow_apcb.apm) &&
+ test_bit_inv(apqi, q->matrix_mdev->shadow_apcb.aqm)) {
+ if (vfio_ap_mdev_unassign_guest_apid(q->matrix_mdev, apid))
+ vfio_ap_mdev_commit_shadow_apcb(q->matrix_mdev);
+ }
+}
+
void vfio_ap_mdev_remove_queue(struct ap_queue *queue)
{
struct vfio_ap_queue *q;
@@ -1707,6 +1786,11 @@ void vfio_ap_mdev_remove_queue(struct ap_queue *queue)
mutex_lock(&matrix_dev->lock);
q = dev_get_drvdata(&queue->ap_dev.device);
+
+ /* Make sure we're not in the middle of an AP configuration change. */
+ if (!(matrix_dev->flags & AP_MATRIX_CFG_CHG))
+ vfio_ap_mdev_hot_unplug_queue(q);
+
dev_set_drvdata(&queue->ap_dev.device, NULL);
apid = AP_QID_CARD(q->apqn);
apqi = AP_QID_QUEUE(q->apqn);
--
2.21.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [PATCH v9 02/15] s390/vfio-ap: use new AP bus interface to search for queue devices
2020-07-20 15:03 ` [PATCH v9 02/15] s390/vfio-ap: use new AP bus interface to search for queue devices Tony Krowiak
@ 2020-07-24 8:38 ` Pierre Morel
2020-07-27 13:42 ` Tony Krowiak
2020-07-27 14:09 ` Tony Krowiak
0 siblings, 2 replies; 21+ messages in thread
From: Pierre Morel @ 2020-07-24 8:38 UTC (permalink / raw)
To: Tony Krowiak, linux-s390, linux-kernel, kvm
Cc: freude, borntraeger, cohuck, mjrosato, pasic, alex.williamson,
kwankhede, fiuczy, kernel test robot
On 2020-07-20 17:03, Tony Krowiak wrote:
> This patch refactor's the vfio_ap device driver to use the AP bus's
> ap_get_qdev() function to retrieve the vfio_ap_queue struct containing
> information about a queue that is bound to the vfio_ap device driver.
> The bus's ap_get_qdev() function retrieves the queue device from a
> hashtable keyed by APQN. This is much more efficient than looping over
> the list of devices attached to the AP bus by several orders of
> magnitude.
The patch does much more than modifying this line. ;)
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
> Reported-by: kernel test robot <lkp@intel.com>
> ---
> drivers/s390/crypto/vfio_ap_drv.c | 27 ++-------
> drivers/s390/crypto/vfio_ap_ops.c | 86 +++++++++++++++------------
> drivers/s390/crypto/vfio_ap_private.h | 8 ++-
> 3 files changed, 59 insertions(+), 62 deletions(-)
>
> diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c
> index f4ceb380dd61..24cdef60039a 100644
> --- a/drivers/s390/crypto/vfio_ap_drv.c
> +++ b/drivers/s390/crypto/vfio_ap_drv.c
> @@ -53,15 +53,9 @@ MODULE_DEVICE_TABLE(vfio_ap, ap_queue_ids);
> */
> static int vfio_ap_queue_dev_probe(struct ap_device *apdev)
> {
> - struct vfio_ap_queue *q;
> -
> - q = kzalloc(sizeof(*q), GFP_KERNEL);
> - if (!q)
> - return -ENOMEM;
> - dev_set_drvdata(&apdev->device, q);
> - q->apqn = to_ap_queue(&apdev->device)->qid;
> - q->saved_isc = VFIO_AP_ISC_INVALID;
> - return 0;
> + struct ap_queue *queue = to_ap_queue(&apdev->device);
> +
> + return vfio_ap_mdev_probe_queue(queue);
> }
You should explain the reason why this function is modified.
>
> /**
> @@ -72,18 +66,9 @@ static int vfio_ap_queue_dev_probe(struct ap_device *apdev)
> */
> static void vfio_ap_queue_dev_remove(struct ap_device *apdev)
> {
> - struct vfio_ap_queue *q;
> - int apid, apqi;
> -
> - mutex_lock(&matrix_dev->lock);
> - q = dev_get_drvdata(&apdev->device);
> - dev_set_drvdata(&apdev->device, NULL);
> - apid = AP_QID_CARD(q->apqn);
> - apqi = AP_QID_QUEUE(q->apqn);
> - vfio_ap_mdev_reset_queue(apid, apqi, 1);
> - vfio_ap_irq_disable(q);
> - kfree(q);
> - mutex_unlock(&matrix_dev->lock);
> + struct ap_queue *queue = to_ap_queue(&apdev->device);
> +
> + vfio_ap_mdev_remove_queue(queue);
> }
... and this one?
>
> static void vfio_ap_matrix_dev_release(struct device *dev)
> diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
> index e0bde8518745..ad3925f04f61 100644
> --- a/drivers/s390/crypto/vfio_ap_ops.c
> +++ b/drivers/s390/crypto/vfio_ap_ops.c
> @@ -26,43 +26,26 @@
>
> static int vfio_ap_mdev_reset_queues(struct mdev_device *mdev);
>
> -static int match_apqn(struct device *dev, const void *data)
> -{
> - struct vfio_ap_queue *q = dev_get_drvdata(dev);
> -
> - return (q->apqn == *(int *)(data)) ? 1 : 0;
> -}
> -
> /**
> - * vfio_ap_get_queue: Retrieve a queue with a specific APQN from a list
> - * @matrix_mdev: the associated mediated matrix
> + * vfio_ap_get_queue: Retrieve a queue with a specific APQN.
> * @apqn: The queue APQN
> *
> - * Retrieve a queue with a specific APQN from the list of the
> - * devices of the vfio_ap_drv.
> - * Verify that the APID and the APQI are set in the matrix.
> + * Retrieve a queue with a specific APQN from the AP queue devices attached to
> + * the AP bus.
> *
> - * Returns the pointer to the associated vfio_ap_queue
> + * Returns the pointer to the vfio_ap_queue with the specified APQN, or NULL.
> */
> -static struct vfio_ap_queue *vfio_ap_get_queue(
> - struct ap_matrix_mdev *matrix_mdev,
> - int apqn)
> +static struct vfio_ap_queue *vfio_ap_get_queue(unsigned long apqn)
> {
> + struct ap_queue *queue;
> struct vfio_ap_queue *q;
> - struct device *dev;
>
> - if (!test_bit_inv(AP_QID_CARD(apqn), matrix_mdev->matrix.apm))
> - return NULL;
> - if (!test_bit_inv(AP_QID_QUEUE(apqn), matrix_mdev->matrix.aqm))
> + queue = ap_get_qdev(apqn);
> + if (!queue)
> return NULL;
>
> - dev = driver_find_device(&matrix_dev->vfio_ap_drv->driver, NULL,
> - &apqn, match_apqn);
> - if (!dev)
> - return NULL;
> - q = dev_get_drvdata(dev);
> - q->matrix_mdev = matrix_mdev;
> - put_device(dev);
> + q = dev_get_drvdata(&queue->ap_dev.device);
> + put_device(&queue->ap_dev.device);
>
> return q;
> }
this function changed a lot too, you should explain the goal in the
patch comment.
...snip...
Regards,
Pierre
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v9 02/15] s390/vfio-ap: use new AP bus interface to search for queue devices
2020-07-24 8:38 ` Pierre Morel
@ 2020-07-27 13:42 ` Tony Krowiak
2020-07-27 14:09 ` Tony Krowiak
1 sibling, 0 replies; 21+ messages in thread
From: Tony Krowiak @ 2020-07-27 13:42 UTC (permalink / raw)
To: Pierre Morel, linux-s390, linux-kernel, kvm
Cc: freude, borntraeger, cohuck, mjrosato, pasic, alex.williamson,
kwankhede, fiuczy, kernel test robot
On 7/24/20 4:38 AM, Pierre Morel wrote:
>
>
> On 2020-07-20 17:03, Tony Krowiak wrote:
>> This patch refactor's the vfio_ap device driver to use the AP bus's
>> ap_get_qdev() function to retrieve the vfio_ap_queue struct containing
>> information about a queue that is bound to the vfio_ap device driver.
>> The bus's ap_get_qdev() function retrieves the queue device from a
>> hashtable keyed by APQN. This is much more efficient than looping over
>> the list of devices attached to the AP bus by several orders of
>> magnitude.
>
> The patch does much more than modifying this line. ;)
Yes it does, I'll be sure to include the additional details.
>
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
>> Reported-by: kernel test robot <lkp@intel.com>
>> ---
>> drivers/s390/crypto/vfio_ap_drv.c | 27 ++-------
>> drivers/s390/crypto/vfio_ap_ops.c | 86 +++++++++++++++------------
>> drivers/s390/crypto/vfio_ap_private.h | 8 ++-
>> 3 files changed, 59 insertions(+), 62 deletions(-)
>>
>> diff --git a/drivers/s390/crypto/vfio_ap_drv.c
>> b/drivers/s390/crypto/vfio_ap_drv.c
>> index f4ceb380dd61..24cdef60039a 100644
>> --- a/drivers/s390/crypto/vfio_ap_drv.c
>> +++ b/drivers/s390/crypto/vfio_ap_drv.c
>> @@ -53,15 +53,9 @@ MODULE_DEVICE_TABLE(vfio_ap, ap_queue_ids);
>> */
>> static int vfio_ap_queue_dev_probe(struct ap_device *apdev)
>> {
>> - struct vfio_ap_queue *q;
>> -
>> - q = kzalloc(sizeof(*q), GFP_KERNEL);
>> - if (!q)
>> - return -ENOMEM;
>> - dev_set_drvdata(&apdev->device, q);
>> - q->apqn = to_ap_queue(&apdev->device)->qid;
>> - q->saved_isc = VFIO_AP_ISC_INVALID;
>> - return 0;
>> + struct ap_queue *queue = to_ap_queue(&apdev->device);
>> +
>> + return vfio_ap_mdev_probe_queue(queue);
>> }
>
> You should explain the reason why this function is modified.
>
>> /**
>> @@ -72,18 +66,9 @@ static int vfio_ap_queue_dev_probe(struct
>> ap_device *apdev)
>> */
>> static void vfio_ap_queue_dev_remove(struct ap_device *apdev)
>> {
>> - struct vfio_ap_queue *q;
>> - int apid, apqi;
>> -
>> - mutex_lock(&matrix_dev->lock);
>> - q = dev_get_drvdata(&apdev->device);
>> - dev_set_drvdata(&apdev->device, NULL);
>> - apid = AP_QID_CARD(q->apqn);
>> - apqi = AP_QID_QUEUE(q->apqn);
>> - vfio_ap_mdev_reset_queue(apid, apqi, 1);
>> - vfio_ap_irq_disable(q);
>> - kfree(q);
>> - mutex_unlock(&matrix_dev->lock);
>> + struct ap_queue *queue = to_ap_queue(&apdev->device);
>> +
>> + vfio_ap_mdev_remove_queue(queue);
>> }
>
> ... and this one?
>
>> static void vfio_ap_matrix_dev_release(struct device *dev)
>> diff --git a/drivers/s390/crypto/vfio_ap_ops.c
>> b/drivers/s390/crypto/vfio_ap_ops.c
>> index e0bde8518745..ad3925f04f61 100644
>> --- a/drivers/s390/crypto/vfio_ap_ops.c
>> +++ b/drivers/s390/crypto/vfio_ap_ops.c
>> @@ -26,43 +26,26 @@
>> static int vfio_ap_mdev_reset_queues(struct mdev_device *mdev);
>> -static int match_apqn(struct device *dev, const void *data)
>> -{
>> - struct vfio_ap_queue *q = dev_get_drvdata(dev);
>> -
>> - return (q->apqn == *(int *)(data)) ? 1 : 0;
>> -}
>> -
>> /**
>> - * vfio_ap_get_queue: Retrieve a queue with a specific APQN from a list
>> - * @matrix_mdev: the associated mediated matrix
>> + * vfio_ap_get_queue: Retrieve a queue with a specific APQN.
>> * @apqn: The queue APQN
>> *
>> - * Retrieve a queue with a specific APQN from the list of the
>> - * devices of the vfio_ap_drv.
>> - * Verify that the APID and the APQI are set in the matrix.
>> + * Retrieve a queue with a specific APQN from the AP queue devices
>> attached to
>> + * the AP bus.
>> *
>> - * Returns the pointer to the associated vfio_ap_queue
>> + * Returns the pointer to the vfio_ap_queue with the specified APQN,
>> or NULL.
>> */
>> -static struct vfio_ap_queue *vfio_ap_get_queue(
>> - struct ap_matrix_mdev *matrix_mdev,
>> - int apqn)
>> +static struct vfio_ap_queue *vfio_ap_get_queue(unsigned long apqn)
>> {
>> + struct ap_queue *queue;
>> struct vfio_ap_queue *q;
>> - struct device *dev;
>> - if (!test_bit_inv(AP_QID_CARD(apqn), matrix_mdev->matrix.apm))
>> - return NULL;
>> - if (!test_bit_inv(AP_QID_QUEUE(apqn), matrix_mdev->matrix.aqm))
>> + queue = ap_get_qdev(apqn);
>> + if (!queue)
>> return NULL;
>> - dev = driver_find_device(&matrix_dev->vfio_ap_drv->driver, NULL,
>> - &apqn, match_apqn);
>> - if (!dev)
>> - return NULL;
>> - q = dev_get_drvdata(dev);
>> - q->matrix_mdev = matrix_mdev;
>> - put_device(dev);
>> + q = dev_get_drvdata(&queue->ap_dev.device);
>> + put_device(&queue->ap_dev.device);
>> return q;
>> }
>
> this function changed a lot too, you should explain the goal in the
> patch comment.
>
> ...snip...
>
> Regards,
> Pierre
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v9 02/15] s390/vfio-ap: use new AP bus interface to search for queue devices
2020-07-24 8:38 ` Pierre Morel
2020-07-27 13:42 ` Tony Krowiak
@ 2020-07-27 14:09 ` Tony Krowiak
1 sibling, 0 replies; 21+ messages in thread
From: Tony Krowiak @ 2020-07-27 14:09 UTC (permalink / raw)
To: Pierre Morel, linux-s390, linux-kernel, kvm
Cc: freude, borntraeger, cohuck, mjrosato, pasic, alex.williamson,
kwankhede, fiuczy, kernel test robot
On 7/24/20 4:38 AM, Pierre Morel wrote:
>
>
> On 2020-07-20 17:03, Tony Krowiak wrote:
>> This patch refactor's the vfio_ap device driver to use the AP bus's
>> ap_get_qdev() function to retrieve the vfio_ap_queue struct containing
>> information about a queue that is bound to the vfio_ap device driver.
>> The bus's ap_get_qdev() function retrieves the queue device from a
>> hashtable keyed by APQN. This is much more efficient than looping over
>> the list of devices attached to the AP bus by several orders of
>> magnitude.
>
> The patch does much more than modifying this line. ;)
>
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
>> Reported-by: kernel test robot <lkp@intel.com>
>> ---
>> drivers/s390/crypto/vfio_ap_drv.c | 27 ++-------
>> drivers/s390/crypto/vfio_ap_ops.c | 86 +++++++++++++++------------
>> drivers/s390/crypto/vfio_ap_private.h | 8 ++-
>> 3 files changed, 59 insertions(+), 62 deletions(-)
>>
>> diff --git a/drivers/s390/crypto/vfio_ap_drv.c
>> b/drivers/s390/crypto/vfio_ap_drv.c
>> index f4ceb380dd61..24cdef60039a 100644
>> --- a/drivers/s390/crypto/vfio_ap_drv.c
>> +++ b/drivers/s390/crypto/vfio_ap_drv.c
>> @@ -53,15 +53,9 @@ MODULE_DEVICE_TABLE(vfio_ap, ap_queue_ids);
>> */
>> static int vfio_ap_queue_dev_probe(struct ap_device *apdev)
>> {
>> - struct vfio_ap_queue *q;
>> -
>> - q = kzalloc(sizeof(*q), GFP_KERNEL);
>> - if (!q)
>> - return -ENOMEM;
>> - dev_set_drvdata(&apdev->device, q);
>> - q->apqn = to_ap_queue(&apdev->device)->qid;
>> - q->saved_isc = VFIO_AP_ISC_INVALID;
>> - return 0;
>> + struct ap_queue *queue = to_ap_queue(&apdev->device);
>> +
>> + return vfio_ap_mdev_probe_queue(queue);
>> }
>
> You should explain the reason why this function is modified.
>
>> /**
>> @@ -72,18 +66,9 @@ static int vfio_ap_queue_dev_probe(struct
>> ap_device *apdev)
>> */
>> static void vfio_ap_queue_dev_remove(struct ap_device *apdev)
>> {
>> - struct vfio_ap_queue *q;
>> - int apid, apqi;
>> -
>> - mutex_lock(&matrix_dev->lock);
>> - q = dev_get_drvdata(&apdev->device);
>> - dev_set_drvdata(&apdev->device, NULL);
>> - apid = AP_QID_CARD(q->apqn);
>> - apqi = AP_QID_QUEUE(q->apqn);
>> - vfio_ap_mdev_reset_queue(apid, apqi, 1);
>> - vfio_ap_irq_disable(q);
>> - kfree(q);
>> - mutex_unlock(&matrix_dev->lock);
>> + struct ap_queue *queue = to_ap_queue(&apdev->device);
>> +
>> + vfio_ap_mdev_remove_queue(queue);
>> }
>
> ... and this one?
>
>> static void vfio_ap_matrix_dev_release(struct device *dev)
>> diff --git a/drivers/s390/crypto/vfio_ap_ops.c
>> b/drivers/s390/crypto/vfio_ap_ops.c
>> index e0bde8518745..ad3925f04f61 100644
>> --- a/drivers/s390/crypto/vfio_ap_ops.c
>> +++ b/drivers/s390/crypto/vfio_ap_ops.c
>> @@ -26,43 +26,26 @@
>> static int vfio_ap_mdev_reset_queues(struct mdev_device *mdev);
>> -static int match_apqn(struct device *dev, const void *data)
>> -{
>> - struct vfio_ap_queue *q = dev_get_drvdata(dev);
>> -
>> - return (q->apqn == *(int *)(data)) ? 1 : 0;
>> -}
>> -
>> /**
>> - * vfio_ap_get_queue: Retrieve a queue with a specific APQN from a list
>> - * @matrix_mdev: the associated mediated matrix
>> + * vfio_ap_get_queue: Retrieve a queue with a specific APQN.
>> * @apqn: The queue APQN
>> *
>> - * Retrieve a queue with a specific APQN from the list of the
>> - * devices of the vfio_ap_drv.
>> - * Verify that the APID and the APQI are set in the matrix.
>> + * Retrieve a queue with a specific APQN from the AP queue devices
>> attached to
>> + * the AP bus.
>> *
>> - * Returns the pointer to the associated vfio_ap_queue
>> + * Returns the pointer to the vfio_ap_queue with the specified APQN,
>> or NULL.
>> */
>> -static struct vfio_ap_queue *vfio_ap_get_queue(
>> - struct ap_matrix_mdev *matrix_mdev,
>> - int apqn)
>> +static struct vfio_ap_queue *vfio_ap_get_queue(unsigned long apqn)
>> {
>> + struct ap_queue *queue;
>> struct vfio_ap_queue *q;
>> - struct device *dev;
>> - if (!test_bit_inv(AP_QID_CARD(apqn), matrix_mdev->matrix.apm))
>> - return NULL;
>> - if (!test_bit_inv(AP_QID_QUEUE(apqn), matrix_mdev->matrix.aqm))
>> + queue = ap_get_qdev(apqn);
>> + if (!queue)
>> return NULL;
>> - dev = driver_find_device(&matrix_dev->vfio_ap_drv->driver, NULL,
>> - &apqn, match_apqn);
>> - if (!dev)
>> - return NULL;
>> - q = dev_get_drvdata(dev);
>> - q->matrix_mdev = matrix_mdev;
>> - put_device(dev);
>> + q = dev_get_drvdata(&queue->ap_dev.device);
>> + put_device(&queue->ap_dev.device);
>> return q;
>> }
>
> this function changed a lot too, you should explain the goal in the
> patch comment.
This is precisely what the current patch comment describes.
>
> ...snip...
>
> Regards,
> Pierre
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v9 00/15] s390/vfio-ap: dynamic configuration support
2020-07-20 15:03 [PATCH v9 00/15] s390/vfio-ap: dynamic configuration support Tony Krowiak
` (14 preceding siblings ...)
2020-07-20 15:03 ` [PATCH v9 15/15] s390/vfio-ap: handle probe/remove not due to host AP config changes Tony Krowiak
@ 2020-08-04 14:28 ` Tony Krowiak
2020-08-10 15:39 ` Tony Krowiak
16 siblings, 0 replies; 21+ messages in thread
From: Tony Krowiak @ 2020-08-04 14:28 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, borntraeger, cohuck, mjrosato, pasic, alex.williamson,
kwankhede, fiuczy
PING!
On 7/20/20 11:03 AM, Tony Krowiak wrote:
> The current design for AP pass-through does not support making dynamic
> changes to the AP matrix of a running guest resulting in a few
> deficiencies this patch series is intended to mitigate:
>
> 1. Adapters, domains and control domains can not be added to or removed
> from a running guest. In order to modify a guest's AP configuration,
> the guest must be terminated; only then can AP resources be assigned
> to or unassigned from the guest's matrix mdev. The new AP
> configuration becomes available to the guest when it is subsequently
> restarted.
>
> 2. The AP bus's /sys/bus/ap/apmask and /sys/bus/ap/aqmask interfaces can
> be modified by a root user without any restrictions. A change to
> either mask can result in AP queue devices being unbound from the
> vfio_ap device driver and bound to a zcrypt device driver even if a
> guest is using the queues, thus giving the host access to the guest's
> private crypto data and vice versa.
>
> 3. The APQNs derived from the Cartesian product of the APIDs of the
> adapters and APQIs of the domains assigned to a matrix mdev must
> reference an AP queue device bound to the vfio_ap device driver. The
> AP architecture allows assignment of AP resources that are not
> available to the system, so this artificial restriction is not
> compliant with the architecture.
>
> 4. The AP configuration profile can be dynamically changed for the linux
> host after a KVM guest is started. For example, a new domain can be
> dynamically added to the configuration profile via the SE or an HMC
> connected to a DPM enabled lpar. Likewise, AP adapters can be
> dynamically configured (online state) and deconfigured (standby state)
> using the SE, an SCLP command or an HMC connected to a DPM enabled
> lpar. This can result in inadvertent sharing of AP queues between the
> guest and host.
>
> 5. A root user can manually unbind an AP queue device representing a
> queue in use by a KVM guest via the vfio_ap device driver's sysfs
> unbind attribute. In this case, the guest will be using a queue that
> is not bound to the driver which violates the device model.
>
> This patch series introduces the following changes to the current design
> to alleviate the shortcomings described above as well as to implement
> more of the AP architecture:
>
> 1. A root user will be prevented from making changes to the AP bus's
> /sys/bus/ap/apmask or /sys/bus/ap/aqmask if the ownership of an APQN
> changes from the vfio_ap device driver to a zcrypt driver when the
> APQN is assigned to a matrix mdev.
>
> 2. Allow a root user to hot plug/unplug AP adapters, domains and control
> domains using the matrix mdev's assign/unassign attributes.
>
> 4. Allow assignment of an AP adapter or domain to a matrix mdev even if
> it results in assignment of an APQN that does not reference an AP
> queue device bound to the vfio_ap device driver, as long as the APQN
> is not reserved for use by the default zcrypt drivers (also known as
> over-provisioning of AP resources). Allowing over-provisioning of AP
> resources better models the architecture which does not preclude
> assigning AP resources that are not yet available in the system. Such
> APQNs, however, will not be assigned to the guest using the matrix
> mdev; only APQNs referencing AP queue devices bound to the vfio_ap
> device driver will actually get assigned to the guest.
>
> 5. Handle dynamic changes to the AP device model.
>
> 1. Rationale for changes to AP bus's apmask/aqmask interfaces:
> ----------------------------------------------------------
> Due to the extremely sensitive nature of cryptographic data, it is
> imperative that great care be taken to ensure that such data is secured.
> Allowing a root user, either inadvertently or maliciously, to configure
> these masks such that a queue is shared between the host and a guest is
> not only avoidable, it is advisable. It was suggested that this scenario
> is better handled in user space with management software, but that does
> not preclude a malicious administrator from using the sysfs interfaces
> to gain access to a guest's crypto data. It was also suggested that this
> scenario could be avoided by taking access to the adapter away from the
> guest and zeroing out the queues prior to the vfio_ap driver releasing the
> device; however, stealing an adapter in use from a guest as a by-product
> of an operation is bad and will likely cause problems for the guest
> unnecessarily. It was decided that the most effective solution with the
> least number of negative side effects is to prevent the situation at the
> source.
>
> 2. Rationale for hot plug/unplug using matrix mdev sysfs interfaces:
> ----------------------------------------------------------------
> Allowing a user to hot plug/unplug AP resources using the matrix mdev
> sysfs interfaces circumvents the need to terminate the guest in order to
> modify its AP configuration. Allowing dynamic configuration makes
> reconfiguring a guest's AP matrix much less disruptive.
>
> 3. Rationale for allowing over-provisioning of AP resources:
> -----------------------------------------------------------
> Allowing assignment of AP resources to a matrix mdev and ultimately to a
> guest better models the AP architecture. The architecture does not
> preclude assignment of unavailable AP resources. If a queue subsequently
> becomes available while a guest using the matrix mdev to which its APQN
> is assigned, the guest will be given access to it. If an APQN
> is dynamically unassigned from the underlying host system, it will
> automatically become unavailable to the guest.
>
> Change log v8-v9:
> ----------------
> * Fixed errors flagged by the kernel test robot
>
> * Fixed issue with guest losing queues when a new queue is probed due to
> manual bind operation.
>
> Change log v7-v8:
> ----------------
> * Now logging a message when an attempt to reserve APQNs for the zcrypt
> drivers will result in taking a queue away from a KVM guest to provide
> the sysadmin a way to ascertain why the sysfs operation failed.
>
> * Created locked and unlocked versions of the ap_parse_mask_str() function.
>
> * Now using new interface provided by an AP bus patch -
> s390/ap: introduce new ap function ap_get_qdev() - to retrieve
> struct ap_queue representing an AP queue device. This patch is not a
> part of this series but is a prerequisite for this series.
>
> Change log v6-v7:
> ----------------
> * Added callbacks to AP bus:
> - on_config_changed: Notifies implementing drivers that
> the AP configuration has changed since last AP device scan.
> - on_scan_complete: Notifies implementing drivers that the device scan
> has completed.
> - implemented on_config_changed and on_scan_complete callbacks for
> vfio_ap device driver.
> - updated vfio_ap device driver's probe and remove callbacks to handle
> dynamic changes to the AP device model.
> * Added code to filter APQNs when assigning AP resources to a KVM guest's
> CRYCB
>
> Change log v5-v6:
> ----------------
> * Fixed a bug in ap_bus.c introduced with patch 2/7 of the v5
> series. Harald Freudenberer pointed out that the mutex lock
> for ap_perms_mutex in the apmask_store and aqmask_store functions
> was not being freed.
>
> * Removed patch 6/7 which added logging to the vfio_ap driver
> to expedite acceptance of this series. The logging will be introduced
> with a separate patch series to allow more time to explore options
> such as DBF logging vs. tracepoints.
>
> * Added 3 patches related to ensuring that APQNs that do not reference
> AP queue devices bound to the vfio_ap device driver are not assigned
> to the guest CRYCB:
>
> Patch 4: Filter CRYCB bits for unavailable queue devices
> Patch 5: sysfs attribute to display the guest CRYCB
> Patch 6: update guest CRYCB in vfio_ap probe and remove callbacks
>
> * Added a patch (Patch 9) to version the vfio_ap module.
>
> * Reshuffled patches to allow the in_use callback implementation to
> invoke the vfio_ap_mdev_verify_no_sharing() function introduced in
> patch 2.
>
> Change log v4-v5:
> ----------------
> * Added a patch to provide kernel s390dbf debug logs for VFIO AP
>
> Change log v3->v4:
> -----------------
> * Restored patches preventing root user from changing ownership of
> APQNs from zcrypt drivers to the vfio_ap driver if the APQN is
> assigned to an mdev.
>
> * No longer enforcing requirement restricting guest access to
> queues represented by a queue device bound to the vfio_ap
> device driver.
>
> * Removed shadow CRYCB and now directly updating the guest CRYCB
> from the matrix mdev's matrix.
>
> * Rebased the patch series on top of 'vfio: ap: AP Queue Interrupt
> Control' patches.
>
> * Disabled bind/unbind sysfs interfaces for vfio_ap driver
>
> Change log v2->v3:
> -----------------
> * Allow guest access to an AP queue only if the queue is bound to
> the vfio_ap device driver.
>
> * Removed the patch to test CRYCB masks before taking the vCPUs
> out of SIE. Now checking the shadow CRYCB in the vfio_ap driver.
>
> Change log v1->v2:
> -----------------
> * Removed patches preventing root user from unbinding AP queues from
> the vfio_ap device driver
> * Introduced a shadow CRYCB in the vfio_ap driver to manage dynamic
> changes to the AP guest configuration due to root user interventions
> or hardware anomalies.
>
> Harald Freudenberger (1):
> s390/zcrypt: Notify driver on config changed and scan complete
> callbacks
>
> Tony Krowiak (14):
> s390/vfio-ap: add version vfio_ap module
> s390/vfio-ap: use new AP bus interface to search for queue devices
> s390/vfio-ap: manage link between queue struct and matrix mdev
> s390/zcrypt: driver callback to indicate resource in use
> s390/vfio-ap: implement in-use callback for vfio_ap driver
> s390/vfio-ap: introduce shadow APCB
> s390/vfio-ap: sysfs attribute to display the guest's matrix
> s390/vfio-ap: filter matrix for unavailable queue devices
> s390/vfio-ap: allow assignment of unavailable AP queues to mdev device
> s390/vfio-ap: allow configuration of matrix mdev in use by a KVM guest
> s390/vfio-ap: allow hot plug/unplug of AP resources using mdev device
> s390/vfio-ap: handle host AP config change notification
> s390/vfio-ap: handle AP bus scan completed notification
> s390/vfio-ap: handle probe/remove not due to host AP config changes
>
> drivers/s390/crypto/ap_bus.c | 323 +++++--
> drivers/s390/crypto/ap_bus.h | 16 +
> drivers/s390/crypto/vfio_ap_drv.c | 36 +-
> drivers/s390/crypto/vfio_ap_ops.c | 1216 ++++++++++++++++++++-----
> drivers/s390/crypto/vfio_ap_private.h | 23 +-
> 5 files changed, 1294 insertions(+), 320 deletions(-)
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v9 00/15] s390/vfio-ap: dynamic configuration support
2020-07-20 15:03 [PATCH v9 00/15] s390/vfio-ap: dynamic configuration support Tony Krowiak
` (15 preceding siblings ...)
2020-08-04 14:28 ` [PATCH v9 00/15] s390/vfio-ap: dynamic configuration support Tony Krowiak
@ 2020-08-10 15:39 ` Tony Krowiak
16 siblings, 0 replies; 21+ messages in thread
From: Tony Krowiak @ 2020-08-10 15:39 UTC (permalink / raw)
To: linux-s390, linux-kernel, kvm
Cc: freude, borntraeger, cohuck, mjrosato, pasic, alex.williamson,
kwankhede, fiuczy
PING, PING
On 7/20/20 11:03 AM, Tony Krowiak wrote:
> The current design for AP pass-through does not support making dynamic
> changes to the AP matrix of a running guest resulting in a few
> deficiencies this patch series is intended to mitigate:
>
> 1. Adapters, domains and control domains can not be added to or removed
> from a running guest. In order to modify a guest's AP configuration,
> the guest must be terminated; only then can AP resources be assigned
> to or unassigned from the guest's matrix mdev. The new AP
> configuration becomes available to the guest when it is subsequently
> restarted.
>
> 2. The AP bus's /sys/bus/ap/apmask and /sys/bus/ap/aqmask interfaces can
> be modified by a root user without any restrictions. A change to
> either mask can result in AP queue devices being unbound from the
> vfio_ap device driver and bound to a zcrypt device driver even if a
> guest is using the queues, thus giving the host access to the guest's
> private crypto data and vice versa.
>
> 3. The APQNs derived from the Cartesian product of the APIDs of the
> adapters and APQIs of the domains assigned to a matrix mdev must
> reference an AP queue device bound to the vfio_ap device driver. The
> AP architecture allows assignment of AP resources that are not
> available to the system, so this artificial restriction is not
> compliant with the architecture.
>
> 4. The AP configuration profile can be dynamically changed for the linux
> host after a KVM guest is started. For example, a new domain can be
> dynamically added to the configuration profile via the SE or an HMC
> connected to a DPM enabled lpar. Likewise, AP adapters can be
> dynamically configured (online state) and deconfigured (standby state)
> using the SE, an SCLP command or an HMC connected to a DPM enabled
> lpar. This can result in inadvertent sharing of AP queues between the
> guest and host.
>
> 5. A root user can manually unbind an AP queue device representing a
> queue in use by a KVM guest via the vfio_ap device driver's sysfs
> unbind attribute. In this case, the guest will be using a queue that
> is not bound to the driver which violates the device model.
>
> This patch series introduces the following changes to the current design
> to alleviate the shortcomings described above as well as to implement
> more of the AP architecture:
>
> 1. A root user will be prevented from making changes to the AP bus's
> /sys/bus/ap/apmask or /sys/bus/ap/aqmask if the ownership of an APQN
> changes from the vfio_ap device driver to a zcrypt driver when the
> APQN is assigned to a matrix mdev.
>
> 2. Allow a root user to hot plug/unplug AP adapters, domains and control
> domains using the matrix mdev's assign/unassign attributes.
>
> 4. Allow assignment of an AP adapter or domain to a matrix mdev even if
> it results in assignment of an APQN that does not reference an AP
> queue device bound to the vfio_ap device driver, as long as the APQN
> is not reserved for use by the default zcrypt drivers (also known as
> over-provisioning of AP resources). Allowing over-provisioning of AP
> resources better models the architecture which does not preclude
> assigning AP resources that are not yet available in the system. Such
> APQNs, however, will not be assigned to the guest using the matrix
> mdev; only APQNs referencing AP queue devices bound to the vfio_ap
> device driver will actually get assigned to the guest.
>
> 5. Handle dynamic changes to the AP device model.
>
> 1. Rationale for changes to AP bus's apmask/aqmask interfaces:
> ----------------------------------------------------------
> Due to the extremely sensitive nature of cryptographic data, it is
> imperative that great care be taken to ensure that such data is secured.
> Allowing a root user, either inadvertently or maliciously, to configure
> these masks such that a queue is shared between the host and a guest is
> not only avoidable, it is advisable. It was suggested that this scenario
> is better handled in user space with management software, but that does
> not preclude a malicious administrator from using the sysfs interfaces
> to gain access to a guest's crypto data. It was also suggested that this
> scenario could be avoided by taking access to the adapter away from the
> guest and zeroing out the queues prior to the vfio_ap driver releasing the
> device; however, stealing an adapter in use from a guest as a by-product
> of an operation is bad and will likely cause problems for the guest
> unnecessarily. It was decided that the most effective solution with the
> least number of negative side effects is to prevent the situation at the
> source.
>
> 2. Rationale for hot plug/unplug using matrix mdev sysfs interfaces:
> ----------------------------------------------------------------
> Allowing a user to hot plug/unplug AP resources using the matrix mdev
> sysfs interfaces circumvents the need to terminate the guest in order to
> modify its AP configuration. Allowing dynamic configuration makes
> reconfiguring a guest's AP matrix much less disruptive.
>
> 3. Rationale for allowing over-provisioning of AP resources:
> -----------------------------------------------------------
> Allowing assignment of AP resources to a matrix mdev and ultimately to a
> guest better models the AP architecture. The architecture does not
> preclude assignment of unavailable AP resources. If a queue subsequently
> becomes available while a guest using the matrix mdev to which its APQN
> is assigned, the guest will be given access to it. If an APQN
> is dynamically unassigned from the underlying host system, it will
> automatically become unavailable to the guest.
>
> Change log v8-v9:
> ----------------
> * Fixed errors flagged by the kernel test robot
>
> * Fixed issue with guest losing queues when a new queue is probed due to
> manual bind operation.
>
> Change log v7-v8:
> ----------------
> * Now logging a message when an attempt to reserve APQNs for the zcrypt
> drivers will result in taking a queue away from a KVM guest to provide
> the sysadmin a way to ascertain why the sysfs operation failed.
>
> * Created locked and unlocked versions of the ap_parse_mask_str() function.
>
> * Now using new interface provided by an AP bus patch -
> s390/ap: introduce new ap function ap_get_qdev() - to retrieve
> struct ap_queue representing an AP queue device. This patch is not a
> part of this series but is a prerequisite for this series.
>
> Change log v6-v7:
> ----------------
> * Added callbacks to AP bus:
> - on_config_changed: Notifies implementing drivers that
> the AP configuration has changed since last AP device scan.
> - on_scan_complete: Notifies implementing drivers that the device scan
> has completed.
> - implemented on_config_changed and on_scan_complete callbacks for
> vfio_ap device driver.
> - updated vfio_ap device driver's probe and remove callbacks to handle
> dynamic changes to the AP device model.
> * Added code to filter APQNs when assigning AP resources to a KVM guest's
> CRYCB
>
> Change log v5-v6:
> ----------------
> * Fixed a bug in ap_bus.c introduced with patch 2/7 of the v5
> series. Harald Freudenberer pointed out that the mutex lock
> for ap_perms_mutex in the apmask_store and aqmask_store functions
> was not being freed.
>
> * Removed patch 6/7 which added logging to the vfio_ap driver
> to expedite acceptance of this series. The logging will be introduced
> with a separate patch series to allow more time to explore options
> such as DBF logging vs. tracepoints.
>
> * Added 3 patches related to ensuring that APQNs that do not reference
> AP queue devices bound to the vfio_ap device driver are not assigned
> to the guest CRYCB:
>
> Patch 4: Filter CRYCB bits for unavailable queue devices
> Patch 5: sysfs attribute to display the guest CRYCB
> Patch 6: update guest CRYCB in vfio_ap probe and remove callbacks
>
> * Added a patch (Patch 9) to version the vfio_ap module.
>
> * Reshuffled patches to allow the in_use callback implementation to
> invoke the vfio_ap_mdev_verify_no_sharing() function introduced in
> patch 2.
>
> Change log v4-v5:
> ----------------
> * Added a patch to provide kernel s390dbf debug logs for VFIO AP
>
> Change log v3->v4:
> -----------------
> * Restored patches preventing root user from changing ownership of
> APQNs from zcrypt drivers to the vfio_ap driver if the APQN is
> assigned to an mdev.
>
> * No longer enforcing requirement restricting guest access to
> queues represented by a queue device bound to the vfio_ap
> device driver.
>
> * Removed shadow CRYCB and now directly updating the guest CRYCB
> from the matrix mdev's matrix.
>
> * Rebased the patch series on top of 'vfio: ap: AP Queue Interrupt
> Control' patches.
>
> * Disabled bind/unbind sysfs interfaces for vfio_ap driver
>
> Change log v2->v3:
> -----------------
> * Allow guest access to an AP queue only if the queue is bound to
> the vfio_ap device driver.
>
> * Removed the patch to test CRYCB masks before taking the vCPUs
> out of SIE. Now checking the shadow CRYCB in the vfio_ap driver.
>
> Change log v1->v2:
> -----------------
> * Removed patches preventing root user from unbinding AP queues from
> the vfio_ap device driver
> * Introduced a shadow CRYCB in the vfio_ap driver to manage dynamic
> changes to the AP guest configuration due to root user interventions
> or hardware anomalies.
>
> Harald Freudenberger (1):
> s390/zcrypt: Notify driver on config changed and scan complete
> callbacks
>
> Tony Krowiak (14):
> s390/vfio-ap: add version vfio_ap module
> s390/vfio-ap: use new AP bus interface to search for queue devices
> s390/vfio-ap: manage link between queue struct and matrix mdev
> s390/zcrypt: driver callback to indicate resource in use
> s390/vfio-ap: implement in-use callback for vfio_ap driver
> s390/vfio-ap: introduce shadow APCB
> s390/vfio-ap: sysfs attribute to display the guest's matrix
> s390/vfio-ap: filter matrix for unavailable queue devices
> s390/vfio-ap: allow assignment of unavailable AP queues to mdev device
> s390/vfio-ap: allow configuration of matrix mdev in use by a KVM guest
> s390/vfio-ap: allow hot plug/unplug of AP resources using mdev device
> s390/vfio-ap: handle host AP config change notification
> s390/vfio-ap: handle AP bus scan completed notification
> s390/vfio-ap: handle probe/remove not due to host AP config changes
>
> drivers/s390/crypto/ap_bus.c | 323 +++++--
> drivers/s390/crypto/ap_bus.h | 16 +
> drivers/s390/crypto/vfio_ap_drv.c | 36 +-
> drivers/s390/crypto/vfio_ap_ops.c | 1216 ++++++++++++++++++++-----
> drivers/s390/crypto/vfio_ap_private.h | 23 +-
> 5 files changed, 1294 insertions(+), 320 deletions(-)
>
^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2020-08-10 15:40 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-07-20 15:03 [PATCH v9 00/15] s390/vfio-ap: dynamic configuration support Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 01/15] s390/vfio-ap: add version vfio_ap module Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 02/15] s390/vfio-ap: use new AP bus interface to search for queue devices Tony Krowiak
2020-07-24 8:38 ` Pierre Morel
2020-07-27 13:42 ` Tony Krowiak
2020-07-27 14:09 ` Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 03/15] s390/vfio-ap: manage link between queue struct and matrix mdev Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 04/15] s390/zcrypt: driver callback to indicate resource in use Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 05/15] s390/vfio-ap: implement in-use callback for vfio_ap driver Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 06/15] s390/vfio-ap: introduce shadow APCB Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 07/15] s390/vfio-ap: sysfs attribute to display the guest's matrix Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 08/15] s390/vfio-ap: filter matrix for unavailable queue devices Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 09/15] s390/vfio-ap: allow assignment of unavailable AP queues to mdev device Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 10/15] s390/vfio-ap: allow configuration of matrix mdev in use by a KVM guest Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 11/15] s390/vfio-ap: allow hot plug/unplug of AP resources using mdev device Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 12/15] s390/zcrypt: Notify driver on config changed and scan complete callbacks Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 13/15] s390/vfio-ap: handle host AP config change notification Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 14/15] s390/vfio-ap: handle AP bus scan completed notification Tony Krowiak
2020-07-20 15:03 ` [PATCH v9 15/15] s390/vfio-ap: handle probe/remove not due to host AP config changes Tony Krowiak
2020-08-04 14:28 ` [PATCH v9 00/15] s390/vfio-ap: dynamic configuration support Tony Krowiak
2020-08-10 15:39 ` Tony Krowiak
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).