From: Tony Krowiak <akrowiak@linux.ibm.com>
To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org,
kvm@vger.kernel.org
Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com,
mjrosato@linux.ibm.com, pasic@linux.ibm.com,
alex.williamson@redhat.com, kwankhede@nvidia.com,
fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com,
hca@linux.ibm.com, gor@linux.ibm.com,
Tony Krowiak <akrowiak@linux.ibm.com>
Subject: [PATCH v11 10/14] s390/vfio-ap: allow hot plug/unplug of AP resources using mdev device
Date: Thu, 22 Oct 2020 13:12:05 -0400 [thread overview]
Message-ID: <20201022171209.19494-11-akrowiak@linux.ibm.com> (raw)
In-Reply-To: <20201022171209.19494-1-akrowiak@linux.ibm.com>
Let's hot plug/unplug adapters, domains and control domains assigned to or
unassigned from an AP matrix mdev device while it is in use by a guest per
the following:
* Hot plug AP adapter:
When the APID of an adapter is assigned to a matrix mdev in use by a KVM
guest, the adapter will be hot plugged into the KVM guest as long as each
APQN derived from the Cartesian product of the APID being assigned and
the APQIs already assigned to the matrix mdev references a queue device
bound to the vfio_ap device driver.
* Hot unplug adapter:
When the APID of an adapter is unassigned from a matrix mdev in use by a
KVM guest, the adapter will be hot unplugged from the KVM guest.
* Hot plug domain:
When the APQI of a domain is assigned to a matrix mdev in use by a KVM
guest, the domain will be hot plugged into the KVM guest as long as each
APQN derived from the Cartesian product of the APQI being assigned and
the APIDs already assigned to the matrix mdev references a queue device
bound to the vfio_ap device driver.
* Hot unplug domain:
When the APQI of a domain is unassigned from a matrix mdev in use by a
KVM guest, the domain will be hot unplugged from the KVM guest
* Hot plug control domain:
When the domain number of a control domain is assigned to a matrix mdev
in use by a KVM guest, the control domain will be hot plugged into the
KVM guest. The AP architecture ensures a guest will only get access to
the control domain if it is in the host's AP configuration, so there is
no risk in hot plugging it; however, it will become automatically
available to the guest when it is added to the host configuration.
* Hot unplug control domain:
When the domain number of a control domain is unassigned from a matrix
mdev in use by a KVM guest, the control domain will be hot unplugged
from the KVM guest.
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
---
drivers/s390/crypto/vfio_ap_ops.c | 148 ++++++++++++++++++++++++------
1 file changed, 119 insertions(+), 29 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index c2c6dcec8829..dae1fba41941 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -517,12 +517,18 @@ static bool vfio_ap_mdev_matrixes_equal(struct ap_matrix *matrix1,
* a matrix mdev's AP configuration and stores the result in the shadow copy of
* the APCB used to supply a KVM guest's AP configuration.
*
+ * Note: Filtering is applied only to adapters and domains. Changes to control
+ * domains will always be reflected in the shadow APCB.
+ *
* @matrix_mdev: the matrix mdev whose AP configuration is to be filtered
+ * @filter_apid: indicates whether APIDs (true) or APQIs (false) shall be
+ * filtered
*
* Returns true if filtering has changed the shadow copy of the APCB used
* to supply a KVM guest's AP configuration; otherwise, returns false.
*/
-static int vfio_ap_mdev_filter_guest_matrix(struct ap_matrix_mdev *matrix_mdev)
+static int vfio_ap_mdev_filter_guest_matrix(struct ap_matrix_mdev *matrix_mdev,
+ bool filter_apid)
{
struct ap_matrix shadow_apcb;
unsigned long apid, apqi, apqn;
@@ -561,9 +567,15 @@ static int vfio_ap_mdev_filter_guest_matrix(struct ap_matrix_mdev *matrix_mdev)
* the APID.
*/
apqn = AP_MKQID(apid, apqi);
+
if (!vfio_ap_mdev_get_queue(matrix_mdev, apqn)) {
- clear_bit_inv(apid, shadow_apcb.apm);
- break;
+ if (filter_apid) {
+ clear_bit_inv(apid, shadow_apcb.apm);
+ break;
+ }
+
+ clear_bit_inv(apqi, shadow_apcb.aqm);
+ continue;
}
}
@@ -723,10 +735,6 @@ static ssize_t assign_adapter_store(struct device *dev,
struct mdev_device *mdev = mdev_from_dev(dev);
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
- /* If the guest is running, disallow assignment of adapter */
- if (matrix_mdev->kvm)
- return -EBUSY;
-
ret = kstrtoul(buf, 0, &apid);
if (ret)
return ret;
@@ -746,12 +754,44 @@ static ssize_t assign_adapter_store(struct device *dev,
}
set_bit_inv(apid, matrix_mdev->matrix.apm);
vfio_ap_mdev_link_queues(matrix_mdev, LINK_APID, apid);
+
+ if (vfio_ap_mdev_has_crycb(matrix_mdev))
+ if (vfio_ap_mdev_filter_guest_matrix(matrix_mdev, true))
+ vfio_ap_mdev_commit_shadow_apcb(matrix_mdev);
mutex_unlock(&matrix_dev->lock);
return count;
}
static DEVICE_ATTR_WO(assign_adapter);
+static bool vfio_ap_mdev_unassign_guest_apid(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long apid)
+{
+ if (vfio_ap_mdev_has_crycb(matrix_mdev)) {
+ if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm)) {
+ clear_bit_inv(apid, matrix_mdev->shadow_apcb.apm);
+
+ /*
+ * If there are no APIDs assigned to the guest, then
+ * the guest will not have access to any queues, so
+ * let's also go ahead and unassign the APQIs. Keeping
+ * them around may yield unpredictable results during
+ * a probe that is not related to a host AP
+ * configuration change (i.e., an AP adapter is
+ * configured online).
+ */
+ if (bitmap_empty(matrix_mdev->shadow_apcb.apm,
+ AP_DEVICES))
+ bitmap_clear(matrix_mdev->shadow_apcb.aqm, 0,
+ AP_DOMAINS);
+
+ return true;
+ }
+ }
+
+ return false;
+}
+
/**
* unassign_adapter_store
*
@@ -778,10 +818,6 @@ static ssize_t unassign_adapter_store(struct device *dev,
struct mdev_device *mdev = mdev_from_dev(dev);
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
- /* If the guest is running, disallow un-assignment of adapter */
- if (matrix_mdev->kvm)
- return -EBUSY;
-
ret = kstrtoul(buf, 0, &apid);
if (ret)
return ret;
@@ -792,6 +828,9 @@ static ssize_t unassign_adapter_store(struct device *dev,
mutex_lock(&matrix_dev->lock);
clear_bit_inv((unsigned long)apid, matrix_mdev->matrix.apm);
vfio_ap_mdev_link_queues(matrix_mdev, UNLINK_APID, apid);
+
+ if (vfio_ap_mdev_unassign_guest_apid(matrix_mdev, apid))
+ vfio_ap_mdev_commit_shadow_apcb(matrix_mdev);
mutex_unlock(&matrix_dev->lock);
return count;
@@ -841,10 +880,6 @@ static ssize_t assign_domain_store(struct device *dev,
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
unsigned long max_apqi = matrix_mdev->matrix.aqm_max;
- /* If the guest is running, disallow assignment of domain */
- if (matrix_mdev->kvm)
- return -EBUSY;
-
ret = kstrtoul(buf, 0, &apqi);
if (ret)
return ret;
@@ -863,12 +898,43 @@ static ssize_t assign_domain_store(struct device *dev,
}
set_bit_inv(apqi, matrix_mdev->matrix.aqm);
vfio_ap_mdev_link_queues(matrix_mdev, LINK_APQI, apqi);
+
+ if (vfio_ap_mdev_has_crycb(matrix_mdev))
+ if (vfio_ap_mdev_filter_guest_matrix(matrix_mdev, false))
+ vfio_ap_mdev_commit_shadow_apcb(matrix_mdev);
mutex_unlock(&matrix_dev->lock);
return count;
}
static DEVICE_ATTR_WO(assign_domain);
+static bool vfio_ap_mdev_unassign_guest_apqi(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long apqi)
+{
+ if (vfio_ap_mdev_has_crycb(matrix_mdev)) {
+ if (test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm)) {
+ clear_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm);
+
+ /*
+ * If there are no APQIs assigned to the guest, then
+ * the guest will not have access to any queues, so
+ * let's also go ahead and unassign the APIDs. Keeping
+ * them around may yield unpredictable results during
+ * a probe that is not related to a host AP
+ * configuration change (i.e., an AP adapter is
+ * configured online).
+ */
+ if (bitmap_empty(matrix_mdev->shadow_apcb.aqm,
+ AP_DOMAINS))
+ bitmap_clear(matrix_mdev->shadow_apcb.apm, 0,
+ AP_DEVICES);
+
+ return true;
+ }
+ }
+
+ return false;
+}
/**
* unassign_domain_store
@@ -896,10 +962,6 @@ static ssize_t unassign_domain_store(struct device *dev,
struct mdev_device *mdev = mdev_from_dev(dev);
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
- /* If the guest is running, disallow un-assignment of domain */
- if (matrix_mdev->kvm)
- return -EBUSY;
-
ret = kstrtoul(buf, 0, &apqi);
if (ret)
return ret;
@@ -910,12 +972,29 @@ static ssize_t unassign_domain_store(struct device *dev,
mutex_lock(&matrix_dev->lock);
clear_bit_inv((unsigned long)apqi, matrix_mdev->matrix.aqm);
vfio_ap_mdev_link_queues(matrix_mdev, UNLINK_APQI, apqi);
+
+ if (vfio_ap_mdev_unassign_guest_apqi(matrix_mdev, apqi))
+ vfio_ap_mdev_commit_shadow_apcb(matrix_mdev);
mutex_unlock(&matrix_dev->lock);
return count;
}
static DEVICE_ATTR_WO(unassign_domain);
+static bool vfio_ap_mdev_assign_guest_cdom(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long domid)
+{
+ if (vfio_ap_mdev_has_crycb(matrix_mdev)) {
+ if (!test_bit_inv(domid, matrix_mdev->shadow_apcb.adm)) {
+ set_bit_inv(domid, matrix_mdev->shadow_apcb.adm);
+
+ return true;
+ }
+ }
+
+ return false;
+}
+
/**
* assign_control_domain_store
*
@@ -941,10 +1020,6 @@ static ssize_t assign_control_domain_store(struct device *dev,
struct mdev_device *mdev = mdev_from_dev(dev);
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
- /* If the guest is running, disallow assignment of control domain */
- if (matrix_mdev->kvm)
- return -EBUSY;
-
ret = kstrtoul(buf, 0, &id);
if (ret)
return ret;
@@ -959,12 +1034,29 @@ static ssize_t assign_control_domain_store(struct device *dev,
*/
mutex_lock(&matrix_dev->lock);
set_bit_inv(id, matrix_mdev->matrix.adm);
+ if (vfio_ap_mdev_assign_guest_cdom(matrix_mdev, id))
+ vfio_ap_mdev_commit_shadow_apcb(matrix_mdev);
mutex_unlock(&matrix_dev->lock);
return count;
}
static DEVICE_ATTR_WO(assign_control_domain);
+static bool
+vfio_ap_mdev_unassign_guest_cdom(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long domid)
+{
+ if (vfio_ap_mdev_has_crycb(matrix_mdev)) {
+ if (test_bit_inv(domid, matrix_mdev->shadow_apcb.adm)) {
+ clear_bit_inv(domid, matrix_mdev->shadow_apcb.adm);
+
+ return true;
+ }
+ }
+
+ return false;
+}
+
/**
* unassign_control_domain_store
*
@@ -991,10 +1083,6 @@ static ssize_t unassign_control_domain_store(struct device *dev,
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
unsigned long max_domid = matrix_mdev->matrix.adm_max;
- /* If the guest is running, disallow un-assignment of control domain */
- if (matrix_mdev->kvm)
- return -EBUSY;
-
ret = kstrtoul(buf, 0, &domid);
if (ret)
return ret;
@@ -1003,6 +1091,8 @@ static ssize_t unassign_control_domain_store(struct device *dev,
mutex_lock(&matrix_dev->lock);
clear_bit_inv(domid, matrix_mdev->matrix.adm);
+ if (vfio_ap_mdev_unassign_guest_cdom(matrix_mdev, domid))
+ vfio_ap_mdev_commit_shadow_apcb(matrix_mdev);
mutex_unlock(&matrix_dev->lock);
return count;
@@ -1216,7 +1306,7 @@ static int vfio_ap_mdev_group_notifier(struct notifier_block *nb,
if (!vfio_ap_mdev_has_crycb(matrix_mdev))
return NOTIFY_DONE;
- if (vfio_ap_mdev_filter_guest_matrix(matrix_mdev))
+ if (vfio_ap_mdev_filter_guest_matrix(matrix_mdev, true))
vfio_ap_mdev_commit_shadow_apcb(matrix_mdev);
return NOTIFY_OK;
@@ -1443,7 +1533,7 @@ static void vfio_ap_mdev_hot_plug_queue(struct vfio_ap_queue *q)
if ((q->matrix_mdev == NULL) || !vfio_ap_mdev_has_crycb(q->matrix_mdev))
return;
- if (vfio_ap_mdev_filter_guest_matrix(q->matrix_mdev))
+ if (vfio_ap_mdev_filter_guest_matrix(q->matrix_mdev, true))
vfio_ap_mdev_commit_shadow_apcb(q->matrix_mdev);
}
--
2.21.1
next prev parent reply other threads:[~2020-10-22 17:12 UTC|newest]
Thread overview: 68+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-10-22 17:11 [PATCH v11 00/14] s390/vfio-ap: dynamic configuration support Tony Krowiak
2020-10-22 17:11 ` [PATCH v11 01/14] s390/vfio-ap: No need to disable IRQ after queue reset Tony Krowiak
2020-10-22 19:44 ` kernel test robot
2020-10-26 16:57 ` Tony Krowiak
2020-10-27 6:48 ` Halil Pasic
2020-10-29 23:29 ` Tony Krowiak
2020-10-30 16:13 ` Tony Krowiak
2020-10-30 17:27 ` Halil Pasic
2020-10-30 20:45 ` Tony Krowiak
2020-10-30 17:42 ` Halil Pasic
2020-10-30 20:37 ` Tony Krowiak
2020-10-31 3:43 ` Halil Pasic
2020-11-02 14:35 ` Tony Krowiak
2020-10-30 17:54 ` Halil Pasic
2020-10-30 20:53 ` Tony Krowiak
2020-10-30 21:13 ` Tony Krowiak
2020-10-30 17:56 ` Halil Pasic
2020-10-30 21:17 ` Tony Krowiak
2020-10-22 17:11 ` [PATCH v11 02/14] 390/vfio-ap: use new AP bus interface to search for queue devices Tony Krowiak
2020-10-27 7:01 ` Halil Pasic
2020-11-02 21:57 ` Tony Krowiak
2020-10-22 17:11 ` [PATCH v11 03/14] s390/vfio-ap: manage link between queue struct and matrix mdev Tony Krowiak
2020-10-27 9:33 ` Halil Pasic
2020-10-22 17:11 ` [PATCH v11 04/14] s390/zcrypt: driver callback to indicate resource in use Tony Krowiak
2020-10-27 13:01 ` Halil Pasic
2020-10-27 16:55 ` Harald Freudenberger
2020-11-13 21:30 ` Tony Krowiak
2020-11-14 0:00 ` Halil Pasic
2020-11-16 16:23 ` Tony Krowiak
2020-10-22 17:12 ` [PATCH v11 05/14] s390/vfio-ap: implement in-use callback for vfio_ap driver Tony Krowiak
2020-10-27 13:27 ` Halil Pasic
2020-11-13 17:14 ` Tony Krowiak
2020-11-13 23:47 ` Halil Pasic
2020-11-16 16:58 ` Tony Krowiak
2020-11-23 17:03 ` Cornelia Huck
2020-11-23 19:23 ` Tony Krowiak
2020-10-22 17:12 ` [PATCH v11 06/14] s390/vfio-ap: introduce shadow APCB Tony Krowiak
2020-10-28 8:11 ` Halil Pasic
2020-11-13 17:18 ` Tony Krowiak
2020-10-22 17:12 ` [PATCH v11 07/14] s390/vfio-ap: sysfs attribute to display the guest's matrix Tony Krowiak
2020-10-28 8:17 ` Halil Pasic
2020-11-13 17:27 ` Tony Krowiak
2020-11-13 23:12 ` Halil Pasic
2020-11-19 18:15 ` Tony Krowiak
2020-10-22 17:12 ` [PATCH v11 08/14] s390/vfio-ap: hot plug/unplug queues on bind/unbind of queue device Tony Krowiak
2020-10-22 20:30 ` kernel test robot
2020-10-26 17:04 ` Tony Krowiak
2020-10-28 13:57 ` Halil Pasic
2020-11-03 22:49 ` Tony Krowiak
2020-11-04 12:52 ` Halil Pasic
2020-11-04 21:20 ` Tony Krowiak
2020-11-05 12:27 ` Halil Pasic
2020-11-13 20:36 ` Tony Krowiak
2020-11-04 13:23 ` Halil Pasic
2020-10-22 17:12 ` [PATCH v11 09/14] s390/vfio-ap: allow assignment of unavailable AP queues to mdev device Tony Krowiak
2020-10-28 15:03 ` Halil Pasic
2020-10-22 17:12 ` Tony Krowiak [this message]
2020-10-22 17:12 ` [PATCH v11 11/14] s390/zcrypt: Notify driver on config changed and scan complete callbacks Tony Krowiak
2020-10-27 17:28 ` Harald Freudenberger
2020-11-13 20:58 ` Tony Krowiak
2020-10-22 17:12 ` [PATCH v11 12/14] s390/vfio-ap: handle host AP config change notification Tony Krowiak
2020-10-22 21:17 ` kernel test robot
2020-10-26 17:07 ` Tony Krowiak
2020-10-26 17:21 ` Tony Krowiak
2020-11-03 9:48 ` kernel test robot
2020-11-13 21:06 ` Tony Krowiak
2020-10-22 17:12 ` [PATCH v11 13/14] s390/vfio-ap: handle AP bus scan completed notification Tony Krowiak
2020-10-22 17:12 ` [PATCH v11 14/14] s390/vfio-ap: update docs to include dynamic config support Tony Krowiak
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201022171209.19494-11-akrowiak@linux.ibm.com \
--to=akrowiak@linux.ibm.com \
--cc=alex.williamson@redhat.com \
--cc=borntraeger@de.ibm.com \
--cc=cohuck@redhat.com \
--cc=david@redhat.com \
--cc=fiuczy@linux.ibm.com \
--cc=frankja@linux.ibm.com \
--cc=freude@linux.ibm.com \
--cc=gor@linux.ibm.com \
--cc=hca@linux.ibm.com \
--cc=kvm@vger.kernel.org \
--cc=kwankhede@nvidia.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-s390@vger.kernel.org \
--cc=mjrosato@linux.ibm.com \
--cc=pasic@linux.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).