linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Tony Krowiak <akrowiak@linux.ibm.com>
To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org,
	kvm@vger.kernel.org
Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com,
	mjrosato@linux.ibm.com, pasic@linux.ibm.com,
	alex.williamson@redhat.com, kwankhede@nvidia.com,
	fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com,
	hca@linux.ibm.com, gor@linux.ibm.com,
	Tony Krowiak <akrowiak@linux.ibm.com>,
	Tony Krowiak <akrowiak@stny.rr.com>
Subject: [PATCH v12 13/17] s390/vfio-ap: hot plug/unplug queues on bind/unbind of queue device
Date: Tue, 24 Nov 2020 16:40:12 -0500	[thread overview]
Message-ID: <20201124214016.3013-14-akrowiak@linux.ibm.com> (raw)
In-Reply-To: <20201124214016.3013-1-akrowiak@linux.ibm.com>

In response to the probe or remove of a queue device, if a KVM guest is
using the matrix mdev to which the APQN of the queue device is assigned,
the vfio_ap device driver must respond accordingly. In an ideal world, the
queue corresponding to the queue device being probed would be hot plugged
into the guest. Likewise, the queue corresponding to the queue device being
removed would be hot unplugged. Unfortunately, the AP architecture
precludes plugging or unplugging individual queues. The queues to which a
guest is granted access are specified as a matrix of adapter and domain
numbers. The Cartesian product of the adapter and domain numbers assigned
to this matrix comprise the AP queue numbers (APQN) to which the guest will
be granted access; therefore, it becomes obvious that assigning a new
adapter or domain number to the matrix may result in multiple APQNs
getting assigned. Likewise, unassigning an adapter or domain number from
the matrix may result in multiple APQNs getting unassigned. Additionally,
in order to enforce the linux device model requirement that a pass-through
device must be bound to the driver facilitating its passthrough, each new
APQN assigned to the guest's matrix must reference a queue device bound to
the vfio_ap device driver. The following sections articulate the design
for this patch.

Probing a queue device:
----------------------
The goal here is to assign the APQN of the queue being probed to the
guest's matrix if possible by adhering to a set of rules:
* The adapter number (APID) will be assigned to the guest matrix iff:
  1. The adapter is in the host's AP configuration
  2. The APID is not yet assigned to the guest's matrix
  3. Each APQN derived from the APID and the domain numbers (APQI) of
     domains already assigned to the guest's matrix references a queue
     device bound to the vfio_ap device driver
* The domain number (APQI) will be assigned to the guest matrix iff:
  1. The domain is in the host's AP configuration
  2. The APQI is not yet assigned to the guest's matrix
  3. Each APQN derived from the APQI and the APIDs of
     adapters already assigned to the guest's matrix references a queue
     device bound to the vfio_ap device driver

Removing a queue device:
-----------------------
Unassigning the adapter number from the guest's matrix will remove access
to all domains on the adapter from the guest. Unassigning the domain
number from the guest's matrix will remove access to that domain on all
adapters assigned to the guest matrix. If both the adapter and domain are
unassigned from the guest's matrix, That will reduce access to every
adapter for the guest. Since an AP adapter card is the actual hardware
device that gets physically plugged/unplugged, unassigning the adapter
number from the guest's matrix makes the most sense here.

Signed-off-by: Tony Krowiak <akrowiak@stny.rr.com>
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_ops.c | 37 +++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 4f96b7861607..1179c6af59c6 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -1508,6 +1508,23 @@ static void vfio_ap_queue_link_mdev(struct vfio_ap_queue *q)
 	}
 }
 
+
+static void vfio_ap_mdev_hot_plug_queue(struct vfio_ap_queue *q)
+{
+	bool hot_plug = false;
+	unsigned long apid = (unsigned long)AP_QID_CARD(q->apqn);
+	unsigned long apqi = (unsigned long)AP_QID_QUEUE(q->apqn);
+
+	if (q->matrix_mdev == NULL)
+		return;
+
+	hot_plug |= vfio_ap_assign_apid_to_apcb(q->matrix_mdev, apid);
+	hot_plug |= vfio_ap_assign_apqi_to_apcb(q->matrix_mdev, apqi);
+
+	if (hot_plug)
+		vfio_ap_mdev_commit_shadow_apcb(q->matrix_mdev);
+}
+
 /**
  * vfio_ap_mdev_probe_queue:
  *
@@ -1526,11 +1543,30 @@ int vfio_ap_mdev_probe_queue(struct ap_device *apdev)
 	q->apqn = to_ap_queue(&apdev->device)->qid;
 	q->saved_isc = VFIO_AP_ISC_INVALID;
 	vfio_ap_queue_link_mdev(q);
+	vfio_ap_mdev_hot_plug_queue(q);
 	mutex_unlock(&matrix_dev->lock);
 
 	return 0;
 }
 
+static void vfio_ap_mdev_hot_unplug_queue(struct vfio_ap_queue *q)
+{
+	unsigned long apid;
+	unsigned long apqi;
+
+	if (q->matrix_mdev == NULL)
+		return;
+
+	apid = AP_QID_CARD(q->apqn);
+	apqi = AP_QID_QUEUE(q->apqn);
+
+	if (test_bit_inv(apid, q->matrix_mdev->shadow_apcb.apm) &&
+	    test_bit_inv(apqi, q->matrix_mdev->shadow_apcb.aqm)) {
+		clear_bit_inv(apid, q->matrix_mdev->shadow_apcb.apm);
+		vfio_ap_mdev_commit_shadow_apcb(q->matrix_mdev);
+	}
+}
+
 /**
  * vfio_ap_mdev_remove_queue:
  *
@@ -1544,6 +1580,7 @@ void vfio_ap_mdev_remove_queue(struct ap_device *apdev)
 
 	mutex_lock(&matrix_dev->lock);
 	q = dev_get_drvdata(&apdev->device);
+	vfio_ap_mdev_hot_unplug_queue(q);
 	dev_set_drvdata(&apdev->device, NULL);
 	apid = AP_QID_CARD(q->apqn);
 	apqi = AP_QID_QUEUE(q->apqn);
-- 
2.21.1


  parent reply	other threads:[~2020-11-24 21:41 UTC|newest]

Thread overview: 48+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-11-24 21:39 [PATCH v12 00/17] s390/vfio-ap: dynamic configuration support Tony Krowiak
2020-11-24 21:40 ` [PATCH v12 01/17] s390/vfio-ap: move probe and remove callbacks to vfio_ap_ops.c Tony Krowiak
2020-11-26  9:35   ` Halil Pasic
2020-11-24 21:40 ` [PATCH v12 02/17] s390/vfio-ap: decrement reference count to KVM Tony Krowiak
2020-11-26  9:42   ` Halil Pasic
2020-11-24 21:40 ` [PATCH v12 03/17] 390/vfio-ap: use new AP bus interface to search for queue devices Tony Krowiak
2020-11-26 10:34   ` Halil Pasic
2020-11-30 14:48     ` Tony Krowiak
2020-11-24 21:40 ` [PATCH v12 04/17] s390/vfio-ap: No need to disable IRQ after queue reset Tony Krowiak
2020-11-26 13:37   ` Halil Pasic
2020-11-24 21:40 ` [PATCH v12 05/17] s390/vfio-ap: manage link between queue struct and matrix mdev Tony Krowiak
2020-11-26 14:08   ` Halil Pasic
2020-12-14 23:37     ` Tony Krowiak
2020-11-26 14:45   ` Halil Pasic
2020-12-14 23:32     ` Tony Krowiak
2020-11-24 21:40 ` [PATCH v12 06/17] s390/zcrypt: driver callback to indicate resource in use Tony Krowiak
2020-11-24 21:40 ` [PATCH v12 07/17] s390/vfio-ap: implement in-use callback for vfio_ap driver Tony Krowiak
2020-11-26 15:54   ` Halil Pasic
2020-12-15  3:52     ` Tony Krowiak
2020-11-24 21:40 ` [PATCH v12 08/17] s390/vfio-ap: introduce shadow APCB Tony Krowiak
2020-11-29  0:44   ` Halil Pasic
2020-11-24 21:40 ` [PATCH v12 09/17] s390/vfio-ap: sysfs attribute to display the guest's matrix Tony Krowiak
2020-11-29  0:49   ` Halil Pasic
2020-12-16 20:00     ` Tony Krowiak
2020-11-24 21:40 ` [PATCH v12 10/17] s390/vfio-ap: initialize the guest apcb Tony Krowiak
2020-11-29  1:09   ` Halil Pasic
2020-12-16 20:08     ` Tony Krowiak
2020-11-24 21:40 ` [PATCH v12 11/17] s390/vfio-ap: allow assignment of unavailable AP queues to mdev device Tony Krowiak
2020-11-29  1:17   ` Halil Pasic
2020-12-16 20:14     ` Tony Krowiak
2020-12-16 22:09       ` Halil Pasic
2020-11-24 21:40 ` [PATCH v12 12/17] s390/vfio-ap: allow hot plug/unplug of AP resources using " Tony Krowiak
2020-11-29  1:52   ` Halil Pasic
2020-11-30 19:36     ` Tony Krowiak
2020-11-30 23:32       ` Halil Pasic
2020-12-01  0:18         ` Tony Krowiak
2020-12-01 10:19           ` Halil Pasic
2020-12-01 17:56         ` Halil Pasic
2020-12-01 22:12           ` Tony Krowiak
2020-12-01 23:14             ` Halil Pasic
2020-11-24 21:40 ` Tony Krowiak [this message]
2020-11-24 21:40 ` [PATCH v12 14/17] s390/zcrypt: Notify driver on config changed and scan complete callbacks Tony Krowiak
     [not found]   ` <20201130101836.0399547c.pasic@linux.ibm.com>
2020-12-09  7:20     ` Harald Freudenberger
2020-12-16 20:32       ` Tony Krowiak
2020-12-16 20:54     ` Tony Krowiak
2020-11-24 21:40 ` [PATCH v12 15/17] s390/vfio-ap: handle host AP config change notification Tony Krowiak
2020-11-24 21:40 ` [PATCH v12 16/17] s390/vfio-ap: handle AP bus scan completed notification Tony Krowiak
2020-11-24 21:40 ` [PATCH v12 17/17] s390/vfio-ap: update docs to include dynamic config support Tony Krowiak

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20201124214016.3013-14-akrowiak@linux.ibm.com \
    --to=akrowiak@linux.ibm.com \
    --cc=akrowiak@stny.rr.com \
    --cc=alex.williamson@redhat.com \
    --cc=borntraeger@de.ibm.com \
    --cc=cohuck@redhat.com \
    --cc=david@redhat.com \
    --cc=fiuczy@linux.ibm.com \
    --cc=frankja@linux.ibm.com \
    --cc=freude@linux.ibm.com \
    --cc=gor@linux.ibm.com \
    --cc=hca@linux.ibm.com \
    --cc=kvm@vger.kernel.org \
    --cc=kwankhede@nvidia.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-s390@vger.kernel.org \
    --cc=mjrosato@linux.ibm.com \
    --cc=pasic@linux.ibm.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).