From: Halil Pasic <pasic@linux.ibm.com>
To: Tony Krowiak <akrowiak@linux.ibm.com>
Cc: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org,
kvm@vger.kernel.org, freude@linux.ibm.com,
borntraeger@de.ibm.com, cohuck@redhat.com,
mjrosato@linux.ibm.com, alex.williamson@redhat.com,
kwankhede@nvidia.com, fiuczy@linux.ibm.com,
frankja@linux.ibm.com, david@redhat.com, hca@linux.ibm.com,
gor@linux.ibm.com
Subject: Re: [PATCH v13 09/15] s390/vfio-ap: allow hot plug/unplug of AP resources using mdev device
Date: Tue, 12 Jan 2021 02:12:51 +0100 [thread overview]
Message-ID: <20210112021251.0d989225.pasic@linux.ibm.com> (raw)
In-Reply-To: <20201223011606.5265-10-akrowiak@linux.ibm.com>
On Tue, 22 Dec 2020 20:16:00 -0500
Tony Krowiak <akrowiak@linux.ibm.com> wrote:
> Let's allow adapters, domains and control domains to be hot plugged into
> and hot unplugged from a KVM guest using a matrix mdev when:
>
> * The adapter, domain or control domain is assigned to or unassigned from
> the matrix mdev
>
> * A queue device with an APQN assigned to the matrix mdev is bound to or
> unbound from the vfio_ap device driver.
>
> Whenever an assignment or unassignment of an adapter, domain or control
> domain is performed as well as when a bind or unbind of a queue device
> is executed, the AP control block (APCB) that supplies the AP configuration
> to a guest is first refreshed. The APCB is refreshed by copying the AP
> configuration from the mdev's matrix to the APCB, then filtering the
> APCB according to the following rules:
>
> * The APID of each adapter and the APQI of each domain that is not in the
> host's AP configuration is filtered out.
>
> * The APID of each adapter comprising an APQN that does not reference a
> queue device bound to the vfio_ap device driver is filtered. The APQNs
> are derived from the Cartesian product of the APID of each adapter and
> APQI of each domain assigned to the mdev's matrix.
>
> After refreshing the APCB, if the mdev is in use by a KVM guest, it is
> hot plugged into the guest to provide access to dynamically provide
> access to the adapters, domains and control domains provided via the
> newly refreshed APCB.
>
> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
> ---
> drivers/s390/crypto/vfio_ap_ops.c | 143 ++++++++++++++++++++++++------
> 1 file changed, 118 insertions(+), 25 deletions(-)
>
> diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
> index 1b1d5975ee0e..843862c88379 100644
> --- a/drivers/s390/crypto/vfio_ap_ops.c
> +++ b/drivers/s390/crypto/vfio_ap_ops.c
> @@ -307,6 +307,88 @@ static void vfio_ap_mdev_commit_shadow_apcb(struct ap_matrix_mdev *matrix_mdev)
> matrix_mdev->shadow_apcb.adm);
> }
>
> +static void vfio_ap_mdev_filter_apcb(struct ap_matrix_mdev *matrix_mdev,
> + struct ap_matrix *shadow_apcb)
> +{
> + int ret;
> + unsigned long apid, apqi, apqn;
> +
> + ret = ap_qci(&matrix_dev->info);
Here we do the qci ourselves, thus the view of vfio_ap and the view
of the ap bus may be different.
> + if (ret)
> + return;
> +
> + memcpy(shadow_apcb, &matrix_mdev->matrix, sizeof(struct ap_matrix));
> +
Why is this memcpy necessary...
> + /*
> + * Copy the adapters, domains and control domains to the shadow_apcb
> + * from the matrix mdev, but only those that are assigned to the host's
> + * AP configuration.
> + */
> + bitmap_and(shadow_apcb->apm, matrix_mdev->matrix.apm,
> + (unsigned long *)matrix_dev->info.apm, AP_DEVICES);
> + bitmap_and(shadow_apcb->aqm, matrix_mdev->matrix.aqm,
> + (unsigned long *)matrix_dev->info.aqm, AP_DOMAINS);
> + bitmap_and(shadow_apcb->adm, matrix_mdev->matrix.adm,
> + (unsigned long *)matrix_dev->info.adm, AP_DOMAINS);
... aren't you overwriting shadow_apcb here anyway?
> +
> + /* If there are no APQNs assigned, then filtering them be unnecessary */
> + if (bitmap_empty(shadow_apcb->apm, AP_DEVICES)) {
> + if (!bitmap_empty(shadow_apcb->aqm, AP_DOMAINS))
> + bitmap_clear(shadow_apcb->aqm, 0, AP_DOMAINS);
> + return;
> + } else if (bitmap_empty(shadow_apcb->aqm, AP_DOMAINS)) {
> + if (!bitmap_empty(shadow_apcb->apm, AP_DEVICES))
> + bitmap_clear(shadow_apcb->apm, 0, AP_DEVICES);
> + return;
> + }
> +
I complained about this before. I still don't understand why do we need
this, but I'm willing to accept it, unless it breaks something later.
BTW I don't think you have to re examine shadow->a[pq]m to tell if empty,
bitmap_and already told you that.
> + for_each_set_bit_inv(apid, shadow_apcb->apm, AP_DEVICES) {
> + for_each_set_bit_inv(apqi, shadow_apcb->aqm, AP_DOMAINS) {
> + /*
> + * If the APQN is not bound to the vfio_ap device
> + * driver, then we can't assign it to the guest's
> + * AP configuration. The AP architecture won't
> + * allow filtering of a single APQN, so if we're
> + * filtering APIDs, then filter the APID; otherwise,
> + * filter the APQI.
> + */
> + apqn = AP_MKQID(apid, apqi);
> + if (!vfio_ap_mdev_get_queue(matrix_mdev, apqn)) {
> + clear_bit_inv(apid, shadow_apcb->apm);
> + break;
> + }
> + }
> + }
> +}
> +
> +/**
> + * vfio_ap_mdev_refresh_apcb
> + *
> + * Filter APQNs assigned to the matrix mdev that do not reference an AP queue
> + * device bound to the vfio_ap device driver.
> + *
> + * @matrix_mdev: the matrix mdev whose AP configuration is to be filtered
> + * @shadow_apcb: the shadow of the KVM guest's APCB (contains AP configuration
> + * for guest)
> + * @filter_apids: boolean value indicating whether the APQNs shall be filtered
> + * by APID (true) or by APQI (false).
> + *
The signature in the doc comment and of the function do not match.
Since none of the complains affects correctness, except maybe for the
qci suff:
Acked-by: Halil Pasic <pasic@linux.ibm.com>
If it's good enough for you, it's good enough for me.
> + * Returns the number of APQNs remaining after filtering is complete.
> + */
> +static void vfio_ap_mdev_refresh_apcb(struct ap_matrix_mdev *matrix_mdev)
> +{
> + struct ap_matrix shadow_apcb;
> +
> + vfio_ap_mdev_filter_apcb(matrix_mdev, &shadow_apcb);
> +
> + if (memcmp(&shadow_apcb, &matrix_mdev->shadow_apcb,
> + sizeof(struct ap_matrix)) != 0) {
> + memcpy(&matrix_mdev->shadow_apcb, &shadow_apcb,
> + sizeof(struct ap_matrix));
> + vfio_ap_mdev_commit_shadow_apcb(matrix_mdev);
> + }
> +}
> +
> static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev)
> {
> struct ap_matrix_mdev *matrix_mdev;
> @@ -552,10 +634,6 @@ static ssize_t assign_adapter_store(struct device *dev,
> struct mdev_device *mdev = mdev_from_dev(dev);
> struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
>
> - /* If the guest is running, disallow assignment of adapter */
> - if (matrix_mdev->kvm)
> - return -EBUSY;
> -
> ret = kstrtoul(buf, 0, &apid);
> if (ret)
> return ret;
> @@ -577,6 +655,7 @@ static ssize_t assign_adapter_store(struct device *dev,
>
> set_bit_inv(apid, matrix_mdev->matrix.apm);
> vfio_ap_mdev_link_adapter(matrix_mdev, apid);
> + vfio_ap_mdev_refresh_apcb(matrix_mdev);
>
> mutex_unlock(&matrix_dev->lock);
>
> @@ -619,10 +698,6 @@ static ssize_t unassign_adapter_store(struct device *dev,
> struct mdev_device *mdev = mdev_from_dev(dev);
> struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
>
> - /* If the guest is running, disallow un-assignment of adapter */
> - if (matrix_mdev->kvm)
> - return -EBUSY;
> -
> ret = kstrtoul(buf, 0, &apid);
> if (ret)
> return ret;
> @@ -633,6 +708,8 @@ static ssize_t unassign_adapter_store(struct device *dev,
> mutex_lock(&matrix_dev->lock);
> clear_bit_inv((unsigned long)apid, matrix_mdev->matrix.apm);
> vfio_ap_mdev_unlink_adapter(matrix_mdev, apid);
> + vfio_ap_mdev_refresh_apcb(matrix_mdev);
> +
> mutex_unlock(&matrix_dev->lock);
>
> return count;
> @@ -691,10 +768,6 @@ static ssize_t assign_domain_store(struct device *dev,
> struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
> unsigned long max_apqi = matrix_mdev->matrix.aqm_max;
>
> - /* If the guest is running, disallow assignment of domain */
> - if (matrix_mdev->kvm)
> - return -EBUSY;
> -
> ret = kstrtoul(buf, 0, &apqi);
> if (ret)
> return ret;
> @@ -715,6 +788,7 @@ static ssize_t assign_domain_store(struct device *dev,
>
> set_bit_inv(apqi, matrix_mdev->matrix.aqm);
> vfio_ap_mdev_link_domain(matrix_mdev, apqi);
> + vfio_ap_mdev_refresh_apcb(matrix_mdev);
>
> mutex_unlock(&matrix_dev->lock);
>
> @@ -757,10 +831,6 @@ static ssize_t unassign_domain_store(struct device *dev,
> struct mdev_device *mdev = mdev_from_dev(dev);
> struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
>
> - /* If the guest is running, disallow un-assignment of domain */
> - if (matrix_mdev->kvm)
> - return -EBUSY;
> -
> ret = kstrtoul(buf, 0, &apqi);
> if (ret)
> return ret;
> @@ -771,12 +841,24 @@ static ssize_t unassign_domain_store(struct device *dev,
> mutex_lock(&matrix_dev->lock);
> clear_bit_inv((unsigned long)apqi, matrix_mdev->matrix.aqm);
> vfio_ap_mdev_unlink_domain(matrix_mdev, apqi);
> + vfio_ap_mdev_refresh_apcb(matrix_mdev);
> +
> mutex_unlock(&matrix_dev->lock);
>
> return count;
> }
> static DEVICE_ATTR_WO(unassign_domain);
>
> +static void vfio_ap_mdev_hot_plug_cdom(struct ap_matrix_mdev *matrix_mdev,
> + unsigned long domid)
> +{
> + if (!test_bit_inv(domid, matrix_mdev->shadow_apcb.adm) &&
> + test_bit_inv(domid, (unsigned long *) matrix_dev->info.adm)) {
> + set_bit_inv(domid, matrix_mdev->shadow_apcb.adm);
> + vfio_ap_mdev_commit_shadow_apcb(matrix_mdev);
> + }
> +}
> +
> /**
> * assign_control_domain_store
> *
> @@ -802,10 +884,6 @@ static ssize_t assign_control_domain_store(struct device *dev,
> struct mdev_device *mdev = mdev_from_dev(dev);
> struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
>
> - /* If the guest is running, disallow assignment of control domain */
> - if (matrix_mdev->kvm)
> - return -EBUSY;
> -
> ret = kstrtoul(buf, 0, &id);
> if (ret)
> return ret;
> @@ -820,12 +898,22 @@ static ssize_t assign_control_domain_store(struct device *dev,
> */
> mutex_lock(&matrix_dev->lock);
> set_bit_inv(id, matrix_mdev->matrix.adm);
> + vfio_ap_mdev_hot_plug_cdom(matrix_mdev, id);
> mutex_unlock(&matrix_dev->lock);
>
> return count;
> }
> static DEVICE_ATTR_WO(assign_control_domain);
>
> +static void vfio_ap_mdev_hot_unplug_cdom(struct ap_matrix_mdev *matrix_mdev,
> + unsigned long domid)
> +{
> + if (test_bit_inv(domid, matrix_mdev->shadow_apcb.adm)) {
> + clear_bit_inv(domid, matrix_mdev->shadow_apcb.adm);
> + vfio_ap_mdev_commit_shadow_apcb(matrix_mdev);
> + }
> +}
> +
> /**
> * unassign_control_domain_store
> *
> @@ -852,10 +940,6 @@ static ssize_t unassign_control_domain_store(struct device *dev,
> struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
> unsigned long max_domid = matrix_mdev->matrix.adm_max;
>
> - /* If the guest is running, disallow un-assignment of control domain */
> - if (matrix_mdev->kvm)
> - return -EBUSY;
> -
> ret = kstrtoul(buf, 0, &domid);
> if (ret)
> return ret;
> @@ -864,6 +948,7 @@ static ssize_t unassign_control_domain_store(struct device *dev,
>
> mutex_lock(&matrix_dev->lock);
> clear_bit_inv(domid, matrix_mdev->matrix.adm);
> + vfio_ap_mdev_hot_unplug_cdom(matrix_mdev, domid);
> mutex_unlock(&matrix_dev->lock);
>
> return count;
> @@ -1089,6 +1174,8 @@ static int vfio_ap_mdev_group_notifier(struct notifier_block *nb,
> matrix_mdev->matrix.aqm,
> matrix_mdev->matrix.adm);
>
> + vfio_ap_mdev_commit_shadow_apcb(matrix_mdev);
> +
> notify_done:
> mutex_unlock(&matrix_dev->lock);
> return notify_rc;
> @@ -1330,6 +1417,8 @@ int vfio_ap_mdev_probe_queue(struct ap_device *apdev)
> q->apqn = to_ap_queue(&apdev->device)->qid;
> q->saved_isc = VFIO_AP_ISC_INVALID;
> vfio_ap_queue_link_mdev(q);
> + if (q->matrix_mdev)
> + vfio_ap_mdev_refresh_apcb(q->matrix_mdev);
> mutex_unlock(&matrix_dev->lock);
>
> return 0;
> @@ -1337,6 +1426,7 @@ int vfio_ap_mdev_probe_queue(struct ap_device *apdev)
>
> void vfio_ap_mdev_remove_queue(struct ap_device *apdev)
> {
> + struct ap_matrix_mdev *matrix_mdev;
> struct vfio_ap_queue *q;
> int apid, apqi;
>
> @@ -1347,8 +1437,11 @@ void vfio_ap_mdev_remove_queue(struct ap_device *apdev)
> apqi = AP_QID_QUEUE(q->apqn);
> vfio_ap_mdev_reset_queue(apid, apqi, 1);
>
> - if (q->matrix_mdev)
> + if (q->matrix_mdev) {
> + matrix_mdev = q->matrix_mdev;
> vfio_ap_mdev_unlink_queue(q);
> + vfio_ap_mdev_refresh_apcb(matrix_mdev);
> + }
>
> kfree(q);
> mutex_unlock(&matrix_dev->lock);
next prev parent reply other threads:[~2021-01-12 1:14 UTC|newest]
Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-12-23 1:15 [PATCH v13 00/15] s390/vfio-ap: dynamic configuration support Tony Krowiak
2020-12-23 1:15 ` [PATCH v13 01/15] s390/vfio-ap: clean up vfio_ap resources when KVM pointer invalidated Tony Krowiak
2020-12-23 1:15 ` [PATCH v13 02/15] s390/vfio-ap: No need to disable IRQ after queue reset Tony Krowiak
2021-01-11 16:32 ` Halil Pasic
2021-01-13 17:06 ` Tony Krowiak
2021-01-13 21:21 ` Halil Pasic
2021-01-14 0:46 ` Tony Krowiak
2021-01-14 3:13 ` Halil Pasic
2020-12-23 1:15 ` [PATCH v13 03/15] s390/vfio-ap: move probe and remove callbacks to vfio_ap_ops.c Tony Krowiak
2020-12-23 1:15 ` [PATCH v13 04/15] s390/vfio-ap: use new AP bus interface to search for queue devices Tony Krowiak
2020-12-23 1:15 ` [PATCH v13 05/15] s390/vfio-ap: manage link between queue struct and matrix mdev Tony Krowiak
2021-01-11 19:17 ` Halil Pasic
2021-01-13 21:41 ` Tony Krowiak
2021-01-14 2:50 ` Halil Pasic
2021-01-14 21:10 ` Tony Krowiak
2020-12-23 1:15 ` [PATCH v13 06/15] s390/vfio-ap: allow assignment of unavailable AP queues to mdev device Tony Krowiak
2021-01-11 20:40 ` Halil Pasic
2021-01-14 17:54 ` Tony Krowiak
2021-01-15 1:08 ` Halil Pasic
2021-01-15 1:44 ` Halil Pasic
2021-03-31 14:36 ` Tony Krowiak
2020-12-23 1:15 ` [PATCH v13 07/15] s390/vfio-ap: introduce shadow APCB Tony Krowiak
2021-01-11 22:50 ` Halil Pasic
2021-01-14 21:35 ` Tony Krowiak
2020-12-23 1:15 ` [PATCH v13 08/15] s390/vfio-ap: sysfs attribute to display the guest's matrix Tony Krowiak
2021-01-11 22:58 ` Halil Pasic
2021-01-28 21:29 ` Tony Krowiak
2020-12-23 1:16 ` [PATCH v13 09/15] s390/vfio-ap: allow hot plug/unplug of AP resources using mdev device Tony Krowiak
2021-01-12 1:12 ` Halil Pasic [this message]
2021-01-12 17:55 ` Halil Pasic
2021-02-01 14:41 ` Tony Krowiak
2021-02-03 23:13 ` Tony Krowiak
2021-02-04 0:21 ` Halil Pasic
2020-12-23 1:16 ` [PATCH v13 10/15] s390/zcrypt: driver callback to indicate resource in use Tony Krowiak
2021-01-12 16:50 ` Halil Pasic
2020-12-23 1:16 ` [PATCH v13 11/15] s390/vfio-ap: implement in-use callback for vfio_ap driver Tony Krowiak
2021-01-12 1:20 ` Halil Pasic
2021-01-12 14:14 ` Matthew Rosato
2021-01-12 16:49 ` Halil Pasic
2020-12-23 1:16 ` [PATCH v13 12/15] s390/zcrypt: Notify driver on config changed and scan complete callbacks Tony Krowiak
2021-01-12 16:58 ` Halil Pasic
2020-12-23 1:16 ` [PATCH v13 13/15] s390/vfio-ap: handle host AP config change notification Tony Krowiak
2021-01-12 18:39 ` Halil Pasic
2020-12-23 1:16 ` [PATCH v13 14/15] s390/vfio-ap: handle AP bus scan completed notification Tony Krowiak
2021-01-12 18:44 ` Halil Pasic
2020-12-23 1:16 ` [PATCH v13 15/15] s390/vfio-ap: update docs to include dynamic config support Tony Krowiak
2021-01-06 15:16 ` [PATCH v13 00/15] s390/vfio-ap: dynamic configuration support Tony Krowiak
2021-01-07 14:41 ` Halil Pasic
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210112021251.0d989225.pasic@linux.ibm.com \
--to=pasic@linux.ibm.com \
--cc=akrowiak@linux.ibm.com \
--cc=alex.williamson@redhat.com \
--cc=borntraeger@de.ibm.com \
--cc=cohuck@redhat.com \
--cc=david@redhat.com \
--cc=fiuczy@linux.ibm.com \
--cc=frankja@linux.ibm.com \
--cc=freude@linux.ibm.com \
--cc=gor@linux.ibm.com \
--cc=hca@linux.ibm.com \
--cc=kvm@vger.kernel.org \
--cc=kwankhede@nvidia.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-s390@vger.kernel.org \
--cc=mjrosato@linux.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).