From: Jason Gunthorpe <jgg@nvidia.com>
To: Christoph Hellwig <hch@lst.de>
Cc: David Airlie <airlied@linux.ie>,
Tony Krowiak <akrowiak@linux.ibm.com>,
Alex Williamson <alex.williamson@redhat.com>,
Christian Borntraeger <borntraeger@de.ibm.com>,
Cornelia Huck <cohuck@redhat.com>,
Jonathan Corbet <corbet@lwn.net>, Daniel Vetter <daniel@ffwll.ch>,
Diana Craciun <diana.craciun@oss.nxp.com>,
dri-devel@lists.freedesktop.org,
Eric Auger <eric.auger@redhat.com>,
Eric Farman <farman@linux.ibm.com>,
Harald Freudenberger <freude@linux.ibm.com>,
Vasily Gorbik <gor@linux.ibm.com>,
Heiko Carstens <hca@linux.ibm.com>,
intel-gfx@lists.freedesktop.org,
intel-gvt-dev@lists.freedesktop.org,
Jani Nikula <jani.nikula@linux.intel.com>,
Jason Herne <jjherne@linux.ibm.com>,
Joonas Lahtinen <joonas.lahtinen@linux.intel.com>,
kvm@vger.kernel.org, Kirti Wankhede <kwankhede@nvidia.com>,
linux-doc@vger.kernel.org, linux-s390@vger.kernel.org,
Matthew Rosato <mjrosato@linux.ibm.com>,
Peter Oberparleiter <oberpar@linux.ibm.com>,
Halil Pasic <pasic@linux.ibm.com>,
Rodrigo Vivi <rodrigo.vivi@intel.com>,
Vineeth Vijayan <vneethv@linux.ibm.com>,
Zhi Wang <zhi.a.wang@intel.com>,
"Raj, Ashok" <ashok.raj@intel.com>,
Leon Romanovsky <leonro@nvidia.com>,
Max Gurtovoy <mgurtovoy@nvidia.com>,
Yishai Hadas <yishaih@nvidia.com>,
Zhenyu Wang <zhenyuw@linux.intel.com>
Subject: Re: [PATCH v2 09/14] vfio/pci: Change vfio_pci_try_bus_reset() to use the dev_set
Date: Fri, 23 Jul 2021 10:30:08 -0300 [thread overview]
Message-ID: <20210723133008.GV1117491@nvidia.com> (raw)
In-Reply-To: <20210723080543.GD2795@lst.de>
On Fri, Jul 23, 2021 at 10:05:43AM +0200, Christoph Hellwig wrote:
> On Tue, Jul 20, 2021 at 02:42:55PM -0300, Jason Gunthorpe wrote:
> > Keep track of all the vfio_devices that have been added to the device set
> > and use this list in vfio_pci_try_bus_reset() instead of trying to work
> > backwards from the pci_device.
> >
> > The dev_set->lock directly prevents devices from joining/leaving the set,
> > which further implies the pci_device cannot change drivers or that the
> > vfio_device be freed, eliminating the need for get/put's.
> >
> > Completeness of the device set can be directly measured by checking if
> > every PCI device in the reset group is also in the device set - which
> > proves that VFIO drivers are attached to everything.
> >
> > This restructuring corrects a call to pci_dev_driver() without holding the
> > device_lock() and removes a hard wiring to &vfio_pci_driver.
> >
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
>
> I think the addition of the list to the dev_set should be a different
> patch. Or maybe even go into the one adding the dev_set concept.
OK
> > +static int vfio_pci_check_all_devices_bound(struct pci_dev *pdev, void *data)
> > {
> > + struct vfio_device_set *dev_set = data;
> > + struct vfio_device *cur;
> >
> > + lockdep_assert_held(&dev_set->lock);
> >
> > + list_for_each_entry(cur, &dev_set->device_list, dev_set_list)
> > + if (cur->dev == &pdev->dev)
> > + return 0;
> > + return -EBUSY;
>
> I don't understand this logic. If there is any device in the set that
> does now have the same struct device we're in trouble? Please clearly
> document what this is trying to do. If the bound in the name makes sense
> you probably want to check the driver instead.
The PCI reset this code is tring to do effects a set of PCI devices,
due to how the HW works.
Along with the vfio_pci_for_each_slot_or_bus() this is computing a set
wise 'is superset' between the list of pci_dev's the reset will affect
(the reset group) and the list of vfio_devices that we have locking
control over to sequence the reset (the dev_set).
If every PCI device we will reset is under the dev_set then we
directly know it is safe to trigger the reset. If any PCI device is
not in this dev_set then we cannot use the reset as we can't know what
will happen to the device that we don't control.
Let's use a different word than bound? vfio_pci_check_device_in_set()?
> > static void vfio_pci_try_bus_reset(struct vfio_pci_device *vdev)
> > {
> > + /* All VFIO devices have a closed FD */
> > + list_for_each_entry(cur, &dev_set->device_list, vdev.dev_set_list)
> > + if (cur->vdev.open_count)
> > + return;
> > +
> > + /* All devices in the group to be reset need VFIO devices */
> > + if (vfio_pci_for_each_slot_or_bus(
> > + vdev->pdev, vfio_pci_check_all_devices_bound, dev_set,
> > + !pci_probe_reset_slot(vdev->pdev->slot)))
> > + return;
> >
> > /* Does at least one need a reset? */
>
> These checks look a little strange, and the comments don't make much
> sense. What about an incremental patch like this?
Sure, it can go in a function
> diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c
> index fbc20f6d2dd412..d8375a5e77e07c 100644
> +++ b/drivers/vfio/pci/vfio_pci.c
> @@ -2188,10 +2188,34 @@ static int vfio_pci_try_zap_and_vma_lock_cb(struct pci_dev *pdev, void *data)
> return 0;
> }
>
> +static struct pci_dev *vfio_pci_reset_target(struct vfio_pci_device *vdev)
> +{
> + struct vfio_device_set *dev_set = vdev->vdev.dev_set;
> + struct vfio_pci_device *cur;
> +
> + /* none of the device is allowed to be currently open: */
> + list_for_each_entry(cur, &dev_set->device_list, vdev.dev_set_list)
> + if (cur->vdev.open_count)
> + return NULL;
> +
> + /* all devices in the group to be reset need to be VFIO devices: */
It is not "need to be VFIO devices" it is "need to be in our
dev_set". Have the PCI dev bound to, say, a mdev VFIO device isn't
good enough.
Thanks,
Jason
next prev parent reply other threads:[~2021-07-23 13:30 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-07-20 17:42 [PATCH v2 00/14] Provide core infrastructure for managing open/release Jason Gunthorpe
2021-07-20 17:42 ` [PATCH v2 01/14] vfio/samples: Remove module get/put Jason Gunthorpe
2021-07-23 7:18 ` Christoph Hellwig
2021-07-20 17:42 ` [PATCH v2 02/14] vfio/mbochs: Fix missing error unwind in mbochs_probe() Jason Gunthorpe
2021-07-20 22:01 ` Alex Williamson
2021-07-20 22:49 ` Jason Gunthorpe
2021-07-20 22:54 ` Alex Williamson
2021-07-21 9:18 ` Cornelia Huck
2021-07-21 9:16 ` Cornelia Huck
2021-07-20 17:42 ` [PATCH v2 03/14] vfio: Introduce a vfio_uninit_group_dev() API call Jason Gunthorpe
2021-07-21 11:33 ` Cornelia Huck
2021-07-23 7:18 ` Christoph Hellwig
2021-07-20 17:42 ` [PATCH v2 04/14] vfio: Provide better generic support for open/release vfio_device_ops Jason Gunthorpe
2021-07-22 14:42 ` Cornelia Huck
2021-07-23 7:39 ` Christoph Hellwig
2021-07-23 14:38 ` Jason Gunthorpe
2021-07-20 17:42 ` [PATCH v2 05/14] vfio/samples: Delete useless open/close Jason Gunthorpe
2021-07-23 7:39 ` Christoph Hellwig
2021-07-20 17:42 ` [PATCH v2 06/14] vfio/fsl: Move to the device set infrastructure Jason Gunthorpe
2021-07-23 7:44 ` Christoph Hellwig
2021-07-23 12:22 ` Jason Gunthorpe
2021-07-23 12:29 ` Christoph Hellwig
2021-07-23 13:11 ` Jason Gunthorpe
2021-07-20 17:42 ` [PATCH v2 07/14] vfio/platform: Use open_device() instead of open coding a refcnt scheme Jason Gunthorpe
2021-07-22 14:48 ` Cornelia Huck
2021-07-23 7:45 ` Christoph Hellwig
2021-07-23 12:23 ` Jason Gunthorpe
2021-07-20 17:42 ` [PATCH v2 08/14] vfio/pci: Move to the device set infrastructure Jason Gunthorpe
2021-07-23 7:47 ` Christoph Hellwig
2021-07-23 12:59 ` Jason Gunthorpe
2021-07-20 17:42 ` [PATCH v2 09/14] vfio/pci: Change vfio_pci_try_bus_reset() to use the dev_set Jason Gunthorpe
2021-07-23 8:05 ` Christoph Hellwig
2021-07-23 13:30 ` Jason Gunthorpe [this message]
2021-07-20 17:42 ` [PATCH v2 10/14] vfio/pci: Reorganize VFIO_DEVICE_PCI_HOT_RESET to use the device set Jason Gunthorpe
2021-07-23 8:12 ` Christoph Hellwig
2021-07-23 13:31 ` Jason Gunthorpe
2021-07-20 17:42 ` [PATCH v2 11/14] vfio/mbochs: Fix close when multiple device FDs are open Jason Gunthorpe
2021-07-23 8:13 ` Christoph Hellwig
2021-07-20 17:42 ` [PATCH v2 12/14] vfio/ap,ccw: Fix open/close " Jason Gunthorpe
2021-07-20 17:42 ` [PATCH v2 13/14] vfio/gvt: " Jason Gunthorpe
2021-07-23 8:14 ` Christoph Hellwig
2021-07-20 17:43 ` [PATCH v2 14/14] vfio: Remove struct vfio_device_ops open/release Jason Gunthorpe
2021-07-23 8:14 ` Christoph Hellwig
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210723133008.GV1117491@nvidia.com \
--to=jgg@nvidia.com \
--cc=airlied@linux.ie \
--cc=akrowiak@linux.ibm.com \
--cc=alex.williamson@redhat.com \
--cc=ashok.raj@intel.com \
--cc=borntraeger@de.ibm.com \
--cc=cohuck@redhat.com \
--cc=corbet@lwn.net \
--cc=daniel@ffwll.ch \
--cc=diana.craciun@oss.nxp.com \
--cc=dri-devel@lists.freedesktop.org \
--cc=eric.auger@redhat.com \
--cc=farman@linux.ibm.com \
--cc=freude@linux.ibm.com \
--cc=gor@linux.ibm.com \
--cc=hca@linux.ibm.com \
--cc=hch@lst.de \
--cc=intel-gfx@lists.freedesktop.org \
--cc=intel-gvt-dev@lists.freedesktop.org \
--cc=jani.nikula@linux.intel.com \
--cc=jjherne@linux.ibm.com \
--cc=joonas.lahtinen@linux.intel.com \
--cc=kvm@vger.kernel.org \
--cc=kwankhede@nvidia.com \
--cc=leonro@nvidia.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-s390@vger.kernel.org \
--cc=mgurtovoy@nvidia.com \
--cc=mjrosato@linux.ibm.com \
--cc=oberpar@linux.ibm.com \
--cc=pasic@linux.ibm.com \
--cc=rodrigo.vivi@intel.com \
--cc=vneethv@linux.ibm.com \
--cc=yishaih@nvidia.com \
--cc=zhenyuw@linux.intel.com \
--cc=zhi.a.wang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).