Re: [PATCH 00/12] Introduce vfio_pci_core subsystem

[Alex Williamson <alex.williamson@redhat.com>]

On Wed, 4 Aug 2021 16:41:34 +0300
Yishai Hadas <yishaih@nvidia.com> wrote:

> On 7/21/2021 7:15 PM, Yishai Hadas wrote:
> > Prologue:
> >
> > This is the second series of three to send the "mlx5_vfio_pci" driver
> > that has been discussed on the list for a while now. It comes on top of
> > the first series (i.e. Reorganize reflck to support splitting vfio_pci)
> > that was sent already and pending merge [1].
> >
> >   - Split vfio_pci into vfio_pci/vfio_pci_core and provide infrastructure
> >     for non-generic VFIO PCI drivers.
> >   - The new driver mlx5_vfio_pci that is a full implementation of
> >     suspend/resume functionality for mlx5 devices.
> >
> > A preview of all the patches can be seen here:
> > https://github.com/jgunthorpe/linux/commits/mlx5_vfio_pci
> >
> > [1] https://lore.kernel.org/dri-devel/0-v2-b6a5582525c9+ff96-vfio_reflck_jgg@nvidia.com/T/#t
> > =====================
> >
> >  From Max Gurtovoy:
> > ====================
> > This series splits the vfio_pci driver into two parts, a PCI driver and
> > a subsystem driver that will also be library of code. The main PCI
> > driver, vfio_pci.ko, will remain as before and it will use the library
> > module vfio_pci_core.ko to help create the vfio_device.
> >
> > This series is intended to solve the issues that were raised in the
> > previous attempts for extending vfio-pci for device specific
> > functionality:
> >
> > 1. https://lore.kernel.org/kvm/20200518024202.13996-1-yan.y.zhao@intel.com
> >     by Yan Zhao
> > 2. https://lore.kernel.org/kvm/20210702095849.1610-1-shameerali.kolothum.thodi@huawei.com
> >     by Longfang Liu
> >
> > Also to support proposed future changes to virtio and other common
> > protocols to support migration:
> >
> > https://lists.oasis-open.org/archives/virtio-comment/202106/msg00044.html
> >
> > This subsystem framework will also ease adding new device specific
> > functionality to VFIO devices in the future by allowing another module
> > to provide the pci_driver that can setup a number of details before
> > registering to the VFIO subsystem, such as injecting its own operations.
> >
> > This series also extends the "driver_override" mechanism. A flag is
> > added for PCI drivers that will declare themselves as "driver_override"
> > capable which sends their match table to the modules.alias file but
> > otherwise leaves them outside of the normal driver core auto-binding
> > world, like vfio_pci.
> >
> > In order to get the best match for "driver_override" drivers, one can
> > create a userspace program to inspect the modules.alias, an example can
> > be found at:
> >
> > https://github.com/maxgurtovoy/linux_tools/blob/main/vfio/bind_vfio_pci_driver.py
> >
> > Which finds the 'best match' according to a simple algorithm: "the
> > driver with the fewest '*' matches wins."
> >
> > For example, the vfio-pci driver will match to any pci device. So it
> > will have the maximal '*' matches.
> >
> > In case we are looking for a match to a mlx5 based device, we'll have a
> > match to vfio-pci.ko and mlx5-vfio-pci.ko. We'll prefer mlx5-vfio-pci.ko
> > since it will have less '*' matches (probably vendor and device IDs will
> > match). This will work in the future for NVMe/Virtio devices that can
> > match according to a class code or other criteria.
> >
> > Yishai
> >
> >
> > Jason Gunthorpe (2):
> >    vfio: Use select for eventfd
> >    vfio: Use kconfig if XX/endif blocks instead of repeating 'depends on'
> >
> > Max Gurtovoy (9):
> >    vfio/pci: Rename vfio_pci.c to vfio_pci_core.c
> >    vfio/pci: Rename vfio_pci_private.h to vfio_pci_core.h
> >    vfio/pci: Rename vfio_pci_device to vfio_pci_core_device
> >    vfio/pci: Rename ops functions to fit core namings
> >    vfio/pci: Include vfio header in vfio_pci_core.h
> >    vfio/pci: Split the pci_driver code out of vfio_pci_core.c
> >    vfio/pci: Move igd initialization to vfio_pci.c
> >    PCI: Add a PCI_ID_F_VFIO_DRIVER_OVERRIDE flag to struct pci_device_id
> >    vfio/pci: Introduce vfio_pci_core.ko
> >
> > Yishai Hadas (1):
> >    vfio/pci: Move module parameters to vfio_pci.c
> >
> >   Documentation/PCI/pci.rst                     |    1 +
> >   drivers/pci/pci-driver.c                      |   25 +-
> >   drivers/vfio/Kconfig                          |   29 +-
> >   drivers/vfio/fsl-mc/Kconfig                   |    3 +-
> >   drivers/vfio/mdev/Kconfig                     |    1 -
> >   drivers/vfio/pci/Kconfig                      |   39 +-
> >   drivers/vfio/pci/Makefile                     |    8 +-
> >   drivers/vfio/pci/vfio_pci.c                   | 2238 +----------------
> >   drivers/vfio/pci/vfio_pci_config.c            |   70 +-
> >   drivers/vfio/pci/vfio_pci_core.c              | 2138 ++++++++++++++++
> >   drivers/vfio/pci/vfio_pci_igd.c               |   19 +-
> >   drivers/vfio/pci/vfio_pci_intrs.c             |   42 +-
> >   drivers/vfio/pci/vfio_pci_rdwr.c              |   18 +-
> >   drivers/vfio/pci/vfio_pci_zdev.c              |    4 +-
> >   drivers/vfio/platform/Kconfig                 |    6 +-
> >   drivers/vfio/platform/reset/Kconfig           |    4 +-
> >   include/linux/mod_devicetable.h               |    7 +
> >   include/linux/pci.h                           |   27 +
> >   .../linux/vfio_pci_core.h                     |   89 +-
> >   scripts/mod/devicetable-offsets.c             |    1 +
> >   scripts/mod/file2alias.c                      |    8 +-
> >   21 files changed, 2496 insertions(+), 2281 deletions(-)
> >   create mode 100644 drivers/vfio/pci/vfio_pci_core.c
> >   rename drivers/vfio/pci/vfio_pci_private.h => include/linux/vfio_pci_core.h (56%)
> >  
> Hi Alex,
> 
> Based on the feedback that we got so far on this series, no functional 
> changes are expected in V2.
> 
> It may include the below minor changes:
> 
> - Drop DRIVER_VERSION as it's useless and not required any more. 
> (Patches #6, #12).
> 
> - Add the sequence of commands/algorithm that is required by userspace 
> to discover the matching driver to the commit message of patch #9.
> 
> Do we need to wait for more feedback or that we are fine to send V2 ?

 - Resolve Kconfig compatibility in patch 12

Patch 9 also depends on an ack from Bjorn, so whether you want to try
to get his buy-in before or after that patch gets updated to clarify
what it's trying to do and why, is up to you.  Thanks,

Alex