RE: [PATCH v2 00/13] vfio_pci: wrap pci device as a mediated device

["Liu, Yi L" <yi.l.liu@intel.com>]

Hi Alex,

Any comments on it? :-) With this version, the vfio-mdev-pci driver
could work with non-singleton groups, also it works with vfio-pci
as well.

Regards,
Yi Liu

> From: Liu, Yi L
> Sent: Thursday, September 5, 2019 3:59 PM
> To: alex.williamson@redhat.com; kwankhede@nvidia.com
> Subject: [PATCH v2 00/13] vfio_pci: wrap pci device as a mediated device
> 
> This patchset aims to add a vfio-pci-like meta driver as a demo
> user of the vfio changes introduced in "vfio/mdev: IOMMU aware
> mediated device" patchset from Baolu Lu. Besides the test purpose,
> per Alex's comments, it could also be a good base driver for
> experimenting with device specific mdev migration.
> 
> Specific interface tested in this proposal:
>  *) int mdev_set_iommu_device(struct device *dev,
>  				struct device *iommu_device)
>     introduced in the patch as below:
>     "[PATCH v5 6/8] vfio/mdev: Add iommu related member in mdev_device"
> 
> Patch Overview:
>  *) patch 1 ~ 7: code refactor for existing vfio-pci module
>                  move the common codes from vfio_pci.c to
>                  vfio_pci_common.c
>  *) patch 8: add protection to perm_bits alloc/free
>  *) patch 9: add vfio-mdev-pci sample driver
>  *) patch 10: refine the sample driver
>  *) patch 11 - 13: make the sample driver work for non-singleton groups
>                    also work for vfio-pci and vfio-mdev-pci mixed usage
>                    includes vfio-mdev-pci driver change and vfio_iommu_type1
>                    changes.
> 
> Links:
>  *) Link of "vfio/mdev: IOMMU aware mediated device"
>          https://lwn.net/Articles/780522/
>  *) Previous versions:
>          RFC v1: https://lkml.org/lkml/2019/3/4/529
>          RFC v2: https://lkml.org/lkml/2019/3/13/113
>          RFC v3: https://lkml.org/lkml/2019/4/24/495
>          Patch v1: https://www.spinics.net/lists/kvm/msg188952.html
>  *) may try it with the codes in below repo
>     current version is branch "v5.3-rc7-pci-mdev":
>          https://github.com/luxis1999/vfio-mdev-pci-sample-driver.git
> 
> Test done on two NICs which share an iommu group.
> 
> -------------------------------------------------------------------
>          |  NIC0           |  NIC1      | vIOMMU  | VMs  | Passtrhu
> -------------------------------------------------------------------
>   Test#0 |  vfio-pci       |  vfio-pci  | no      |  1   | pass
> -------------------------------------------------------------------
>   Test#1 |  vfio-pci       |  vfio-pci  | no      |  2   | fail[1]
> -------------------------------------------------------------------
>   Test#2 |  vfio-pci       |  vfio-pci  | yes     |  1   | fail[2]
> -------------------------------------------------------------------
>   Test#3 |  vfio-pci-mdev  |  vfio-pci  | no      |  1   | pass
> -------------------------------------------------------------------
>   Test#4 |  vfio-pci-mdev  |  vfio-pci  | no      |  2   | fail[3]
> -------------------------------------------------------------------
>   Test#5 |  vfio-pci-mdev  |  vfio-pci  | yes     |  1   | fail[4]
> -------------------------------------------------------------------
> Tips:
> [1] qemu-system-x86_64: -device vfio-pci,host=01:00.1,id=hostdev0,addr=0x6:
>      vfio 0000:01:00.1: failed to open /dev/vfio/1: Device or resource busy
> [2] qemu-system-x86_64: -device vfio-pci,host=01:00.1,id=hostdev0,addr=0x6:
>      vfio 0000:01:00.1: group 1 used in multiple address spaces
> [3] qemu-system-x86_64: -device vfio-pci,host=01:00.1,id=hostdev0,addr=0x6:
>      vfio 0000:01:00.1: failed to setup container for group 1: Failed to set
>      iommu for container: Device or resource busy
> [4] qemu-system-x86_64: -device vfio-pci,host=01:00.1,id=hostdev0,addr=0x6:
>      vfio 0000:01:00.1: failed to setup container for group 1: Failed to set
>      iommu for container: Device or resource busy
>     Or
>     qemu-system-x86_64: -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/
>      83b8f4f2-509f-382f-3c1e-e6bfe0fa1003: vfio 83b8f4f2-509f-382f-3c1e-
>      e6bfe0fa1003: failed to setup container for group 11: Failed to set iommu
>      for container: Device or resource busy
> Some other tests are not listed. Like bind NIC0 to vfio-pci-mdev and try to
> passthru it with "vfio-pci,host=01:00.0", kernel will throw a warn log and
> fail the operation.
> 
> Please feel free give your comments.
> 
> Thanks,
> Yi Liu
> 
> Change log:
>   patch v1 -> patch v2:
>   - the sample driver implementation refined
>   - the sample driver can work on non-singleton iommu groups
>   - the sample driver can work with vfio-pci, devices from a non-singleton
>     group can either be bound to vfio-mdev-pci or vfio-pci, and the
>     assignment of this group still follows current vfio assignment rule.
> 
>   RFC v3 -> patch v1:
>   - split the patchset from 3 patches to 9 patches to better demonstrate
>     the changes step by step
> 
>   v2->v3:
>   - use vfio-mdev-pci instead of vfio-pci-mdev
>   - place the new driver under drivers/vfio/pci while define
>     Kconfig in samples/Kconfig to clarify it is a sample driver
> 
>   v1->v2:
>   - instead of adding kernel option to existing vfio-pci
>     module in v1, v2 follows Alex's suggestion to add a
>     separate vfio-pci-mdev module.
>   - new patchset subject: "vfio/pci: wrap pci device as a mediated device"
> 
> Alex Williamson (1):
>   samples: refine vfio-mdev-pci driver
> 
> Liu Yi L (12):
>   vfio_pci: move vfio_pci_is_vga/vfio_vga_disabled to header
>   vfio_pci: refine user config reference in vfio-pci module
>   vfio_pci: refine vfio_pci_driver reference in vfio_pci.c
>   vfio_pci: make common functions be extern
>   vfio_pci: duplicate vfio_pci.c
>   vfio_pci: shrink vfio_pci_common.c
>   vfio_pci: shrink vfio_pci.c
>   vfio/pci: protect cap/ecap_perm bits alloc/free with atomic op
>   samples: add vfio-mdev-pci driver
>   samples/vfio-mdev-pci: call vfio_add_group_dev()
>   vfio/type1: use iommu_attach_group() for wrapping PF/VF as mdev
>   vfio/type1: track iommu backed group attach
> 
>  drivers/vfio/pci/Makefile           |    9 +-
>  drivers/vfio/pci/vfio_mdev_pci.c    |  497 ++++++++++++
>  drivers/vfio/pci/vfio_pci.c         | 1449 +---------------------------------
>  drivers/vfio/pci/vfio_pci_common.c  | 1455
> +++++++++++++++++++++++++++++++++++
>  drivers/vfio/pci/vfio_pci_config.c  |    9 +
>  drivers/vfio/pci/vfio_pci_private.h |   36 +
>  drivers/vfio/vfio_iommu_type1.c     |  185 ++++-
>  samples/Kconfig                     |   11 +
>  8 files changed, 2194 insertions(+), 1457 deletions(-)
>  create mode 100644 drivers/vfio/pci/vfio_mdev_pci.c
>  create mode 100644 drivers/vfio/pci/vfio_pci_common.c
> 
> --
> 2.7.4