[PATCH rdma-core 00/27] Introduce mlx5 user space driver over VFIO

[Yishai Hadas <yishaih@nvidia.com>]

This series introduces mlx5 user space driver over VFIO.

This enables an application to take full ownership on the opened device and run
any firmware command (e.g. port up/down) without any concern to hurt someone
else.

The application look and feel is like regular RDMA application over DEVX, it
uses verbs API to open/close a device and then mostly uses DEVX APIs to
interact with the device.

To achieve it, few mlx5 DV APIs were introduced, it includes:
- An API to get ibv_device for a given mlx5 PCI name.
- APIs to manage device specific events.

Detailed man pages were added to describe the expected usage of those APIs.

The mlx5 VFIO driver implemented the basic verbs APIs as of managing MR/PD and
the DEVX APIs which are required to write an RDMA application.

The VFIO uAPIs were used to setup mlx5 vfio context by reading the device
initialization segment, setup DMA and enables the device command interface.

In addition, the series includes pyverbs stuff which runs DEVX like testing
over RDMA and VFIO mlx5 devices.

Some extra documentation of the benefits and the motivation to use VFIO can be
found here [1].

PR was sent [2].

[1] https://www.kernel.org/doc/html/latest/driver-api/vfio.html
[2] https://github.com/linux-rdma/rdma-core/pull/1034

Yishai

Edward Srouji (10):
  pyverbs: Support DevX UMEM registration
  pyverbs/mlx5: Support EQN querying
  pyverbs/mlx5: Support more DevX objects
  pyverbs: Add auxiliary memory functions
  pyverbs/mlx5: Add support to extract mlx5dv objects
  pyverbs/mlx5: Wrap mlx5_cqe64 struct and add enums
  tests: Add MAC address to the tests' args
  tests: Add mlx5 DevX data path test
  pyverbs/mlx5: Support mlx5 devices over VFIO
  tests: Add a test for mlx5 over VFIO

Maor Gottlieb (1):
  mlx5: Enable debug functionality for vfio

Mark Zhang (5):
  util: Add interval_set support
  mlx5: Support fast teardown over vfio
  mlx5: VFIO poll_health support
  mlx5: Set DV context ops
  mlx5: Implement mlx5dv devx_obj APIs over vfio

Yishai Hadas (11):
  Update kernel headers
  mlx5: Introduce mlx5dv_get_vfio_device_list()
  verbs: Enable verbs_open_device() to work over non sysfs devices
  mlx5: Setup mlx5 vfio context
  mlx5: Add mlx5_vfio_cmd_exec() support
  mlx5: vfio setup function support
  mlx5: vfio setup basic caps
  mlx5: Enable interrupt command mode over vfio
  mlx5: Introduce vfio APIs to process events
  mlx5: Implement basic verbs operation for PD and MR over vfio
  mlx5: Support initial DEVX/DV APIs over vfio

 Documentation/pyverbs.md                           |   34 +
 debian/ibverbs-providers.symbols                   |    4 +
 kernel-headers/CMakeLists.txt                      |    4 +
 kernel-headers/linux/vfio.h                        | 1374 ++++++++
 libibverbs/device.c                                |   39 +-
 libibverbs/sysfs.c                                 |    5 +
 providers/mlx5/CMakeLists.txt                      |    3 +-
 providers/mlx5/dr_rule.c                           |   10 +-
 providers/mlx5/libmlx5.map                         |    7 +
 providers/mlx5/man/CMakeLists.txt                  |    3 +
 .../mlx5/man/mlx5dv_get_vfio_device_list.3.md      |   64 +
 providers/mlx5/man/mlx5dv_vfio_get_events_fd.3.md  |   41 +
 providers/mlx5/man/mlx5dv_vfio_process_events.3.md |   43 +
 providers/mlx5/mlx5.c                              |  376 ++-
 providers/mlx5/mlx5.h                              |  187 +-
 providers/mlx5/mlx5_ifc.h                          | 1206 ++++++-
 providers/mlx5/mlx5_vfio.c                         | 3379 ++++++++++++++++++++
 providers/mlx5/mlx5_vfio.h                         |  329 ++
 providers/mlx5/mlx5dv.h                            |   25 +
 providers/mlx5/verbs.c                             |  966 +++++-
 pyverbs/CMakeLists.txt                             |    7 +
 pyverbs/dma_util.pyx                               |   25 +
 pyverbs/mem_alloc.pyx                              |   46 +-
 pyverbs/pd.pyx                                     |    4 +
 pyverbs/providers/mlx5/CMakeLists.txt              |    4 +-
 pyverbs/providers/mlx5/libmlx5.pxd                 |  103 +-
 pyverbs/providers/mlx5/mlx5_vfio.pxd               |   15 +
 pyverbs/providers/mlx5/mlx5_vfio.pyx               |  116 +
 pyverbs/providers/mlx5/mlx5dv.pxd                  |   20 +
 pyverbs/providers/mlx5/mlx5dv.pyx                  |  277 +-
 pyverbs/providers/mlx5/mlx5dv_enums.pxd            |   42 +
 pyverbs/providers/mlx5/mlx5dv_objects.pxd          |   28 +
 pyverbs/providers/mlx5/mlx5dv_objects.pyx          |  214 ++
 tests/CMakeLists.txt                               |    3 +
 tests/args_parser.py                               |    5 +
 tests/base.py                                      |   14 +-
 tests/mlx5_base.py                                 |  460 ++-
 tests/mlx5_prm_structs.py                          | 1046 ++++++
 tests/test_mlx5_devx.py                            |   23 +
 tests/test_mlx5_vfio.py                            |  104 +
 util/CMakeLists.txt                                |    2 +
 util/interval_set.c                                |  208 ++
 util/interval_set.h                                |   77 +
 util/util.h                                        |    5 +
 44 files changed, 10650 insertions(+), 297 deletions(-)
 create mode 100644 kernel-headers/linux/vfio.h
 create mode 100644 providers/mlx5/man/mlx5dv_get_vfio_device_list.3.md
 create mode 100644 providers/mlx5/man/mlx5dv_vfio_get_events_fd.3.md
 create mode 100644 providers/mlx5/man/mlx5dv_vfio_process_events.3.md
 create mode 100644 providers/mlx5/mlx5_vfio.c
 create mode 100644 providers/mlx5/mlx5_vfio.h
 create mode 100644 pyverbs/dma_util.pyx
 create mode 100644 pyverbs/providers/mlx5/mlx5_vfio.pxd
 create mode 100644 pyverbs/providers/mlx5/mlx5_vfio.pyx
 create mode 100644 pyverbs/providers/mlx5/mlx5dv_objects.pxd
 create mode 100644 pyverbs/providers/mlx5/mlx5dv_objects.pyx
 create mode 100644 tests/mlx5_prm_structs.py
 create mode 100644 tests/test_mlx5_devx.py
 create mode 100644 tests/test_mlx5_vfio.py
 create mode 100644 util/interval_set.c
 create mode 100644 util/interval_set.h

-- 
1.8.3.1