From: Yishai Hadas <yishaih@nvidia.com>
To: <linux-rdma@vger.kernel.org>
Cc: <jgg@nvidia.com>, <maorg@nvidia.com>, <markzhang@nvidia.com>,
<edwards@nvidia.com>
Subject: Re: [PATCH rdma-core 00/27] Introduce mlx5 user space driver over VFIO
Date: Sun, 1 Aug 2021 11:00:05 +0300 [thread overview]
Message-ID: <c8b13b98-0c68-939d-0a6c-801beaef7d3e@nvidia.com> (raw)
In-Reply-To: <20210720081647.1980-1-yishaih@nvidia.com>
On 7/20/2021 11:16 AM, Yishai Hadas wrote:
> This series introduces mlx5 user space driver over VFIO.
>
> This enables an application to take full ownership on the opened device and run
> any firmware command (e.g. port up/down) without any concern to hurt someone
> else.
>
> The application look and feel is like regular RDMA application over DEVX, it
> uses verbs API to open/close a device and then mostly uses DEVX APIs to
> interact with the device.
>
> To achieve it, few mlx5 DV APIs were introduced, it includes:
> - An API to get ibv_device for a given mlx5 PCI name.
> - APIs to manage device specific events.
>
> Detailed man pages were added to describe the expected usage of those APIs.
>
> The mlx5 VFIO driver implemented the basic verbs APIs as of managing MR/PD and
> the DEVX APIs which are required to write an RDMA application.
>
> The VFIO uAPIs were used to setup mlx5 vfio context by reading the device
> initialization segment, setup DMA and enables the device command interface.
>
> In addition, the series includes pyverbs stuff which runs DEVX like testing
> over RDMA and VFIO mlx5 devices.
>
> Some extra documentation of the benefits and the motivation to use VFIO can be
> found here [1].
>
> PR was sent [2].
>
> [1] https://www.kernel.org/doc/html/latest/driver-api/vfio.html
> [2] https://github.com/linux-rdma/rdma-core/pull/1034
>
> Yishai
>
> Edward Srouji (10):
> pyverbs: Support DevX UMEM registration
> pyverbs/mlx5: Support EQN querying
> pyverbs/mlx5: Support more DevX objects
> pyverbs: Add auxiliary memory functions
> pyverbs/mlx5: Add support to extract mlx5dv objects
> pyverbs/mlx5: Wrap mlx5_cqe64 struct and add enums
> tests: Add MAC address to the tests' args
> tests: Add mlx5 DevX data path test
> pyverbs/mlx5: Support mlx5 devices over VFIO
> tests: Add a test for mlx5 over VFIO
>
> Maor Gottlieb (1):
> mlx5: Enable debug functionality for vfio
>
> Mark Zhang (5):
> util: Add interval_set support
> mlx5: Support fast teardown over vfio
> mlx5: VFIO poll_health support
> mlx5: Set DV context ops
> mlx5: Implement mlx5dv devx_obj APIs over vfio
>
> Yishai Hadas (11):
> Update kernel headers
> mlx5: Introduce mlx5dv_get_vfio_device_list()
> verbs: Enable verbs_open_device() to work over non sysfs devices
> mlx5: Setup mlx5 vfio context
> mlx5: Add mlx5_vfio_cmd_exec() support
> mlx5: vfio setup function support
> mlx5: vfio setup basic caps
> mlx5: Enable interrupt command mode over vfio
> mlx5: Introduce vfio APIs to process events
> mlx5: Implement basic verbs operation for PD and MR over vfio
> mlx5: Support initial DEVX/DV APIs over vfio
>
> Documentation/pyverbs.md | 34 +
> debian/ibverbs-providers.symbols | 4 +
> kernel-headers/CMakeLists.txt | 4 +
> kernel-headers/linux/vfio.h | 1374 ++++++++
> libibverbs/device.c | 39 +-
> libibverbs/sysfs.c | 5 +
> providers/mlx5/CMakeLists.txt | 3 +-
> providers/mlx5/dr_rule.c | 10 +-
> providers/mlx5/libmlx5.map | 7 +
> providers/mlx5/man/CMakeLists.txt | 3 +
> .../mlx5/man/mlx5dv_get_vfio_device_list.3.md | 64 +
> providers/mlx5/man/mlx5dv_vfio_get_events_fd.3.md | 41 +
> providers/mlx5/man/mlx5dv_vfio_process_events.3.md | 43 +
> providers/mlx5/mlx5.c | 376 ++-
> providers/mlx5/mlx5.h | 187 +-
> providers/mlx5/mlx5_ifc.h | 1206 ++++++-
> providers/mlx5/mlx5_vfio.c | 3379 ++++++++++++++++++++
> providers/mlx5/mlx5_vfio.h | 329 ++
> providers/mlx5/mlx5dv.h | 25 +
> providers/mlx5/verbs.c | 966 +++++-
> pyverbs/CMakeLists.txt | 7 +
> pyverbs/dma_util.pyx | 25 +
> pyverbs/mem_alloc.pyx | 46 +-
> pyverbs/pd.pyx | 4 +
> pyverbs/providers/mlx5/CMakeLists.txt | 4 +-
> pyverbs/providers/mlx5/libmlx5.pxd | 103 +-
> pyverbs/providers/mlx5/mlx5_vfio.pxd | 15 +
> pyverbs/providers/mlx5/mlx5_vfio.pyx | 116 +
> pyverbs/providers/mlx5/mlx5dv.pxd | 20 +
> pyverbs/providers/mlx5/mlx5dv.pyx | 277 +-
> pyverbs/providers/mlx5/mlx5dv_enums.pxd | 42 +
> pyverbs/providers/mlx5/mlx5dv_objects.pxd | 28 +
> pyverbs/providers/mlx5/mlx5dv_objects.pyx | 214 ++
> tests/CMakeLists.txt | 3 +
> tests/args_parser.py | 5 +
> tests/base.py | 14 +-
> tests/mlx5_base.py | 460 ++-
> tests/mlx5_prm_structs.py | 1046 ++++++
> tests/test_mlx5_devx.py | 23 +
> tests/test_mlx5_vfio.py | 104 +
> util/CMakeLists.txt | 2 +
> util/interval_set.c | 208 ++
> util/interval_set.h | 77 +
> util/util.h | 5 +
> 44 files changed, 10650 insertions(+), 297 deletions(-)
> create mode 100644 kernel-headers/linux/vfio.h
> create mode 100644 providers/mlx5/man/mlx5dv_get_vfio_device_list.3.md
> create mode 100644 providers/mlx5/man/mlx5dv_vfio_get_events_fd.3.md
> create mode 100644 providers/mlx5/man/mlx5dv_vfio_process_events.3.md
> create mode 100644 providers/mlx5/mlx5_vfio.c
> create mode 100644 providers/mlx5/mlx5_vfio.h
> create mode 100644 pyverbs/dma_util.pyx
> create mode 100644 pyverbs/providers/mlx5/mlx5_vfio.pxd
> create mode 100644 pyverbs/providers/mlx5/mlx5_vfio.pyx
> create mode 100644 pyverbs/providers/mlx5/mlx5dv_objects.pxd
> create mode 100644 pyverbs/providers/mlx5/mlx5dv_objects.pyx
> create mode 100644 tests/mlx5_prm_structs.py
> create mode 100644 tests/test_mlx5_devx.py
> create mode 100644 tests/test_mlx5_vfio.py
> create mode 100644 util/interval_set.c
> create mode 100644 util/interval_set.h
>
The PR was merged.
Yishai
prev parent reply other threads:[~2021-08-01 8:01 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-07-20 8:16 [PATCH rdma-core 00/27] Introduce mlx5 user space driver over VFIO Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 01/27] Update kernel headers Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 02/27] mlx5: Introduce mlx5dv_get_vfio_device_list() Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 03/27] mlx5: Enable debug functionality for vfio Yishai Hadas
2021-07-20 8:51 ` Leon Romanovsky
2021-07-20 9:27 ` Yishai Hadas
2021-07-20 12:27 ` Leon Romanovsky
2021-07-20 14:57 ` Yishai Hadas
2021-07-21 7:05 ` Gal Pressman
2021-07-21 7:58 ` Yishai Hadas
2021-07-21 8:51 ` Gal Pressman
2021-07-20 8:16 ` [PATCH rdma-core 04/27] util: Add interval_set support Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 05/27] verbs: Enable verbs_open_device() to work over non sysfs devices Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 06/27] mlx5: Setup mlx5 vfio context Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 07/27] mlx5: Add mlx5_vfio_cmd_exec() support Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 08/27] mlx5: vfio setup function support Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 09/27] mlx5: vfio setup basic caps Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 10/27] mlx5: Support fast teardown over vfio Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 11/27] mlx5: Enable interrupt command mode " Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 12/27] mlx5: Introduce vfio APIs to process events Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 13/27] mlx5: VFIO poll_health support Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 14/27] mlx5: Implement basic verbs operation for PD and MR over vfio Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 15/27] mlx5: Set DV context ops Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 16/27] mlx5: Support initial DEVX/DV APIs over vfio Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 17/27] mlx5: Implement mlx5dv devx_obj " Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 18/27] pyverbs: Support DevX UMEM registration Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 19/27] pyverbs/mlx5: Support EQN querying Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 20/27] pyverbs/mlx5: Support more DevX objects Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 21/27] pyverbs: Add auxiliary memory functions Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 22/27] pyverbs/mlx5: Add support to extract mlx5dv objects Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 23/27] pyverbs/mlx5: Wrap mlx5_cqe64 struct and add enums Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 24/27] tests: Add MAC address to the tests' args Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 25/27] tests: Add mlx5 DevX data path test Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 26/27] pyverbs/mlx5: Support mlx5 devices over VFIO Yishai Hadas
2021-07-20 8:16 ` [PATCH rdma-core 27/27] tests: Add a test for mlx5 " Yishai Hadas
2021-08-01 8:00 ` Yishai Hadas [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=c8b13b98-0c68-939d-0a6c-801beaef7d3e@nvidia.com \
--to=yishaih@nvidia.com \
--cc=edwards@nvidia.com \
--cc=jgg@nvidia.com \
--cc=linux-rdma@vger.kernel.org \
--cc=maorg@nvidia.com \
--cc=markzhang@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).