From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.7 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS, UNPARSEABLE_RELAY,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01B02C433E0 for ; Wed, 17 Jun 2020 07:46:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D96E5208D5 for ; Wed, 17 Jun 2020 07:46:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726303AbgFQHqd (ORCPT ); Wed, 17 Jun 2020 03:46:33 -0400 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:37711 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726369AbgFQHqc (ORCPT ); Wed, 17 Jun 2020 03:46:32 -0400 Received: from Internal Mail-Server by MTLPINE1 (envelope-from yishaih@mellanox.com) with SMTP; 17 Jun 2020 10:46:28 +0300 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [10.7.2.17]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 05H7kSYZ017624; Wed, 17 Jun 2020 10:46:28 +0300 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [127.0.0.1]) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8) with ESMTP id 05H7kSQe007157; Wed, 17 Jun 2020 10:46:28 +0300 Received: (from yishaih@localhost) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8/Submit) id 05H7kSDc007156; Wed, 17 Jun 2020 10:46:28 +0300 From: Yishai Hadas To: linux-rdma@vger.kernel.org Cc: jgg@mellanox.com, yishaih@mellanox.com, maorg@mellanox.com Subject: [PATCH rdma-core 00/13] verbs: Introduce import verbs for device, PD, MR Date: Wed, 17 Jun 2020 10:45:43 +0300 Message-Id: <1592379956-7043-1-git-send-email-yishaih@mellanox.com> X-Mailer: git-send-email 1.8.2.3 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org This series Introduces import verbs for device, PD, MR which enables processes to share their ibv_context and then share PD(s) and MR(s) that are associated with. This functionality enables utilizing and optimizing some application flows, few examples below. Any solution which is a single business logic based on multi-process design needs this. Example include NGINX, with TCP load balancing, sharing the RSS indirection table with RQ per process. HPC frameworks with multi-rank (process) solution on single hosts. UCX can share IB resources using the shared PD and can help dispatch data to multiple processes/MR's in single RDMA operation. Also, there are use cases when a primary processes registered a large shared memory range, and each worker process spawned will create a private QP on the shared PD, and use the shared MR to save the registration time per-process. As part of this series was added also some pyverbs stuff to support and demonstrate some usage of the APIs. The verbs APIs were introduced in the mailing list by the below RFC [1], the matching kernel series was sent to rdma-next, PR: https://github.com/linux-rdma/rdma-core/pull/776 [1] https://patchwork.kernel.org/patch/11540665/ Yishai Edward Srouji (3): pyverbs: Support verbs import APIs Documentation: Add usage example for verbs import tests: Add a shared PD Pyverbs test Yishai Hadas (10): Update kernel headers verbs: Close async_fd only when it was previously created verbs: Introduce ibv_import_device() verb verbs: Handle async FD on an imported device mlx5: Refactor mlx5_alloc_context() mlx5: Implement the import device functionality verbs: Introduce ibv_import/unimport_pd() verbs mlx5: Implement the import/unimport PD verbs verbs: Introduce ibv_import/unimport_mr() verbs mlx5: Implement the import/unimport MR verbs Documentation/pyverbs.md | 40 ++++ debian/libibverbs1.symbols | 6 + kernel-headers/rdma/ib_user_ioctl_cmds.h | 15 ++ kernel-headers/rdma/mlx5-abi.h | 9 +- kernel-headers/rdma/mlx5_user_ioctl_cmds.h | 14 ++ kernel-headers/rdma/rdma_netlink.h | 8 + kernel-headers/rdma/rdma_user_cm.h | 11 +- kernel-headers/rdma/rdma_user_ioctl_cmds.h | 2 +- libibverbs/CMakeLists.txt | 2 +- libibverbs/cmd_cq.c | 9 +- libibverbs/cmd_device.c | 32 ++- libibverbs/cmd_mr.c | 35 ++++ libibverbs/cmd_qp.c | 4 + libibverbs/cmd_srq.c | 4 + libibverbs/cmd_wq.c | 4 + libibverbs/device.c | 73 ++++++- libibverbs/driver.h | 14 ++ libibverbs/dummy_ops.c | 30 +++ libibverbs/ibverbs.h | 1 + libibverbs/libibverbs.map.in | 11 + libibverbs/man/CMakeLists.txt | 5 + libibverbs/man/ibv_import_device.3.md | 48 +++++ libibverbs/man/ibv_import_mr.3.md | 63 ++++++ libibverbs/man/ibv_import_pd.3.md | 57 ++++++ libibverbs/verbs.c | 30 +++ libibverbs/verbs.h | 26 +++ providers/mlx5/mlx5.c | 317 ++++++++++++++++++----------- providers/mlx5/mlx5.h | 6 + providers/mlx5/verbs.c | 78 ++++++- pyverbs/device.pyx | 12 +- pyverbs/libibverbs.pxd | 5 + pyverbs/mr.pxd | 1 + pyverbs/mr.pyx | 60 +++++- pyverbs/pd.pxd | 1 + pyverbs/pd.pyx | 37 +++- tests/CMakeLists.txt | 1 + tests/base.py | 11 +- tests/test_shared_pd.py | 95 +++++++++ 38 files changed, 1022 insertions(+), 155 deletions(-) create mode 100644 libibverbs/man/ibv_import_device.3.md create mode 100644 libibverbs/man/ibv_import_mr.3.md create mode 100644 libibverbs/man/ibv_import_pd.3.md create mode 100644 tests/test_shared_pd.py -- 1.8.3.1