linux-rdma.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Yishai Hadas <yishaih@mellanox.com>
To: linux-rdma@vger.kernel.org
Cc: jgg@mellanox.com, yishaih@mellanox.com, maorg@mellanox.com,
	Edward Srouji <edwards@mellanox.com>
Subject: [PATCH rdma-core 12/13] Documentation: Add usage example for verbs import
Date: Wed, 17 Jun 2020 10:45:55 +0300	[thread overview]
Message-ID: <1592379956-7043-13-git-send-email-yishaih@mellanox.com> (raw)
In-Reply-To: <1592379956-7043-1-git-send-email-yishaih@mellanox.com>

From: Edward Srouji <edwards@mellanox.com>

Add a documentation for shared PD and verbs import usage in Pyverbs.
This includes a code snippet to demonstrate a usage example.

Reviewed-by: Ido Kalir <idok@mellanox.com>
Signed-off-by: Edward Srouji <edwards@mellanox.com>
---
 Documentation/pyverbs.md | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/Documentation/pyverbs.md b/Documentation/pyverbs.md
index 77a5406..2516b10 100755
--- a/Documentation/pyverbs.md
+++ b/Documentation/pyverbs.md
@@ -626,3 +626,43 @@ ctx = Context(name='rocep0s8f0')
 uar = Mlx5UAR(ctx)
 uar.close()
 ```
+
+##### Import device, PD and MR
+Importing a device, PD and MR enables processes to share their context and then
+share PDs and MRs that is associated with.
+A process creates a device and then uses some of the Linux systems calls to dup
+its 'cmd_fd' member which lets other process to obtain ownership.
+Once other process obtains the 'cmd_fd' it can import the device, then PD(s) and
+MR(s) to share these objects.
+Like in C, Pyverbs users are responsible for unimporting the imported objects
+(which will also close the Pyverbs instance in our case) after they finish using
+them, and they have to sync between the different processes in order to
+coordinate the closure of the objects.
+Unlike in C, closing the underlying objects is currently supported only via the
+"original" object (meaning only by the process that creates them) and not via
+the imported object. This limitation is made because currently there's no
+reference or relation between different Pyverbs objects in different processes.
+But it's doable and might be added in the future.
+Here is a demonstration of importing a device, PD and MR in one process.
+```python
+from pyverbs.device import Context
+from pyverbs.pd import PD
+from pyverbs.mr import MR
+import pyverbs.enums as e
+import os
+
+ctx = Context(name='ibp0s8f0')
+pd = PD(ctx)
+mr = MR(pd, 100, e.IBV_ACCESS_LOCAL_WRITE)
+cmd_fd_dup = os.dup(ctx.cmd_fd)
+improted_ctx = Context(cmd_fd=cmd_fd_dup)
+imported_pd = PD(improted_ctx, handle=pd.handle)
+imported_mr = MR(imported_pd, handle=mr.handle)
+# MRs can be created as usual on the imported PD
+secondary_mr = MR(imported_pd, 100, e.IBV_ACCESS_REMOTE_READ)
+# Must manually unimport the imported objects (which close the object and frees
+# other resources that use them) before closing the "original" objects.
+# This prevents unexpected behaviours caused by the GC.
+imported_mr.unimport()
+imported_pd.unimport()
+```
-- 
1.8.3.1


  parent reply	other threads:[~2020-06-17  7:46 UTC|newest]

Thread overview: 30+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-06-17  7:45 [PATCH rdma-core 00/13] verbs: Introduce import verbs for device, PD, MR Yishai Hadas
2020-06-17  7:45 ` [PATCH rdma-core 01/13] Update kernel headers Yishai Hadas
2020-06-17  7:45 ` [PATCH rdma-core 02/13] verbs: Close async_fd only when it was previously created Yishai Hadas
2020-06-17  7:45 ` [PATCH rdma-core 03/13] verbs: Introduce ibv_import_device() verb Yishai Hadas
2020-06-19 12:29   ` Jason Gunthorpe
2020-06-21  7:01     ` Yishai Hadas
2020-06-22 12:52       ` Jason Gunthorpe
2020-06-23 13:06         ` Yishai Hadas
2020-06-17  7:45 ` [PATCH rdma-core 04/13] verbs: Handle async FD on an imported device Yishai Hadas
2020-06-19 12:33   ` Jason Gunthorpe
2020-06-21  9:08     ` Yishai Hadas
2020-06-23 17:34       ` Jason Gunthorpe
2020-06-24  7:22         ` Yishai Hadas
2020-06-17  7:45 ` [PATCH rdma-core 05/13] mlx5: Refactor mlx5_alloc_context() Yishai Hadas
2020-06-17  7:45 ` [PATCH rdma-core 06/13] mlx5: Implement the import device functionality Yishai Hadas
2020-06-17  7:45 ` [PATCH rdma-core 07/13] verbs: Introduce ibv_import/unimport_pd() verbs Yishai Hadas
2020-06-19 12:48   ` Jason Gunthorpe
2020-06-21  8:30     ` Yishai Hadas
2020-06-17  7:45 ` [PATCH rdma-core 08/13] mlx5: Implement the import/unimport PD verbs Yishai Hadas
2020-06-17  7:45 ` [PATCH rdma-core 09/13] verbs: Introduce ibv_import/unimport_mr() verbs Yishai Hadas
2020-06-17  7:45 ` [PATCH rdma-core 10/13] mlx5: Implement the import/unimport MR verbs Yishai Hadas
2020-06-19 12:50   ` Jason Gunthorpe
2020-06-21  8:44     ` Yishai Hadas
2020-06-23 17:33       ` Jason Gunthorpe
2020-06-24  7:20         ` Yishai Hadas
2020-07-01 12:28         ` Yishai Hadas
2020-07-02 17:42           ` Jason Gunthorpe
2020-06-17  7:45 ` [PATCH rdma-core 11/13] pyverbs: Support verbs import APIs Yishai Hadas
2020-06-17  7:45 ` Yishai Hadas [this message]
2020-06-17  7:45 ` [PATCH rdma-core 13/13] tests: Add a shared PD Pyverbs test Yishai Hadas

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1592379956-7043-13-git-send-email-yishaih@mellanox.com \
    --to=yishaih@mellanox.com \
    --cc=edwards@mellanox.com \
    --cc=jgg@mellanox.com \
    --cc=linux-rdma@vger.kernel.org \
    --cc=maorg@mellanox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).