All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH rdma-core v7 0/6] Add user space dma-buf support
@ 2021-01-25 19:56 ` Jianxin Xiong
  0 siblings, 0 replies; 42+ messages in thread
From: Jianxin Xiong @ 2021-01-25 19:56 UTC (permalink / raw)
  To: linux-rdma, dri-devel
  Cc: Jianxin Xiong, Doug Ledford, Jason Gunthorpe, Leon Romanovsky,
	Sumit Semwal, Christian Koenig, Daniel Vetter, Edward Srouji,
	Yishai Hadas

This is the seventh version of the patch series. Change log:

v7:
* Rebase to the latest rdma-core master (commit a4885eda9addc4)
* Rerun kernel-headers/update against linux-rdma for-next so that the
  kernel commit id in the commit message is correct

v6: https://www.spinics.net/lists/linux-rdma/msg99221.html
* Rebase to the latest rdma-core master (commit 14006f2f841b0c)
* Update the ABI symbol version to match new package version; also bump
  the private ABI version because new function has been added to the
  provider interface
* Avoid changing 'struct ibv_context_ops' by replacing SET_OP() with
  SET_PRIV_OP_IC()
* Replace sprintf() with snprintf()
* Keep the ops in verbs_set_ops() sorted
* Fix some styling issues: extra spaces, struct 0-initialization, error
  checking control flow

v5: https://www.spinics.net/lists/linux-rdma/msg99015.html
* Use a different mr_type for dmabuf so that ibv_dofork_range() is not
  called inside ibv_dereg_mr() for dmabuf based mr

v4: https://www.spinics.net/lists/linux-rdma/msg98135.html
* Rework the cmake funciton rdma_cython_module to support both single
  source (.pyx) and multiple source (.pyx + [.c]*) scenarios instead
  of using two separate functions
* Rename 'dri_*' to 'drm_*' for the dmabuf allocation interface
* Add option to dmabuf allocation routine to allow allocation from GTT
  instead of VRAM
* Add proper CPU access flags when allocating dmabufs
* Remove 'radeon' driver support from the dmabuf allocation routines
* Add comand line arguments to the tests for selecting GPU unit and
  setting the option for allocating from GTT

v3: https://www.spinics.net/lists/linux-rdma/msg98059.html
* Add parameter 'iova' to the new ibv_reg_dmabuf_mr() API
* Change the way of allocating dma-buf object - use /dev/dri/renderD*
  instead of /dev/dri/card* and use GEM object instead of dumb buffer
* Add cmake function to allow building modules with mixed cython and C
  source files
* Add new tests that use dma-buf MRs for send/recv and rdma traffic
* Skip dma-buf tests on unsupported systems
* Remove some use of random values in the new tests
* Add dealloc() and close() methods to the new classes
* Replace string.format with f-string in python code
* Fix some coding style issues: spacing, indentation, typo, comments

v2: https://www.spinics.net/lists/linux-rdma/msg97936.html
* Put the kernel header updates into a separate commit
* Add comments for the data structure used in python ioctl calls
* Fix issues related to symbol versioning
* Fix styling issues: extra spaces, unncecessary variable, typo
* Fix an inproper error code usage
* Put the new op into ibv_context_ops instead if verbs_context

v1: https://www.spinics.net/lists/linux-rdma/msg97865.html
* Add user space API for registering dma-buf based memory regions
* Update pyverbs with the new API
* Add new tests

This is the user space counter-part of the kernel patch set to add
dma-buf support to the RDMA subsystem.

This series consists of six patches. The first patch updates the
kernel headers for dma-buf support. Patch 2 adds the new API function
and updates the man pages. Patch 3 implements the new API in the mlx5
provider. Patch 4 adds new class definitions to pyverbs for the new API.
Patch 5 adds a set of new tests for the new API. Patch 6 fixes bug in
the utility code of the tests.

Pull request at github: https://github.com/linux-rdma/rdma-core/pull/895

Jianxin Xiong (6):
  Update kernel headers
  verbs: Support dma-buf based memory region
  mlx5: Support dma-buf based memory region
  pyverbs: Add dma-buf based MR support
  tests: Add tests for dma-buf based memory regions
  tests: Bug fix for get_access_flags()

 buildlib/pyverbs_functions.cmake         |  78 ++++++---
 debian/libibverbs1.symbols               |   2 +
 kernel-headers/rdma/ib_user_ioctl_cmds.h |  14 ++
 libibverbs/CMakeLists.txt                |   2 +-
 libibverbs/cmd_mr.c                      |  38 +++++
 libibverbs/driver.h                      |   8 +
 libibverbs/dummy_ops.c                   |  11 ++
 libibverbs/libibverbs.map.in             |   6 +
 libibverbs/man/ibv_reg_mr.3              |  27 ++-
 libibverbs/verbs.c                       |  19 +++
 libibverbs/verbs.h                       |   7 +
 providers/mlx5/mlx5.c                    |   2 +
 providers/mlx5/mlx5.h                    |   3 +
 providers/mlx5/verbs.c                   |  22 +++
 pyverbs/CMakeLists.txt                   |  11 +-
 pyverbs/dmabuf.pxd                       |  15 ++
 pyverbs/dmabuf.pyx                       |  73 ++++++++
 pyverbs/dmabuf_alloc.c                   | 278 +++++++++++++++++++++++++++++++
 pyverbs/dmabuf_alloc.h                   |  19 +++
 pyverbs/libibverbs.pxd                   |   2 +
 pyverbs/mr.pxd                           |   6 +
 pyverbs/mr.pyx                           | 105 +++++++++++-
 tests/args_parser.py                     |   4 +
 tests/test_mr.py                         | 266 ++++++++++++++++++++++++++++-
 tests/utils.py                           |  30 +++-
 25 files changed, 1012 insertions(+), 36 deletions(-)
 create mode 100644 pyverbs/dmabuf.pxd
 create mode 100644 pyverbs/dmabuf.pyx
 create mode 100644 pyverbs/dmabuf_alloc.c
 create mode 100644 pyverbs/dmabuf_alloc.h

-- 
1.8.3.1


^ permalink raw reply	[flat|nested] 42+ messages in thread
* [PATCH rdma-core v7 0/6] Add user space dma-buf support
@ 2021-01-25 19:56 ` Jianxin Xiong
  0 siblings, 0 replies; 42+ messages in thread
From: Jianxin Xiong @ 2021-01-25 19:56 UTC (permalink / raw)
  To: linux-rdma, dri-devel
  Cc: Yishai Hadas, Leon Romanovsky, Edward Srouji, Jason Gunthorpe,
	Doug Ledford, Daniel Vetter, Christian Koenig, Jianxin Xiong

This is the seventh version of the patch series. Change log:

v7:
* Rebase to the latest rdma-core master (commit a4885eda9addc4)
* Rerun kernel-headers/update against linux-rdma for-next so that the
  kernel commit id in the commit message is correct

v6: https://www.spinics.net/lists/linux-rdma/msg99221.html
* Rebase to the latest rdma-core master (commit 14006f2f841b0c)
* Update the ABI symbol version to match new package version; also bump
  the private ABI version because new function has been added to the
  provider interface
* Avoid changing 'struct ibv_context_ops' by replacing SET_OP() with
  SET_PRIV_OP_IC()
* Replace sprintf() with snprintf()
* Keep the ops in verbs_set_ops() sorted
* Fix some styling issues: extra spaces, struct 0-initialization, error
  checking control flow

v5: https://www.spinics.net/lists/linux-rdma/msg99015.html
* Use a different mr_type for dmabuf so that ibv_dofork_range() is not
  called inside ibv_dereg_mr() for dmabuf based mr

v4: https://www.spinics.net/lists/linux-rdma/msg98135.html
* Rework the cmake funciton rdma_cython_module to support both single
  source (.pyx) and multiple source (.pyx + [.c]*) scenarios instead
  of using two separate functions
* Rename 'dri_*' to 'drm_*' for the dmabuf allocation interface
* Add option to dmabuf allocation routine to allow allocation from GTT
  instead of VRAM
* Add proper CPU access flags when allocating dmabufs
* Remove 'radeon' driver support from the dmabuf allocation routines
* Add comand line arguments to the tests for selecting GPU unit and
  setting the option for allocating from GTT

v3: https://www.spinics.net/lists/linux-rdma/msg98059.html
* Add parameter 'iova' to the new ibv_reg_dmabuf_mr() API
* Change the way of allocating dma-buf object - use /dev/dri/renderD*
  instead of /dev/dri/card* and use GEM object instead of dumb buffer
* Add cmake function to allow building modules with mixed cython and C
  source files
* Add new tests that use dma-buf MRs for send/recv and rdma traffic
* Skip dma-buf tests on unsupported systems
* Remove some use of random values in the new tests
* Add dealloc() and close() methods to the new classes
* Replace string.format with f-string in python code
* Fix some coding style issues: spacing, indentation, typo, comments

v2: https://www.spinics.net/lists/linux-rdma/msg97936.html
* Put the kernel header updates into a separate commit
* Add comments for the data structure used in python ioctl calls
* Fix issues related to symbol versioning
* Fix styling issues: extra spaces, unncecessary variable, typo
* Fix an inproper error code usage
* Put the new op into ibv_context_ops instead if verbs_context

v1: https://www.spinics.net/lists/linux-rdma/msg97865.html
* Add user space API for registering dma-buf based memory regions
* Update pyverbs with the new API
* Add new tests

This is the user space counter-part of the kernel patch set to add
dma-buf support to the RDMA subsystem.

This series consists of six patches. The first patch updates the
kernel headers for dma-buf support. Patch 2 adds the new API function
and updates the man pages. Patch 3 implements the new API in the mlx5
provider. Patch 4 adds new class definitions to pyverbs for the new API.
Patch 5 adds a set of new tests for the new API. Patch 6 fixes bug in
the utility code of the tests.

Pull request at github: https://github.com/linux-rdma/rdma-core/pull/895

Jianxin Xiong (6):
  Update kernel headers
  verbs: Support dma-buf based memory region
  mlx5: Support dma-buf based memory region
  pyverbs: Add dma-buf based MR support
  tests: Add tests for dma-buf based memory regions
  tests: Bug fix for get_access_flags()

 buildlib/pyverbs_functions.cmake         |  78 ++++++---
 debian/libibverbs1.symbols               |   2 +
 kernel-headers/rdma/ib_user_ioctl_cmds.h |  14 ++
 libibverbs/CMakeLists.txt                |   2 +-
 libibverbs/cmd_mr.c                      |  38 +++++
 libibverbs/driver.h                      |   8 +
 libibverbs/dummy_ops.c                   |  11 ++
 libibverbs/libibverbs.map.in             |   6 +
 libibverbs/man/ibv_reg_mr.3              |  27 ++-
 libibverbs/verbs.c                       |  19 +++
 libibverbs/verbs.h                       |   7 +
 providers/mlx5/mlx5.c                    |   2 +
 providers/mlx5/mlx5.h                    |   3 +
 providers/mlx5/verbs.c                   |  22 +++
 pyverbs/CMakeLists.txt                   |  11 +-
 pyverbs/dmabuf.pxd                       |  15 ++
 pyverbs/dmabuf.pyx                       |  73 ++++++++
 pyverbs/dmabuf_alloc.c                   | 278 +++++++++++++++++++++++++++++++
 pyverbs/dmabuf_alloc.h                   |  19 +++
 pyverbs/libibverbs.pxd                   |   2 +
 pyverbs/mr.pxd                           |   6 +
 pyverbs/mr.pyx                           | 105 +++++++++++-
 tests/args_parser.py                     |   4 +
 tests/test_mr.py                         | 266 ++++++++++++++++++++++++++++-
 tests/utils.py                           |  30 +++-
 25 files changed, 1012 insertions(+), 36 deletions(-)
 create mode 100644 pyverbs/dmabuf.pxd
 create mode 100644 pyverbs/dmabuf.pyx
 create mode 100644 pyverbs/dmabuf_alloc.c
 create mode 100644 pyverbs/dmabuf_alloc.h

-- 
1.8.3.1

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

^ permalink raw reply	[flat|nested] 42+ messages in thread
* [PATCH rdma-core v7 1/6] Update kernel headers
  2021-01-25 19:56 ` Jianxin Xiong
@ 2021-01-25 19:56   ` Jianxin Xiong
  -1 siblings, 0 replies; 42+ messages in thread
From: Jianxin Xiong @ 2021-01-25 19:56 UTC (permalink / raw)
  To: linux-rdma, dri-devel
  Cc: Jianxin Xiong, Doug Ledford, Jason Gunthorpe, Leon Romanovsky,
	Sumit Semwal, Christian Koenig, Daniel Vetter, Edward Srouji,
	Yishai Hadas

To commit bfe0cc6eb249 ("RDMA/uverbs: Add uverbs command for dma-buf based
MR registration").

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
 kernel-headers/rdma/ib_user_ioctl_cmds.h | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/kernel-headers/rdma/ib_user_ioctl_cmds.h b/kernel-headers/rdma/ib_user_ioctl_cmds.h
index 7968a18..dafc7eb 100644
--- a/kernel-headers/rdma/ib_user_ioctl_cmds.h
+++ b/kernel-headers/rdma/ib_user_ioctl_cmds.h
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2018, Mellanox Technologies inc.  All rights reserved.
+ * Copyright (c) 2020, Intel Corporation. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -251,6 +252,7 @@ enum uverbs_methods_mr {
 	UVERBS_METHOD_MR_DESTROY,
 	UVERBS_METHOD_ADVISE_MR,
 	UVERBS_METHOD_QUERY_MR,
+	UVERBS_METHOD_REG_DMABUF_MR,
 };
 
 enum uverbs_attrs_mr_destroy_ids {
@@ -272,6 +274,18 @@ enum uverbs_attrs_query_mr_cmd_attr_ids {
 	UVERBS_ATTR_QUERY_MR_RESP_IOVA,
 };
 
+enum uverbs_attrs_reg_dmabuf_mr_cmd_attr_ids {
+	UVERBS_ATTR_REG_DMABUF_MR_HANDLE,
+	UVERBS_ATTR_REG_DMABUF_MR_PD_HANDLE,
+	UVERBS_ATTR_REG_DMABUF_MR_OFFSET,
+	UVERBS_ATTR_REG_DMABUF_MR_LENGTH,
+	UVERBS_ATTR_REG_DMABUF_MR_IOVA,
+	UVERBS_ATTR_REG_DMABUF_MR_FD,
+	UVERBS_ATTR_REG_DMABUF_MR_ACCESS_FLAGS,
+	UVERBS_ATTR_REG_DMABUF_MR_RESP_LKEY,
+	UVERBS_ATTR_REG_DMABUF_MR_RESP_RKEY,
+};
+
 enum uverbs_attrs_create_counters_cmd_attr_ids {
 	UVERBS_ATTR_CREATE_COUNTERS_HANDLE,
 };
-- 
1.8.3.1


^ permalink raw reply related	[flat|nested] 42+ messages in thread
* [PATCH rdma-core v7 1/6] Update kernel headers
@ 2021-01-25 19:56   ` Jianxin Xiong
  0 siblings, 0 replies; 42+ messages in thread
From: Jianxin Xiong @ 2021-01-25 19:56 UTC (permalink / raw)
  To: linux-rdma, dri-devel
  Cc: Yishai Hadas, Leon Romanovsky, Edward Srouji, Jason Gunthorpe,
	Doug Ledford, Daniel Vetter, Christian Koenig, Jianxin Xiong

To commit bfe0cc6eb249 ("RDMA/uverbs: Add uverbs command for dma-buf based
MR registration").

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
 kernel-headers/rdma/ib_user_ioctl_cmds.h | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/kernel-headers/rdma/ib_user_ioctl_cmds.h b/kernel-headers/rdma/ib_user_ioctl_cmds.h
index 7968a18..dafc7eb 100644
--- a/kernel-headers/rdma/ib_user_ioctl_cmds.h
+++ b/kernel-headers/rdma/ib_user_ioctl_cmds.h
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2018, Mellanox Technologies inc.  All rights reserved.
+ * Copyright (c) 2020, Intel Corporation. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -251,6 +252,7 @@ enum uverbs_methods_mr {
 	UVERBS_METHOD_MR_DESTROY,
 	UVERBS_METHOD_ADVISE_MR,
 	UVERBS_METHOD_QUERY_MR,
+	UVERBS_METHOD_REG_DMABUF_MR,
 };
 
 enum uverbs_attrs_mr_destroy_ids {
@@ -272,6 +274,18 @@ enum uverbs_attrs_query_mr_cmd_attr_ids {
 	UVERBS_ATTR_QUERY_MR_RESP_IOVA,
 };
 
+enum uverbs_attrs_reg_dmabuf_mr_cmd_attr_ids {
+	UVERBS_ATTR_REG_DMABUF_MR_HANDLE,
+	UVERBS_ATTR_REG_DMABUF_MR_PD_HANDLE,
+	UVERBS_ATTR_REG_DMABUF_MR_OFFSET,
+	UVERBS_ATTR_REG_DMABUF_MR_LENGTH,
+	UVERBS_ATTR_REG_DMABUF_MR_IOVA,
+	UVERBS_ATTR_REG_DMABUF_MR_FD,
+	UVERBS_ATTR_REG_DMABUF_MR_ACCESS_FLAGS,
+	UVERBS_ATTR_REG_DMABUF_MR_RESP_LKEY,
+	UVERBS_ATTR_REG_DMABUF_MR_RESP_RKEY,
+};
+
 enum uverbs_attrs_create_counters_cmd_attr_ids {
 	UVERBS_ATTR_CREATE_COUNTERS_HANDLE,
 };
-- 
1.8.3.1

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

^ permalink raw reply related	[flat|nested] 42+ messages in thread
* [PATCH rdma-core v7 2/6] verbs: Support dma-buf based memory region
  2021-01-25 19:56 ` Jianxin Xiong
@ 2021-01-25 19:56   ` Jianxin Xiong
  -1 siblings, 0 replies; 42+ messages in thread
From: Jianxin Xiong @ 2021-01-25 19:56 UTC (permalink / raw)
  To: linux-rdma, dri-devel
  Cc: Jianxin Xiong, Doug Ledford, Jason Gunthorpe, Leon Romanovsky,
	Sumit Semwal, Christian Koenig, Daniel Vetter, Edward Srouji,
	Yishai Hadas

Add new API function and new provider method for registering dma-buf
based memory region. Update the man page and bump the API version.

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
 debian/libibverbs1.symbols   |  2 ++
 libibverbs/CMakeLists.txt    |  2 +-
 libibverbs/cmd_mr.c          | 38 ++++++++++++++++++++++++++++++++++++++
 libibverbs/driver.h          |  8 ++++++++
 libibverbs/dummy_ops.c       | 11 +++++++++++
 libibverbs/libibverbs.map.in |  6 ++++++
 libibverbs/man/ibv_reg_mr.3  | 27 +++++++++++++++++++++++++--
 libibverbs/verbs.c           | 19 +++++++++++++++++++
 libibverbs/verbs.h           |  7 +++++++
 9 files changed, 117 insertions(+), 3 deletions(-)

diff --git a/debian/libibverbs1.symbols b/debian/libibverbs1.symbols
index 7f59d0a..6cf5c2f 100644
--- a/debian/libibverbs1.symbols
+++ b/debian/libibverbs1.symbols
@@ -9,6 +9,7 @@ libibverbs.so.1 libibverbs1 #MINVER#
  IBVERBS_1.9@IBVERBS_1.9 30
  IBVERBS_1.10@IBVERBS_1.10 31
  IBVERBS_1.11@IBVERBS_1.11 32
+ IBVERBS_1.12@IBVERBS_1.12 34
  (symver)IBVERBS_PRIVATE_34 34
  _ibv_query_gid_ex@IBVERBS_1.11 32
  _ibv_query_gid_table@IBVERBS_1.11 32
@@ -99,6 +100,7 @@ libibverbs.so.1 libibverbs1 #MINVER#
  ibv_rate_to_mbps@IBVERBS_1.1 1.1.8
  ibv_rate_to_mult@IBVERBS_1.0 1.1.6
  ibv_read_sysfs_file@IBVERBS_1.0 1.1.6
+ ibv_reg_dmabuf_mr@IBVERBS_1.12 34
  ibv_reg_mr@IBVERBS_1.0 1.1.6
  ibv_reg_mr@IBVERBS_1.1 1.1.6
  ibv_reg_mr_iova@IBVERBS_1.7 25
diff --git a/libibverbs/CMakeLists.txt b/libibverbs/CMakeLists.txt
index 0fe4256..d075225 100644
--- a/libibverbs/CMakeLists.txt
+++ b/libibverbs/CMakeLists.txt
@@ -21,7 +21,7 @@ configure_file("libibverbs.map.in"
 
 rdma_library(ibverbs "${CMAKE_CURRENT_BINARY_DIR}/libibverbs.map"
   # See Documentation/versioning.md
-  1 1.11.${PACKAGE_VERSION}
+  1 1.12.${PACKAGE_VERSION}
   all_providers.c
   cmd.c
   cmd_ah.c
diff --git a/libibverbs/cmd_mr.c b/libibverbs/cmd_mr.c
index 42dbe42..af0fad7 100644
--- a/libibverbs/cmd_mr.c
+++ b/libibverbs/cmd_mr.c
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2018 Mellanox Technologies, Ltd.  All rights reserved.
+ * Copyright (c) 2020 Intel Corporation.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -116,3 +117,40 @@ int ibv_cmd_query_mr(struct ibv_pd *pd, struct verbs_mr *vmr,
 	return 0;
 }
 
+int ibv_cmd_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+			  uint64_t iova, int fd, int access,
+			  struct verbs_mr *vmr)
+{
+	DECLARE_COMMAND_BUFFER(cmdb, UVERBS_OBJECT_MR,
+			       UVERBS_METHOD_REG_DMABUF_MR,
+			       9);
+	struct ib_uverbs_attr *handle;
+	uint32_t lkey, rkey;
+	int ret;
+
+	handle = fill_attr_out_obj(cmdb, UVERBS_ATTR_REG_DMABUF_MR_HANDLE);
+	fill_attr_out_ptr(cmdb, UVERBS_ATTR_REG_DMABUF_MR_RESP_LKEY, &lkey);
+	fill_attr_out_ptr(cmdb, UVERBS_ATTR_REG_DMABUF_MR_RESP_RKEY, &rkey);
+
+	fill_attr_in_obj(cmdb, UVERBS_ATTR_REG_DMABUF_MR_PD_HANDLE, pd->handle);
+	fill_attr_in_uint64(cmdb, UVERBS_ATTR_REG_DMABUF_MR_OFFSET, offset);
+	fill_attr_in_uint64(cmdb, UVERBS_ATTR_REG_DMABUF_MR_LENGTH, length);
+	fill_attr_in_uint64(cmdb, UVERBS_ATTR_REG_DMABUF_MR_IOVA, iova);
+	fill_attr_in_uint32(cmdb, UVERBS_ATTR_REG_DMABUF_MR_FD, fd);
+	fill_attr_in_uint32(cmdb, UVERBS_ATTR_REG_DMABUF_MR_ACCESS_FLAGS, access);
+
+	ret = execute_ioctl(pd->context, cmdb);
+	if (ret)
+		return errno;
+
+	vmr->ibv_mr.handle = read_attr_obj(UVERBS_ATTR_REG_DMABUF_MR_HANDLE,
+					   handle);
+	vmr->ibv_mr.context = pd->context;
+	vmr->ibv_mr.lkey = lkey;
+	vmr->ibv_mr.rkey = rkey;
+	vmr->ibv_mr.pd = pd;
+	vmr->ibv_mr.addr = (void *)offset;
+	vmr->ibv_mr.length = length;
+	vmr->mr_type = IBV_MR_TYPE_DMABUF_MR;
+	return 0;
+}
diff --git a/libibverbs/driver.h b/libibverbs/driver.h
index 9bbd3b4..3965fa3 100644
--- a/libibverbs/driver.h
+++ b/libibverbs/driver.h
@@ -2,6 +2,7 @@
  * Copyright (c) 2004, 2005 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005, 2006 Cisco Systems, Inc.  All rights reserved.
  * Copyright (c) 2005 PathScale, Inc.  All rights reserved.
+ * Copyright (c) 2020 Intel Corporation. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -91,6 +92,7 @@ enum ibv_mr_type {
 	IBV_MR_TYPE_MR,
 	IBV_MR_TYPE_NULL_MR,
 	IBV_MR_TYPE_IMPORTED_MR,
+	IBV_MR_TYPE_DMABUF_MR,
 };
 
 struct verbs_mr {
@@ -375,6 +377,9 @@ struct verbs_context_ops {
 	struct ibv_mr *(*reg_dm_mr)(struct ibv_pd *pd, struct ibv_dm *dm,
 				    uint64_t dm_offset, size_t length,
 				    unsigned int access);
+	struct ibv_mr *(*reg_dmabuf_mr)(struct ibv_pd *pd, uint64_t offset,
+					size_t length, uint64_t iova,
+					int fd, int access);
 	struct ibv_mr *(*reg_mr)(struct ibv_pd *pd, void *addr, size_t length,
 				 uint64_t hca_va, int access);
 	int (*req_notify_cq)(struct ibv_cq *cq, int solicited_only);
@@ -493,6 +498,9 @@ int ibv_cmd_advise_mr(struct ibv_pd *pd,
 		      uint32_t flags,
 		      struct ibv_sge *sg_list,
 		      uint32_t num_sge);
+int ibv_cmd_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+			  uint64_t iova, int fd, int access,
+			  struct verbs_mr *vmr);
 int ibv_cmd_alloc_mw(struct ibv_pd *pd, enum ibv_mw_type type,
 		     struct ibv_mw *mw, struct ibv_alloc_mw *cmd,
 		     size_t cmd_size,
diff --git a/libibverbs/dummy_ops.c b/libibverbs/dummy_ops.c
index b6f272d..a4769ad 100644
--- a/libibverbs/dummy_ops.c
+++ b/libibverbs/dummy_ops.c
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2017 Mellanox Technologies, Inc.  All rights reserved.
+ * Copyright (c) 2020 Intel Corporation.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -435,6 +436,14 @@ static struct ibv_mr *reg_mr(struct ibv_pd *pd, void *addr, size_t length,
 	return NULL;
 }
 
+static struct ibv_mr *reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset,
+				    size_t length, uint64_t iova,
+				    int fd, int access)
+{
+	errno = EOPNOTSUPP;
+	return NULL;
+}
+
 static int req_notify_cq(struct ibv_cq *cq, int solicited_only)
 {
 	return EOPNOTSUPP;
@@ -542,6 +551,7 @@ const struct verbs_context_ops verbs_dummy_ops = {
 	query_srq,
 	read_counters,
 	reg_dm_mr,
+	reg_dmabuf_mr,
 	reg_mr,
 	req_notify_cq,
 	rereg_mr,
@@ -663,6 +673,7 @@ void verbs_set_ops(struct verbs_context *vctx,
 	SET_OP(vctx, read_counters);
 	SET_PRIV_OP(ctx, query_srq);
 	SET_OP(vctx, reg_dm_mr);
+	SET_PRIV_OP_IC(vctx, reg_dmabuf_mr);
 	SET_PRIV_OP(ctx, reg_mr);
 	SET_OP(ctx, req_notify_cq);
 	SET_PRIV_OP(ctx, rereg_mr);
diff --git a/libibverbs/libibverbs.map.in b/libibverbs/libibverbs.map.in
index d124004..745146d 100644
--- a/libibverbs/libibverbs.map.in
+++ b/libibverbs/libibverbs.map.in
@@ -148,6 +148,11 @@ IBVERBS_1.11 {
 		_ibv_query_gid_table;
 } IBVERBS_1.10;
 
+IBVERBS_1.12 {
+	global:
+		ibv_reg_dmabuf_mr;
+} IBVERBS_1.11;
+
 /* If any symbols in this stanza change ABI then the entire staza gets a new symbol
    version. See the top level CMakeLists.txt for this setting. */
 
@@ -211,6 +216,7 @@ IBVERBS_PRIVATE_@IBVERBS_PABI_VERSION@ {
 		ibv_cmd_query_srq;
 		ibv_cmd_read_counters;
 		ibv_cmd_reg_dm_mr;
+		ibv_cmd_reg_dmabuf_mr;
 		ibv_cmd_reg_mr;
 		ibv_cmd_req_notify_cq;
 		ibv_cmd_rereg_mr;
diff --git a/libibverbs/man/ibv_reg_mr.3 b/libibverbs/man/ibv_reg_mr.3
index 2bfc955..a522527 100644
--- a/libibverbs/man/ibv_reg_mr.3
+++ b/libibverbs/man/ibv_reg_mr.3
@@ -3,7 +3,7 @@
 .\"
 .TH IBV_REG_MR 3 2006-10-31 libibverbs "Libibverbs Programmer's Manual"
 .SH "NAME"
-ibv_reg_mr, ibv_reg_mr_iova, ibv_dereg_mr \- register or deregister a memory region (MR)
+ibv_reg_mr, ibv_reg_mr_iova, ibv_reg_dmabuf_mr, ibv_dereg_mr \- register or deregister a memory region (MR)
 .SH "SYNOPSIS"
 .nf
 .B #include <infiniband/verbs.h>
@@ -15,6 +15,10 @@ ibv_reg_mr, ibv_reg_mr_iova, ibv_dereg_mr \- register or deregister a memory reg
 .BI "                               size_t " "length" ", uint64_t " "hca_va" ,
 .BI "                               int " "access" );
 .sp
+.BI "struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd " "*pd" ", uint64_t " "offset" ,
+.BI "                                 size_t " "length" ", uint64_t " "iova" ,
+.BI "                                 int " "fd" ", int " "access" );
+.sp
 .BI "int ibv_dereg_mr(struct ibv_mr " "*mr" );
 .fi
 .SH "DESCRIPTION"
@@ -71,11 +75,30 @@ a lkey or rkey. The offset in the memory region is computed as 'addr +
 (iova - hca_va)'. Specifying 0 for hca_va has the same effect as
 IBV_ACCESS_ZERO_BASED.
 .PP
+.B ibv_reg_dmabuf_mr()
+registers a dma-buf based memory region (MR) associated with the protection domain
+.I pd\fR.
+The MR starts at
+.I offset
+of the dma-buf and its size is
+.I length\fR.
+The dma-buf is identified by the file descriptor
+.I fd\fR.
+The argument
+.I iova
+specifies the virtual base address of the MR when accessed through a lkey or rkey.
+It must have the same page offset as
+.I offset\fR.
+The argument
+.I access
+describes the desired memory protection attributes; it is similar to the ibv_reg_mr case except that only the following flags are supported:
+.B IBV_ACCESS_LOCAL_WRITE, IBV_ACCESS_REMOTE_WRITE, IBV_ACCESS_REMOTE_READ, IBV_ACCESS_REMOTE_ATOMIC, IBV_ACCESS_RELAXED_ORDERING.
+.PP
 .B ibv_dereg_mr()
 deregisters the MR
 .I mr\fR.
 .SH "RETURN VALUE"
-.B ibv_reg_mr() / ibv_reg_mr_iova()
+.B ibv_reg_mr() / ibv_reg_mr_iova() / ibv_reg_dmabuf_mr()
 returns a pointer to the registered MR, or NULL if the request fails.
 The local key (\fBL_Key\fR) field
 .B lkey
diff --git a/libibverbs/verbs.c b/libibverbs/verbs.c
index 4fb3893..b93046a 100644
--- a/libibverbs/verbs.c
+++ b/libibverbs/verbs.c
@@ -1,6 +1,7 @@
 /*
  * Copyright (c) 2005 Topspin Communications.  All rights reserved.
  * Copyright (c) 2006, 2007 Cisco Systems, Inc.  All rights reserved.
+ * Copyright (c) 2020 Intel Corperation.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -370,6 +371,24 @@ void ibv_unimport_mr(struct ibv_mr *mr)
 	get_ops(mr->context)->unimport_mr(mr);
 }
 
+struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset,
+				 size_t length, uint64_t iova, int fd,
+				 int access)
+{
+	struct ibv_mr *mr;
+
+	mr = get_ops(pd->context)->reg_dmabuf_mr(pd, offset, length, iova,
+						 fd, access);
+	if (!mr)
+		return NULL;
+
+	mr->context = pd->context;
+	mr->pd = pd;
+	mr->addr = (void *)offset;
+	mr->length = length;
+	return mr;
+}
+
 LATEST_SYMVER_FUNC(ibv_rereg_mr, 1_1, "IBVERBS_1.1",
 		   int,
 		   struct ibv_mr *mr, int flags,
diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h
index da782f2..656b0f9 100644
--- a/libibverbs/verbs.h
+++ b/libibverbs/verbs.h
@@ -3,6 +3,7 @@
  * Copyright (c) 2004, 2011-2012 Intel Corporation.  All rights reserved.
  * Copyright (c) 2005, 2006, 2007 Cisco Systems, Inc.  All rights reserved.
  * Copyright (c) 2005 PathScale, Inc.  All rights reserved.
+ * Copyright (c) 2020 Intel Corporation.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -2536,6 +2537,12 @@ __ibv_reg_mr_iova(struct ibv_pd *pd, void *addr, size_t length, uint64_t iova,
 			  __builtin_constant_p(                                \
 				  ((access) & IBV_ACCESS_OPTIONAL_RANGE) == 0))
 
+/**
+ * ibv_reg_dmabuf_mr - Register a dambuf-based memory region
+ */
+struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+				 uint64_t iova, int fd, int access);
+
 enum ibv_rereg_mr_err_code {
 	/* Old MR is valid, invalid input */
 	IBV_REREG_MR_ERR_INPUT = -1,
-- 
1.8.3.1


^ permalink raw reply related	[flat|nested] 42+ messages in thread
* [PATCH rdma-core v7 2/6] verbs: Support dma-buf based memory region
@ 2021-01-25 19:56   ` Jianxin Xiong
  0 siblings, 0 replies; 42+ messages in thread
From: Jianxin Xiong @ 2021-01-25 19:56 UTC (permalink / raw)
  To: linux-rdma, dri-devel
  Cc: Yishai Hadas, Leon Romanovsky, Edward Srouji, Jason Gunthorpe,
	Doug Ledford, Daniel Vetter, Christian Koenig, Jianxin Xiong

Add new API function and new provider method for registering dma-buf
based memory region. Update the man page and bump the API version.

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
 debian/libibverbs1.symbols   |  2 ++
 libibverbs/CMakeLists.txt    |  2 +-
 libibverbs/cmd_mr.c          | 38 ++++++++++++++++++++++++++++++++++++++
 libibverbs/driver.h          |  8 ++++++++
 libibverbs/dummy_ops.c       | 11 +++++++++++
 libibverbs/libibverbs.map.in |  6 ++++++
 libibverbs/man/ibv_reg_mr.3  | 27 +++++++++++++++++++++++++--
 libibverbs/verbs.c           | 19 +++++++++++++++++++
 libibverbs/verbs.h           |  7 +++++++
 9 files changed, 117 insertions(+), 3 deletions(-)

diff --git a/debian/libibverbs1.symbols b/debian/libibverbs1.symbols
index 7f59d0a..6cf5c2f 100644
--- a/debian/libibverbs1.symbols
+++ b/debian/libibverbs1.symbols
@@ -9,6 +9,7 @@ libibverbs.so.1 libibverbs1 #MINVER#
  IBVERBS_1.9@IBVERBS_1.9 30
  IBVERBS_1.10@IBVERBS_1.10 31
  IBVERBS_1.11@IBVERBS_1.11 32
+ IBVERBS_1.12@IBVERBS_1.12 34
  (symver)IBVERBS_PRIVATE_34 34
  _ibv_query_gid_ex@IBVERBS_1.11 32
  _ibv_query_gid_table@IBVERBS_1.11 32
@@ -99,6 +100,7 @@ libibverbs.so.1 libibverbs1 #MINVER#
  ibv_rate_to_mbps@IBVERBS_1.1 1.1.8
  ibv_rate_to_mult@IBVERBS_1.0 1.1.6
  ibv_read_sysfs_file@IBVERBS_1.0 1.1.6
+ ibv_reg_dmabuf_mr@IBVERBS_1.12 34
  ibv_reg_mr@IBVERBS_1.0 1.1.6
  ibv_reg_mr@IBVERBS_1.1 1.1.6
  ibv_reg_mr_iova@IBVERBS_1.7 25
diff --git a/libibverbs/CMakeLists.txt b/libibverbs/CMakeLists.txt
index 0fe4256..d075225 100644
--- a/libibverbs/CMakeLists.txt
+++ b/libibverbs/CMakeLists.txt
@@ -21,7 +21,7 @@ configure_file("libibverbs.map.in"
 
 rdma_library(ibverbs "${CMAKE_CURRENT_BINARY_DIR}/libibverbs.map"
   # See Documentation/versioning.md
-  1 1.11.${PACKAGE_VERSION}
+  1 1.12.${PACKAGE_VERSION}
   all_providers.c
   cmd.c
   cmd_ah.c
diff --git a/libibverbs/cmd_mr.c b/libibverbs/cmd_mr.c
index 42dbe42..af0fad7 100644
--- a/libibverbs/cmd_mr.c
+++ b/libibverbs/cmd_mr.c
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2018 Mellanox Technologies, Ltd.  All rights reserved.
+ * Copyright (c) 2020 Intel Corporation.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -116,3 +117,40 @@ int ibv_cmd_query_mr(struct ibv_pd *pd, struct verbs_mr *vmr,
 	return 0;
 }
 
+int ibv_cmd_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+			  uint64_t iova, int fd, int access,
+			  struct verbs_mr *vmr)
+{
+	DECLARE_COMMAND_BUFFER(cmdb, UVERBS_OBJECT_MR,
+			       UVERBS_METHOD_REG_DMABUF_MR,
+			       9);
+	struct ib_uverbs_attr *handle;
+	uint32_t lkey, rkey;
+	int ret;
+
+	handle = fill_attr_out_obj(cmdb, UVERBS_ATTR_REG_DMABUF_MR_HANDLE);
+	fill_attr_out_ptr(cmdb, UVERBS_ATTR_REG_DMABUF_MR_RESP_LKEY, &lkey);
+	fill_attr_out_ptr(cmdb, UVERBS_ATTR_REG_DMABUF_MR_RESP_RKEY, &rkey);
+
+	fill_attr_in_obj(cmdb, UVERBS_ATTR_REG_DMABUF_MR_PD_HANDLE, pd->handle);
+	fill_attr_in_uint64(cmdb, UVERBS_ATTR_REG_DMABUF_MR_OFFSET, offset);
+	fill_attr_in_uint64(cmdb, UVERBS_ATTR_REG_DMABUF_MR_LENGTH, length);
+	fill_attr_in_uint64(cmdb, UVERBS_ATTR_REG_DMABUF_MR_IOVA, iova);
+	fill_attr_in_uint32(cmdb, UVERBS_ATTR_REG_DMABUF_MR_FD, fd);
+	fill_attr_in_uint32(cmdb, UVERBS_ATTR_REG_DMABUF_MR_ACCESS_FLAGS, access);
+
+	ret = execute_ioctl(pd->context, cmdb);
+	if (ret)
+		return errno;
+
+	vmr->ibv_mr.handle = read_attr_obj(UVERBS_ATTR_REG_DMABUF_MR_HANDLE,
+					   handle);
+	vmr->ibv_mr.context = pd->context;
+	vmr->ibv_mr.lkey = lkey;
+	vmr->ibv_mr.rkey = rkey;
+	vmr->ibv_mr.pd = pd;
+	vmr->ibv_mr.addr = (void *)offset;
+	vmr->ibv_mr.length = length;
+	vmr->mr_type = IBV_MR_TYPE_DMABUF_MR;
+	return 0;
+}
diff --git a/libibverbs/driver.h b/libibverbs/driver.h
index 9bbd3b4..3965fa3 100644
--- a/libibverbs/driver.h
+++ b/libibverbs/driver.h
@@ -2,6 +2,7 @@
  * Copyright (c) 2004, 2005 Topspin Communications.  All rights reserved.
  * Copyright (c) 2005, 2006 Cisco Systems, Inc.  All rights reserved.
  * Copyright (c) 2005 PathScale, Inc.  All rights reserved.
+ * Copyright (c) 2020 Intel Corporation. All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -91,6 +92,7 @@ enum ibv_mr_type {
 	IBV_MR_TYPE_MR,
 	IBV_MR_TYPE_NULL_MR,
 	IBV_MR_TYPE_IMPORTED_MR,
+	IBV_MR_TYPE_DMABUF_MR,
 };
 
 struct verbs_mr {
@@ -375,6 +377,9 @@ struct verbs_context_ops {
 	struct ibv_mr *(*reg_dm_mr)(struct ibv_pd *pd, struct ibv_dm *dm,
 				    uint64_t dm_offset, size_t length,
 				    unsigned int access);
+	struct ibv_mr *(*reg_dmabuf_mr)(struct ibv_pd *pd, uint64_t offset,
+					size_t length, uint64_t iova,
+					int fd, int access);
 	struct ibv_mr *(*reg_mr)(struct ibv_pd *pd, void *addr, size_t length,
 				 uint64_t hca_va, int access);
 	int (*req_notify_cq)(struct ibv_cq *cq, int solicited_only);
@@ -493,6 +498,9 @@ int ibv_cmd_advise_mr(struct ibv_pd *pd,
 		      uint32_t flags,
 		      struct ibv_sge *sg_list,
 		      uint32_t num_sge);
+int ibv_cmd_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+			  uint64_t iova, int fd, int access,
+			  struct verbs_mr *vmr);
 int ibv_cmd_alloc_mw(struct ibv_pd *pd, enum ibv_mw_type type,
 		     struct ibv_mw *mw, struct ibv_alloc_mw *cmd,
 		     size_t cmd_size,
diff --git a/libibverbs/dummy_ops.c b/libibverbs/dummy_ops.c
index b6f272d..a4769ad 100644
--- a/libibverbs/dummy_ops.c
+++ b/libibverbs/dummy_ops.c
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2017 Mellanox Technologies, Inc.  All rights reserved.
+ * Copyright (c) 2020 Intel Corporation.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -435,6 +436,14 @@ static struct ibv_mr *reg_mr(struct ibv_pd *pd, void *addr, size_t length,
 	return NULL;
 }
 
+static struct ibv_mr *reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset,
+				    size_t length, uint64_t iova,
+				    int fd, int access)
+{
+	errno = EOPNOTSUPP;
+	return NULL;
+}
+
 static int req_notify_cq(struct ibv_cq *cq, int solicited_only)
 {
 	return EOPNOTSUPP;
@@ -542,6 +551,7 @@ const struct verbs_context_ops verbs_dummy_ops = {
 	query_srq,
 	read_counters,
 	reg_dm_mr,
+	reg_dmabuf_mr,
 	reg_mr,
 	req_notify_cq,
 	rereg_mr,
@@ -663,6 +673,7 @@ void verbs_set_ops(struct verbs_context *vctx,
 	SET_OP(vctx, read_counters);
 	SET_PRIV_OP(ctx, query_srq);
 	SET_OP(vctx, reg_dm_mr);
+	SET_PRIV_OP_IC(vctx, reg_dmabuf_mr);
 	SET_PRIV_OP(ctx, reg_mr);
 	SET_OP(ctx, req_notify_cq);
 	SET_PRIV_OP(ctx, rereg_mr);
diff --git a/libibverbs/libibverbs.map.in b/libibverbs/libibverbs.map.in
index d124004..745146d 100644
--- a/libibverbs/libibverbs.map.in
+++ b/libibverbs/libibverbs.map.in
@@ -148,6 +148,11 @@ IBVERBS_1.11 {
 		_ibv_query_gid_table;
 } IBVERBS_1.10;
 
+IBVERBS_1.12 {
+	global:
+		ibv_reg_dmabuf_mr;
+} IBVERBS_1.11;
+
 /* If any symbols in this stanza change ABI then the entire staza gets a new symbol
    version. See the top level CMakeLists.txt for this setting. */
 
@@ -211,6 +216,7 @@ IBVERBS_PRIVATE_@IBVERBS_PABI_VERSION@ {
 		ibv_cmd_query_srq;
 		ibv_cmd_read_counters;
 		ibv_cmd_reg_dm_mr;
+		ibv_cmd_reg_dmabuf_mr;
 		ibv_cmd_reg_mr;
 		ibv_cmd_req_notify_cq;
 		ibv_cmd_rereg_mr;
diff --git a/libibverbs/man/ibv_reg_mr.3 b/libibverbs/man/ibv_reg_mr.3
index 2bfc955..a522527 100644
--- a/libibverbs/man/ibv_reg_mr.3
+++ b/libibverbs/man/ibv_reg_mr.3
@@ -3,7 +3,7 @@
 .\"
 .TH IBV_REG_MR 3 2006-10-31 libibverbs "Libibverbs Programmer's Manual"
 .SH "NAME"
-ibv_reg_mr, ibv_reg_mr_iova, ibv_dereg_mr \- register or deregister a memory region (MR)
+ibv_reg_mr, ibv_reg_mr_iova, ibv_reg_dmabuf_mr, ibv_dereg_mr \- register or deregister a memory region (MR)
 .SH "SYNOPSIS"
 .nf
 .B #include <infiniband/verbs.h>
@@ -15,6 +15,10 @@ ibv_reg_mr, ibv_reg_mr_iova, ibv_dereg_mr \- register or deregister a memory reg
 .BI "                               size_t " "length" ", uint64_t " "hca_va" ,
 .BI "                               int " "access" );
 .sp
+.BI "struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd " "*pd" ", uint64_t " "offset" ,
+.BI "                                 size_t " "length" ", uint64_t " "iova" ,
+.BI "                                 int " "fd" ", int " "access" );
+.sp
 .BI "int ibv_dereg_mr(struct ibv_mr " "*mr" );
 .fi
 .SH "DESCRIPTION"
@@ -71,11 +75,30 @@ a lkey or rkey. The offset in the memory region is computed as 'addr +
 (iova - hca_va)'. Specifying 0 for hca_va has the same effect as
 IBV_ACCESS_ZERO_BASED.
 .PP
+.B ibv_reg_dmabuf_mr()
+registers a dma-buf based memory region (MR) associated with the protection domain
+.I pd\fR.
+The MR starts at
+.I offset
+of the dma-buf and its size is
+.I length\fR.
+The dma-buf is identified by the file descriptor
+.I fd\fR.
+The argument
+.I iova
+specifies the virtual base address of the MR when accessed through a lkey or rkey.
+It must have the same page offset as
+.I offset\fR.
+The argument
+.I access
+describes the desired memory protection attributes; it is similar to the ibv_reg_mr case except that only the following flags are supported:
+.B IBV_ACCESS_LOCAL_WRITE, IBV_ACCESS_REMOTE_WRITE, IBV_ACCESS_REMOTE_READ, IBV_ACCESS_REMOTE_ATOMIC, IBV_ACCESS_RELAXED_ORDERING.
+.PP
 .B ibv_dereg_mr()
 deregisters the MR
 .I mr\fR.
 .SH "RETURN VALUE"
-.B ibv_reg_mr() / ibv_reg_mr_iova()
+.B ibv_reg_mr() / ibv_reg_mr_iova() / ibv_reg_dmabuf_mr()
 returns a pointer to the registered MR, or NULL if the request fails.
 The local key (\fBL_Key\fR) field
 .B lkey
diff --git a/libibverbs/verbs.c b/libibverbs/verbs.c
index 4fb3893..b93046a 100644
--- a/libibverbs/verbs.c
+++ b/libibverbs/verbs.c
@@ -1,6 +1,7 @@
 /*
  * Copyright (c) 2005 Topspin Communications.  All rights reserved.
  * Copyright (c) 2006, 2007 Cisco Systems, Inc.  All rights reserved.
+ * Copyright (c) 2020 Intel Corperation.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -370,6 +371,24 @@ void ibv_unimport_mr(struct ibv_mr *mr)
 	get_ops(mr->context)->unimport_mr(mr);
 }
 
+struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset,
+				 size_t length, uint64_t iova, int fd,
+				 int access)
+{
+	struct ibv_mr *mr;
+
+	mr = get_ops(pd->context)->reg_dmabuf_mr(pd, offset, length, iova,
+						 fd, access);
+	if (!mr)
+		return NULL;
+
+	mr->context = pd->context;
+	mr->pd = pd;
+	mr->addr = (void *)offset;
+	mr->length = length;
+	return mr;
+}
+
 LATEST_SYMVER_FUNC(ibv_rereg_mr, 1_1, "IBVERBS_1.1",
 		   int,
 		   struct ibv_mr *mr, int flags,
diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h
index da782f2..656b0f9 100644
--- a/libibverbs/verbs.h
+++ b/libibverbs/verbs.h
@@ -3,6 +3,7 @@
  * Copyright (c) 2004, 2011-2012 Intel Corporation.  All rights reserved.
  * Copyright (c) 2005, 2006, 2007 Cisco Systems, Inc.  All rights reserved.
  * Copyright (c) 2005 PathScale, Inc.  All rights reserved.
+ * Copyright (c) 2020 Intel Corporation.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -2536,6 +2537,12 @@ __ibv_reg_mr_iova(struct ibv_pd *pd, void *addr, size_t length, uint64_t iova,
 			  __builtin_constant_p(                                \
 				  ((access) & IBV_ACCESS_OPTIONAL_RANGE) == 0))
 
+/**
+ * ibv_reg_dmabuf_mr - Register a dambuf-based memory region
+ */
+struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+				 uint64_t iova, int fd, int access);
+
 enum ibv_rereg_mr_err_code {
 	/* Old MR is valid, invalid input */
 	IBV_REREG_MR_ERR_INPUT = -1,
-- 
1.8.3.1

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

^ permalink raw reply related	[flat|nested] 42+ messages in thread
* [PATCH rdma-core v7 3/6] mlx5: Support dma-buf based memory region
  2021-01-25 19:56 ` Jianxin Xiong
@ 2021-01-25 19:56   ` Jianxin Xiong
  -1 siblings, 0 replies; 42+ messages in thread
From: Jianxin Xiong @ 2021-01-25 19:56 UTC (permalink / raw)
  To: linux-rdma, dri-devel
  Cc: Jianxin Xiong, Doug Ledford, Jason Gunthorpe, Leon Romanovsky,
	Sumit Semwal, Christian Koenig, Daniel Vetter, Edward Srouji,
	Yishai Hadas

Implement the new provider method for registering dma-buf based memory
regions.

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
 providers/mlx5/mlx5.c  |  2 ++
 providers/mlx5/mlx5.h  |  3 +++
 providers/mlx5/verbs.c | 22 ++++++++++++++++++++++
 3 files changed, 27 insertions(+)

diff --git a/providers/mlx5/mlx5.c b/providers/mlx5/mlx5.c
index a2a1696..b3c49af 100644
--- a/providers/mlx5/mlx5.c
+++ b/providers/mlx5/mlx5.c
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2012 Mellanox Technologies, Inc.  All rights reserved.
+ * Copyright (c) 2020 Intel Corporation.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -95,6 +96,7 @@ static const struct verbs_context_ops mlx5_ctx_common_ops = {
 	.async_event   = mlx5_async_event,
 	.dealloc_pd    = mlx5_free_pd,
 	.reg_mr	       = mlx5_reg_mr,
+	.reg_dmabuf_mr = mlx5_reg_dmabuf_mr,
 	.rereg_mr      = mlx5_rereg_mr,
 	.dereg_mr      = mlx5_dereg_mr,
 	.alloc_mw      = mlx5_alloc_mw,
diff --git a/providers/mlx5/mlx5.h b/providers/mlx5/mlx5.h
index bafe077..decc7f8 100644
--- a/providers/mlx5/mlx5.h
+++ b/providers/mlx5/mlx5.h
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2012 Mellanox Technologies, Inc.  All rights reserved.
+ * Copyright (c) 2020 Intel Corporation.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -941,6 +942,8 @@ void mlx5_async_event(struct ibv_context *context,
 struct ibv_mr *mlx5_alloc_null_mr(struct ibv_pd *pd);
 struct ibv_mr *mlx5_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
 			   uint64_t hca_va, int access);
+struct ibv_mr *mlx5_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+				  uint64_t iova, int fd, int access);
 int mlx5_rereg_mr(struct verbs_mr *mr, int flags, struct ibv_pd *pd, void *addr,
 		  size_t length, int access);
 int mlx5_dereg_mr(struct verbs_mr *mr);
diff --git a/providers/mlx5/verbs.c b/providers/mlx5/verbs.c
index 30c7b6e..7790b50 100644
--- a/providers/mlx5/verbs.c
+++ b/providers/mlx5/verbs.c
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2012 Mellanox Technologies, Inc.  All rights reserved.
+ * Copyright (c) 2020 Intel Corporation.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -628,6 +629,27 @@ struct ibv_mr *mlx5_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
 	return &mr->vmr.ibv_mr;
 }
 
+struct ibv_mr *mlx5_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+				  uint64_t iova, int fd, int acc)
+{
+	struct mlx5_mr *mr;
+	int ret;
+
+	mr = calloc(1, sizeof(*mr));
+	if (!mr)
+		return NULL;
+
+	ret = ibv_cmd_reg_dmabuf_mr(pd, offset, length, iova, fd, acc,
+				    &mr->vmr);
+	if (ret) {
+		free(mr);
+		return NULL;
+	}
+	mr->alloc_flags = acc;
+
+	return &mr->vmr.ibv_mr;
+}
+
 struct ibv_mr *mlx5_alloc_null_mr(struct ibv_pd *pd)
 {
 	struct mlx5_mr *mr;
-- 
1.8.3.1


^ permalink raw reply related	[flat|nested] 42+ messages in thread
* [PATCH rdma-core v7 3/6] mlx5: Support dma-buf based memory region
@ 2021-01-25 19:56   ` Jianxin Xiong
  0 siblings, 0 replies; 42+ messages in thread
From: Jianxin Xiong @ 2021-01-25 19:56 UTC (permalink / raw)
  To: linux-rdma, dri-devel
  Cc: Yishai Hadas, Leon Romanovsky, Edward Srouji, Jason Gunthorpe,
	Doug Ledford, Daniel Vetter, Christian Koenig, Jianxin Xiong

Implement the new provider method for registering dma-buf based memory
regions.

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
 providers/mlx5/mlx5.c  |  2 ++
 providers/mlx5/mlx5.h  |  3 +++
 providers/mlx5/verbs.c | 22 ++++++++++++++++++++++
 3 files changed, 27 insertions(+)

diff --git a/providers/mlx5/mlx5.c b/providers/mlx5/mlx5.c
index a2a1696..b3c49af 100644
--- a/providers/mlx5/mlx5.c
+++ b/providers/mlx5/mlx5.c
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2012 Mellanox Technologies, Inc.  All rights reserved.
+ * Copyright (c) 2020 Intel Corporation.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -95,6 +96,7 @@ static const struct verbs_context_ops mlx5_ctx_common_ops = {
 	.async_event   = mlx5_async_event,
 	.dealloc_pd    = mlx5_free_pd,
 	.reg_mr	       = mlx5_reg_mr,
+	.reg_dmabuf_mr = mlx5_reg_dmabuf_mr,
 	.rereg_mr      = mlx5_rereg_mr,
 	.dereg_mr      = mlx5_dereg_mr,
 	.alloc_mw      = mlx5_alloc_mw,
diff --git a/providers/mlx5/mlx5.h b/providers/mlx5/mlx5.h
index bafe077..decc7f8 100644
--- a/providers/mlx5/mlx5.h
+++ b/providers/mlx5/mlx5.h
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2012 Mellanox Technologies, Inc.  All rights reserved.
+ * Copyright (c) 2020 Intel Corporation.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -941,6 +942,8 @@ void mlx5_async_event(struct ibv_context *context,
 struct ibv_mr *mlx5_alloc_null_mr(struct ibv_pd *pd);
 struct ibv_mr *mlx5_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
 			   uint64_t hca_va, int access);
+struct ibv_mr *mlx5_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+				  uint64_t iova, int fd, int access);
 int mlx5_rereg_mr(struct verbs_mr *mr, int flags, struct ibv_pd *pd, void *addr,
 		  size_t length, int access);
 int mlx5_dereg_mr(struct verbs_mr *mr);
diff --git a/providers/mlx5/verbs.c b/providers/mlx5/verbs.c
index 30c7b6e..7790b50 100644
--- a/providers/mlx5/verbs.c
+++ b/providers/mlx5/verbs.c
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2012 Mellanox Technologies, Inc.  All rights reserved.
+ * Copyright (c) 2020 Intel Corporation.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -628,6 +629,27 @@ struct ibv_mr *mlx5_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
 	return &mr->vmr.ibv_mr;
 }
 
+struct ibv_mr *mlx5_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+				  uint64_t iova, int fd, int acc)
+{
+	struct mlx5_mr *mr;
+	int ret;
+
+	mr = calloc(1, sizeof(*mr));
+	if (!mr)
+		return NULL;
+
+	ret = ibv_cmd_reg_dmabuf_mr(pd, offset, length, iova, fd, acc,
+				    &mr->vmr);
+	if (ret) {
+		free(mr);
+		return NULL;
+	}
+	mr->alloc_flags = acc;
+
+	return &mr->vmr.ibv_mr;
+}
+
 struct ibv_mr *mlx5_alloc_null_mr(struct ibv_pd *pd)
 {
 	struct mlx5_mr *mr;
-- 
1.8.3.1

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

^ permalink raw reply related	[flat|nested] 42+ messages in thread
* [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
  2021-01-25 19:56 ` Jianxin Xiong
@ 2021-01-25 19:57   ` Jianxin Xiong
  -1 siblings, 0 replies; 42+ messages in thread
From: Jianxin Xiong @ 2021-01-25 19:57 UTC (permalink / raw)
  To: linux-rdma, dri-devel
  Cc: Jianxin Xiong, Doug Ledford, Jason Gunthorpe, Leon Romanovsky,
	Sumit Semwal, Christian Koenig, Daniel Vetter, Edward Srouji,
	Yishai Hadas

Define a new sub-class of 'MR' that uses dma-buf object for the memory
region. Define a new class 'DmaBuf' as a wrapper for dma-buf allocation
mechanism implemented in C.

Update the cmake function for cython modules to allow building modules
with mixed cython and c source files.

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
 buildlib/pyverbs_functions.cmake |  78 +++++++----
 pyverbs/CMakeLists.txt           |  11 +-
 pyverbs/dmabuf.pxd               |  15 +++
 pyverbs/dmabuf.pyx               |  73 ++++++++++
 pyverbs/dmabuf_alloc.c           | 278 +++++++++++++++++++++++++++++++++++++++
 pyverbs/dmabuf_alloc.h           |  19 +++
 pyverbs/libibverbs.pxd           |   2 +
 pyverbs/mr.pxd                   |   6 +
 pyverbs/mr.pyx                   | 105 ++++++++++++++-
 9 files changed, 557 insertions(+), 30 deletions(-)
 create mode 100644 pyverbs/dmabuf.pxd
 create mode 100644 pyverbs/dmabuf.pyx
 create mode 100644 pyverbs/dmabuf_alloc.c
 create mode 100644 pyverbs/dmabuf_alloc.h

diff --git a/buildlib/pyverbs_functions.cmake b/buildlib/pyverbs_functions.cmake
index 953cec2..0792410 100644
--- a/buildlib/pyverbs_functions.cmake
+++ b/buildlib/pyverbs_functions.cmake
@@ -1,35 +1,61 @@
 # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
 # Copyright (c) 2018, Mellanox Technologies. All rights reserved.  See COPYING file
+# Copyright (c) 2020, Intel Corporation. All rights reserved.  See COPYING file
+
+function(build_module_from_cfiles PY_MODULE MODULE_NAME ALL_CFILES LINKER_FLAGS)
+  string(REGEX REPLACE "\\.so$" "" SONAME "${MODULE_NAME}${CMAKE_PYTHON_SO_SUFFIX}")
+  add_library(${SONAME} SHARED ${ALL_CFILES})
+  set_target_properties(${SONAME} PROPERTIES
+    COMPILE_FLAGS "${CMAKE_C_FLAGS} -fPIC -fno-strict-aliasing -Wno-unused-function -Wno-redundant-decls -Wno-shadow -Wno-cast-function-type -Wno-implicit-fallthrough -Wno-unknown-warning -Wno-unknown-warning-option -Wno-deprecated-declarations ${NO_VAR_TRACKING_FLAGS}"
+    LIBRARY_OUTPUT_DIRECTORY "${BUILD_PYTHON}/${PY_MODULE}"
+    PREFIX "")
+  target_link_libraries(${SONAME} LINK_PRIVATE ${PYTHON_LIBRARIES} ibverbs rdmacm ${LINKER_FLAGS})
+  install(TARGETS ${SONAME}
+    DESTINATION ${CMAKE_INSTALL_PYTHON_ARCH_LIB}/${PY_MODULE})
+endfunction()
 
 function(rdma_cython_module PY_MODULE LINKER_FLAGS)
-  foreach(PYX_FILE ${ARGN})
-    get_filename_component(FILENAME ${PYX_FILE} NAME_WE)
-    get_filename_component(DIR ${PYX_FILE} DIRECTORY)
-	if (DIR)
-		set(PYX "${CMAKE_CURRENT_SOURCE_DIR}/${DIR}/${FILENAME}.pyx")
-	else()
-	    set(PYX "${CMAKE_CURRENT_SOURCE_DIR}/${FILENAME}.pyx")
-	endif()
-    set(CFILE "${CMAKE_CURRENT_BINARY_DIR}/${FILENAME}.c")
-    include_directories(${PYTHON_INCLUDE_DIRS})
-    add_custom_command(
-      OUTPUT "${CFILE}"
-      MAIN_DEPENDENCY "${PYX}"
-      COMMAND ${CYTHON_EXECUTABLE} "${PYX}" -o "${CFILE}"
-      "-I${PYTHON_INCLUDE_DIRS}"
-      COMMENT "Cythonizing ${PYX}"
+  set(ALL_CFILES "")
+  set(MODULE_NAME "")
+  foreach(SRC_FILE ${ARGN})
+    get_filename_component(FILENAME ${SRC_FILE} NAME_WE)
+    get_filename_component(DIR ${SRC_FILE} DIRECTORY)
+    get_filename_component(EXT ${SRC_FILE} EXT)
+    if (DIR)
+      set(SRC_PATH "${CMAKE_CURRENT_SOURCE_DIR}/${DIR}")
+    else()
+      set(SRC_PATH "${CMAKE_CURRENT_SOURCE_DIR}")
+    endif()
+    if (${EXT} STREQUAL ".pyx")
+      # each .pyx file starts a new module, finish the previous module first
+      if (ALL_CFILES AND MODULE_NAME)
+        build_module_from_cfiles(${PY_MODULE} ${MODULE_NAME} "${ALL_CFILES}" "${LINKER_FLAGS}")
+      endif()
+      set(PYX "${SRC_PATH}/${FILENAME}.pyx")
+      set(CFILE "${CMAKE_CURRENT_BINARY_DIR}/${FILENAME}.c")
+      include_directories(${PYTHON_INCLUDE_DIRS})
+      add_custom_command(
+        OUTPUT "${CFILE}"
+        MAIN_DEPENDENCY "${PYX}"
+        COMMAND ${CYTHON_EXECUTABLE} "${PYX}" -o "${CFILE}"
+        "-I${PYTHON_INCLUDE_DIRS}"
+        COMMENT "Cythonizing ${PYX}"
       )
-
-    string(REGEX REPLACE "\\.so$" "" SONAME "${FILENAME}${CMAKE_PYTHON_SO_SUFFIX}")
-    add_library(${SONAME} SHARED ${CFILE})
-    set_target_properties(${SONAME} PROPERTIES
-      COMPILE_FLAGS "${CMAKE_C_FLAGS} -fPIC -fno-strict-aliasing -Wno-unused-function -Wno-redundant-decls -Wno-shadow -Wno-cast-function-type -Wno-implicit-fallthrough -Wno-unknown-warning -Wno-unknown-warning-option -Wno-deprecated-declarations ${NO_VAR_TRACKING_FLAGS}"
-      LIBRARY_OUTPUT_DIRECTORY "${BUILD_PYTHON}/${PY_MODULE}"
-      PREFIX "")
-    target_link_libraries(${SONAME} LINK_PRIVATE ${PYTHON_LIBRARIES} ibverbs rdmacm ${LINKER_FLAGS})
-    install(TARGETS ${SONAME}
-      DESTINATION ${CMAKE_INSTALL_PYTHON_ARCH_LIB}/${PY_MODULE})
+      set(MODULE_NAME ${FILENAME})
+      set(ALL_CFILES "${CFILE}")
+    elseif(${EXT} STREQUAL ".c")
+      # .c files belong to the same module as the most recent .pyx file,
+      # ignored if appearing before all .pyx files
+      set(CFILE "${SRC_PATH}/${FILENAME}.c")
+      set(ALL_CFILES "${ALL_CFILES};${CFILE}")
+    else()
+      continue()
+    endif()
   endforeach()
+  # finish the last module
+  if (ALL_CFILES AND MODULE_NAME)
+    build_module_from_cfiles(${PY_MODULE} ${MODULE_NAME} "${ALL_CFILES}" "${LINKER_FLAGS}")
+  endif()
 endfunction()
 
 function(rdma_python_module PY_MODULE)
diff --git a/pyverbs/CMakeLists.txt b/pyverbs/CMakeLists.txt
index 9542c4b..6fd7625 100644
--- a/pyverbs/CMakeLists.txt
+++ b/pyverbs/CMakeLists.txt
@@ -1,5 +1,10 @@
 # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
 # Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file
+# Copyright (c) 2020, Intel Corporation. All rights reserved. See COPYING file
+
+publish_internal_headers(""
+  dmabuf_alloc.h
+)
 
 rdma_cython_module(pyverbs ""
   addr.pyx
@@ -8,15 +13,17 @@ rdma_cython_module(pyverbs ""
   cmid.pyx
   cq.pyx
   device.pyx
+  dmabuf.pyx
+  dmabuf_alloc.c
   enums.pyx
   mem_alloc.pyx
   mr.pyx
   pd.pyx
   qp.pyx
+  srq.pyx
   wr.pyx
   xrcd.pyx
-  srq.pyx
-  )
+)
 
 rdma_python_module(pyverbs
   __init__.py
diff --git a/pyverbs/dmabuf.pxd b/pyverbs/dmabuf.pxd
new file mode 100644
index 0000000..a063acb
--- /dev/null
+++ b/pyverbs/dmabuf.pxd
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
+# Copyright (c) 2020, Intel Corporation. All rights reserved. See COPYING file
+
+#cython: language_level=3
+
+cdef class DmaBuf:
+    cdef int drm_fd
+    cdef int handle
+    cdef int fd
+    cdef unsigned long size
+    cdef unsigned long map_offset
+    cdef void *dmabuf
+    cdef object dmabuf_mrs
+    cdef add_ref(self, obj)
+    cpdef close(self)
diff --git a/pyverbs/dmabuf.pyx b/pyverbs/dmabuf.pyx
new file mode 100644
index 0000000..b9406bd
--- /dev/null
+++ b/pyverbs/dmabuf.pyx
@@ -0,0 +1,73 @@
+# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
+# Copyright (c) 2020, Intel Corporation. All rights reserved. See COPYING file
+
+#cython: language_level=3
+
+import weakref
+
+from pyverbs.base cimport close_weakrefs
+from pyverbs.base import PyverbsRDMAErrno
+from pyverbs.mr cimport DmaBufMR
+
+cdef extern from "dmabuf_alloc.h":
+    cdef struct dmabuf:
+        pass
+    dmabuf *dmabuf_alloc(unsigned long size, int unit, int gtt)
+    void dmabuf_free(dmabuf *dmabuf)
+    int dmabuf_get_drm_fd(dmabuf *dmabuf)
+    int dmabuf_get_fd(dmabuf *dmabuf)
+    unsigned long dmabuf_get_offset(dmabuf *dmabuf)
+
+
+cdef class DmaBuf:
+    def __init__(self, size, unit=0, gtt=0):
+        """
+        Allocate DmaBuf object from a GPU device. This is done through the
+        DRI device interface. Usually this requires the effective user id
+        being a member of the 'render' group.
+        :param size: The size (in number of bytes) of the buffer.
+        :param unit: The unit number of the GPU to allocate the buffer from.
+        :param gtt: Allocate from GTT instead of VRAM.
+        :return: The newly created DmaBuf object on success.
+        """
+        self.dmabuf_mrs = weakref.WeakSet()
+        self.dmabuf = dmabuf_alloc(size, unit, gtt)
+        if self.dmabuf == NULL:
+            raise PyverbsRDMAErrno(f'Failed to allocate dmabuf of size {size} on unit {unit}')
+        self.drm_fd = dmabuf_get_drm_fd(<dmabuf *>self.dmabuf)
+        self.fd = dmabuf_get_fd(<dmabuf *>self.dmabuf)
+        self.map_offset = dmabuf_get_offset(<dmabuf *>self.dmabuf)
+
+    def __dealloc__(self):
+        self.close()
+
+    cpdef close(self):
+        if self.dmabuf == NULL:
+            return None
+        close_weakrefs([self.dmabuf_mrs])
+        dmabuf_free(<dmabuf *>self.dmabuf)
+        self.dmabuf = NULL
+
+    cdef add_ref(self, obj):
+        if isinstance(obj, DmaBufMR):
+            self.dmabuf_mrs.add(obj)
+
+    @property
+    def drm_fd(self):
+        return self.drm_fd
+
+    @property
+    def handle(self):
+        return self.handle
+
+    @property
+    def fd(self):
+        return self.fd
+
+    @property
+    def size(self):
+        return self.size
+
+    @property
+    def map_offset(self):
+        return self.map_offset
diff --git a/pyverbs/dmabuf_alloc.c b/pyverbs/dmabuf_alloc.c
new file mode 100644
index 0000000..05eae75
--- /dev/null
+++ b/pyverbs/dmabuf_alloc.c
@@ -0,0 +1,278 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/*
+ * Copyright 2020 Intel Corporation. All rights reserved. See COPYING file
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <unistd.h>
+#include <string.h>
+#include <errno.h>
+#include <drm/drm.h>
+#include <drm/i915_drm.h>
+#include <drm/amdgpu_drm.h>
+#include <drm/radeon_drm.h>
+#include <fcntl.h>
+#include <sys/ioctl.h>
+#include "dmabuf_alloc.h"
+
+/*
+ * Abstraction of the buffer allocation mechanism using the DRM interface.
+ * The interface is accessed by ioctl() calls over the '/dev/dri/renderD*'
+ * device. Successful access usually requires the effective user id being
+ * in the 'render' group.
+ */
+
+struct drm {
+	int fd;
+	int (*alloc)(struct drm *drm, uint64_t size, uint32_t *handle, int gtt);
+	int (*mmap_offset)(struct drm *drm, uint32_t handle, uint64_t *offset);
+};
+
+static int i915_alloc(struct drm *drm, uint64_t size, uint32_t *handle, int gtt)
+{
+	struct drm_i915_gem_create gem_create = {};
+	int err;
+
+	gem_create.size = size;
+	err = ioctl(drm->fd, DRM_IOCTL_I915_GEM_CREATE, &gem_create);
+	if (err)
+		return err;
+
+	*handle = gem_create.handle;
+	return 0;
+}
+
+static int amdgpu_alloc(struct drm *drm, size_t size, uint32_t *handle, int gtt)
+{
+	union drm_amdgpu_gem_create gem_create = {{}};
+	int err;
+
+	gem_create.in.bo_size = size;
+	if (gtt) {
+		gem_create.in.domains = AMDGPU_GEM_DOMAIN_GTT;
+		gem_create.in.domain_flags = AMDGPU_GEM_CREATE_CPU_GTT_USWC;
+	} else {
+		gem_create.in.domains = AMDGPU_GEM_DOMAIN_VRAM;
+		gem_create.in.domain_flags =
+			AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED;
+	}
+	err = ioctl(drm->fd, DRM_IOCTL_AMDGPU_GEM_CREATE, &gem_create);
+	if (err)
+		return err;
+
+	*handle = gem_create.out.handle;
+	return 0;
+}
+
+static int i915_mmap_offset(struct drm *drm, uint32_t handle, uint64_t *offset)
+{
+	struct drm_i915_gem_mmap_gtt gem_mmap = {};
+	int err;
+
+	gem_mmap.handle = handle;
+	err = ioctl(drm->fd, DRM_IOCTL_I915_GEM_MMAP_GTT, &gem_mmap);
+	if (err)
+		return err;
+
+	*offset = gem_mmap.offset;
+	return 0;
+}
+
+static int amdgpu_mmap_offset(struct drm *drm, uint32_t handle,
+			      uint64_t *offset)
+{
+	union drm_amdgpu_gem_mmap gem_mmap = {{}};
+	int err;
+
+	gem_mmap.in.handle = handle;
+	err = ioctl(drm->fd, DRM_IOCTL_AMDGPU_GEM_MMAP, &gem_mmap);
+	if (err)
+		return err;
+
+	*offset = gem_mmap.out.addr_ptr;
+	return 0;
+}
+
+static struct drm *drm_open(int unit)
+{
+	char path[32];
+	struct drm_version version = {};
+	char name[16] = {};
+	int err;
+	struct drm *drm;
+
+	drm = malloc(sizeof(*drm));
+	if (!drm)
+		return NULL;
+
+	snprintf(path, sizeof(path), "/dev/dri/renderD%d", unit + 128);
+
+	drm->fd = open(path, O_RDWR);
+	if (drm->fd < 0)
+		goto out_free;
+
+	version.name = name;
+	version.name_len = 16;
+	err = ioctl(drm->fd, DRM_IOCTL_VERSION, &version);
+	if (err)
+		goto out_close;
+
+	if (!strcmp(name, "amdgpu")) {
+		drm->alloc = amdgpu_alloc;
+		drm->mmap_offset = amdgpu_mmap_offset;
+	} else if (!strcmp(name, "i915")) {
+		drm->alloc = i915_alloc;
+		drm->mmap_offset = i915_mmap_offset;
+	} else {
+		errno = EOPNOTSUPP;
+		goto out_close;
+	}
+	return drm;
+
+out_close:
+	close(drm->fd);
+
+out_free:
+	free(drm);
+	return NULL;
+}
+
+static void drm_close(struct drm *drm)
+{
+	if (!drm || drm->fd < 0)
+		return;
+
+	close(drm->fd);
+	free(drm);
+}
+
+static void drm_free_buf(struct drm *drm, uint32_t handle)
+{
+	struct drm_gem_close close = {};
+
+	close.handle = handle;
+	ioctl(drm->fd, DRM_IOCTL_GEM_CLOSE, &close);
+}
+
+static int drm_alloc_buf(struct drm *drm, size_t size, uint32_t *handle,
+			 int *fd, int gtt)
+{
+	struct drm_prime_handle prime_handle = {};
+	int err;
+
+	if (!drm || drm->fd < 0)
+		return -EINVAL;
+
+	err = drm->alloc(drm, size, handle, gtt);
+	if (err)
+		return err;
+
+	prime_handle.handle = *handle;
+	prime_handle.flags = O_RDWR;
+	err = ioctl(drm->fd, DRM_IOCTL_PRIME_HANDLE_TO_FD, &prime_handle);
+	if (err) {
+		drm_free_buf(drm, *handle);
+		return err;
+	}
+
+	*fd = prime_handle.fd;
+	return 0;
+}
+
+static int drm_map_buf(struct drm *drm, uint32_t handle, uint64_t *offset)
+{
+	if (!drm || drm->fd < 0)
+		return -EINVAL;
+
+	return drm->mmap_offset(drm, handle, offset);
+}
+
+/*
+ * Abstraction of dmabuf object, allocated using the DRI abstraction defined
+ * above.
+ */
+
+struct dmabuf {
+	struct drm *drm;
+	int fd;
+	uint32_t handle;
+	uint64_t map_offset;
+};
+
+/*
+ * dmabuf_alloc - allocate a dmabuf from GPU
+ * @size - byte size of the buffer to allocate
+ * @unit - the GPU unit to use
+ * @gtt - if true, allocate from GTT instead of VRAM
+ */
+struct dmabuf *dmabuf_alloc(uint64_t size, int unit, int gtt)
+{
+	struct dmabuf *dmabuf;
+	int err;
+
+	dmabuf = malloc(sizeof(*dmabuf));
+	if (!dmabuf)
+		return NULL;
+
+	dmabuf->drm = drm_open(unit);
+	if (!dmabuf->drm)
+		goto out_free;
+
+	err = drm_alloc_buf(dmabuf->drm, size, &dmabuf->handle, &dmabuf->fd, gtt);
+	if (err)
+		goto out_close;
+
+	err = drm_map_buf(dmabuf->drm, dmabuf->handle, &dmabuf->map_offset);
+	if (err)
+		goto out_free_buf;
+
+	return dmabuf;
+
+out_free_buf:
+	drm_free_buf(dmabuf->drm, dmabuf->handle);
+
+out_close:
+	drm_close(dmabuf->drm);
+
+out_free:
+	free(dmabuf);
+	return NULL;
+}
+
+void dmabuf_free(struct dmabuf *dmabuf)
+{
+	if (!dmabuf)
+		return;
+
+	close(dmabuf->fd);
+	drm_free_buf(dmabuf->drm, dmabuf->handle);
+	drm_close(dmabuf->drm);
+	free(dmabuf);
+}
+
+int dmabuf_get_drm_fd(struct dmabuf *dmabuf)
+{
+	if (!dmabuf || !dmabuf->drm)
+		return -1;
+
+	return dmabuf->drm->fd;
+}
+
+int dmabuf_get_fd(struct dmabuf *dmabuf)
+{
+	if (!dmabuf)
+		return -1;
+
+	return dmabuf->fd;
+}
+
+uint64_t dmabuf_get_offset(struct dmabuf *dmabuf)
+{
+	if (!dmabuf)
+		return -1;
+
+	return dmabuf->map_offset;
+}
+
diff --git a/pyverbs/dmabuf_alloc.h b/pyverbs/dmabuf_alloc.h
new file mode 100644
index 0000000..f1b03c5
--- /dev/null
+++ b/pyverbs/dmabuf_alloc.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/*
+ * Copyright 2020 Intel Corporation. All rights reserved. See COPYING file
+ */
+
+#ifndef _DMABUF_ALLOC_H_
+#define _DMABUF_ALLOC_H_
+
+#include <stdint.h>
+
+struct dmabuf;
+
+struct dmabuf *dmabuf_alloc(uint64_t size, int unit, int gtt);
+void dmabuf_free(struct dmabuf *dmabuf);
+int dmabuf_get_drm_fd(struct dmabuf *dmabuf);
+int dmabuf_get_fd(struct dmabuf *dmabuf);
+uint64_t dmabuf_get_offset(struct dmabuf *dmabuf);
+
+#endif /* _DMABUF_ALLOC_H_ */
diff --git a/pyverbs/libibverbs.pxd b/pyverbs/libibverbs.pxd
index 5005650..d960d31 100644
--- a/pyverbs/libibverbs.pxd
+++ b/pyverbs/libibverbs.pxd
@@ -507,6 +507,8 @@ cdef extern from 'infiniband/verbs.h':
     ibv_pd *ibv_alloc_pd(ibv_context *context)
     int ibv_dealloc_pd(ibv_pd *pd)
     ibv_mr *ibv_reg_mr(ibv_pd *pd, void *addr, size_t length, int access)
+    ibv_mr *ibv_reg_dmabuf_mr(ibv_pd *pd, uint64_t offset, size_t length,
+                              uint64_t iova, int fd, int access)
     int ibv_rereg_mr(ibv_mr *mr, int flags, ibv_pd *pd, void *addr,
                      size_t length, int access)
     int ibv_dereg_mr(ibv_mr *mr)
diff --git a/pyverbs/mr.pxd b/pyverbs/mr.pxd
index ebe8ada..d9a79ff 100644
--- a/pyverbs/mr.pxd
+++ b/pyverbs/mr.pxd
@@ -1,5 +1,6 @@
 # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
 # Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file
+# Copyright (c) 2020, Intel Corporation. All rights reserved. See COPYING file
 
 #cython: language_level=3
 
@@ -33,3 +34,8 @@ cdef class MW(PyverbsCM):
 
 cdef class DMMR(MR):
     cdef object dm
+
+cdef class DmaBufMR(MR):
+    cdef object dmabuf
+    cdef unsigned long offset
+    cdef object is_dmabuf_internal
diff --git a/pyverbs/mr.pyx b/pyverbs/mr.pyx
index b0f106d..aad47e2 100644
--- a/pyverbs/mr.pyx
+++ b/pyverbs/mr.pyx
@@ -1,11 +1,12 @@
 # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
 # Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file
+# Copyright (c) 2020, Intel Corporation. All rights reserved. See COPYING file
 
 import resource
 import logging
 
 from posix.mman cimport mmap, munmap, MAP_PRIVATE, PROT_READ, PROT_WRITE, \
-    MAP_ANONYMOUS, MAP_HUGETLB
+    MAP_ANONYMOUS, MAP_HUGETLB, MAP_SHARED
 from pyverbs.pyverbs_error import PyverbsError, PyverbsRDMAError, \
     PyverbsUserError
 from libc.stdint cimport uintptr_t, SIZE_MAX
@@ -15,9 +16,10 @@ from posix.stdlib cimport posix_memalign
 from libc.string cimport memcpy, memset
 cimport pyverbs.libibverbs_enums as e
 from pyverbs.device cimport DM
-from libc.stdlib cimport free
+from libc.stdlib cimport free, malloc
 from .cmid cimport CMID
 from .pd cimport PD
+from .dmabuf cimport DmaBuf
 
 cdef extern from 'sys/mman.h':
     cdef void* MAP_FAILED
@@ -380,6 +382,105 @@ cdef class DMMR(MR):
     cpdef read(self, length, offset):
         return self.dm.copy_from_dm(offset, length)
 
+cdef class DmaBufMR(MR):
+    def __init__(self, PD pd not None, length, access, DmaBuf dmabuf=None,
+                 offset=0, unit=0, gtt=0):
+        """
+        Initializes a DmaBufMR (DMA-BUF Memory Region) of the given length
+        and access flags using the given PD and DmaBuf objects.
+        :param pd: A PD object
+        :param length: Length in bytes
+        :param access: Access flags, see ibv_access_flags enum
+        :param dmabuf: A DmaBuf object. One will be allocated if absent
+        :param offset: Byte offset from the beginning of the dma-buf
+        :param unit: GPU unit for internal dmabuf allocation
+        :param gtt: If true allocate internal dmabuf from GTT instead of VRAM
+        :return: The newly created DMABUFMR
+        """
+        self.logger = logging.getLogger(self.__class__.__name__)
+        if dmabuf is None:
+            self.is_dmabuf_internal = True
+            dmabuf = DmaBuf(length + offset, unit, gtt)
+        self.mr = v.ibv_reg_dmabuf_mr(pd.pd, offset, length, offset, dmabuf.fd, access)
+        if self.mr == NULL:
+            raise PyverbsRDMAErrno(f'Failed to register a dma-buf MR. length: {length}, access flags: {access}')
+        super().__init__(pd, length, access)
+        self.pd = pd
+        self.dmabuf = dmabuf
+        self.offset = offset
+        pd.add_ref(self)
+        dmabuf.add_ref(self)
+        self.logger.debug(f'Registered dma-buf ibv_mr. Length: {length}, access flags {access}')
+
+    def __dealloc__(self):
+        self.close()
+
+    cpdef close(self):
+        """
+        Closes the underlying C object of the MR and frees the memory allocated.
+        :return: None
+        """
+        if self.mr != NULL:
+            self.logger.debug('Closing dma-buf MR')
+            rc = v.ibv_dereg_mr(self.mr)
+            if rc != 0:
+                raise PyverbsRDMAError('Failed to dereg dma-buf MR', rc)
+            self.pd = None
+            self.mr = NULL
+            # Set self.mr to NULL before closing dmabuf because this method is
+            # re-entered when close_weakrefs() is called inside dmabuf.close().
+            if self.is_dmabuf_internal:
+                self.dmabuf.close()
+            self.dmabuf = None
+
+    @property
+    def offset(self):
+        return self.offset
+
+    @property
+    def dmabuf(self):
+        return self.dmabuf
+
+    def write(self, data, length, offset=0):
+        """
+        Write user data to the dma-buf backing the MR
+        :param data: User data to write
+        :param length: Length of the data to write
+        :param offset: Writing offset
+        :return: None
+        """
+        if isinstance(data, str):
+            data = data.encode()
+        cdef int off = offset + self.offset
+        cdef void *buf = mmap(NULL, length + off, PROT_READ | PROT_WRITE,
+                              MAP_SHARED, self.dmabuf.drm_fd,
+                              self.dmabuf.map_offset)
+        if buf == MAP_FAILED:
+            raise PyverbsError(f'Failed to map dma-buf of size {length}')
+        memcpy(<char*>(buf + off), <char *>data, length)
+        munmap(buf, length + off)
+
+    cpdef read(self, length, offset):
+        """
+        Reads data from the dma-buf backing the MR
+        :param length: Length of data to read
+        :param offset: Reading offset
+        :return: The data on the buffer in the requested offset
+        """
+        cdef int off = offset + self.offset
+        cdef void *buf = mmap(NULL, length + off, PROT_READ | PROT_WRITE,
+                              MAP_SHARED, self.dmabuf.drm_fd,
+                              self.dmabuf.map_offset)
+        if buf == MAP_FAILED:
+            raise PyverbsError(f'Failed to map dma-buf of size {length}')
+        cdef char *data =<char*>malloc(length)
+        memset(data, 0, length)
+        memcpy(data, <char*>(buf + off), length)
+        munmap(buf, length + off)
+        res = data[:length]
+        free(data)
+        return res
+
 
 def mwtype2str(mw_type):
     mw_types = {1:'IBV_MW_TYPE_1', 2:'IBV_MW_TYPE_2'}
-- 
1.8.3.1


^ permalink raw reply related	[flat|nested] 42+ messages in thread
* [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
@ 2021-01-25 19:57   ` Jianxin Xiong
  0 siblings, 0 replies; 42+ messages in thread
From: Jianxin Xiong @ 2021-01-25 19:57 UTC (permalink / raw)
  To: linux-rdma, dri-devel
  Cc: Yishai Hadas, Leon Romanovsky, Edward Srouji, Jason Gunthorpe,
	Doug Ledford, Daniel Vetter, Christian Koenig, Jianxin Xiong

Define a new sub-class of 'MR' that uses dma-buf object for the memory
region. Define a new class 'DmaBuf' as a wrapper for dma-buf allocation
mechanism implemented in C.

Update the cmake function for cython modules to allow building modules
with mixed cython and c source files.

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
 buildlib/pyverbs_functions.cmake |  78 +++++++----
 pyverbs/CMakeLists.txt           |  11 +-
 pyverbs/dmabuf.pxd               |  15 +++
 pyverbs/dmabuf.pyx               |  73 ++++++++++
 pyverbs/dmabuf_alloc.c           | 278 +++++++++++++++++++++++++++++++++++++++
 pyverbs/dmabuf_alloc.h           |  19 +++
 pyverbs/libibverbs.pxd           |   2 +
 pyverbs/mr.pxd                   |   6 +
 pyverbs/mr.pyx                   | 105 ++++++++++++++-
 9 files changed, 557 insertions(+), 30 deletions(-)
 create mode 100644 pyverbs/dmabuf.pxd
 create mode 100644 pyverbs/dmabuf.pyx
 create mode 100644 pyverbs/dmabuf_alloc.c
 create mode 100644 pyverbs/dmabuf_alloc.h

diff --git a/buildlib/pyverbs_functions.cmake b/buildlib/pyverbs_functions.cmake
index 953cec2..0792410 100644
--- a/buildlib/pyverbs_functions.cmake
+++ b/buildlib/pyverbs_functions.cmake
@@ -1,35 +1,61 @@
 # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
 # Copyright (c) 2018, Mellanox Technologies. All rights reserved.  See COPYING file
+# Copyright (c) 2020, Intel Corporation. All rights reserved.  See COPYING file
+
+function(build_module_from_cfiles PY_MODULE MODULE_NAME ALL_CFILES LINKER_FLAGS)
+  string(REGEX REPLACE "\\.so$" "" SONAME "${MODULE_NAME}${CMAKE_PYTHON_SO_SUFFIX}")
+  add_library(${SONAME} SHARED ${ALL_CFILES})
+  set_target_properties(${SONAME} PROPERTIES
+    COMPILE_FLAGS "${CMAKE_C_FLAGS} -fPIC -fno-strict-aliasing -Wno-unused-function -Wno-redundant-decls -Wno-shadow -Wno-cast-function-type -Wno-implicit-fallthrough -Wno-unknown-warning -Wno-unknown-warning-option -Wno-deprecated-declarations ${NO_VAR_TRACKING_FLAGS}"
+    LIBRARY_OUTPUT_DIRECTORY "${BUILD_PYTHON}/${PY_MODULE}"
+    PREFIX "")
+  target_link_libraries(${SONAME} LINK_PRIVATE ${PYTHON_LIBRARIES} ibverbs rdmacm ${LINKER_FLAGS})
+  install(TARGETS ${SONAME}
+    DESTINATION ${CMAKE_INSTALL_PYTHON_ARCH_LIB}/${PY_MODULE})
+endfunction()
 
 function(rdma_cython_module PY_MODULE LINKER_FLAGS)
-  foreach(PYX_FILE ${ARGN})
-    get_filename_component(FILENAME ${PYX_FILE} NAME_WE)
-    get_filename_component(DIR ${PYX_FILE} DIRECTORY)
-	if (DIR)
-		set(PYX "${CMAKE_CURRENT_SOURCE_DIR}/${DIR}/${FILENAME}.pyx")
-	else()
-	    set(PYX "${CMAKE_CURRENT_SOURCE_DIR}/${FILENAME}.pyx")
-	endif()
-    set(CFILE "${CMAKE_CURRENT_BINARY_DIR}/${FILENAME}.c")
-    include_directories(${PYTHON_INCLUDE_DIRS})
-    add_custom_command(
-      OUTPUT "${CFILE}"
-      MAIN_DEPENDENCY "${PYX}"
-      COMMAND ${CYTHON_EXECUTABLE} "${PYX}" -o "${CFILE}"
-      "-I${PYTHON_INCLUDE_DIRS}"
-      COMMENT "Cythonizing ${PYX}"
+  set(ALL_CFILES "")
+  set(MODULE_NAME "")
+  foreach(SRC_FILE ${ARGN})
+    get_filename_component(FILENAME ${SRC_FILE} NAME_WE)
+    get_filename_component(DIR ${SRC_FILE} DIRECTORY)
+    get_filename_component(EXT ${SRC_FILE} EXT)
+    if (DIR)
+      set(SRC_PATH "${CMAKE_CURRENT_SOURCE_DIR}/${DIR}")
+    else()
+      set(SRC_PATH "${CMAKE_CURRENT_SOURCE_DIR}")
+    endif()
+    if (${EXT} STREQUAL ".pyx")
+      # each .pyx file starts a new module, finish the previous module first
+      if (ALL_CFILES AND MODULE_NAME)
+        build_module_from_cfiles(${PY_MODULE} ${MODULE_NAME} "${ALL_CFILES}" "${LINKER_FLAGS}")
+      endif()
+      set(PYX "${SRC_PATH}/${FILENAME}.pyx")
+      set(CFILE "${CMAKE_CURRENT_BINARY_DIR}/${FILENAME}.c")
+      include_directories(${PYTHON_INCLUDE_DIRS})
+      add_custom_command(
+        OUTPUT "${CFILE}"
+        MAIN_DEPENDENCY "${PYX}"
+        COMMAND ${CYTHON_EXECUTABLE} "${PYX}" -o "${CFILE}"
+        "-I${PYTHON_INCLUDE_DIRS}"
+        COMMENT "Cythonizing ${PYX}"
       )
-
-    string(REGEX REPLACE "\\.so$" "" SONAME "${FILENAME}${CMAKE_PYTHON_SO_SUFFIX}")
-    add_library(${SONAME} SHARED ${CFILE})
-    set_target_properties(${SONAME} PROPERTIES
-      COMPILE_FLAGS "${CMAKE_C_FLAGS} -fPIC -fno-strict-aliasing -Wno-unused-function -Wno-redundant-decls -Wno-shadow -Wno-cast-function-type -Wno-implicit-fallthrough -Wno-unknown-warning -Wno-unknown-warning-option -Wno-deprecated-declarations ${NO_VAR_TRACKING_FLAGS}"
-      LIBRARY_OUTPUT_DIRECTORY "${BUILD_PYTHON}/${PY_MODULE}"
-      PREFIX "")
-    target_link_libraries(${SONAME} LINK_PRIVATE ${PYTHON_LIBRARIES} ibverbs rdmacm ${LINKER_FLAGS})
-    install(TARGETS ${SONAME}
-      DESTINATION ${CMAKE_INSTALL_PYTHON_ARCH_LIB}/${PY_MODULE})
+      set(MODULE_NAME ${FILENAME})
+      set(ALL_CFILES "${CFILE}")
+    elseif(${EXT} STREQUAL ".c")
+      # .c files belong to the same module as the most recent .pyx file,
+      # ignored if appearing before all .pyx files
+      set(CFILE "${SRC_PATH}/${FILENAME}.c")
+      set(ALL_CFILES "${ALL_CFILES};${CFILE}")
+    else()
+      continue()
+    endif()
   endforeach()
+  # finish the last module
+  if (ALL_CFILES AND MODULE_NAME)
+    build_module_from_cfiles(${PY_MODULE} ${MODULE_NAME} "${ALL_CFILES}" "${LINKER_FLAGS}")
+  endif()
 endfunction()
 
 function(rdma_python_module PY_MODULE)
diff --git a/pyverbs/CMakeLists.txt b/pyverbs/CMakeLists.txt
index 9542c4b..6fd7625 100644
--- a/pyverbs/CMakeLists.txt
+++ b/pyverbs/CMakeLists.txt
@@ -1,5 +1,10 @@
 # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
 # Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file
+# Copyright (c) 2020, Intel Corporation. All rights reserved. See COPYING file
+
+publish_internal_headers(""
+  dmabuf_alloc.h
+)
 
 rdma_cython_module(pyverbs ""
   addr.pyx
@@ -8,15 +13,17 @@ rdma_cython_module(pyverbs ""
   cmid.pyx
   cq.pyx
   device.pyx
+  dmabuf.pyx
+  dmabuf_alloc.c
   enums.pyx
   mem_alloc.pyx
   mr.pyx
   pd.pyx
   qp.pyx
+  srq.pyx
   wr.pyx
   xrcd.pyx
-  srq.pyx
-  )
+)
 
 rdma_python_module(pyverbs
   __init__.py
diff --git a/pyverbs/dmabuf.pxd b/pyverbs/dmabuf.pxd
new file mode 100644
index 0000000..a063acb
--- /dev/null
+++ b/pyverbs/dmabuf.pxd
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
+# Copyright (c) 2020, Intel Corporation. All rights reserved. See COPYING file
+
+#cython: language_level=3
+
+cdef class DmaBuf:
+    cdef int drm_fd
+    cdef int handle
+    cdef int fd
+    cdef unsigned long size
+    cdef unsigned long map_offset
+    cdef void *dmabuf
+    cdef object dmabuf_mrs
+    cdef add_ref(self, obj)
+    cpdef close(self)
diff --git a/pyverbs/dmabuf.pyx b/pyverbs/dmabuf.pyx
new file mode 100644
index 0000000..b9406bd
--- /dev/null
+++ b/pyverbs/dmabuf.pyx
@@ -0,0 +1,73 @@
+# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
+# Copyright (c) 2020, Intel Corporation. All rights reserved. See COPYING file
+
+#cython: language_level=3
+
+import weakref
+
+from pyverbs.base cimport close_weakrefs
+from pyverbs.base import PyverbsRDMAErrno
+from pyverbs.mr cimport DmaBufMR
+
+cdef extern from "dmabuf_alloc.h":
+    cdef struct dmabuf:
+        pass
+    dmabuf *dmabuf_alloc(unsigned long size, int unit, int gtt)
+    void dmabuf_free(dmabuf *dmabuf)
+    int dmabuf_get_drm_fd(dmabuf *dmabuf)
+    int dmabuf_get_fd(dmabuf *dmabuf)
+    unsigned long dmabuf_get_offset(dmabuf *dmabuf)
+
+
+cdef class DmaBuf:
+    def __init__(self, size, unit=0, gtt=0):
+        """
+        Allocate DmaBuf object from a GPU device. This is done through the
+        DRI device interface. Usually this requires the effective user id
+        being a member of the 'render' group.
+        :param size: The size (in number of bytes) of the buffer.
+        :param unit: The unit number of the GPU to allocate the buffer from.
+        :param gtt: Allocate from GTT instead of VRAM.
+        :return: The newly created DmaBuf object on success.
+        """
+        self.dmabuf_mrs = weakref.WeakSet()
+        self.dmabuf = dmabuf_alloc(size, unit, gtt)
+        if self.dmabuf == NULL:
+            raise PyverbsRDMAErrno(f'Failed to allocate dmabuf of size {size} on unit {unit}')
+        self.drm_fd = dmabuf_get_drm_fd(<dmabuf *>self.dmabuf)
+        self.fd = dmabuf_get_fd(<dmabuf *>self.dmabuf)
+        self.map_offset = dmabuf_get_offset(<dmabuf *>self.dmabuf)
+
+    def __dealloc__(self):
+        self.close()
+
+    cpdef close(self):
+        if self.dmabuf == NULL:
+            return None
+        close_weakrefs([self.dmabuf_mrs])
+        dmabuf_free(<dmabuf *>self.dmabuf)
+        self.dmabuf = NULL
+
+    cdef add_ref(self, obj):
+        if isinstance(obj, DmaBufMR):
+            self.dmabuf_mrs.add(obj)
+
+    @property
+    def drm_fd(self):
+        return self.drm_fd
+
+    @property
+    def handle(self):
+        return self.handle
+
+    @property
+    def fd(self):
+        return self.fd
+
+    @property
+    def size(self):
+        return self.size
+
+    @property
+    def map_offset(self):
+        return self.map_offset
diff --git a/pyverbs/dmabuf_alloc.c b/pyverbs/dmabuf_alloc.c
new file mode 100644
index 0000000..05eae75
--- /dev/null
+++ b/pyverbs/dmabuf_alloc.c
@@ -0,0 +1,278 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/*
+ * Copyright 2020 Intel Corporation. All rights reserved. See COPYING file
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <unistd.h>
+#include <string.h>
+#include <errno.h>
+#include <drm/drm.h>
+#include <drm/i915_drm.h>
+#include <drm/amdgpu_drm.h>
+#include <drm/radeon_drm.h>
+#include <fcntl.h>
+#include <sys/ioctl.h>
+#include "dmabuf_alloc.h"
+
+/*
+ * Abstraction of the buffer allocation mechanism using the DRM interface.
+ * The interface is accessed by ioctl() calls over the '/dev/dri/renderD*'
+ * device. Successful access usually requires the effective user id being
+ * in the 'render' group.
+ */
+
+struct drm {
+	int fd;
+	int (*alloc)(struct drm *drm, uint64_t size, uint32_t *handle, int gtt);
+	int (*mmap_offset)(struct drm *drm, uint32_t handle, uint64_t *offset);
+};
+
+static int i915_alloc(struct drm *drm, uint64_t size, uint32_t *handle, int gtt)
+{
+	struct drm_i915_gem_create gem_create = {};
+	int err;
+
+	gem_create.size = size;
+	err = ioctl(drm->fd, DRM_IOCTL_I915_GEM_CREATE, &gem_create);
+	if (err)
+		return err;
+
+	*handle = gem_create.handle;
+	return 0;
+}
+
+static int amdgpu_alloc(struct drm *drm, size_t size, uint32_t *handle, int gtt)
+{
+	union drm_amdgpu_gem_create gem_create = {{}};
+	int err;
+
+	gem_create.in.bo_size = size;
+	if (gtt) {
+		gem_create.in.domains = AMDGPU_GEM_DOMAIN_GTT;
+		gem_create.in.domain_flags = AMDGPU_GEM_CREATE_CPU_GTT_USWC;
+	} else {
+		gem_create.in.domains = AMDGPU_GEM_DOMAIN_VRAM;
+		gem_create.in.domain_flags =
+			AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED;
+	}
+	err = ioctl(drm->fd, DRM_IOCTL_AMDGPU_GEM_CREATE, &gem_create);
+	if (err)
+		return err;
+
+	*handle = gem_create.out.handle;
+	return 0;
+}
+
+static int i915_mmap_offset(struct drm *drm, uint32_t handle, uint64_t *offset)
+{
+	struct drm_i915_gem_mmap_gtt gem_mmap = {};
+	int err;
+
+	gem_mmap.handle = handle;
+	err = ioctl(drm->fd, DRM_IOCTL_I915_GEM_MMAP_GTT, &gem_mmap);
+	if (err)
+		return err;
+
+	*offset = gem_mmap.offset;
+	return 0;
+}
+
+static int amdgpu_mmap_offset(struct drm *drm, uint32_t handle,
+			      uint64_t *offset)
+{
+	union drm_amdgpu_gem_mmap gem_mmap = {{}};
+	int err;
+
+	gem_mmap.in.handle = handle;
+	err = ioctl(drm->fd, DRM_IOCTL_AMDGPU_GEM_MMAP, &gem_mmap);
+	if (err)
+		return err;
+
+	*offset = gem_mmap.out.addr_ptr;
+	return 0;
+}
+
+static struct drm *drm_open(int unit)
+{
+	char path[32];
+	struct drm_version version = {};
+	char name[16] = {};
+	int err;
+	struct drm *drm;
+
+	drm = malloc(sizeof(*drm));
+	if (!drm)
+		return NULL;
+
+	snprintf(path, sizeof(path), "/dev/dri/renderD%d", unit + 128);
+
+	drm->fd = open(path, O_RDWR);
+	if (drm->fd < 0)
+		goto out_free;
+
+	version.name = name;
+	version.name_len = 16;
+	err = ioctl(drm->fd, DRM_IOCTL_VERSION, &version);
+	if (err)
+		goto out_close;
+
+	if (!strcmp(name, "amdgpu")) {
+		drm->alloc = amdgpu_alloc;
+		drm->mmap_offset = amdgpu_mmap_offset;
+	} else if (!strcmp(name, "i915")) {
+		drm->alloc = i915_alloc;
+		drm->mmap_offset = i915_mmap_offset;
+	} else {
+		errno = EOPNOTSUPP;
+		goto out_close;
+	}
+	return drm;
+
+out_close:
+	close(drm->fd);
+
+out_free:
+	free(drm);
+	return NULL;
+}
+
+static void drm_close(struct drm *drm)
+{
+	if (!drm || drm->fd < 0)
+		return;
+
+	close(drm->fd);
+	free(drm);
+}
+
+static void drm_free_buf(struct drm *drm, uint32_t handle)
+{
+	struct drm_gem_close close = {};
+
+	close.handle = handle;
+	ioctl(drm->fd, DRM_IOCTL_GEM_CLOSE, &close);
+}
+
+static int drm_alloc_buf(struct drm *drm, size_t size, uint32_t *handle,
+			 int *fd, int gtt)
+{
+	struct drm_prime_handle prime_handle = {};
+	int err;
+
+	if (!drm || drm->fd < 0)
+		return -EINVAL;
+
+	err = drm->alloc(drm, size, handle, gtt);
+	if (err)
+		return err;
+
+	prime_handle.handle = *handle;
+	prime_handle.flags = O_RDWR;
+	err = ioctl(drm->fd, DRM_IOCTL_PRIME_HANDLE_TO_FD, &prime_handle);
+	if (err) {
+		drm_free_buf(drm, *handle);
+		return err;
+	}
+
+	*fd = prime_handle.fd;
+	return 0;
+}
+
+static int drm_map_buf(struct drm *drm, uint32_t handle, uint64_t *offset)
+{
+	if (!drm || drm->fd < 0)
+		return -EINVAL;
+
+	return drm->mmap_offset(drm, handle, offset);
+}
+
+/*
+ * Abstraction of dmabuf object, allocated using the DRI abstraction defined
+ * above.
+ */
+
+struct dmabuf {
+	struct drm *drm;
+	int fd;
+	uint32_t handle;
+	uint64_t map_offset;
+};
+
+/*
+ * dmabuf_alloc - allocate a dmabuf from GPU
+ * @size - byte size of the buffer to allocate
+ * @unit - the GPU unit to use
+ * @gtt - if true, allocate from GTT instead of VRAM
+ */
+struct dmabuf *dmabuf_alloc(uint64_t size, int unit, int gtt)
+{
+	struct dmabuf *dmabuf;
+	int err;
+
+	dmabuf = malloc(sizeof(*dmabuf));
+	if (!dmabuf)
+		return NULL;
+
+	dmabuf->drm = drm_open(unit);
+	if (!dmabuf->drm)
+		goto out_free;
+
+	err = drm_alloc_buf(dmabuf->drm, size, &dmabuf->handle, &dmabuf->fd, gtt);
+	if (err)
+		goto out_close;
+
+	err = drm_map_buf(dmabuf->drm, dmabuf->handle, &dmabuf->map_offset);
+	if (err)
+		goto out_free_buf;
+
+	return dmabuf;
+
+out_free_buf:
+	drm_free_buf(dmabuf->drm, dmabuf->handle);
+
+out_close:
+	drm_close(dmabuf->drm);
+
+out_free:
+	free(dmabuf);
+	return NULL;
+}
+
+void dmabuf_free(struct dmabuf *dmabuf)
+{
+	if (!dmabuf)
+		return;
+
+	close(dmabuf->fd);
+	drm_free_buf(dmabuf->drm, dmabuf->handle);
+	drm_close(dmabuf->drm);
+	free(dmabuf);
+}
+
+int dmabuf_get_drm_fd(struct dmabuf *dmabuf)
+{
+	if (!dmabuf || !dmabuf->drm)
+		return -1;
+
+	return dmabuf->drm->fd;
+}
+
+int dmabuf_get_fd(struct dmabuf *dmabuf)
+{
+	if (!dmabuf)
+		return -1;
+
+	return dmabuf->fd;
+}
+
+uint64_t dmabuf_get_offset(struct dmabuf *dmabuf)
+{
+	if (!dmabuf)
+		return -1;
+
+	return dmabuf->map_offset;
+}
+
diff --git a/pyverbs/dmabuf_alloc.h b/pyverbs/dmabuf_alloc.h
new file mode 100644
index 0000000..f1b03c5
--- /dev/null
+++ b/pyverbs/dmabuf_alloc.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/*
+ * Copyright 2020 Intel Corporation. All rights reserved. See COPYING file
+ */
+
+#ifndef _DMABUF_ALLOC_H_
+#define _DMABUF_ALLOC_H_
+
+#include <stdint.h>
+
+struct dmabuf;
+
+struct dmabuf *dmabuf_alloc(uint64_t size, int unit, int gtt);
+void dmabuf_free(struct dmabuf *dmabuf);
+int dmabuf_get_drm_fd(struct dmabuf *dmabuf);
+int dmabuf_get_fd(struct dmabuf *dmabuf);
+uint64_t dmabuf_get_offset(struct dmabuf *dmabuf);
+
+#endif /* _DMABUF_ALLOC_H_ */
diff --git a/pyverbs/libibverbs.pxd b/pyverbs/libibverbs.pxd
index 5005650..d960d31 100644
--- a/pyverbs/libibverbs.pxd
+++ b/pyverbs/libibverbs.pxd
@@ -507,6 +507,8 @@ cdef extern from 'infiniband/verbs.h':
     ibv_pd *ibv_alloc_pd(ibv_context *context)
     int ibv_dealloc_pd(ibv_pd *pd)
     ibv_mr *ibv_reg_mr(ibv_pd *pd, void *addr, size_t length, int access)
+    ibv_mr *ibv_reg_dmabuf_mr(ibv_pd *pd, uint64_t offset, size_t length,
+                              uint64_t iova, int fd, int access)
     int ibv_rereg_mr(ibv_mr *mr, int flags, ibv_pd *pd, void *addr,
                      size_t length, int access)
     int ibv_dereg_mr(ibv_mr *mr)
diff --git a/pyverbs/mr.pxd b/pyverbs/mr.pxd
index ebe8ada..d9a79ff 100644
--- a/pyverbs/mr.pxd
+++ b/pyverbs/mr.pxd
@@ -1,5 +1,6 @@
 # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
 # Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file
+# Copyright (c) 2020, Intel Corporation. All rights reserved. See COPYING file
 
 #cython: language_level=3
 
@@ -33,3 +34,8 @@ cdef class MW(PyverbsCM):
 
 cdef class DMMR(MR):
     cdef object dm
+
+cdef class DmaBufMR(MR):
+    cdef object dmabuf
+    cdef unsigned long offset
+    cdef object is_dmabuf_internal
diff --git a/pyverbs/mr.pyx b/pyverbs/mr.pyx
index b0f106d..aad47e2 100644
--- a/pyverbs/mr.pyx
+++ b/pyverbs/mr.pyx
@@ -1,11 +1,12 @@
 # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
 # Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file
+# Copyright (c) 2020, Intel Corporation. All rights reserved. See COPYING file
 
 import resource
 import logging
 
 from posix.mman cimport mmap, munmap, MAP_PRIVATE, PROT_READ, PROT_WRITE, \
-    MAP_ANONYMOUS, MAP_HUGETLB
+    MAP_ANONYMOUS, MAP_HUGETLB, MAP_SHARED
 from pyverbs.pyverbs_error import PyverbsError, PyverbsRDMAError, \
     PyverbsUserError
 from libc.stdint cimport uintptr_t, SIZE_MAX
@@ -15,9 +16,10 @@ from posix.stdlib cimport posix_memalign
 from libc.string cimport memcpy, memset
 cimport pyverbs.libibverbs_enums as e
 from pyverbs.device cimport DM
-from libc.stdlib cimport free
+from libc.stdlib cimport free, malloc
 from .cmid cimport CMID
 from .pd cimport PD
+from .dmabuf cimport DmaBuf
 
 cdef extern from 'sys/mman.h':
     cdef void* MAP_FAILED
@@ -380,6 +382,105 @@ cdef class DMMR(MR):
     cpdef read(self, length, offset):
         return self.dm.copy_from_dm(offset, length)
 
+cdef class DmaBufMR(MR):
+    def __init__(self, PD pd not None, length, access, DmaBuf dmabuf=None,
+                 offset=0, unit=0, gtt=0):
+        """
+        Initializes a DmaBufMR (DMA-BUF Memory Region) of the given length
+        and access flags using the given PD and DmaBuf objects.
+        :param pd: A PD object
+        :param length: Length in bytes
+        :param access: Access flags, see ibv_access_flags enum
+        :param dmabuf: A DmaBuf object. One will be allocated if absent
+        :param offset: Byte offset from the beginning of the dma-buf
+        :param unit: GPU unit for internal dmabuf allocation
+        :param gtt: If true allocate internal dmabuf from GTT instead of VRAM
+        :return: The newly created DMABUFMR
+        """
+        self.logger = logging.getLogger(self.__class__.__name__)
+        if dmabuf is None:
+            self.is_dmabuf_internal = True
+            dmabuf = DmaBuf(length + offset, unit, gtt)
+        self.mr = v.ibv_reg_dmabuf_mr(pd.pd, offset, length, offset, dmabuf.fd, access)
+        if self.mr == NULL:
+            raise PyverbsRDMAErrno(f'Failed to register a dma-buf MR. length: {length}, access flags: {access}')
+        super().__init__(pd, length, access)
+        self.pd = pd
+        self.dmabuf = dmabuf
+        self.offset = offset
+        pd.add_ref(self)
+        dmabuf.add_ref(self)
+        self.logger.debug(f'Registered dma-buf ibv_mr. Length: {length}, access flags {access}')
+
+    def __dealloc__(self):
+        self.close()
+
+    cpdef close(self):
+        """
+        Closes the underlying C object of the MR and frees the memory allocated.
+        :return: None
+        """
+        if self.mr != NULL:
+            self.logger.debug('Closing dma-buf MR')
+            rc = v.ibv_dereg_mr(self.mr)
+            if rc != 0:
+                raise PyverbsRDMAError('Failed to dereg dma-buf MR', rc)
+            self.pd = None
+            self.mr = NULL
+            # Set self.mr to NULL before closing dmabuf because this method is
+            # re-entered when close_weakrefs() is called inside dmabuf.close().
+            if self.is_dmabuf_internal:
+                self.dmabuf.close()
+            self.dmabuf = None
+
+    @property
+    def offset(self):
+        return self.offset
+
+    @property
+    def dmabuf(self):
+        return self.dmabuf
+
+    def write(self, data, length, offset=0):
+        """
+        Write user data to the dma-buf backing the MR
+        :param data: User data to write
+        :param length: Length of the data to write
+        :param offset: Writing offset
+        :return: None
+        """
+        if isinstance(data, str):
+            data = data.encode()
+        cdef int off = offset + self.offset
+        cdef void *buf = mmap(NULL, length + off, PROT_READ | PROT_WRITE,
+                              MAP_SHARED, self.dmabuf.drm_fd,
+                              self.dmabuf.map_offset)
+        if buf == MAP_FAILED:
+            raise PyverbsError(f'Failed to map dma-buf of size {length}')
+        memcpy(<char*>(buf + off), <char *>data, length)
+        munmap(buf, length + off)
+
+    cpdef read(self, length, offset):
+        """
+        Reads data from the dma-buf backing the MR
+        :param length: Length of data to read
+        :param offset: Reading offset
+        :return: The data on the buffer in the requested offset
+        """
+        cdef int off = offset + self.offset
+        cdef void *buf = mmap(NULL, length + off, PROT_READ | PROT_WRITE,
+                              MAP_SHARED, self.dmabuf.drm_fd,
+                              self.dmabuf.map_offset)
+        if buf == MAP_FAILED:
+            raise PyverbsError(f'Failed to map dma-buf of size {length}')
+        cdef char *data =<char*>malloc(length)
+        memset(data, 0, length)
+        memcpy(data, <char*>(buf + off), length)
+        munmap(buf, length + off)
+        res = data[:length]
+        free(data)
+        return res
+
 
 def mwtype2str(mw_type):
     mw_types = {1:'IBV_MW_TYPE_1', 2:'IBV_MW_TYPE_2'}
-- 
1.8.3.1

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

^ permalink raw reply related	[flat|nested] 42+ messages in thread
* [PATCH rdma-core v7 5/6] tests: Add tests for dma-buf based memory regions
  2021-01-25 19:56 ` Jianxin Xiong
@ 2021-01-25 19:57   ` Jianxin Xiong
  -1 siblings, 0 replies; 42+ messages in thread
From: Jianxin Xiong @ 2021-01-25 19:57 UTC (permalink / raw)
  To: linux-rdma, dri-devel
  Cc: Jianxin Xiong, Doug Ledford, Jason Gunthorpe, Leon Romanovsky,
	Sumit Semwal, Christian Koenig, Daniel Vetter, Edward Srouji,
	Yishai Hadas

Define a set of unit tests similar to regular MR tests and a set of
tests for send/recv and rdma traffic using dma-buf MRs. Add a utility
function to generate access flags for dma-buf based MRs because the
set of supported flags is smaller.

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
 tests/args_parser.py |   4 +
 tests/test_mr.py     | 266 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 tests/utils.py       |  26 +++++
 3 files changed, 295 insertions(+), 1 deletion(-)

diff --git a/tests/args_parser.py b/tests/args_parser.py
index 446535a..5bc53b0 100644
--- a/tests/args_parser.py
+++ b/tests/args_parser.py
@@ -19,6 +19,10 @@ class ArgsParser(object):
         parser.add_argument('--port',
                             help='Use port <port> of RDMA device', type=int,
                             default=1)
+        parser.add_argument('--gpu', nargs='?', type=int, const=0, default=0,
+                            help='GPU unit to allocate dmabuf from')
+        parser.add_argument('--gtt', action='store_true', default=False,
+                            help='Allocate dmabuf from GTT instead of VRAM')
         parser.add_argument('-v', '--verbose', dest='verbosity',
                             action='store_const',
                             const=2, help='Verbose output')
diff --git a/tests/test_mr.py b/tests/test_mr.py
index b88ad23..03a645f 100644
--- a/tests/test_mr.py
+++ b/tests/test_mr.py
@@ -1,5 +1,6 @@
 # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
 # Copyright (c) 2019 Mellanox Technologies, Inc. All rights reserved. See COPYING file
+# Copyright (c) 2020 Intel Corporation. All rights reserved. See COPYING file
 """
 Test module for pyverbs' mr module.
 """
@@ -9,15 +10,18 @@ import errno
 
 from tests.base import PyverbsAPITestCase, RCResources, RDMATestCase
 from pyverbs.pyverbs_error import PyverbsRDMAError, PyverbsError
-from pyverbs.mr import MR, MW, DMMR, MWBindInfo, MWBind
+from pyverbs.mr import MR, MW, DMMR, DmaBufMR, MWBindInfo, MWBind
 from pyverbs.qp import QPCap, QPInitAttr, QPAttr, QP
 from pyverbs.mem_alloc import posix_memalign, free
 from pyverbs.wr import SendWR
+from pyverbs.dmabuf import DmaBuf
 import pyverbs.device as d
 from pyverbs.pd import PD
 import pyverbs.enums as e
 import tests.utils as u
 
+MAX_IO_LEN = 1048576
+
 
 class MRRes(RCResources):
     def __init__(self, dev_name, ib_port, gid_index,
@@ -423,3 +427,263 @@ class DMMRTest(PyverbsAPITestCase):
                         dm_mr = DMMR(pd, dm_mr_len, e.IBV_ACCESS_ZERO_BASED,
                                      dm=dm, offset=dm_mr_offset)
                         dm_mr.close()
+
+
+def check_dmabuf_support(unit=0):
+    """
+    Check if dma-buf allocation is supported by the system.
+    Skip the test on failure.
+    """
+    device_num = 128 + unit
+    try:
+        DmaBuf(1, unit=unit)
+    except PyverbsRDMAError as ex:
+        if ex.error_code == errno.ENOENT:
+            raise unittest.SkipTest(f'Device /dev/dri/renderD{device_num} is not present')
+        if ex.error_code == errno.EACCES:
+            raise unittest.SkipTest(f'Lack of permission to access /dev/dri/renderD{device_num}')
+        if ex.error_code == errno.EOPNOTSUPP:
+            raise unittest.SkipTest(f'Allocating dmabuf is not supported by /dev/dri/renderD{device_num}')
+
+
+def check_dmabuf_mr_support(pd, unit=0):
+    """
+    Check if dma-buf MR registration is supported by the driver.
+    Skip the test on failure
+    """
+    try:
+        DmaBufMR(pd, 1, 0, unit=unit)
+    except PyverbsRDMAError as ex:
+        if ex.error_code == errno.EOPNOTSUPP:
+            raise unittest.SkipTest('Reg dma-buf MR is not supported by the RDMA driver')
+
+
+class DmaBufMRTest(PyverbsAPITestCase):
+    """
+    Test various functionalities of the DmaBufMR class.
+    """
+    def setUp(self):
+        super().setUp()
+        self.unit = self.config['gpu']
+        self.gtt = self.config['gtt']
+
+    def test_dmabuf_reg_mr(self):
+        """
+        Test ibv_reg_dmabuf_mr()
+        """
+        check_dmabuf_support(self.unit)
+        for ctx, attr, attr_ex in self.devices:
+            with PD(ctx) as pd:
+                check_dmabuf_mr_support(pd, self.unit)
+                flags = u.get_dmabuf_access_flags(ctx)
+                for f in flags:
+                    len = u.get_mr_length()
+                    for off in [0, len//2]:
+                        with DmaBufMR(pd, len, f, offset=off, unit=self.unit,
+                                      gtt=self.gtt) as mr:
+                            pass
+
+    def test_dmabuf_dereg_mr(self):
+        """
+        Test ibv_dereg_mr() with DmaBufMR
+        """
+        check_dmabuf_support(self.unit)
+        for ctx, attr, attr_ex in self.devices:
+            with PD(ctx) as pd:
+                check_dmabuf_mr_support(pd, self.unit)
+                flags = u.get_dmabuf_access_flags(ctx)
+                for f in flags:
+                    len = u.get_mr_length()
+                    for off in [0, len//2]:
+                        with DmaBufMR(pd, len, f, offset=off, unit=self.unit,
+                                      gtt=self.gtt) as mr:
+                            mr.close()
+
+    def test_dmabuf_dereg_mr_twice(self):
+        """
+        Verify that explicit call to DmaBufMR's close() doesn't fail
+        """
+        check_dmabuf_support(self.unit)
+        for ctx, attr, attr_ex in self.devices:
+            with PD(ctx) as pd:
+                check_dmabuf_mr_support(pd, self.unit)
+                flags = u.get_dmabuf_access_flags(ctx)
+                for f in flags:
+                    len = u.get_mr_length()
+                    for off in [0, len//2]:
+                        with DmaBufMR(pd, len, f, offset=off, unit=self.unit,
+                                      gtt=self.gtt) as mr:
+                            # Pyverbs supports multiple destruction of objects,
+                            # we are not expecting an exception here.
+                            mr.close()
+                            mr.close()
+
+    def test_dmabuf_reg_mr_bad_flags(self):
+        """
+        Verify that illegal flags combination fails as expected
+        """
+        check_dmabuf_support(self.unit)
+        for ctx, attr, attr_ex in self.devices:
+            with PD(ctx) as pd:
+                check_dmabuf_mr_support(pd, self.unit)
+                for i in range(5):
+                    flags = random.sample([e.IBV_ACCESS_REMOTE_WRITE,
+                                           e.IBV_ACCESS_REMOTE_ATOMIC],
+                                          random.randint(1, 2))
+                    mr_flags = 0
+                    for i in flags:
+                        mr_flags += i.value
+                    try:
+                        DmaBufMR(pd, u.get_mr_length(), mr_flags,
+                                 unit=self.unit, gtt=self.gtt)
+                    except PyverbsRDMAError as err:
+                        assert 'Failed to register a dma-buf MR' in err.args[0]
+                    else:
+                        raise PyverbsRDMAError('Registered a dma-buf MR with illegal falgs')
+
+    def test_dmabuf_write(self):
+        """
+        Test writing to DmaBufMR's buffer
+        """
+        check_dmabuf_support(self.unit)
+        for ctx, attr, attr_ex in self.devices:
+            with PD(ctx) as pd:
+                check_dmabuf_mr_support(pd, self.unit)
+                for i in range(10):
+                    mr_len = u.get_mr_length()
+                    flags = u.get_dmabuf_access_flags(ctx)
+                    for f in flags:
+                        for mr_off in [0, mr_len//2]:
+                            with DmaBufMR(pd, mr_len, f, offset=mr_off,
+                                          unit=self.unit, gtt=self.gtt) as mr:
+                                write_len = min(random.randint(1, MAX_IO_LEN),
+                                                mr_len)
+                                mr.write('a' * write_len, write_len)
+
+    def test_dmabuf_read(self):
+        """
+        Test reading from DmaBufMR's buffer
+        """
+        check_dmabuf_support(self.unit)
+        for ctx, attr, attr_ex in self.devices:
+            with PD(ctx) as pd:
+                check_dmabuf_mr_support(pd, self.unit)
+                for i in range(10):
+                    mr_len = u.get_mr_length()
+                    flags = u.get_dmabuf_access_flags(ctx)
+                    for f in flags:
+                        for mr_off in [0, mr_len//2]:
+                            with DmaBufMR(pd, mr_len, f, offset=mr_off,
+                                          unit=self.unit, gtt=self.gtt) as mr:
+                                write_len = min(random.randint(1, MAX_IO_LEN),
+                                                mr_len)
+                                write_str = 'a' * write_len
+                                mr.write(write_str, write_len)
+                                read_len = random.randint(1, write_len)
+                                offset = random.randint(0, write_len-read_len)
+                                read_str = mr.read(read_len, offset).decode()
+                                assert read_str in write_str
+
+    def test_dmabuf_lkey(self):
+        """
+        Test reading lkey property
+        """
+        check_dmabuf_support(self.unit)
+        for ctx, attr, attr_ex in self.devices:
+            with PD(ctx) as pd:
+                check_dmabuf_mr_support(pd, self.unit)
+                length = u.get_mr_length()
+                flags = u.get_dmabuf_access_flags(ctx)
+                for f in flags:
+                    with DmaBufMR(pd, length, f, unit=self.unit,
+                                  gtt=self.gtt) as mr:
+                        mr.lkey
+
+    def test_dmabuf_rkey(self):
+        """
+        Test reading rkey property
+        """
+        check_dmabuf_support(self.unit)
+        for ctx, attr, attr_ex in self.devices:
+            with PD(ctx) as pd:
+                check_dmabuf_mr_support(pd, self.unit)
+                length = u.get_mr_length()
+                flags = u.get_dmabuf_access_flags(ctx)
+                for f in flags:
+                    with DmaBufMR(pd, length, f, unit=self.unit,
+                                  gtt=self.gtt) as mr:
+                        mr.rkey
+
+
+class DmaBufRC(RCResources):
+    def __init__(self, dev_name, ib_port, gid_index, unit, gtt):
+        """
+        Initialize an DmaBufRC object.
+        :param dev_name: Device name to be used
+        :param ib_port: IB port of the device to use
+        :param gid_index: Which GID index to use
+        :param unit: GPU unit to allocate dmabuf from
+        :gtt: Allocate dmabuf from GTT instead og VRAM
+        """
+        self.unit = unit
+        self.gtt = gtt
+        super(DmaBufRC, self).__init__(dev_name=dev_name, ib_port=ib_port,
+                                       gid_index=gid_index)
+
+    def create_mr(self):
+        check_dmabuf_support(self.unit)
+        check_dmabuf_mr_support(self.pd, self.unit)
+        access = e.IBV_ACCESS_LOCAL_WRITE | e.IBV_ACCESS_REMOTE_WRITE
+        mr = DmaBufMR(self.pd, self.msg_size, access, unit=self.unit,
+                      gtt=self.gtt)
+        self.mr = mr
+
+    def create_qp_attr(self):
+        qp_attr = QPAttr(port_num=self.ib_port)
+        qp_access = e.IBV_ACCESS_LOCAL_WRITE | e.IBV_ACCESS_REMOTE_WRITE
+        qp_attr.qp_access_flags = qp_access
+        return qp_attr
+
+
+class DmaBufTestCase(RDMATestCase):
+    def setUp(self):
+        super(DmaBufTestCase, self).setUp()
+        self.iters = 100
+        self.unit = self.config['gpu']
+        self.gtt = self.config['gtt']
+
+    def create_players(self, resource, **resource_arg):
+        """
+        Init dma-buf tests resources.
+        :param resource: The RDMA resources to use. A class of type
+                         BaseResources.
+        :param resource_arg: Dict of args that specify the resource specific
+                             attributes.
+        :return: The (client, server) resources.
+        """
+        client = resource(**self.dev_info, **resource_arg)
+        server = resource(**self.dev_info, **resource_arg)
+        client.pre_run(server.psns, server.qps_num)
+        server.pre_run(client.psns, client.qps_num)
+        return client, server
+
+    def test_dmabuf_rc_traffic(self):
+        """
+        Test send/recv using dma-buf MR over RC
+        """
+        client, server = self.create_players(DmaBufRC, unit=self.unit,
+                                             gtt=self.gtt)
+        u.traffic(client, server, self.iters, self.gid_index, self.ib_port)
+
+    def test_dmabuf_rdma_traffic(self):
+        """
+        Test rdma write using dma-buf MR
+        """
+        client, server = self.create_players(DmaBufRC, unit=self.unit,
+                                             gtt=self.gtt)
+        server.rkey = client.mr.rkey
+        server.remote_addr = client.mr.offset
+        client.rkey = server.mr.rkey
+        client.remote_addr = server.mr.offset
+        u.rdma_traffic(client, server, self.iters, self.gid_index, self.ib_port,
+                       send_op=e.IBV_WR_RDMA_WRITE)
diff --git a/tests/utils.py b/tests/utils.py
index e2ab669..8546329 100644
--- a/tests/utils.py
+++ b/tests/utils.py
@@ -1,5 +1,6 @@
 # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
 # Copyright (c) 2019 Mellanox Technologies, Inc. All rights reserved.  See COPYING file
+# Copyright (c) 2020 Intel Corporation. All rights reserved. See COPYING file
 """
 Provide some useful helper function for pyverbs' tests.
 """
@@ -96,6 +97,31 @@ def get_access_flags(ctx):
     return arr
 
 
+def get_dmabuf_access_flags(ctx):
+    """
+    Similar to get_access_flags, except that dma-buf MR only support
+    a subset of the flags.
+    :param ctx: Device Context to check capabilities
+    :return: A random legal value for MR flags
+    """
+    attr = ctx.query_device()
+    vals = [e.IBV_ACCESS_LOCAL_WRITE, e.IBV_ACCESS_REMOTE_WRITE,
+            e.IBV_ACCESS_REMOTE_READ, e.IBV_ACCESS_REMOTE_ATOMIC,
+            e.IBV_ACCESS_RELAXED_ORDERING]
+    if not attr.atomic_caps & e.IBV_ATOMIC_HCA:
+        vals.remove(e.IBV_ACCESS_REMOTE_ATOMIC)
+    arr = []
+    for i in range(1, len(vals)):
+        tmp = list(com(vals, i))
+        tmp = filter(filter_illegal_access_flags, tmp)
+        for t in tmp:  # Iterate legal combinations and bitwise OR them
+            val = 0
+            for flag in t:
+                val += flag.value
+            arr.append(val)
+    return arr
+
+
 def get_dm_attrs(dm_len):
     """
     Initializes an AllocDmAttr member with the given length and random
-- 
1.8.3.1


^ permalink raw reply related	[flat|nested] 42+ messages in thread
* [PATCH rdma-core v7 5/6] tests: Add tests for dma-buf based memory regions
@ 2021-01-25 19:57   ` Jianxin Xiong
  0 siblings, 0 replies; 42+ messages in thread
From: Jianxin Xiong @ 2021-01-25 19:57 UTC (permalink / raw)
  To: linux-rdma, dri-devel
  Cc: Yishai Hadas, Leon Romanovsky, Edward Srouji, Jason Gunthorpe,
	Doug Ledford, Daniel Vetter, Christian Koenig, Jianxin Xiong

Define a set of unit tests similar to regular MR tests and a set of
tests for send/recv and rdma traffic using dma-buf MRs. Add a utility
function to generate access flags for dma-buf based MRs because the
set of supported flags is smaller.

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
 tests/args_parser.py |   4 +
 tests/test_mr.py     | 266 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 tests/utils.py       |  26 +++++
 3 files changed, 295 insertions(+), 1 deletion(-)

diff --git a/tests/args_parser.py b/tests/args_parser.py
index 446535a..5bc53b0 100644
--- a/tests/args_parser.py
+++ b/tests/args_parser.py
@@ -19,6 +19,10 @@ class ArgsParser(object):
         parser.add_argument('--port',
                             help='Use port <port> of RDMA device', type=int,
                             default=1)
+        parser.add_argument('--gpu', nargs='?', type=int, const=0, default=0,
+                            help='GPU unit to allocate dmabuf from')
+        parser.add_argument('--gtt', action='store_true', default=False,
+                            help='Allocate dmabuf from GTT instead of VRAM')
         parser.add_argument('-v', '--verbose', dest='verbosity',
                             action='store_const',
                             const=2, help='Verbose output')
diff --git a/tests/test_mr.py b/tests/test_mr.py
index b88ad23..03a645f 100644
--- a/tests/test_mr.py
+++ b/tests/test_mr.py
@@ -1,5 +1,6 @@
 # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
 # Copyright (c) 2019 Mellanox Technologies, Inc. All rights reserved. See COPYING file
+# Copyright (c) 2020 Intel Corporation. All rights reserved. See COPYING file
 """
 Test module for pyverbs' mr module.
 """
@@ -9,15 +10,18 @@ import errno
 
 from tests.base import PyverbsAPITestCase, RCResources, RDMATestCase
 from pyverbs.pyverbs_error import PyverbsRDMAError, PyverbsError
-from pyverbs.mr import MR, MW, DMMR, MWBindInfo, MWBind
+from pyverbs.mr import MR, MW, DMMR, DmaBufMR, MWBindInfo, MWBind
 from pyverbs.qp import QPCap, QPInitAttr, QPAttr, QP
 from pyverbs.mem_alloc import posix_memalign, free
 from pyverbs.wr import SendWR
+from pyverbs.dmabuf import DmaBuf
 import pyverbs.device as d
 from pyverbs.pd import PD
 import pyverbs.enums as e
 import tests.utils as u
 
+MAX_IO_LEN = 1048576
+
 
 class MRRes(RCResources):
     def __init__(self, dev_name, ib_port, gid_index,
@@ -423,3 +427,263 @@ class DMMRTest(PyverbsAPITestCase):
                         dm_mr = DMMR(pd, dm_mr_len, e.IBV_ACCESS_ZERO_BASED,
                                      dm=dm, offset=dm_mr_offset)
                         dm_mr.close()
+
+
+def check_dmabuf_support(unit=0):
+    """
+    Check if dma-buf allocation is supported by the system.
+    Skip the test on failure.
+    """
+    device_num = 128 + unit
+    try:
+        DmaBuf(1, unit=unit)
+    except PyverbsRDMAError as ex:
+        if ex.error_code == errno.ENOENT:
+            raise unittest.SkipTest(f'Device /dev/dri/renderD{device_num} is not present')
+        if ex.error_code == errno.EACCES:
+            raise unittest.SkipTest(f'Lack of permission to access /dev/dri/renderD{device_num}')
+        if ex.error_code == errno.EOPNOTSUPP:
+            raise unittest.SkipTest(f'Allocating dmabuf is not supported by /dev/dri/renderD{device_num}')
+
+
+def check_dmabuf_mr_support(pd, unit=0):
+    """
+    Check if dma-buf MR registration is supported by the driver.
+    Skip the test on failure
+    """
+    try:
+        DmaBufMR(pd, 1, 0, unit=unit)
+    except PyverbsRDMAError as ex:
+        if ex.error_code == errno.EOPNOTSUPP:
+            raise unittest.SkipTest('Reg dma-buf MR is not supported by the RDMA driver')
+
+
+class DmaBufMRTest(PyverbsAPITestCase):
+    """
+    Test various functionalities of the DmaBufMR class.
+    """
+    def setUp(self):
+        super().setUp()
+        self.unit = self.config['gpu']
+        self.gtt = self.config['gtt']
+
+    def test_dmabuf_reg_mr(self):
+        """
+        Test ibv_reg_dmabuf_mr()
+        """
+        check_dmabuf_support(self.unit)
+        for ctx, attr, attr_ex in self.devices:
+            with PD(ctx) as pd:
+                check_dmabuf_mr_support(pd, self.unit)
+                flags = u.get_dmabuf_access_flags(ctx)
+                for f in flags:
+                    len = u.get_mr_length()
+                    for off in [0, len//2]:
+                        with DmaBufMR(pd, len, f, offset=off, unit=self.unit,
+                                      gtt=self.gtt) as mr:
+                            pass
+
+    def test_dmabuf_dereg_mr(self):
+        """
+        Test ibv_dereg_mr() with DmaBufMR
+        """
+        check_dmabuf_support(self.unit)
+        for ctx, attr, attr_ex in self.devices:
+            with PD(ctx) as pd:
+                check_dmabuf_mr_support(pd, self.unit)
+                flags = u.get_dmabuf_access_flags(ctx)
+                for f in flags:
+                    len = u.get_mr_length()
+                    for off in [0, len//2]:
+                        with DmaBufMR(pd, len, f, offset=off, unit=self.unit,
+                                      gtt=self.gtt) as mr:
+                            mr.close()
+
+    def test_dmabuf_dereg_mr_twice(self):
+        """
+        Verify that explicit call to DmaBufMR's close() doesn't fail
+        """
+        check_dmabuf_support(self.unit)
+        for ctx, attr, attr_ex in self.devices:
+            with PD(ctx) as pd:
+                check_dmabuf_mr_support(pd, self.unit)
+                flags = u.get_dmabuf_access_flags(ctx)
+                for f in flags:
+                    len = u.get_mr_length()
+                    for off in [0, len//2]:
+                        with DmaBufMR(pd, len, f, offset=off, unit=self.unit,
+                                      gtt=self.gtt) as mr:
+                            # Pyverbs supports multiple destruction of objects,
+                            # we are not expecting an exception here.
+                            mr.close()
+                            mr.close()
+
+    def test_dmabuf_reg_mr_bad_flags(self):
+        """
+        Verify that illegal flags combination fails as expected
+        """
+        check_dmabuf_support(self.unit)
+        for ctx, attr, attr_ex in self.devices:
+            with PD(ctx) as pd:
+                check_dmabuf_mr_support(pd, self.unit)
+                for i in range(5):
+                    flags = random.sample([e.IBV_ACCESS_REMOTE_WRITE,
+                                           e.IBV_ACCESS_REMOTE_ATOMIC],
+                                          random.randint(1, 2))
+                    mr_flags = 0
+                    for i in flags:
+                        mr_flags += i.value
+                    try:
+                        DmaBufMR(pd, u.get_mr_length(), mr_flags,
+                                 unit=self.unit, gtt=self.gtt)
+                    except PyverbsRDMAError as err:
+                        assert 'Failed to register a dma-buf MR' in err.args[0]
+                    else:
+                        raise PyverbsRDMAError('Registered a dma-buf MR with illegal falgs')
+
+    def test_dmabuf_write(self):
+        """
+        Test writing to DmaBufMR's buffer
+        """
+        check_dmabuf_support(self.unit)
+        for ctx, attr, attr_ex in self.devices:
+            with PD(ctx) as pd:
+                check_dmabuf_mr_support(pd, self.unit)
+                for i in range(10):
+                    mr_len = u.get_mr_length()
+                    flags = u.get_dmabuf_access_flags(ctx)
+                    for f in flags:
+                        for mr_off in [0, mr_len//2]:
+                            with DmaBufMR(pd, mr_len, f, offset=mr_off,
+                                          unit=self.unit, gtt=self.gtt) as mr:
+                                write_len = min(random.randint(1, MAX_IO_LEN),
+                                                mr_len)
+                                mr.write('a' * write_len, write_len)
+
+    def test_dmabuf_read(self):
+        """
+        Test reading from DmaBufMR's buffer
+        """
+        check_dmabuf_support(self.unit)
+        for ctx, attr, attr_ex in self.devices:
+            with PD(ctx) as pd:
+                check_dmabuf_mr_support(pd, self.unit)
+                for i in range(10):
+                    mr_len = u.get_mr_length()
+                    flags = u.get_dmabuf_access_flags(ctx)
+                    for f in flags:
+                        for mr_off in [0, mr_len//2]:
+                            with DmaBufMR(pd, mr_len, f, offset=mr_off,
+                                          unit=self.unit, gtt=self.gtt) as mr:
+                                write_len = min(random.randint(1, MAX_IO_LEN),
+                                                mr_len)
+                                write_str = 'a' * write_len
+                                mr.write(write_str, write_len)
+                                read_len = random.randint(1, write_len)
+                                offset = random.randint(0, write_len-read_len)
+                                read_str = mr.read(read_len, offset).decode()
+                                assert read_str in write_str
+
+    def test_dmabuf_lkey(self):
+        """
+        Test reading lkey property
+        """
+        check_dmabuf_support(self.unit)
+        for ctx, attr, attr_ex in self.devices:
+            with PD(ctx) as pd:
+                check_dmabuf_mr_support(pd, self.unit)
+                length = u.get_mr_length()
+                flags = u.get_dmabuf_access_flags(ctx)
+                for f in flags:
+                    with DmaBufMR(pd, length, f, unit=self.unit,
+                                  gtt=self.gtt) as mr:
+                        mr.lkey
+
+    def test_dmabuf_rkey(self):
+        """
+        Test reading rkey property
+        """
+        check_dmabuf_support(self.unit)
+        for ctx, attr, attr_ex in self.devices:
+            with PD(ctx) as pd:
+                check_dmabuf_mr_support(pd, self.unit)
+                length = u.get_mr_length()
+                flags = u.get_dmabuf_access_flags(ctx)
+                for f in flags:
+                    with DmaBufMR(pd, length, f, unit=self.unit,
+                                  gtt=self.gtt) as mr:
+                        mr.rkey
+
+
+class DmaBufRC(RCResources):
+    def __init__(self, dev_name, ib_port, gid_index, unit, gtt):
+        """
+        Initialize an DmaBufRC object.
+        :param dev_name: Device name to be used
+        :param ib_port: IB port of the device to use
+        :param gid_index: Which GID index to use
+        :param unit: GPU unit to allocate dmabuf from
+        :gtt: Allocate dmabuf from GTT instead og VRAM
+        """
+        self.unit = unit
+        self.gtt = gtt
+        super(DmaBufRC, self).__init__(dev_name=dev_name, ib_port=ib_port,
+                                       gid_index=gid_index)
+
+    def create_mr(self):
+        check_dmabuf_support(self.unit)
+        check_dmabuf_mr_support(self.pd, self.unit)
+        access = e.IBV_ACCESS_LOCAL_WRITE | e.IBV_ACCESS_REMOTE_WRITE
+        mr = DmaBufMR(self.pd, self.msg_size, access, unit=self.unit,
+                      gtt=self.gtt)
+        self.mr = mr
+
+    def create_qp_attr(self):
+        qp_attr = QPAttr(port_num=self.ib_port)
+        qp_access = e.IBV_ACCESS_LOCAL_WRITE | e.IBV_ACCESS_REMOTE_WRITE
+        qp_attr.qp_access_flags = qp_access
+        return qp_attr
+
+
+class DmaBufTestCase(RDMATestCase):
+    def setUp(self):
+        super(DmaBufTestCase, self).setUp()
+        self.iters = 100
+        self.unit = self.config['gpu']
+        self.gtt = self.config['gtt']
+
+    def create_players(self, resource, **resource_arg):
+        """
+        Init dma-buf tests resources.
+        :param resource: The RDMA resources to use. A class of type
+                         BaseResources.
+        :param resource_arg: Dict of args that specify the resource specific
+                             attributes.
+        :return: The (client, server) resources.
+        """
+        client = resource(**self.dev_info, **resource_arg)
+        server = resource(**self.dev_info, **resource_arg)
+        client.pre_run(server.psns, server.qps_num)
+        server.pre_run(client.psns, client.qps_num)
+        return client, server
+
+    def test_dmabuf_rc_traffic(self):
+        """
+        Test send/recv using dma-buf MR over RC
+        """
+        client, server = self.create_players(DmaBufRC, unit=self.unit,
+                                             gtt=self.gtt)
+        u.traffic(client, server, self.iters, self.gid_index, self.ib_port)
+
+    def test_dmabuf_rdma_traffic(self):
+        """
+        Test rdma write using dma-buf MR
+        """
+        client, server = self.create_players(DmaBufRC, unit=self.unit,
+                                             gtt=self.gtt)
+        server.rkey = client.mr.rkey
+        server.remote_addr = client.mr.offset
+        client.rkey = server.mr.rkey
+        client.remote_addr = server.mr.offset
+        u.rdma_traffic(client, server, self.iters, self.gid_index, self.ib_port,
+                       send_op=e.IBV_WR_RDMA_WRITE)
diff --git a/tests/utils.py b/tests/utils.py
index e2ab669..8546329 100644
--- a/tests/utils.py
+++ b/tests/utils.py
@@ -1,5 +1,6 @@
 # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
 # Copyright (c) 2019 Mellanox Technologies, Inc. All rights reserved.  See COPYING file
+# Copyright (c) 2020 Intel Corporation. All rights reserved. See COPYING file
 """
 Provide some useful helper function for pyverbs' tests.
 """
@@ -96,6 +97,31 @@ def get_access_flags(ctx):
     return arr
 
 
+def get_dmabuf_access_flags(ctx):
+    """
+    Similar to get_access_flags, except that dma-buf MR only support
+    a subset of the flags.
+    :param ctx: Device Context to check capabilities
+    :return: A random legal value for MR flags
+    """
+    attr = ctx.query_device()
+    vals = [e.IBV_ACCESS_LOCAL_WRITE, e.IBV_ACCESS_REMOTE_WRITE,
+            e.IBV_ACCESS_REMOTE_READ, e.IBV_ACCESS_REMOTE_ATOMIC,
+            e.IBV_ACCESS_RELAXED_ORDERING]
+    if not attr.atomic_caps & e.IBV_ATOMIC_HCA:
+        vals.remove(e.IBV_ACCESS_REMOTE_ATOMIC)
+    arr = []
+    for i in range(1, len(vals)):
+        tmp = list(com(vals, i))
+        tmp = filter(filter_illegal_access_flags, tmp)
+        for t in tmp:  # Iterate legal combinations and bitwise OR them
+            val = 0
+            for flag in t:
+                val += flag.value
+            arr.append(val)
+    return arr
+
+
 def get_dm_attrs(dm_len):
     """
     Initializes an AllocDmAttr member with the given length and random
-- 
1.8.3.1

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

^ permalink raw reply related	[flat|nested] 42+ messages in thread
* [PATCH rdma-core v7 6/6] tests: Bug fix for get_access_flags()
  2021-01-25 19:56 ` Jianxin Xiong
@ 2021-01-25 19:57   ` Jianxin Xiong
  -1 siblings, 0 replies; 42+ messages in thread
From: Jianxin Xiong @ 2021-01-25 19:57 UTC (permalink / raw)
  To: linux-rdma, dri-devel
  Cc: Jianxin Xiong, Doug Ledford, Jason Gunthorpe, Leon Romanovsky,
	Sumit Semwal, Christian Koenig, Daniel Vetter, Edward Srouji,
	Yishai Hadas

The filter definition is wrong and causes get_access_flags() always
returning empty list. As the result the MR tests using this function
are effectively skipped (but report success).

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
 tests/utils.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/utils.py b/tests/utils.py
index 8546329..c41cb3b 100644
--- a/tests/utils.py
+++ b/tests/utils.py
@@ -58,8 +58,8 @@ def filter_illegal_access_flags(element):
     :param element: A list of access flags to check
     :return: True if this list is legal, else False
     """
-    if e.IBV_ACCESS_REMOTE_ATOMIC in element or e.IBV_ACCESS_REMOTE_WRITE:
-        if e.IBV_ACCESS_LOCAL_WRITE:
+    if e.IBV_ACCESS_REMOTE_ATOMIC in element or e.IBV_ACCESS_REMOTE_WRITE in element:
+        if not e.IBV_ACCESS_LOCAL_WRITE in element:
             return False
     return True
 
-- 
1.8.3.1


^ permalink raw reply related	[flat|nested] 42+ messages in thread
* [PATCH rdma-core v7 6/6] tests: Bug fix for get_access_flags()
@ 2021-01-25 19:57   ` Jianxin Xiong
  0 siblings, 0 replies; 42+ messages in thread
From: Jianxin Xiong @ 2021-01-25 19:57 UTC (permalink / raw)
  To: linux-rdma, dri-devel
  Cc: Yishai Hadas, Leon Romanovsky, Edward Srouji, Jason Gunthorpe,
	Doug Ledford, Daniel Vetter, Christian Koenig, Jianxin Xiong

The filter definition is wrong and causes get_access_flags() always
returning empty list. As the result the MR tests using this function
are effectively skipped (but report success).

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
 tests/utils.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/utils.py b/tests/utils.py
index 8546329..c41cb3b 100644
--- a/tests/utils.py
+++ b/tests/utils.py
@@ -58,8 +58,8 @@ def filter_illegal_access_flags(element):
     :param element: A list of access flags to check
     :return: True if this list is legal, else False
     """
-    if e.IBV_ACCESS_REMOTE_ATOMIC in element or e.IBV_ACCESS_REMOTE_WRITE:
-        if e.IBV_ACCESS_LOCAL_WRITE:
+    if e.IBV_ACCESS_REMOTE_ATOMIC in element or e.IBV_ACCESS_REMOTE_WRITE in element:
+        if not e.IBV_ACCESS_LOCAL_WRITE in element:
             return False
     return True
 
-- 
1.8.3.1

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

^ permalink raw reply related	[flat|nested] 42+ messages in thread
* Re: [PATCH rdma-core v7 5/6] tests: Add tests for dma-buf based memory regions
  2021-01-25 19:57   ` Jianxin Xiong
@ 2021-01-31  8:44     ` John Hubbard
  -1 siblings, 0 replies; 42+ messages in thread
From: John Hubbard @ 2021-01-31  8:44 UTC (permalink / raw)
  To: Jianxin Xiong, linux-rdma, dri-devel
  Cc: Doug Ledford, Jason Gunthorpe, Leon Romanovsky, Sumit Semwal,
	Christian Koenig, Daniel Vetter, Edward Srouji, Yishai Hadas

On 1/25/21 11:57 AM, Jianxin Xiong wrote:
> Define a set of unit tests similar to regular MR tests and a set of
> tests for send/recv and rdma traffic using dma-buf MRs. Add a utility
> function to generate access flags for dma-buf based MRs because the
> set of supported flags is smaller.
> 
> Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>

Hi Jianxin,

It's awesome to see a GPU to IB test suite here!

> ---
>   tests/args_parser.py |   4 +
>   tests/test_mr.py     | 266 ++++++++++++++++++++++++++++++++++++++++++++++++++-
>   tests/utils.py       |  26 +++++
>   3 files changed, 295 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/args_parser.py b/tests/args_parser.py
> index 446535a..5bc53b0 100644
> --- a/tests/args_parser.py
> +++ b/tests/args_parser.py
> @@ -19,6 +19,10 @@ class ArgsParser(object):
>           parser.add_argument('--port',
>                               help='Use port <port> of RDMA device', type=int,
>                               default=1)
> +        parser.add_argument('--gpu', nargs='?', type=int, const=0, default=0,
> +                            help='GPU unit to allocate dmabuf from')
> +        parser.add_argument('--gtt', action='store_true', default=False,
> +                            help='Allocate dmabuf from GTT instead of VRAM')

Just to be kind to non-GPU people, how about:

	s/GTT/GTT (Graphics Translation Table)/


>           parser.add_argument('-v', '--verbose', dest='verbosity',
>                               action='store_const',
>                               const=2, help='Verbose output')
> diff --git a/tests/test_mr.py b/tests/test_mr.py
> index b88ad23..03a645f 100644
> --- a/tests/test_mr.py
> +++ b/tests/test_mr.py
> @@ -1,5 +1,6 @@
>   # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
>   # Copyright (c) 2019 Mellanox Technologies, Inc. All rights reserved. See COPYING file
> +# Copyright (c) 2020 Intel Corporation. All rights reserved. See COPYING file
>   """
>   Test module for pyverbs' mr module.
>   """
> @@ -9,15 +10,18 @@ import errno
>   
>   from tests.base import PyverbsAPITestCase, RCResources, RDMATestCase
>   from pyverbs.pyverbs_error import PyverbsRDMAError, PyverbsError
> -from pyverbs.mr import MR, MW, DMMR, MWBindInfo, MWBind
> +from pyverbs.mr import MR, MW, DMMR, DmaBufMR, MWBindInfo, MWBind
>   from pyverbs.qp import QPCap, QPInitAttr, QPAttr, QP
>   from pyverbs.mem_alloc import posix_memalign, free
>   from pyverbs.wr import SendWR
> +from pyverbs.dmabuf import DmaBuf
>   import pyverbs.device as d
>   from pyverbs.pd import PD
>   import pyverbs.enums as e
>   import tests.utils as u
>   
> +MAX_IO_LEN = 1048576
> +
>   
>   class MRRes(RCResources):
>       def __init__(self, dev_name, ib_port, gid_index,
> @@ -423,3 +427,263 @@ class DMMRTest(PyverbsAPITestCase):
>                           dm_mr = DMMR(pd, dm_mr_len, e.IBV_ACCESS_ZERO_BASED,
>                                        dm=dm, offset=dm_mr_offset)
>                           dm_mr.close()
> +
> +
> +def check_dmabuf_support(unit=0):
> +    """
> +    Check if dma-buf allocation is supported by the system.
> +    Skip the test on failure.
> +    """
> +    device_num = 128 + unit
> +    try:
> +        DmaBuf(1, unit=unit)

unit?? This is a GPU, never anything else! Let's s/unit/gpu/ throughout, yes?

thanks,
-- 
John Hubbard
NVIDIA

> +    except PyverbsRDMAError as ex:
> +        if ex.error_code == errno.ENOENT:
> +            raise unittest.SkipTest(f'Device /dev/dri/renderD{device_num} is not present')
> +        if ex.error_code == errno.EACCES:
> +            raise unittest.SkipTest(f'Lack of permission to access /dev/dri/renderD{device_num}')
> +        if ex.error_code == errno.EOPNOTSUPP:
> +            raise unittest.SkipTest(f'Allocating dmabuf is not supported by /dev/dri/renderD{device_num}')
> +
> +
> +def check_dmabuf_mr_support(pd, unit=0):
> +    """
> +    Check if dma-buf MR registration is supported by the driver.
> +    Skip the test on failure
> +    """
> +    try:
> +        DmaBufMR(pd, 1, 0, unit=unit)
> +    except PyverbsRDMAError as ex:
> +        if ex.error_code == errno.EOPNOTSUPP:
> +            raise unittest.SkipTest('Reg dma-buf MR is not supported by the RDMA driver')
> +
> +
> +class DmaBufMRTest(PyverbsAPITestCase):
> +    """
> +    Test various functionalities of the DmaBufMR class.
> +    """
> +    def setUp(self):
> +        super().setUp()
> +        self.unit = self.config['gpu']
> +        self.gtt = self.config['gtt']
> +
> +    def test_dmabuf_reg_mr(self):
> +        """
> +        Test ibv_reg_dmabuf_mr()
> +        """
> +        check_dmabuf_support(self.unit)
> +        for ctx, attr, attr_ex in self.devices:
> +            with PD(ctx) as pd:
> +                check_dmabuf_mr_support(pd, self.unit)
> +                flags = u.get_dmabuf_access_flags(ctx)
> +                for f in flags:
> +                    len = u.get_mr_length()
> +                    for off in [0, len//2]:
> +                        with DmaBufMR(pd, len, f, offset=off, unit=self.unit,
> +                                      gtt=self.gtt) as mr:
> +                            pass
> +
> +    def test_dmabuf_dereg_mr(self):
> +        """
> +        Test ibv_dereg_mr() with DmaBufMR
> +        """
> +        check_dmabuf_support(self.unit)
> +        for ctx, attr, attr_ex in self.devices:
> +            with PD(ctx) as pd:
> +                check_dmabuf_mr_support(pd, self.unit)
> +                flags = u.get_dmabuf_access_flags(ctx)
> +                for f in flags:
> +                    len = u.get_mr_length()
> +                    for off in [0, len//2]:
> +                        with DmaBufMR(pd, len, f, offset=off, unit=self.unit,
> +                                      gtt=self.gtt) as mr:
> +                            mr.close()
> +
> +    def test_dmabuf_dereg_mr_twice(self):
> +        """
> +        Verify that explicit call to DmaBufMR's close() doesn't fail
> +        """
> +        check_dmabuf_support(self.unit)
> +        for ctx, attr, attr_ex in self.devices:
> +            with PD(ctx) as pd:
> +                check_dmabuf_mr_support(pd, self.unit)
> +                flags = u.get_dmabuf_access_flags(ctx)
> +                for f in flags:
> +                    len = u.get_mr_length()
> +                    for off in [0, len//2]:
> +                        with DmaBufMR(pd, len, f, offset=off, unit=self.unit,
> +                                      gtt=self.gtt) as mr:
> +                            # Pyverbs supports multiple destruction of objects,
> +                            # we are not expecting an exception here.
> +                            mr.close()
> +                            mr.close()
> +
> +    def test_dmabuf_reg_mr_bad_flags(self):
> +        """
> +        Verify that illegal flags combination fails as expected
> +        """
> +        check_dmabuf_support(self.unit)
> +        for ctx, attr, attr_ex in self.devices:
> +            with PD(ctx) as pd:
> +                check_dmabuf_mr_support(pd, self.unit)
> +                for i in range(5):
> +                    flags = random.sample([e.IBV_ACCESS_REMOTE_WRITE,
> +                                           e.IBV_ACCESS_REMOTE_ATOMIC],
> +                                          random.randint(1, 2))
> +                    mr_flags = 0
> +                    for i in flags:
> +                        mr_flags += i.value
> +                    try:
> +                        DmaBufMR(pd, u.get_mr_length(), mr_flags,
> +                                 unit=self.unit, gtt=self.gtt)
> +                    except PyverbsRDMAError as err:
> +                        assert 'Failed to register a dma-buf MR' in err.args[0]
> +                    else:
> +                        raise PyverbsRDMAError('Registered a dma-buf MR with illegal falgs')
> +
> +    def test_dmabuf_write(self):
> +        """
> +        Test writing to DmaBufMR's buffer
> +        """
> +        check_dmabuf_support(self.unit)
> +        for ctx, attr, attr_ex in self.devices:
> +            with PD(ctx) as pd:
> +                check_dmabuf_mr_support(pd, self.unit)
> +                for i in range(10):
> +                    mr_len = u.get_mr_length()
> +                    flags = u.get_dmabuf_access_flags(ctx)
> +                    for f in flags:
> +                        for mr_off in [0, mr_len//2]:
> +                            with DmaBufMR(pd, mr_len, f, offset=mr_off,
> +                                          unit=self.unit, gtt=self.gtt) as mr:
> +                                write_len = min(random.randint(1, MAX_IO_LEN),
> +                                                mr_len)
> +                                mr.write('a' * write_len, write_len)
> +
> +    def test_dmabuf_read(self):
> +        """
> +        Test reading from DmaBufMR's buffer
> +        """
> +        check_dmabuf_support(self.unit)
> +        for ctx, attr, attr_ex in self.devices:
> +            with PD(ctx) as pd:
> +                check_dmabuf_mr_support(pd, self.unit)
> +                for i in range(10):
> +                    mr_len = u.get_mr_length()
> +                    flags = u.get_dmabuf_access_flags(ctx)
> +                    for f in flags:
> +                        for mr_off in [0, mr_len//2]:
> +                            with DmaBufMR(pd, mr_len, f, offset=mr_off,
> +                                          unit=self.unit, gtt=self.gtt) as mr:
> +                                write_len = min(random.randint(1, MAX_IO_LEN),
> +                                                mr_len)
> +                                write_str = 'a' * write_len
> +                                mr.write(write_str, write_len)
> +                                read_len = random.randint(1, write_len)
> +                                offset = random.randint(0, write_len-read_len)
> +                                read_str = mr.read(read_len, offset).decode()
> +                                assert read_str in write_str
> +
> +    def test_dmabuf_lkey(self):
> +        """
> +        Test reading lkey property
> +        """
> +        check_dmabuf_support(self.unit)
> +        for ctx, attr, attr_ex in self.devices:
> +            with PD(ctx) as pd:
> +                check_dmabuf_mr_support(pd, self.unit)
> +                length = u.get_mr_length()
> +                flags = u.get_dmabuf_access_flags(ctx)
> +                for f in flags:
> +                    with DmaBufMR(pd, length, f, unit=self.unit,
> +                                  gtt=self.gtt) as mr:
> +                        mr.lkey
> +
> +    def test_dmabuf_rkey(self):
> +        """
> +        Test reading rkey property
> +        """
> +        check_dmabuf_support(self.unit)
> +        for ctx, attr, attr_ex in self.devices:
> +            with PD(ctx) as pd:
> +                check_dmabuf_mr_support(pd, self.unit)
> +                length = u.get_mr_length()
> +                flags = u.get_dmabuf_access_flags(ctx)
> +                for f in flags:
> +                    with DmaBufMR(pd, length, f, unit=self.unit,
> +                                  gtt=self.gtt) as mr:
> +                        mr.rkey
> +
> +
> +class DmaBufRC(RCResources):
> +    def __init__(self, dev_name, ib_port, gid_index, unit, gtt):
> +        """
> +        Initialize an DmaBufRC object.
> +        :param dev_name: Device name to be used
> +        :param ib_port: IB port of the device to use
> +        :param gid_index: Which GID index to use
> +        :param unit: GPU unit to allocate dmabuf from
> +        :gtt: Allocate dmabuf from GTT instead og VRAM
> +        """
> +        self.unit = unit
> +        self.gtt = gtt
> +        super(DmaBufRC, self).__init__(dev_name=dev_name, ib_port=ib_port,
> +                                       gid_index=gid_index)
> +
> +    def create_mr(self):
> +        check_dmabuf_support(self.unit)
> +        check_dmabuf_mr_support(self.pd, self.unit)
> +        access = e.IBV_ACCESS_LOCAL_WRITE | e.IBV_ACCESS_REMOTE_WRITE
> +        mr = DmaBufMR(self.pd, self.msg_size, access, unit=self.unit,
> +                      gtt=self.gtt)
> +        self.mr = mr
> +
> +    def create_qp_attr(self):
> +        qp_attr = QPAttr(port_num=self.ib_port)
> +        qp_access = e.IBV_ACCESS_LOCAL_WRITE | e.IBV_ACCESS_REMOTE_WRITE
> +        qp_attr.qp_access_flags = qp_access
> +        return qp_attr
> +
> +
> +class DmaBufTestCase(RDMATestCase):
> +    def setUp(self):
> +        super(DmaBufTestCase, self).setUp()
> +        self.iters = 100
> +        self.unit = self.config['gpu']
> +        self.gtt = self.config['gtt']
> +
> +    def create_players(self, resource, **resource_arg):
> +        """
> +        Init dma-buf tests resources.
> +        :param resource: The RDMA resources to use. A class of type
> +                         BaseResources.
> +        :param resource_arg: Dict of args that specify the resource specific
> +                             attributes.
> +        :return: The (client, server) resources.
> +        """
> +        client = resource(**self.dev_info, **resource_arg)
> +        server = resource(**self.dev_info, **resource_arg)
> +        client.pre_run(server.psns, server.qps_num)
> +        server.pre_run(client.psns, client.qps_num)
> +        return client, server
> +
> +    def test_dmabuf_rc_traffic(self):
> +        """
> +        Test send/recv using dma-buf MR over RC
> +        """
> +        client, server = self.create_players(DmaBufRC, unit=self.unit,
> +                                             gtt=self.gtt)
> +        u.traffic(client, server, self.iters, self.gid_index, self.ib_port)
> +
> +    def test_dmabuf_rdma_traffic(self):
> +        """
> +        Test rdma write using dma-buf MR
> +        """
> +        client, server = self.create_players(DmaBufRC, unit=self.unit,
> +                                             gtt=self.gtt)
> +        server.rkey = client.mr.rkey
> +        server.remote_addr = client.mr.offset
> +        client.rkey = server.mr.rkey
> +        client.remote_addr = server.mr.offset
> +        u.rdma_traffic(client, server, self.iters, self.gid_index, self.ib_port,
> +                       send_op=e.IBV_WR_RDMA_WRITE)
> diff --git a/tests/utils.py b/tests/utils.py
> index e2ab669..8546329 100644
> --- a/tests/utils.py
> +++ b/tests/utils.py
> @@ -1,5 +1,6 @@
>   # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
>   # Copyright (c) 2019 Mellanox Technologies, Inc. All rights reserved.  See COPYING file
> +# Copyright (c) 2020 Intel Corporation. All rights reserved. See COPYING file
>   """
>   Provide some useful helper function for pyverbs' tests.
>   """
> @@ -96,6 +97,31 @@ def get_access_flags(ctx):
>       return arr
>   
>   
> +def get_dmabuf_access_flags(ctx):
> +    """
> +    Similar to get_access_flags, except that dma-buf MR only support
> +    a subset of the flags.
> +    :param ctx: Device Context to check capabilities
> +    :return: A random legal value for MR flags
> +    """
> +    attr = ctx.query_device()
> +    vals = [e.IBV_ACCESS_LOCAL_WRITE, e.IBV_ACCESS_REMOTE_WRITE,
> +            e.IBV_ACCESS_REMOTE_READ, e.IBV_ACCESS_REMOTE_ATOMIC,
> +            e.IBV_ACCESS_RELAXED_ORDERING]
> +    if not attr.atomic_caps & e.IBV_ATOMIC_HCA:
> +        vals.remove(e.IBV_ACCESS_REMOTE_ATOMIC)
> +    arr = []
> +    for i in range(1, len(vals)):
> +        tmp = list(com(vals, i))
> +        tmp = filter(filter_illegal_access_flags, tmp)
> +        for t in tmp:  # Iterate legal combinations and bitwise OR them
> +            val = 0
> +            for flag in t:
> +                val += flag.value
> +            arr.append(val)
> +    return arr
> +
> +
>   def get_dm_attrs(dm_len):
>       """
>       Initializes an AllocDmAttr member with the given length and random
> 


^ permalink raw reply	[flat|nested] 42+ messages in thread
* Re: [PATCH rdma-core v7 5/6] tests: Add tests for dma-buf based memory regions
@ 2021-01-31  8:44     ` John Hubbard
  0 siblings, 0 replies; 42+ messages in thread
From: John Hubbard @ 2021-01-31  8:44 UTC (permalink / raw)
  To: Jianxin Xiong, linux-rdma, dri-devel
  Cc: Yishai Hadas, Leon Romanovsky, Edward Srouji, Christian Koenig,
	Jason Gunthorpe, Doug Ledford, Daniel Vetter

On 1/25/21 11:57 AM, Jianxin Xiong wrote:
> Define a set of unit tests similar to regular MR tests and a set of
> tests for send/recv and rdma traffic using dma-buf MRs. Add a utility
> function to generate access flags for dma-buf based MRs because the
> set of supported flags is smaller.
> 
> Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>

Hi Jianxin,

It's awesome to see a GPU to IB test suite here!

> ---
>   tests/args_parser.py |   4 +
>   tests/test_mr.py     | 266 ++++++++++++++++++++++++++++++++++++++++++++++++++-
>   tests/utils.py       |  26 +++++
>   3 files changed, 295 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/args_parser.py b/tests/args_parser.py
> index 446535a..5bc53b0 100644
> --- a/tests/args_parser.py
> +++ b/tests/args_parser.py
> @@ -19,6 +19,10 @@ class ArgsParser(object):
>           parser.add_argument('--port',
>                               help='Use port <port> of RDMA device', type=int,
>                               default=1)
> +        parser.add_argument('--gpu', nargs='?', type=int, const=0, default=0,
> +                            help='GPU unit to allocate dmabuf from')
> +        parser.add_argument('--gtt', action='store_true', default=False,
> +                            help='Allocate dmabuf from GTT instead of VRAM')

Just to be kind to non-GPU people, how about:

	s/GTT/GTT (Graphics Translation Table)/


>           parser.add_argument('-v', '--verbose', dest='verbosity',
>                               action='store_const',
>                               const=2, help='Verbose output')
> diff --git a/tests/test_mr.py b/tests/test_mr.py
> index b88ad23..03a645f 100644
> --- a/tests/test_mr.py
> +++ b/tests/test_mr.py
> @@ -1,5 +1,6 @@
>   # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
>   # Copyright (c) 2019 Mellanox Technologies, Inc. All rights reserved. See COPYING file
> +# Copyright (c) 2020 Intel Corporation. All rights reserved. See COPYING file
>   """
>   Test module for pyverbs' mr module.
>   """
> @@ -9,15 +10,18 @@ import errno
>   
>   from tests.base import PyverbsAPITestCase, RCResources, RDMATestCase
>   from pyverbs.pyverbs_error import PyverbsRDMAError, PyverbsError
> -from pyverbs.mr import MR, MW, DMMR, MWBindInfo, MWBind
> +from pyverbs.mr import MR, MW, DMMR, DmaBufMR, MWBindInfo, MWBind
>   from pyverbs.qp import QPCap, QPInitAttr, QPAttr, QP
>   from pyverbs.mem_alloc import posix_memalign, free
>   from pyverbs.wr import SendWR
> +from pyverbs.dmabuf import DmaBuf
>   import pyverbs.device as d
>   from pyverbs.pd import PD
>   import pyverbs.enums as e
>   import tests.utils as u
>   
> +MAX_IO_LEN = 1048576
> +
>   
>   class MRRes(RCResources):
>       def __init__(self, dev_name, ib_port, gid_index,
> @@ -423,3 +427,263 @@ class DMMRTest(PyverbsAPITestCase):
>                           dm_mr = DMMR(pd, dm_mr_len, e.IBV_ACCESS_ZERO_BASED,
>                                        dm=dm, offset=dm_mr_offset)
>                           dm_mr.close()
> +
> +
> +def check_dmabuf_support(unit=0):
> +    """
> +    Check if dma-buf allocation is supported by the system.
> +    Skip the test on failure.
> +    """
> +    device_num = 128 + unit
> +    try:
> +        DmaBuf(1, unit=unit)

unit?? This is a GPU, never anything else! Let's s/unit/gpu/ throughout, yes?

thanks,
-- 
John Hubbard
NVIDIA

> +    except PyverbsRDMAError as ex:
> +        if ex.error_code == errno.ENOENT:
> +            raise unittest.SkipTest(f'Device /dev/dri/renderD{device_num} is not present')
> +        if ex.error_code == errno.EACCES:
> +            raise unittest.SkipTest(f'Lack of permission to access /dev/dri/renderD{device_num}')
> +        if ex.error_code == errno.EOPNOTSUPP:
> +            raise unittest.SkipTest(f'Allocating dmabuf is not supported by /dev/dri/renderD{device_num}')
> +
> +
> +def check_dmabuf_mr_support(pd, unit=0):
> +    """
> +    Check if dma-buf MR registration is supported by the driver.
> +    Skip the test on failure
> +    """
> +    try:
> +        DmaBufMR(pd, 1, 0, unit=unit)
> +    except PyverbsRDMAError as ex:
> +        if ex.error_code == errno.EOPNOTSUPP:
> +            raise unittest.SkipTest('Reg dma-buf MR is not supported by the RDMA driver')
> +
> +
> +class DmaBufMRTest(PyverbsAPITestCase):
> +    """
> +    Test various functionalities of the DmaBufMR class.
> +    """
> +    def setUp(self):
> +        super().setUp()
> +        self.unit = self.config['gpu']
> +        self.gtt = self.config['gtt']
> +
> +    def test_dmabuf_reg_mr(self):
> +        """
> +        Test ibv_reg_dmabuf_mr()
> +        """
> +        check_dmabuf_support(self.unit)
> +        for ctx, attr, attr_ex in self.devices:
> +            with PD(ctx) as pd:
> +                check_dmabuf_mr_support(pd, self.unit)
> +                flags = u.get_dmabuf_access_flags(ctx)
> +                for f in flags:
> +                    len = u.get_mr_length()
> +                    for off in [0, len//2]:
> +                        with DmaBufMR(pd, len, f, offset=off, unit=self.unit,
> +                                      gtt=self.gtt) as mr:
> +                            pass
> +
> +    def test_dmabuf_dereg_mr(self):
> +        """
> +        Test ibv_dereg_mr() with DmaBufMR
> +        """
> +        check_dmabuf_support(self.unit)
> +        for ctx, attr, attr_ex in self.devices:
> +            with PD(ctx) as pd:
> +                check_dmabuf_mr_support(pd, self.unit)
> +                flags = u.get_dmabuf_access_flags(ctx)
> +                for f in flags:
> +                    len = u.get_mr_length()
> +                    for off in [0, len//2]:
> +                        with DmaBufMR(pd, len, f, offset=off, unit=self.unit,
> +                                      gtt=self.gtt) as mr:
> +                            mr.close()
> +
> +    def test_dmabuf_dereg_mr_twice(self):
> +        """
> +        Verify that explicit call to DmaBufMR's close() doesn't fail
> +        """
> +        check_dmabuf_support(self.unit)
> +        for ctx, attr, attr_ex in self.devices:
> +            with PD(ctx) as pd:
> +                check_dmabuf_mr_support(pd, self.unit)
> +                flags = u.get_dmabuf_access_flags(ctx)
> +                for f in flags:
> +                    len = u.get_mr_length()
> +                    for off in [0, len//2]:
> +                        with DmaBufMR(pd, len, f, offset=off, unit=self.unit,
> +                                      gtt=self.gtt) as mr:
> +                            # Pyverbs supports multiple destruction of objects,
> +                            # we are not expecting an exception here.
> +                            mr.close()
> +                            mr.close()
> +
> +    def test_dmabuf_reg_mr_bad_flags(self):
> +        """
> +        Verify that illegal flags combination fails as expected
> +        """
> +        check_dmabuf_support(self.unit)
> +        for ctx, attr, attr_ex in self.devices:
> +            with PD(ctx) as pd:
> +                check_dmabuf_mr_support(pd, self.unit)
> +                for i in range(5):
> +                    flags = random.sample([e.IBV_ACCESS_REMOTE_WRITE,
> +                                           e.IBV_ACCESS_REMOTE_ATOMIC],
> +                                          random.randint(1, 2))
> +                    mr_flags = 0
> +                    for i in flags:
> +                        mr_flags += i.value
> +                    try:
> +                        DmaBufMR(pd, u.get_mr_length(), mr_flags,
> +                                 unit=self.unit, gtt=self.gtt)
> +                    except PyverbsRDMAError as err:
> +                        assert 'Failed to register a dma-buf MR' in err.args[0]
> +                    else:
> +                        raise PyverbsRDMAError('Registered a dma-buf MR with illegal falgs')
> +
> +    def test_dmabuf_write(self):
> +        """
> +        Test writing to DmaBufMR's buffer
> +        """
> +        check_dmabuf_support(self.unit)
> +        for ctx, attr, attr_ex in self.devices:
> +            with PD(ctx) as pd:
> +                check_dmabuf_mr_support(pd, self.unit)
> +                for i in range(10):
> +                    mr_len = u.get_mr_length()
> +                    flags = u.get_dmabuf_access_flags(ctx)
> +                    for f in flags:
> +                        for mr_off in [0, mr_len//2]:
> +                            with DmaBufMR(pd, mr_len, f, offset=mr_off,
> +                                          unit=self.unit, gtt=self.gtt) as mr:
> +                                write_len = min(random.randint(1, MAX_IO_LEN),
> +                                                mr_len)
> +                                mr.write('a' * write_len, write_len)
> +
> +    def test_dmabuf_read(self):
> +        """
> +        Test reading from DmaBufMR's buffer
> +        """
> +        check_dmabuf_support(self.unit)
> +        for ctx, attr, attr_ex in self.devices:
> +            with PD(ctx) as pd:
> +                check_dmabuf_mr_support(pd, self.unit)
> +                for i in range(10):
> +                    mr_len = u.get_mr_length()
> +                    flags = u.get_dmabuf_access_flags(ctx)
> +                    for f in flags:
> +                        for mr_off in [0, mr_len//2]:
> +                            with DmaBufMR(pd, mr_len, f, offset=mr_off,
> +                                          unit=self.unit, gtt=self.gtt) as mr:
> +                                write_len = min(random.randint(1, MAX_IO_LEN),
> +                                                mr_len)
> +                                write_str = 'a' * write_len
> +                                mr.write(write_str, write_len)
> +                                read_len = random.randint(1, write_len)
> +                                offset = random.randint(0, write_len-read_len)
> +                                read_str = mr.read(read_len, offset).decode()
> +                                assert read_str in write_str
> +
> +    def test_dmabuf_lkey(self):
> +        """
> +        Test reading lkey property
> +        """
> +        check_dmabuf_support(self.unit)
> +        for ctx, attr, attr_ex in self.devices:
> +            with PD(ctx) as pd:
> +                check_dmabuf_mr_support(pd, self.unit)
> +                length = u.get_mr_length()
> +                flags = u.get_dmabuf_access_flags(ctx)
> +                for f in flags:
> +                    with DmaBufMR(pd, length, f, unit=self.unit,
> +                                  gtt=self.gtt) as mr:
> +                        mr.lkey
> +
> +    def test_dmabuf_rkey(self):
> +        """
> +        Test reading rkey property
> +        """
> +        check_dmabuf_support(self.unit)
> +        for ctx, attr, attr_ex in self.devices:
> +            with PD(ctx) as pd:
> +                check_dmabuf_mr_support(pd, self.unit)
> +                length = u.get_mr_length()
> +                flags = u.get_dmabuf_access_flags(ctx)
> +                for f in flags:
> +                    with DmaBufMR(pd, length, f, unit=self.unit,
> +                                  gtt=self.gtt) as mr:
> +                        mr.rkey
> +
> +
> +class DmaBufRC(RCResources):
> +    def __init__(self, dev_name, ib_port, gid_index, unit, gtt):
> +        """
> +        Initialize an DmaBufRC object.
> +        :param dev_name: Device name to be used
> +        :param ib_port: IB port of the device to use
> +        :param gid_index: Which GID index to use
> +        :param unit: GPU unit to allocate dmabuf from
> +        :gtt: Allocate dmabuf from GTT instead og VRAM
> +        """
> +        self.unit = unit
> +        self.gtt = gtt
> +        super(DmaBufRC, self).__init__(dev_name=dev_name, ib_port=ib_port,
> +                                       gid_index=gid_index)
> +
> +    def create_mr(self):
> +        check_dmabuf_support(self.unit)
> +        check_dmabuf_mr_support(self.pd, self.unit)
> +        access = e.IBV_ACCESS_LOCAL_WRITE | e.IBV_ACCESS_REMOTE_WRITE
> +        mr = DmaBufMR(self.pd, self.msg_size, access, unit=self.unit,
> +                      gtt=self.gtt)
> +        self.mr = mr
> +
> +    def create_qp_attr(self):
> +        qp_attr = QPAttr(port_num=self.ib_port)
> +        qp_access = e.IBV_ACCESS_LOCAL_WRITE | e.IBV_ACCESS_REMOTE_WRITE
> +        qp_attr.qp_access_flags = qp_access
> +        return qp_attr
> +
> +
> +class DmaBufTestCase(RDMATestCase):
> +    def setUp(self):
> +        super(DmaBufTestCase, self).setUp()
> +        self.iters = 100
> +        self.unit = self.config['gpu']
> +        self.gtt = self.config['gtt']
> +
> +    def create_players(self, resource, **resource_arg):
> +        """
> +        Init dma-buf tests resources.
> +        :param resource: The RDMA resources to use. A class of type
> +                         BaseResources.
> +        :param resource_arg: Dict of args that specify the resource specific
> +                             attributes.
> +        :return: The (client, server) resources.
> +        """
> +        client = resource(**self.dev_info, **resource_arg)
> +        server = resource(**self.dev_info, **resource_arg)
> +        client.pre_run(server.psns, server.qps_num)
> +        server.pre_run(client.psns, client.qps_num)
> +        return client, server
> +
> +    def test_dmabuf_rc_traffic(self):
> +        """
> +        Test send/recv using dma-buf MR over RC
> +        """
> +        client, server = self.create_players(DmaBufRC, unit=self.unit,
> +                                             gtt=self.gtt)
> +        u.traffic(client, server, self.iters, self.gid_index, self.ib_port)
> +
> +    def test_dmabuf_rdma_traffic(self):
> +        """
> +        Test rdma write using dma-buf MR
> +        """
> +        client, server = self.create_players(DmaBufRC, unit=self.unit,
> +                                             gtt=self.gtt)
> +        server.rkey = client.mr.rkey
> +        server.remote_addr = client.mr.offset
> +        client.rkey = server.mr.rkey
> +        client.remote_addr = server.mr.offset
> +        u.rdma_traffic(client, server, self.iters, self.gid_index, self.ib_port,
> +                       send_op=e.IBV_WR_RDMA_WRITE)
> diff --git a/tests/utils.py b/tests/utils.py
> index e2ab669..8546329 100644
> --- a/tests/utils.py
> +++ b/tests/utils.py
> @@ -1,5 +1,6 @@
>   # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
>   # Copyright (c) 2019 Mellanox Technologies, Inc. All rights reserved.  See COPYING file
> +# Copyright (c) 2020 Intel Corporation. All rights reserved. See COPYING file
>   """
>   Provide some useful helper function for pyverbs' tests.
>   """
> @@ -96,6 +97,31 @@ def get_access_flags(ctx):
>       return arr
>   
>   
> +def get_dmabuf_access_flags(ctx):
> +    """
> +    Similar to get_access_flags, except that dma-buf MR only support
> +    a subset of the flags.
> +    :param ctx: Device Context to check capabilities
> +    :return: A random legal value for MR flags
> +    """
> +    attr = ctx.query_device()
> +    vals = [e.IBV_ACCESS_LOCAL_WRITE, e.IBV_ACCESS_REMOTE_WRITE,
> +            e.IBV_ACCESS_REMOTE_READ, e.IBV_ACCESS_REMOTE_ATOMIC,
> +            e.IBV_ACCESS_RELAXED_ORDERING]
> +    if not attr.atomic_caps & e.IBV_ATOMIC_HCA:
> +        vals.remove(e.IBV_ACCESS_REMOTE_ATOMIC)
> +    arr = []
> +    for i in range(1, len(vals)):
> +        tmp = list(com(vals, i))
> +        tmp = filter(filter_illegal_access_flags, tmp)
> +        for t in tmp:  # Iterate legal combinations and bitwise OR them
> +            val = 0
> +            for flag in t:
> +                val += flag.value
> +            arr.append(val)
> +    return arr
> +
> +
>   def get_dm_attrs(dm_len):
>       """
>       Initializes an AllocDmAttr member with the given length and random
> 

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

^ permalink raw reply	[flat|nested] 42+ messages in thread
* Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
  2021-01-25 19:57   ` Jianxin Xiong
@ 2021-01-31 15:31     ` Gal Pressman
  -1 siblings, 0 replies; 42+ messages in thread
From: Gal Pressman @ 2021-01-31 15:31 UTC (permalink / raw)
  To: Jianxin Xiong, linux-rdma, dri-devel
  Cc: Doug Ledford, Jason Gunthorpe, Leon Romanovsky, Sumit Semwal,
	Christian Koenig, Daniel Vetter, Edward Srouji, Yishai Hadas

On 25/01/2021 21:57, Jianxin Xiong wrote:
> Define a new sub-class of 'MR' that uses dma-buf object for the memory
> region. Define a new class 'DmaBuf' as a wrapper for dma-buf allocation
> mechanism implemented in C.
> 
> Update the cmake function for cython modules to allow building modules
> with mixed cython and c source files.
> 
> Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
> ---
>  buildlib/pyverbs_functions.cmake |  78 +++++++----
>  pyverbs/CMakeLists.txt           |  11 +-
>  pyverbs/dmabuf.pxd               |  15 +++
>  pyverbs/dmabuf.pyx               |  73 ++++++++++
>  pyverbs/dmabuf_alloc.c           | 278 +++++++++++++++++++++++++++++++++++++++
>  pyverbs/dmabuf_alloc.h           |  19 +++
>  pyverbs/libibverbs.pxd           |   2 +
>  pyverbs/mr.pxd                   |   6 +
>  pyverbs/mr.pyx                   | 105 ++++++++++++++-
>  9 files changed, 557 insertions(+), 30 deletions(-)
>  create mode 100644 pyverbs/dmabuf.pxd
>  create mode 100644 pyverbs/dmabuf.pyx
>  create mode 100644 pyverbs/dmabuf_alloc.c
>  create mode 100644 pyverbs/dmabuf_alloc.h
> 
> diff --git a/buildlib/pyverbs_functions.cmake b/buildlib/pyverbs_functions.cmake
> index 953cec2..0792410 100644
> --- a/buildlib/pyverbs_functions.cmake
> +++ b/buildlib/pyverbs_functions.cmake
> @@ -1,35 +1,61 @@
>  # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
>  # Copyright (c) 2018, Mellanox Technologies. All rights reserved.  See COPYING file
> +# Copyright (c) 2020, Intel Corporation. All rights reserved.  See COPYING file
> +
> +function(build_module_from_cfiles PY_MODULE MODULE_NAME ALL_CFILES LINKER_FLAGS)
> +  string(REGEX REPLACE "\\.so$" "" SONAME "${MODULE_NAME}${CMAKE_PYTHON_SO_SUFFIX}")
> +  add_library(${SONAME} SHARED ${ALL_CFILES})
> +  set_target_properties(${SONAME} PROPERTIES
> +    COMPILE_FLAGS "${CMAKE_C_FLAGS} -fPIC -fno-strict-aliasing -Wno-unused-function -Wno-redundant-decls -Wno-shadow -Wno-cast-function-type -Wno-implicit-fallthrough -Wno-unknown-warning -Wno-unknown-warning-option -Wno-deprecated-declarations ${NO_VAR_TRACKING_FLAGS}"
> +    LIBRARY_OUTPUT_DIRECTORY "${BUILD_PYTHON}/${PY_MODULE}"
> +    PREFIX "")
> +  target_link_libraries(${SONAME} LINK_PRIVATE ${PYTHON_LIBRARIES} ibverbs rdmacm ${LINKER_FLAGS})
> +  install(TARGETS ${SONAME}
> +    DESTINATION ${CMAKE_INSTALL_PYTHON_ARCH_LIB}/${PY_MODULE})
> +endfunction()
>  
>  function(rdma_cython_module PY_MODULE LINKER_FLAGS)
> -  foreach(PYX_FILE ${ARGN})
> -    get_filename_component(FILENAME ${PYX_FILE} NAME_WE)
> -    get_filename_component(DIR ${PYX_FILE} DIRECTORY)
> -	if (DIR)
> -		set(PYX "${CMAKE_CURRENT_SOURCE_DIR}/${DIR}/${FILENAME}.pyx")
> -	else()
> -	    set(PYX "${CMAKE_CURRENT_SOURCE_DIR}/${FILENAME}.pyx")
> -	endif()
> -    set(CFILE "${CMAKE_CURRENT_BINARY_DIR}/${FILENAME}.c")
> -    include_directories(${PYTHON_INCLUDE_DIRS})
> -    add_custom_command(
> -      OUTPUT "${CFILE}"
> -      MAIN_DEPENDENCY "${PYX}"
> -      COMMAND ${CYTHON_EXECUTABLE} "${PYX}" -o "${CFILE}"
> -      "-I${PYTHON_INCLUDE_DIRS}"
> -      COMMENT "Cythonizing ${PYX}"
> +  set(ALL_CFILES "")
> +  set(MODULE_NAME "")
> +  foreach(SRC_FILE ${ARGN})
> +    get_filename_component(FILENAME ${SRC_FILE} NAME_WE)
> +    get_filename_component(DIR ${SRC_FILE} DIRECTORY)
> +    get_filename_component(EXT ${SRC_FILE} EXT)
> +    if (DIR)
> +      set(SRC_PATH "${CMAKE_CURRENT_SOURCE_DIR}/${DIR}")
> +    else()
> +      set(SRC_PATH "${CMAKE_CURRENT_SOURCE_DIR}")
> +    endif()
> +    if (${EXT} STREQUAL ".pyx")
> +      # each .pyx file starts a new module, finish the previous module first
> +      if (ALL_CFILES AND MODULE_NAME)
> +        build_module_from_cfiles(${PY_MODULE} ${MODULE_NAME} "${ALL_CFILES}" "${LINKER_FLAGS}")
> +      endif()
> +      set(PYX "${SRC_PATH}/${FILENAME}.pyx")
> +      set(CFILE "${CMAKE_CURRENT_BINARY_DIR}/${FILENAME}.c")
> +      include_directories(${PYTHON_INCLUDE_DIRS})
> +      add_custom_command(
> +        OUTPUT "${CFILE}"
> +        MAIN_DEPENDENCY "${PYX}"
> +        COMMAND ${CYTHON_EXECUTABLE} "${PYX}" -o "${CFILE}"
> +        "-I${PYTHON_INCLUDE_DIRS}"
> +        COMMENT "Cythonizing ${PYX}"
>        )
> -
> -    string(REGEX REPLACE "\\.so$" "" SONAME "${FILENAME}${CMAKE_PYTHON_SO_SUFFIX}")
> -    add_library(${SONAME} SHARED ${CFILE})
> -    set_target_properties(${SONAME} PROPERTIES
> -      COMPILE_FLAGS "${CMAKE_C_FLAGS} -fPIC -fno-strict-aliasing -Wno-unused-function -Wno-redundant-decls -Wno-shadow -Wno-cast-function-type -Wno-implicit-fallthrough -Wno-unknown-warning -Wno-unknown-warning-option -Wno-deprecated-declarations ${NO_VAR_TRACKING_FLAGS}"
> -      LIBRARY_OUTPUT_DIRECTORY "${BUILD_PYTHON}/${PY_MODULE}"
> -      PREFIX "")
> -    target_link_libraries(${SONAME} LINK_PRIVATE ${PYTHON_LIBRARIES} ibverbs rdmacm ${LINKER_FLAGS})
> -    install(TARGETS ${SONAME}
> -      DESTINATION ${CMAKE_INSTALL_PYTHON_ARCH_LIB}/${PY_MODULE})
> +      set(MODULE_NAME ${FILENAME})
> +      set(ALL_CFILES "${CFILE}")
> +    elseif(${EXT} STREQUAL ".c")
> +      # .c files belong to the same module as the most recent .pyx file,
> +      # ignored if appearing before all .pyx files
> +      set(CFILE "${SRC_PATH}/${FILENAME}.c")
> +      set(ALL_CFILES "${ALL_CFILES};${CFILE}")
> +    else()
> +      continue()
> +    endif()
>    endforeach()
> +  # finish the last module
> +  if (ALL_CFILES AND MODULE_NAME)
> +    build_module_from_cfiles(${PY_MODULE} ${MODULE_NAME} "${ALL_CFILES}" "${LINKER_FLAGS}")
> +  endif()
>  endfunction()
>  
>  function(rdma_python_module PY_MODULE)
> diff --git a/pyverbs/CMakeLists.txt b/pyverbs/CMakeLists.txt
> index 9542c4b..6fd7625 100644
> --- a/pyverbs/CMakeLists.txt
> +++ b/pyverbs/CMakeLists.txt
> @@ -1,5 +1,10 @@
>  # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
>  # Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file
> +# Copyright (c) 2020, Intel Corporation. All rights reserved. See COPYING file
> +
> +publish_internal_headers(""
> +  dmabuf_alloc.h
> +)
>  
>  rdma_cython_module(pyverbs ""
>    addr.pyx
> @@ -8,15 +13,17 @@ rdma_cython_module(pyverbs ""
>    cmid.pyx
>    cq.pyx
>    device.pyx
> +  dmabuf.pyx
> +  dmabuf_alloc.c
>    enums.pyx
>    mem_alloc.pyx
>    mr.pyx
>    pd.pyx
>    qp.pyx
> +  srq.pyx
>    wr.pyx
>    xrcd.pyx
> -  srq.pyx
> -  )
> +)
>  
>  rdma_python_module(pyverbs
>    __init__.py
> diff --git a/pyverbs/dmabuf.pxd b/pyverbs/dmabuf.pxd
> new file mode 100644
> index 0000000..a063acb
> --- /dev/null
> +++ b/pyverbs/dmabuf.pxd
> @@ -0,0 +1,15 @@
> +# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
> +# Copyright (c) 2020, Intel Corporation. All rights reserved. See COPYING file
> +
> +#cython: language_level=3
> +
> +cdef class DmaBuf:
> +    cdef int drm_fd
> +    cdef int handle
> +    cdef int fd
> +    cdef unsigned long size
> +    cdef unsigned long map_offset
> +    cdef void *dmabuf
> +    cdef object dmabuf_mrs
> +    cdef add_ref(self, obj)
> +    cpdef close(self)
> diff --git a/pyverbs/dmabuf.pyx b/pyverbs/dmabuf.pyx
> new file mode 100644
> index 0000000..b9406bd
> --- /dev/null
> +++ b/pyverbs/dmabuf.pyx
> @@ -0,0 +1,73 @@
> +# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
> +# Copyright (c) 2020, Intel Corporation. All rights reserved. See COPYING file
> +
> +#cython: language_level=3
> +
> +import weakref
> +
> +from pyverbs.base cimport close_weakrefs
> +from pyverbs.base import PyverbsRDMAErrno
> +from pyverbs.mr cimport DmaBufMR
> +
> +cdef extern from "dmabuf_alloc.h":
> +    cdef struct dmabuf:
> +        pass
> +    dmabuf *dmabuf_alloc(unsigned long size, int unit, int gtt)
> +    void dmabuf_free(dmabuf *dmabuf)
> +    int dmabuf_get_drm_fd(dmabuf *dmabuf)
> +    int dmabuf_get_fd(dmabuf *dmabuf)
> +    unsigned long dmabuf_get_offset(dmabuf *dmabuf)
> +
> +
> +cdef class DmaBuf:
> +    def __init__(self, size, unit=0, gtt=0):
> +        """
> +        Allocate DmaBuf object from a GPU device. This is done through the
> +        DRI device interface. Usually this requires the effective user id
> +        being a member of the 'render' group.
> +        :param size: The size (in number of bytes) of the buffer.
> +        :param unit: The unit number of the GPU to allocate the buffer from.
> +        :param gtt: Allocate from GTT instead of VRAM.
> +        :return: The newly created DmaBuf object on success.
> +        """
> +        self.dmabuf_mrs = weakref.WeakSet()
> +        self.dmabuf = dmabuf_alloc(size, unit, gtt)
> +        if self.dmabuf == NULL:
> +            raise PyverbsRDMAErrno(f'Failed to allocate dmabuf of size {size} on unit {unit}')
> +        self.drm_fd = dmabuf_get_drm_fd(<dmabuf *>self.dmabuf)
> +        self.fd = dmabuf_get_fd(<dmabuf *>self.dmabuf)
> +        self.map_offset = dmabuf_get_offset(<dmabuf *>self.dmabuf)
> +
> +    def __dealloc__(self):
> +        self.close()
> +
> +    cpdef close(self):
> +        if self.dmabuf == NULL:
> +            return None
> +        close_weakrefs([self.dmabuf_mrs])
> +        dmabuf_free(<dmabuf *>self.dmabuf)
> +        self.dmabuf = NULL
> +
> +    cdef add_ref(self, obj):
> +        if isinstance(obj, DmaBufMR):
> +            self.dmabuf_mrs.add(obj)
> +
> +    @property
> +    def drm_fd(self):
> +        return self.drm_fd
> +
> +    @property
> +    def handle(self):
> +        return self.handle
> +
> +    @property
> +    def fd(self):
> +        return self.fd
> +
> +    @property
> +    def size(self):
> +        return self.size
> +
> +    @property
> +    def map_offset(self):
> +        return self.map_offset
> diff --git a/pyverbs/dmabuf_alloc.c b/pyverbs/dmabuf_alloc.c
> new file mode 100644
> index 0000000..05eae75
> --- /dev/null
> +++ b/pyverbs/dmabuf_alloc.c
> @@ -0,0 +1,278 @@
> +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> +/*
> + * Copyright 2020 Intel Corporation. All rights reserved. See COPYING file
> + */
> +
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <stdint.h>
> +#include <unistd.h>
> +#include <string.h>
> +#include <errno.h>
> +#include <drm/drm.h>
> +#include <drm/i915_drm.h>
> +#include <drm/amdgpu_drm.h>
> +#include <drm/radeon_drm.h>

I assume these should come from the kernel headers package, right?
We're getting new compilation errors on SLES 15 for some reason, although the
package is installed.

FAILED: pyverbs/CMakeFiles/dmabuf.cpython-36m-x86_64-linux-gnu.dir/dmabuf_alloc.c.o 
/usr/bin/cc -Ddmabuf_cpython_36m_x86_64_linux_gnu_EXPORTS -Iinclude -I/usr/include/libnl3 -I/usr/include/python3.6m -Wall -Wextra -Wno-sign-compare -Wno-unused-parameter -Wmissing-prototypes -Wmissing-declarations -Wwrite-strings -Wformat=2 -Wformat-nonliteral -Wdate-time -Wnested-externs -Wshadow -Wstrict-prototypes -Wold-style-definition -Wredundant-decls -O2 -g  -fPIC    -Wall -Wextra -Wno-sign-compare -Wno-unused-parameter -Wmissing-prototypes -Wmissing-declarations -Wwrite-strings -Wformat=2 -Wformat-nonliteral -Wdate-time -Wnested-externs -Wshadow -Wstrict-prototypes -Wold-style-definition -Wredundant-decls -fPIC -fno-strict-aliasing -Wno-unused-function -Wno-redundant-decls -Wno-shadow -Wno-cast-function-type -Wno-implicit-fallthrough -Wno-unknown-warning -Wno-unknown-warning-option -Wno-deprecated-declarations  -fno-var-tracking-assignments -std=gnu11 -MD -MT pyverbs/CMakeFiles/dmabuf.cpython-36m-x86_64-linux-gnu.dir/dmabuf_alloc.c.o -MF pyverbs/CMakeFiles/dmabuf.cpython-36m-x86_64-linux-gnu.dir/dmabuf_alloc.c.o.d -o pyverbs/CMakeFiles/dmabuf.cpython-36m-x86_64-linux-gnu.dir/dmabuf_alloc.c.o   -c ../pyverbs/dmabuf_alloc.c
 ../pyverbs/dmabuf_alloc.c:12:10: fatal error: drm/drm.h: No such file or directory
  #include <drm/drm.h>
           ^~~~~~~~~~~
 compilation terminated.

^ permalink raw reply	[flat|nested] 42+ messages in thread
* Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
@ 2021-01-31 15:31     ` Gal Pressman
  0 siblings, 0 replies; 42+ messages in thread
From: Gal Pressman @ 2021-01-31 15:31 UTC (permalink / raw)
  To: Jianxin Xiong, linux-rdma, dri-devel
  Cc: Yishai Hadas, Leon Romanovsky, Edward Srouji, Christian Koenig,
	Jason Gunthorpe, Doug Ledford, Daniel Vetter

On 25/01/2021 21:57, Jianxin Xiong wrote:
> Define a new sub-class of 'MR' that uses dma-buf object for the memory
> region. Define a new class 'DmaBuf' as a wrapper for dma-buf allocation
> mechanism implemented in C.
> 
> Update the cmake function for cython modules to allow building modules
> with mixed cython and c source files.
> 
> Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
> ---
>  buildlib/pyverbs_functions.cmake |  78 +++++++----
>  pyverbs/CMakeLists.txt           |  11 +-
>  pyverbs/dmabuf.pxd               |  15 +++
>  pyverbs/dmabuf.pyx               |  73 ++++++++++
>  pyverbs/dmabuf_alloc.c           | 278 +++++++++++++++++++++++++++++++++++++++
>  pyverbs/dmabuf_alloc.h           |  19 +++
>  pyverbs/libibverbs.pxd           |   2 +
>  pyverbs/mr.pxd                   |   6 +
>  pyverbs/mr.pyx                   | 105 ++++++++++++++-
>  9 files changed, 557 insertions(+), 30 deletions(-)
>  create mode 100644 pyverbs/dmabuf.pxd
>  create mode 100644 pyverbs/dmabuf.pyx
>  create mode 100644 pyverbs/dmabuf_alloc.c
>  create mode 100644 pyverbs/dmabuf_alloc.h
> 
> diff --git a/buildlib/pyverbs_functions.cmake b/buildlib/pyverbs_functions.cmake
> index 953cec2..0792410 100644
> --- a/buildlib/pyverbs_functions.cmake
> +++ b/buildlib/pyverbs_functions.cmake
> @@ -1,35 +1,61 @@
>  # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
>  # Copyright (c) 2018, Mellanox Technologies. All rights reserved.  See COPYING file
> +# Copyright (c) 2020, Intel Corporation. All rights reserved.  See COPYING file
> +
> +function(build_module_from_cfiles PY_MODULE MODULE_NAME ALL_CFILES LINKER_FLAGS)
> +  string(REGEX REPLACE "\\.so$" "" SONAME "${MODULE_NAME}${CMAKE_PYTHON_SO_SUFFIX}")
> +  add_library(${SONAME} SHARED ${ALL_CFILES})
> +  set_target_properties(${SONAME} PROPERTIES
> +    COMPILE_FLAGS "${CMAKE_C_FLAGS} -fPIC -fno-strict-aliasing -Wno-unused-function -Wno-redundant-decls -Wno-shadow -Wno-cast-function-type -Wno-implicit-fallthrough -Wno-unknown-warning -Wno-unknown-warning-option -Wno-deprecated-declarations ${NO_VAR_TRACKING_FLAGS}"
> +    LIBRARY_OUTPUT_DIRECTORY "${BUILD_PYTHON}/${PY_MODULE}"
> +    PREFIX "")
> +  target_link_libraries(${SONAME} LINK_PRIVATE ${PYTHON_LIBRARIES} ibverbs rdmacm ${LINKER_FLAGS})
> +  install(TARGETS ${SONAME}
> +    DESTINATION ${CMAKE_INSTALL_PYTHON_ARCH_LIB}/${PY_MODULE})
> +endfunction()
>  
>  function(rdma_cython_module PY_MODULE LINKER_FLAGS)
> -  foreach(PYX_FILE ${ARGN})
> -    get_filename_component(FILENAME ${PYX_FILE} NAME_WE)
> -    get_filename_component(DIR ${PYX_FILE} DIRECTORY)
> -	if (DIR)
> -		set(PYX "${CMAKE_CURRENT_SOURCE_DIR}/${DIR}/${FILENAME}.pyx")
> -	else()
> -	    set(PYX "${CMAKE_CURRENT_SOURCE_DIR}/${FILENAME}.pyx")
> -	endif()
> -    set(CFILE "${CMAKE_CURRENT_BINARY_DIR}/${FILENAME}.c")
> -    include_directories(${PYTHON_INCLUDE_DIRS})
> -    add_custom_command(
> -      OUTPUT "${CFILE}"
> -      MAIN_DEPENDENCY "${PYX}"
> -      COMMAND ${CYTHON_EXECUTABLE} "${PYX}" -o "${CFILE}"
> -      "-I${PYTHON_INCLUDE_DIRS}"
> -      COMMENT "Cythonizing ${PYX}"
> +  set(ALL_CFILES "")
> +  set(MODULE_NAME "")
> +  foreach(SRC_FILE ${ARGN})
> +    get_filename_component(FILENAME ${SRC_FILE} NAME_WE)
> +    get_filename_component(DIR ${SRC_FILE} DIRECTORY)
> +    get_filename_component(EXT ${SRC_FILE} EXT)
> +    if (DIR)
> +      set(SRC_PATH "${CMAKE_CURRENT_SOURCE_DIR}/${DIR}")
> +    else()
> +      set(SRC_PATH "${CMAKE_CURRENT_SOURCE_DIR}")
> +    endif()
> +    if (${EXT} STREQUAL ".pyx")
> +      # each .pyx file starts a new module, finish the previous module first
> +      if (ALL_CFILES AND MODULE_NAME)
> +        build_module_from_cfiles(${PY_MODULE} ${MODULE_NAME} "${ALL_CFILES}" "${LINKER_FLAGS}")
> +      endif()
> +      set(PYX "${SRC_PATH}/${FILENAME}.pyx")
> +      set(CFILE "${CMAKE_CURRENT_BINARY_DIR}/${FILENAME}.c")
> +      include_directories(${PYTHON_INCLUDE_DIRS})
> +      add_custom_command(
> +        OUTPUT "${CFILE}"
> +        MAIN_DEPENDENCY "${PYX}"
> +        COMMAND ${CYTHON_EXECUTABLE} "${PYX}" -o "${CFILE}"
> +        "-I${PYTHON_INCLUDE_DIRS}"
> +        COMMENT "Cythonizing ${PYX}"
>        )
> -
> -    string(REGEX REPLACE "\\.so$" "" SONAME "${FILENAME}${CMAKE_PYTHON_SO_SUFFIX}")
> -    add_library(${SONAME} SHARED ${CFILE})
> -    set_target_properties(${SONAME} PROPERTIES
> -      COMPILE_FLAGS "${CMAKE_C_FLAGS} -fPIC -fno-strict-aliasing -Wno-unused-function -Wno-redundant-decls -Wno-shadow -Wno-cast-function-type -Wno-implicit-fallthrough -Wno-unknown-warning -Wno-unknown-warning-option -Wno-deprecated-declarations ${NO_VAR_TRACKING_FLAGS}"
> -      LIBRARY_OUTPUT_DIRECTORY "${BUILD_PYTHON}/${PY_MODULE}"
> -      PREFIX "")
> -    target_link_libraries(${SONAME} LINK_PRIVATE ${PYTHON_LIBRARIES} ibverbs rdmacm ${LINKER_FLAGS})
> -    install(TARGETS ${SONAME}
> -      DESTINATION ${CMAKE_INSTALL_PYTHON_ARCH_LIB}/${PY_MODULE})
> +      set(MODULE_NAME ${FILENAME})
> +      set(ALL_CFILES "${CFILE}")
> +    elseif(${EXT} STREQUAL ".c")
> +      # .c files belong to the same module as the most recent .pyx file,
> +      # ignored if appearing before all .pyx files
> +      set(CFILE "${SRC_PATH}/${FILENAME}.c")
> +      set(ALL_CFILES "${ALL_CFILES};${CFILE}")
> +    else()
> +      continue()
> +    endif()
>    endforeach()
> +  # finish the last module
> +  if (ALL_CFILES AND MODULE_NAME)
> +    build_module_from_cfiles(${PY_MODULE} ${MODULE_NAME} "${ALL_CFILES}" "${LINKER_FLAGS}")
> +  endif()
>  endfunction()
>  
>  function(rdma_python_module PY_MODULE)
> diff --git a/pyverbs/CMakeLists.txt b/pyverbs/CMakeLists.txt
> index 9542c4b..6fd7625 100644
> --- a/pyverbs/CMakeLists.txt
> +++ b/pyverbs/CMakeLists.txt
> @@ -1,5 +1,10 @@
>  # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
>  # Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file
> +# Copyright (c) 2020, Intel Corporation. All rights reserved. See COPYING file
> +
> +publish_internal_headers(""
> +  dmabuf_alloc.h
> +)
>  
>  rdma_cython_module(pyverbs ""
>    addr.pyx
> @@ -8,15 +13,17 @@ rdma_cython_module(pyverbs ""
>    cmid.pyx
>    cq.pyx
>    device.pyx
> +  dmabuf.pyx
> +  dmabuf_alloc.c
>    enums.pyx
>    mem_alloc.pyx
>    mr.pyx
>    pd.pyx
>    qp.pyx
> +  srq.pyx
>    wr.pyx
>    xrcd.pyx
> -  srq.pyx
> -  )
> +)
>  
>  rdma_python_module(pyverbs
>    __init__.py
> diff --git a/pyverbs/dmabuf.pxd b/pyverbs/dmabuf.pxd
> new file mode 100644
> index 0000000..a063acb
> --- /dev/null
> +++ b/pyverbs/dmabuf.pxd
> @@ -0,0 +1,15 @@
> +# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
> +# Copyright (c) 2020, Intel Corporation. All rights reserved. See COPYING file
> +
> +#cython: language_level=3
> +
> +cdef class DmaBuf:
> +    cdef int drm_fd
> +    cdef int handle
> +    cdef int fd
> +    cdef unsigned long size
> +    cdef unsigned long map_offset
> +    cdef void *dmabuf
> +    cdef object dmabuf_mrs
> +    cdef add_ref(self, obj)
> +    cpdef close(self)
> diff --git a/pyverbs/dmabuf.pyx b/pyverbs/dmabuf.pyx
> new file mode 100644
> index 0000000..b9406bd
> --- /dev/null
> +++ b/pyverbs/dmabuf.pyx
> @@ -0,0 +1,73 @@
> +# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
> +# Copyright (c) 2020, Intel Corporation. All rights reserved. See COPYING file
> +
> +#cython: language_level=3
> +
> +import weakref
> +
> +from pyverbs.base cimport close_weakrefs
> +from pyverbs.base import PyverbsRDMAErrno
> +from pyverbs.mr cimport DmaBufMR
> +
> +cdef extern from "dmabuf_alloc.h":
> +    cdef struct dmabuf:
> +        pass
> +    dmabuf *dmabuf_alloc(unsigned long size, int unit, int gtt)
> +    void dmabuf_free(dmabuf *dmabuf)
> +    int dmabuf_get_drm_fd(dmabuf *dmabuf)
> +    int dmabuf_get_fd(dmabuf *dmabuf)
> +    unsigned long dmabuf_get_offset(dmabuf *dmabuf)
> +
> +
> +cdef class DmaBuf:
> +    def __init__(self, size, unit=0, gtt=0):
> +        """
> +        Allocate DmaBuf object from a GPU device. This is done through the
> +        DRI device interface. Usually this requires the effective user id
> +        being a member of the 'render' group.
> +        :param size: The size (in number of bytes) of the buffer.
> +        :param unit: The unit number of the GPU to allocate the buffer from.
> +        :param gtt: Allocate from GTT instead of VRAM.
> +        :return: The newly created DmaBuf object on success.
> +        """
> +        self.dmabuf_mrs = weakref.WeakSet()
> +        self.dmabuf = dmabuf_alloc(size, unit, gtt)
> +        if self.dmabuf == NULL:
> +            raise PyverbsRDMAErrno(f'Failed to allocate dmabuf of size {size} on unit {unit}')
> +        self.drm_fd = dmabuf_get_drm_fd(<dmabuf *>self.dmabuf)
> +        self.fd = dmabuf_get_fd(<dmabuf *>self.dmabuf)
> +        self.map_offset = dmabuf_get_offset(<dmabuf *>self.dmabuf)
> +
> +    def __dealloc__(self):
> +        self.close()
> +
> +    cpdef close(self):
> +        if self.dmabuf == NULL:
> +            return None
> +        close_weakrefs([self.dmabuf_mrs])
> +        dmabuf_free(<dmabuf *>self.dmabuf)
> +        self.dmabuf = NULL
> +
> +    cdef add_ref(self, obj):
> +        if isinstance(obj, DmaBufMR):
> +            self.dmabuf_mrs.add(obj)
> +
> +    @property
> +    def drm_fd(self):
> +        return self.drm_fd
> +
> +    @property
> +    def handle(self):
> +        return self.handle
> +
> +    @property
> +    def fd(self):
> +        return self.fd
> +
> +    @property
> +    def size(self):
> +        return self.size
> +
> +    @property
> +    def map_offset(self):
> +        return self.map_offset
> diff --git a/pyverbs/dmabuf_alloc.c b/pyverbs/dmabuf_alloc.c
> new file mode 100644
> index 0000000..05eae75
> --- /dev/null
> +++ b/pyverbs/dmabuf_alloc.c
> @@ -0,0 +1,278 @@
> +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> +/*
> + * Copyright 2020 Intel Corporation. All rights reserved. See COPYING file
> + */
> +
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <stdint.h>
> +#include <unistd.h>
> +#include <string.h>
> +#include <errno.h>
> +#include <drm/drm.h>
> +#include <drm/i915_drm.h>
> +#include <drm/amdgpu_drm.h>
> +#include <drm/radeon_drm.h>

I assume these should come from the kernel headers package, right?
We're getting new compilation errors on SLES 15 for some reason, although the
package is installed.

FAILED: pyverbs/CMakeFiles/dmabuf.cpython-36m-x86_64-linux-gnu.dir/dmabuf_alloc.c.o 
/usr/bin/cc -Ddmabuf_cpython_36m_x86_64_linux_gnu_EXPORTS -Iinclude -I/usr/include/libnl3 -I/usr/include/python3.6m -Wall -Wextra -Wno-sign-compare -Wno-unused-parameter -Wmissing-prototypes -Wmissing-declarations -Wwrite-strings -Wformat=2 -Wformat-nonliteral -Wdate-time -Wnested-externs -Wshadow -Wstrict-prototypes -Wold-style-definition -Wredundant-decls -O2 -g  -fPIC    -Wall -Wextra -Wno-sign-compare -Wno-unused-parameter -Wmissing-prototypes -Wmissing-declarations -Wwrite-strings -Wformat=2 -Wformat-nonliteral -Wdate-time -Wnested-externs -Wshadow -Wstrict-prototypes -Wold-style-definition -Wredundant-decls -fPIC -fno-strict-aliasing -Wno-unused-function -Wno-redundant-decls -Wno-shadow -Wno-cast-function-type -Wno-implicit-fallthrough -Wno-unknown-warning -Wno-unknown-warning-option -Wno-deprecated-declarations  -fno-var-tracking-assignments -std=gnu11 -MD -MT pyverbs/CMakeFiles/dmabuf.cpython-36m-x86_64-linux-gnu.dir/dmabuf_alloc.c.o -MF pyverbs/CMakeFiles/dmabuf.cpython-36m-
 x86_64-linux-gnu.dir/dmabuf_alloc.c.o.d -o pyverbs/CMakeFiles/dmabuf.cpython-36m-x86_64-linux-gnu.dir/dmabuf_alloc.c.o   -c ../pyverbs/dmabuf_alloc.c
 ../pyverbs/dmabuf_alloc.c:12:10: fatal error: drm/drm.h: No such file or directory
  #include <drm/drm.h>
           ^~~~~~~~~~~
 compilation terminated.
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

^ permalink raw reply	[flat|nested] 42+ messages in thread
* Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
  2021-01-31 15:31     ` Gal Pressman
@ 2021-02-01  6:16       ` Leon Romanovsky
  -1 siblings, 0 replies; 42+ messages in thread
From: Leon Romanovsky @ 2021-02-01  6:16 UTC (permalink / raw)
  To: Gal Pressman, Jianxin Xiong
  Cc: linux-rdma, dri-devel, Doug Ledford, Jason Gunthorpe,
	Sumit Semwal, Christian Koenig, Daniel Vetter, Edward Srouji,
	Yishai Hadas

On Sun, Jan 31, 2021 at 05:31:16PM +0200, Gal Pressman wrote:
> On 25/01/2021 21:57, Jianxin Xiong wrote:
> > Define a new sub-class of 'MR' that uses dma-buf object for the memory
> > region. Define a new class 'DmaBuf' as a wrapper for dma-buf allocation
> > mechanism implemented in C.
> >
> > Update the cmake function for cython modules to allow building modules
> > with mixed cython and c source files.
> >
> > Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
> > ---
> >  buildlib/pyverbs_functions.cmake |  78 +++++++----
> >  pyverbs/CMakeLists.txt           |  11 +-
> >  pyverbs/dmabuf.pxd               |  15 +++
> >  pyverbs/dmabuf.pyx               |  73 ++++++++++
> >  pyverbs/dmabuf_alloc.c           | 278 +++++++++++++++++++++++++++++++++++++++
> >  pyverbs/dmabuf_alloc.h           |  19 +++
> >  pyverbs/libibverbs.pxd           |   2 +
> >  pyverbs/mr.pxd                   |   6 +
> >  pyverbs/mr.pyx                   | 105 ++++++++++++++-
> >  9 files changed, 557 insertions(+), 30 deletions(-)
> >  create mode 100644 pyverbs/dmabuf.pxd
> >  create mode 100644 pyverbs/dmabuf.pyx
> >  create mode 100644 pyverbs/dmabuf_alloc.c
> >  create mode 100644 pyverbs/dmabuf_alloc.h

<...>

> > index 0000000..05eae75
> > --- /dev/null
> > +++ b/pyverbs/dmabuf_alloc.c
> > @@ -0,0 +1,278 @@
> > +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> > +/*
> > + * Copyright 2020 Intel Corporation. All rights reserved. See COPYING file
> > + */
> > +
> > +#include <stdio.h>
> > +#include <stdlib.h>
> > +#include <stdint.h>
> > +#include <unistd.h>
> > +#include <string.h>
> > +#include <errno.h>
> > +#include <drm/drm.h>
> > +#include <drm/i915_drm.h>
> > +#include <drm/amdgpu_drm.h>
> > +#include <drm/radeon_drm.h>
>
> I assume these should come from the kernel headers package, right?

This is gross, all kernel headers should be placed in kernel-headers/*
and "update" script needs to be extended to take drm/* files too :(.

Jianxin, are you fixing it?

Thanks

^ permalink raw reply	[flat|nested] 42+ messages in thread
* Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
@ 2021-02-01  6:16       ` Leon Romanovsky
  0 siblings, 0 replies; 42+ messages in thread
From: Leon Romanovsky @ 2021-02-01  6:16 UTC (permalink / raw)
  To: Gal Pressman, Jianxin Xiong
  Cc: Yishai Hadas, linux-rdma, Edward Srouji, dri-devel,
	Christian Koenig, Jason Gunthorpe, Doug Ledford, Daniel Vetter

On Sun, Jan 31, 2021 at 05:31:16PM +0200, Gal Pressman wrote:
> On 25/01/2021 21:57, Jianxin Xiong wrote:
> > Define a new sub-class of 'MR' that uses dma-buf object for the memory
> > region. Define a new class 'DmaBuf' as a wrapper for dma-buf allocation
> > mechanism implemented in C.
> >
> > Update the cmake function for cython modules to allow building modules
> > with mixed cython and c source files.
> >
> > Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
> > ---
> >  buildlib/pyverbs_functions.cmake |  78 +++++++----
> >  pyverbs/CMakeLists.txt           |  11 +-
> >  pyverbs/dmabuf.pxd               |  15 +++
> >  pyverbs/dmabuf.pyx               |  73 ++++++++++
> >  pyverbs/dmabuf_alloc.c           | 278 +++++++++++++++++++++++++++++++++++++++
> >  pyverbs/dmabuf_alloc.h           |  19 +++
> >  pyverbs/libibverbs.pxd           |   2 +
> >  pyverbs/mr.pxd                   |   6 +
> >  pyverbs/mr.pyx                   | 105 ++++++++++++++-
> >  9 files changed, 557 insertions(+), 30 deletions(-)
> >  create mode 100644 pyverbs/dmabuf.pxd
> >  create mode 100644 pyverbs/dmabuf.pyx
> >  create mode 100644 pyverbs/dmabuf_alloc.c
> >  create mode 100644 pyverbs/dmabuf_alloc.h

<...>

> > index 0000000..05eae75
> > --- /dev/null
> > +++ b/pyverbs/dmabuf_alloc.c
> > @@ -0,0 +1,278 @@
> > +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> > +/*
> > + * Copyright 2020 Intel Corporation. All rights reserved. See COPYING file
> > + */
> > +
> > +#include <stdio.h>
> > +#include <stdlib.h>
> > +#include <stdint.h>
> > +#include <unistd.h>
> > +#include <string.h>
> > +#include <errno.h>
> > +#include <drm/drm.h>
> > +#include <drm/i915_drm.h>
> > +#include <drm/amdgpu_drm.h>
> > +#include <drm/radeon_drm.h>
>
> I assume these should come from the kernel headers package, right?

This is gross, all kernel headers should be placed in kernel-headers/*
and "update" script needs to be extended to take drm/* files too :(.

Jianxin, are you fixing it?

Thanks
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

^ permalink raw reply	[flat|nested] 42+ messages in thread
* Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
  2021-02-01  6:16       ` Leon Romanovsky
@ 2021-02-01 14:10         ` Daniel Vetter
  -1 siblings, 0 replies; 42+ messages in thread
From: Daniel Vetter @ 2021-02-01 14:10 UTC (permalink / raw)
  To: Leon Romanovsky
  Cc: Gal Pressman, Jianxin Xiong, Yishai Hadas, linux-rdma,
	Edward Srouji, dri-devel, Christian Koenig, Jason Gunthorpe,
	Doug Ledford, Daniel Vetter

On Mon, Feb 1, 2021 at 7:16 AM Leon Romanovsky <leon@kernel.org> wrote:
>
> On Sun, Jan 31, 2021 at 05:31:16PM +0200, Gal Pressman wrote:
> > On 25/01/2021 21:57, Jianxin Xiong wrote:
> > > Define a new sub-class of 'MR' that uses dma-buf object for the memory
> > > region. Define a new class 'DmaBuf' as a wrapper for dma-buf allocation
> > > mechanism implemented in C.
> > >
> > > Update the cmake function for cython modules to allow building modules
> > > with mixed cython and c source files.
> > >
> > > Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
> > > ---
> > >  buildlib/pyverbs_functions.cmake |  78 +++++++----
> > >  pyverbs/CMakeLists.txt           |  11 +-
> > >  pyverbs/dmabuf.pxd               |  15 +++
> > >  pyverbs/dmabuf.pyx               |  73 ++++++++++
> > >  pyverbs/dmabuf_alloc.c           | 278 +++++++++++++++++++++++++++++++++++++++
> > >  pyverbs/dmabuf_alloc.h           |  19 +++
> > >  pyverbs/libibverbs.pxd           |   2 +
> > >  pyverbs/mr.pxd                   |   6 +
> > >  pyverbs/mr.pyx                   | 105 ++++++++++++++-
> > >  9 files changed, 557 insertions(+), 30 deletions(-)
> > >  create mode 100644 pyverbs/dmabuf.pxd
> > >  create mode 100644 pyverbs/dmabuf.pyx
> > >  create mode 100644 pyverbs/dmabuf_alloc.c
> > >  create mode 100644 pyverbs/dmabuf_alloc.h
>
> <...>
>
> > > index 0000000..05eae75
> > > --- /dev/null
> > > +++ b/pyverbs/dmabuf_alloc.c
> > > @@ -0,0 +1,278 @@
> > > +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> > > +/*
> > > + * Copyright 2020 Intel Corporation. All rights reserved. See COPYING file
> > > + */
> > > +
> > > +#include <stdio.h>
> > > +#include <stdlib.h>
> > > +#include <stdint.h>
> > > +#include <unistd.h>
> > > +#include <string.h>
> > > +#include <errno.h>
> > > +#include <drm/drm.h>
> > > +#include <drm/i915_drm.h>
> > > +#include <drm/amdgpu_drm.h>
> > > +#include <drm/radeon_drm.h>
> >
> > I assume these should come from the kernel headers package, right?
>
> This is gross, all kernel headers should be placed in kernel-headers/*
> and "update" script needs to be extended to take drm/* files too :(.

drm kernel headers are in the libdrm package. You need that anyway for
doing the ioctls (if you don't hand-roll the restarting yourself).

Also our userspace has gone over to just outright copying the driver
headers. Not the generic headers, but for the rendering side of gpus,
which is the topic here, there's really not much generic stuff.

> Jianxin, are you fixing it?

So fix is either to depend upon libdrm for building, or have copies of
the headers included in the package for the i915/amdgpu/radeon headers
(drm/drm.h probably not so good idea).

Cheers, Daniel
>
> Thanks
> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel



-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

^ permalink raw reply	[flat|nested] 42+ messages in thread
* Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
@ 2021-02-01 14:10         ` Daniel Vetter
  0 siblings, 0 replies; 42+ messages in thread
From: Daniel Vetter @ 2021-02-01 14:10 UTC (permalink / raw)
  To: Leon Romanovsky
  Cc: Yishai Hadas, linux-rdma, Edward Srouji, Gal Pressman, dri-devel,
	Jason Gunthorpe, Doug Ledford, Daniel Vetter, Christian Koenig,
	Jianxin Xiong

On Mon, Feb 1, 2021 at 7:16 AM Leon Romanovsky <leon@kernel.org> wrote:
>
> On Sun, Jan 31, 2021 at 05:31:16PM +0200, Gal Pressman wrote:
> > On 25/01/2021 21:57, Jianxin Xiong wrote:
> > > Define a new sub-class of 'MR' that uses dma-buf object for the memory
> > > region. Define a new class 'DmaBuf' as a wrapper for dma-buf allocation
> > > mechanism implemented in C.
> > >
> > > Update the cmake function for cython modules to allow building modules
> > > with mixed cython and c source files.
> > >
> > > Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
> > > ---
> > >  buildlib/pyverbs_functions.cmake |  78 +++++++----
> > >  pyverbs/CMakeLists.txt           |  11 +-
> > >  pyverbs/dmabuf.pxd               |  15 +++
> > >  pyverbs/dmabuf.pyx               |  73 ++++++++++
> > >  pyverbs/dmabuf_alloc.c           | 278 +++++++++++++++++++++++++++++++++++++++
> > >  pyverbs/dmabuf_alloc.h           |  19 +++
> > >  pyverbs/libibverbs.pxd           |   2 +
> > >  pyverbs/mr.pxd                   |   6 +
> > >  pyverbs/mr.pyx                   | 105 ++++++++++++++-
> > >  9 files changed, 557 insertions(+), 30 deletions(-)
> > >  create mode 100644 pyverbs/dmabuf.pxd
> > >  create mode 100644 pyverbs/dmabuf.pyx
> > >  create mode 100644 pyverbs/dmabuf_alloc.c
> > >  create mode 100644 pyverbs/dmabuf_alloc.h
>
> <...>
>
> > > index 0000000..05eae75
> > > --- /dev/null
> > > +++ b/pyverbs/dmabuf_alloc.c
> > > @@ -0,0 +1,278 @@
> > > +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> > > +/*
> > > + * Copyright 2020 Intel Corporation. All rights reserved. See COPYING file
> > > + */
> > > +
> > > +#include <stdio.h>
> > > +#include <stdlib.h>
> > > +#include <stdint.h>
> > > +#include <unistd.h>
> > > +#include <string.h>
> > > +#include <errno.h>
> > > +#include <drm/drm.h>
> > > +#include <drm/i915_drm.h>
> > > +#include <drm/amdgpu_drm.h>
> > > +#include <drm/radeon_drm.h>
> >
> > I assume these should come from the kernel headers package, right?
>
> This is gross, all kernel headers should be placed in kernel-headers/*
> and "update" script needs to be extended to take drm/* files too :(.

drm kernel headers are in the libdrm package. You need that anyway for
doing the ioctls (if you don't hand-roll the restarting yourself).

Also our userspace has gone over to just outright copying the driver
headers. Not the generic headers, but for the rendering side of gpus,
which is the topic here, there's really not much generic stuff.

> Jianxin, are you fixing it?

So fix is either to depend upon libdrm for building, or have copies of
the headers included in the package for the i915/amdgpu/radeon headers
(drm/drm.h probably not so good idea).

Cheers, Daniel
>
> Thanks
> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel



-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

^ permalink raw reply	[flat|nested] 42+ messages in thread
* Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
  2021-02-01 14:10         ` Daniel Vetter
@ 2021-02-01 15:29           ` Jason Gunthorpe
  -1 siblings, 0 replies; 42+ messages in thread
From: Jason Gunthorpe @ 2021-02-01 15:29 UTC (permalink / raw)
  To: Daniel Vetter
  Cc: Leon Romanovsky, Gal Pressman, Jianxin Xiong, Yishai Hadas,
	linux-rdma, Edward Srouji, dri-devel, Christian Koenig,
	Doug Ledford, Daniel Vetter

On Mon, Feb 01, 2021 at 03:10:00PM +0100, Daniel Vetter wrote:
> On Mon, Feb 1, 2021 at 7:16 AM Leon Romanovsky <leon@kernel.org> wrote:
> >
> > On Sun, Jan 31, 2021 at 05:31:16PM +0200, Gal Pressman wrote:
> > > On 25/01/2021 21:57, Jianxin Xiong wrote:
> > > > Define a new sub-class of 'MR' that uses dma-buf object for the memory
> > > > region. Define a new class 'DmaBuf' as a wrapper for dma-buf allocation
> > > > mechanism implemented in C.
> > > >
> > > > Update the cmake function for cython modules to allow building modules
> > > > with mixed cython and c source files.
> > > >
> > > > Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
> > > >  buildlib/pyverbs_functions.cmake |  78 +++++++----
> > > >  pyverbs/CMakeLists.txt           |  11 +-
> > > >  pyverbs/dmabuf.pxd               |  15 +++
> > > >  pyverbs/dmabuf.pyx               |  73 ++++++++++
> > > >  pyverbs/dmabuf_alloc.c           | 278 +++++++++++++++++++++++++++++++++++++++
> > > >  pyverbs/dmabuf_alloc.h           |  19 +++
> > > >  pyverbs/libibverbs.pxd           |   2 +
> > > >  pyverbs/mr.pxd                   |   6 +
> > > >  pyverbs/mr.pyx                   | 105 ++++++++++++++-
> > > >  9 files changed, 557 insertions(+), 30 deletions(-)
> > > >  create mode 100644 pyverbs/dmabuf.pxd
> > > >  create mode 100644 pyverbs/dmabuf.pyx
> > > >  create mode 100644 pyverbs/dmabuf_alloc.c
> > > >  create mode 100644 pyverbs/dmabuf_alloc.h
> >
> > <...>
> >
> > > > index 0000000..05eae75
> > > > +++ b/pyverbs/dmabuf_alloc.c
> > > > @@ -0,0 +1,278 @@
> > > > +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> > > > +/*
> > > > + * Copyright 2020 Intel Corporation. All rights reserved. See COPYING file
> > > > + */
> > > > +
> > > > +#include <stdio.h>
> > > > +#include <stdlib.h>
> > > > +#include <stdint.h>
> > > > +#include <unistd.h>
> > > > +#include <string.h>
> > > > +#include <errno.h>
> > > > +#include <drm/drm.h>
> > > > +#include <drm/i915_drm.h>
> > > > +#include <drm/amdgpu_drm.h>
> > > > +#include <drm/radeon_drm.h>
> > >
> > > I assume these should come from the kernel headers package, right?
> >
> > This is gross, all kernel headers should be placed in kernel-headers/*
> > and "update" script needs to be extended to take drm/* files too :(.
> 
> drm kernel headers are in the libdrm package. You need that anyway for
> doing the ioctls (if you don't hand-roll the restarting yourself).
> 
> Also our userspace has gone over to just outright copying the driver
> headers. Not the generic headers, but for the rendering side of gpus,
> which is the topic here, there's really not much generic stuff.
> 
> > Jianxin, are you fixing it?
> 
> So fix is either to depend upon libdrm for building, or have copies of
> the headers included in the package for the i915/amdgpu/radeon headers
> (drm/drm.h probably not so good idea).

We should have a cmake test to not build the drm parts if it can't be
built, and pyverbs should skip the tests.

Jason

^ permalink raw reply	[flat|nested] 42+ messages in thread
* Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
@ 2021-02-01 15:29           ` Jason Gunthorpe
  0 siblings, 0 replies; 42+ messages in thread
From: Jason Gunthorpe @ 2021-02-01 15:29 UTC (permalink / raw)
  To: Daniel Vetter
  Cc: Yishai Hadas, Leon Romanovsky, linux-rdma, Edward Srouji,
	Gal Pressman, dri-devel, Doug Ledford, Daniel Vetter,
	Christian Koenig, Jianxin Xiong

On Mon, Feb 01, 2021 at 03:10:00PM +0100, Daniel Vetter wrote:
> On Mon, Feb 1, 2021 at 7:16 AM Leon Romanovsky <leon@kernel.org> wrote:
> >
> > On Sun, Jan 31, 2021 at 05:31:16PM +0200, Gal Pressman wrote:
> > > On 25/01/2021 21:57, Jianxin Xiong wrote:
> > > > Define a new sub-class of 'MR' that uses dma-buf object for the memory
> > > > region. Define a new class 'DmaBuf' as a wrapper for dma-buf allocation
> > > > mechanism implemented in C.
> > > >
> > > > Update the cmake function for cython modules to allow building modules
> > > > with mixed cython and c source files.
> > > >
> > > > Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
> > > >  buildlib/pyverbs_functions.cmake |  78 +++++++----
> > > >  pyverbs/CMakeLists.txt           |  11 +-
> > > >  pyverbs/dmabuf.pxd               |  15 +++
> > > >  pyverbs/dmabuf.pyx               |  73 ++++++++++
> > > >  pyverbs/dmabuf_alloc.c           | 278 +++++++++++++++++++++++++++++++++++++++
> > > >  pyverbs/dmabuf_alloc.h           |  19 +++
> > > >  pyverbs/libibverbs.pxd           |   2 +
> > > >  pyverbs/mr.pxd                   |   6 +
> > > >  pyverbs/mr.pyx                   | 105 ++++++++++++++-
> > > >  9 files changed, 557 insertions(+), 30 deletions(-)
> > > >  create mode 100644 pyverbs/dmabuf.pxd
> > > >  create mode 100644 pyverbs/dmabuf.pyx
> > > >  create mode 100644 pyverbs/dmabuf_alloc.c
> > > >  create mode 100644 pyverbs/dmabuf_alloc.h
> >
> > <...>
> >
> > > > index 0000000..05eae75
> > > > +++ b/pyverbs/dmabuf_alloc.c
> > > > @@ -0,0 +1,278 @@
> > > > +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> > > > +/*
> > > > + * Copyright 2020 Intel Corporation. All rights reserved. See COPYING file
> > > > + */
> > > > +
> > > > +#include <stdio.h>
> > > > +#include <stdlib.h>
> > > > +#include <stdint.h>
> > > > +#include <unistd.h>
> > > > +#include <string.h>
> > > > +#include <errno.h>
> > > > +#include <drm/drm.h>
> > > > +#include <drm/i915_drm.h>
> > > > +#include <drm/amdgpu_drm.h>
> > > > +#include <drm/radeon_drm.h>
> > >
> > > I assume these should come from the kernel headers package, right?
> >
> > This is gross, all kernel headers should be placed in kernel-headers/*
> > and "update" script needs to be extended to take drm/* files too :(.
> 
> drm kernel headers are in the libdrm package. You need that anyway for
> doing the ioctls (if you don't hand-roll the restarting yourself).
> 
> Also our userspace has gone over to just outright copying the driver
> headers. Not the generic headers, but for the rendering side of gpus,
> which is the topic here, there's really not much generic stuff.
> 
> > Jianxin, are you fixing it?
> 
> So fix is either to depend upon libdrm for building, or have copies of
> the headers included in the package for the i915/amdgpu/radeon headers
> (drm/drm.h probably not so good idea).

We should have a cmake test to not build the drm parts if it can't be
built, and pyverbs should skip the tests.

Jason
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

^ permalink raw reply	[flat|nested] 42+ messages in thread
* RE: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
  2021-02-01 15:29           ` Jason Gunthorpe
@ 2021-02-01 17:03             ` Xiong, Jianxin
  -1 siblings, 0 replies; 42+ messages in thread
From: Xiong, Jianxin @ 2021-02-01 17:03 UTC (permalink / raw)
  To: Jason Gunthorpe, Daniel Vetter
  Cc: Leon Romanovsky, Gal Pressman, Yishai Hadas, linux-rdma,
	Edward Srouji, dri-devel, Christian Koenig, Doug Ledford, Vetter,
	Daniel

> -----Original Message-----
> From: Jason Gunthorpe <jgg@ziepe.ca>
> Sent: Monday, February 01, 2021 7:29 AM
> To: Daniel Vetter <daniel@ffwll.ch>
> Cc: Leon Romanovsky <leon@kernel.org>; Gal Pressman <galpress@amazon.com>; Xiong, Jianxin <jianxin.xiong@intel.com>; Yishai Hadas
> <yishaih@nvidia.com>; linux-rdma <linux-rdma@vger.kernel.org>; Edward Srouji <edwards@nvidia.com>; dri-devel <dri-
> devel@lists.freedesktop.org>; Christian Koenig <christian.koenig@amd.com>; Doug Ledford <dledford@redhat.com>; Vetter, Daniel
> <daniel.vetter@intel.com>
> Subject: Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
> 
> On Mon, Feb 01, 2021 at 03:10:00PM +0100, Daniel Vetter wrote:
> > On Mon, Feb 1, 2021 at 7:16 AM Leon Romanovsky <leon@kernel.org> wrote:
> > >
> > > On Sun, Jan 31, 2021 at 05:31:16PM +0200, Gal Pressman wrote:
> > > > On 25/01/2021 21:57, Jianxin Xiong wrote:
> > > > > Define a new sub-class of 'MR' that uses dma-buf object for the
> > > > > memory region. Define a new class 'DmaBuf' as a wrapper for
> > > > > dma-buf allocation mechanism implemented in C.
> > > > >
> > > > > Update the cmake function for cython modules to allow building
> > > > > modules with mixed cython and c source files.
> > > > >
> > > > > Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
> > > > > buildlib/pyverbs_functions.cmake |  78 +++++++----
> > > > >  pyverbs/CMakeLists.txt           |  11 +-
> > > > >  pyverbs/dmabuf.pxd               |  15 +++
> > > > >  pyverbs/dmabuf.pyx               |  73 ++++++++++
> > > > >  pyverbs/dmabuf_alloc.c           | 278 +++++++++++++++++++++++++++++++++++++++
> > > > >  pyverbs/dmabuf_alloc.h           |  19 +++
> > > > >  pyverbs/libibverbs.pxd           |   2 +
> > > > >  pyverbs/mr.pxd                   |   6 +
> > > > >  pyverbs/mr.pyx                   | 105 ++++++++++++++-
> > > > >  9 files changed, 557 insertions(+), 30 deletions(-)  create
> > > > > mode 100644 pyverbs/dmabuf.pxd  create mode 100644
> > > > > pyverbs/dmabuf.pyx  create mode 100644 pyverbs/dmabuf_alloc.c
> > > > > create mode 100644 pyverbs/dmabuf_alloc.h
> > >
> > > <...>
> > >
> > > > > index 0000000..05eae75
> > > > > +++ b/pyverbs/dmabuf_alloc.c
> > > > > @@ -0,0 +1,278 @@
> > > > > +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> > > > > +/*
> > > > > + * Copyright 2020 Intel Corporation. All rights reserved. See
> > > > > +COPYING file  */
> > > > > +
> > > > > +#include <stdio.h>
> > > > > +#include <stdlib.h>
> > > > > +#include <stdint.h>
> > > > > +#include <unistd.h>
> > > > > +#include <string.h>
> > > > > +#include <errno.h>
> > > > > +#include <drm/drm.h>
> > > > > +#include <drm/i915_drm.h>
> > > > > +#include <drm/amdgpu_drm.h>
> > > > > +#include <drm/radeon_drm.h>
> > > >
> > > > I assume these should come from the kernel headers package, right?
> > >
> > > This is gross, all kernel headers should be placed in
> > > kernel-headers/* and "update" script needs to be extended to take drm/* files too :(.
> >
> > drm kernel headers are in the libdrm package. You need that anyway for
> > doing the ioctls (if you don't hand-roll the restarting yourself).
> >
> > Also our userspace has gone over to just outright copying the driver
> > headers. Not the generic headers, but for the rendering side of gpus,
> > which is the topic here, there's really not much generic stuff.
> >
> > > Jianxin, are you fixing it?
> >
> > So fix is either to depend upon libdrm for building, or have copies of
> > the headers included in the package for the i915/amdgpu/radeon headers
> > (drm/drm.h probably not so good idea).
> 
> We should have a cmake test to not build the drm parts if it can't be built, and pyverbs should skip the tests.
> 

Yes, I will add a test for that. Also, on SLES, the headers could be under /usr/include/libdrm instead of /usr/include/drm. The make test should check that and use proper path. 

^ permalink raw reply	[flat|nested] 42+ messages in thread
* RE: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
@ 2021-02-01 17:03             ` Xiong, Jianxin
  0 siblings, 0 replies; 42+ messages in thread
From: Xiong, Jianxin @ 2021-02-01 17:03 UTC (permalink / raw)
  To: Jason Gunthorpe, Daniel Vetter
  Cc: Yishai Hadas, Leon Romanovsky, linux-rdma, Edward Srouji,
	Gal Pressman, dri-devel, Doug Ledford, Vetter,  Daniel,
	Christian Koenig

> -----Original Message-----
> From: Jason Gunthorpe <jgg@ziepe.ca>
> Sent: Monday, February 01, 2021 7:29 AM
> To: Daniel Vetter <daniel@ffwll.ch>
> Cc: Leon Romanovsky <leon@kernel.org>; Gal Pressman <galpress@amazon.com>; Xiong, Jianxin <jianxin.xiong@intel.com>; Yishai Hadas
> <yishaih@nvidia.com>; linux-rdma <linux-rdma@vger.kernel.org>; Edward Srouji <edwards@nvidia.com>; dri-devel <dri-
> devel@lists.freedesktop.org>; Christian Koenig <christian.koenig@amd.com>; Doug Ledford <dledford@redhat.com>; Vetter, Daniel
> <daniel.vetter@intel.com>
> Subject: Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
> 
> On Mon, Feb 01, 2021 at 03:10:00PM +0100, Daniel Vetter wrote:
> > On Mon, Feb 1, 2021 at 7:16 AM Leon Romanovsky <leon@kernel.org> wrote:
> > >
> > > On Sun, Jan 31, 2021 at 05:31:16PM +0200, Gal Pressman wrote:
> > > > On 25/01/2021 21:57, Jianxin Xiong wrote:
> > > > > Define a new sub-class of 'MR' that uses dma-buf object for the
> > > > > memory region. Define a new class 'DmaBuf' as a wrapper for
> > > > > dma-buf allocation mechanism implemented in C.
> > > > >
> > > > > Update the cmake function for cython modules to allow building
> > > > > modules with mixed cython and c source files.
> > > > >
> > > > > Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
> > > > > buildlib/pyverbs_functions.cmake |  78 +++++++----
> > > > >  pyverbs/CMakeLists.txt           |  11 +-
> > > > >  pyverbs/dmabuf.pxd               |  15 +++
> > > > >  pyverbs/dmabuf.pyx               |  73 ++++++++++
> > > > >  pyverbs/dmabuf_alloc.c           | 278 +++++++++++++++++++++++++++++++++++++++
> > > > >  pyverbs/dmabuf_alloc.h           |  19 +++
> > > > >  pyverbs/libibverbs.pxd           |   2 +
> > > > >  pyverbs/mr.pxd                   |   6 +
> > > > >  pyverbs/mr.pyx                   | 105 ++++++++++++++-
> > > > >  9 files changed, 557 insertions(+), 30 deletions(-)  create
> > > > > mode 100644 pyverbs/dmabuf.pxd  create mode 100644
> > > > > pyverbs/dmabuf.pyx  create mode 100644 pyverbs/dmabuf_alloc.c
> > > > > create mode 100644 pyverbs/dmabuf_alloc.h
> > >
> > > <...>
> > >
> > > > > index 0000000..05eae75
> > > > > +++ b/pyverbs/dmabuf_alloc.c
> > > > > @@ -0,0 +1,278 @@
> > > > > +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> > > > > +/*
> > > > > + * Copyright 2020 Intel Corporation. All rights reserved. See
> > > > > +COPYING file  */
> > > > > +
> > > > > +#include <stdio.h>
> > > > > +#include <stdlib.h>
> > > > > +#include <stdint.h>
> > > > > +#include <unistd.h>
> > > > > +#include <string.h>
> > > > > +#include <errno.h>
> > > > > +#include <drm/drm.h>
> > > > > +#include <drm/i915_drm.h>
> > > > > +#include <drm/amdgpu_drm.h>
> > > > > +#include <drm/radeon_drm.h>
> > > >
> > > > I assume these should come from the kernel headers package, right?
> > >
> > > This is gross, all kernel headers should be placed in
> > > kernel-headers/* and "update" script needs to be extended to take drm/* files too :(.
> >
> > drm kernel headers are in the libdrm package. You need that anyway for
> > doing the ioctls (if you don't hand-roll the restarting yourself).
> >
> > Also our userspace has gone over to just outright copying the driver
> > headers. Not the generic headers, but for the rendering side of gpus,
> > which is the topic here, there's really not much generic stuff.
> >
> > > Jianxin, are you fixing it?
> >
> > So fix is either to depend upon libdrm for building, or have copies of
> > the headers included in the package for the i915/amdgpu/radeon headers
> > (drm/drm.h probably not so good idea).
> 
> We should have a cmake test to not build the drm parts if it can't be built, and pyverbs should skip the tests.
> 

Yes, I will add a test for that. Also, on SLES, the headers could be under /usr/include/libdrm instead of /usr/include/drm. The make test should check that and use proper path. 
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

^ permalink raw reply	[flat|nested] 42+ messages in thread
* RE: [PATCH rdma-core v7 5/6] tests: Add tests for dma-buf based memory regions
  2021-01-31  8:44     ` John Hubbard
@ 2021-02-01 17:12       ` Xiong, Jianxin
  -1 siblings, 0 replies; 42+ messages in thread
From: Xiong, Jianxin @ 2021-02-01 17:12 UTC (permalink / raw)
  To: John Hubbard, linux-rdma, dri-devel
  Cc: Doug Ledford, Jason Gunthorpe, Leon Romanovsky, Sumit Semwal,
	Christian Koenig, Vetter, Daniel, Edward Srouji, Yishai Hadas

> -----Original Message-----
> From: John Hubbard <jhubbard@nvidia.com>
> Sent: Sunday, January 31, 2021 12:45 AM
> To: Xiong, Jianxin <jianxin.xiong@intel.com>; linux-rdma@vger.kernel.org; dri-devel@lists.freedesktop.org
> Cc: Doug Ledford <dledford@redhat.com>; Jason Gunthorpe <jgg@ziepe.ca>; Leon Romanovsky <leon@kernel.org>; Sumit Semwal
> <sumit.semwal@linaro.org>; Christian Koenig <christian.koenig@amd.com>; Vetter, Daniel <daniel.vetter@intel.com>; Edward Srouji
> <edwards@nvidia.com>; Yishai Hadas <yishaih@nvidia.com>
> Subject: Re: [PATCH rdma-core v7 5/6] tests: Add tests for dma-buf based memory regions
> 
> On 1/25/21 11:57 AM, Jianxin Xiong wrote:
> > Define a set of unit tests similar to regular MR tests and a set of
> > tests for send/recv and rdma traffic using dma-buf MRs. Add a utility
> > function to generate access flags for dma-buf based MRs because the
> > set of supported flags is smaller.
> >
> > Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
> 
> Hi Jianxin,
> 
> It's awesome to see a GPU to IB test suite here!
> 
> > ---
> >   tests/args_parser.py |   4 +
> >   tests/test_mr.py     | 266 ++++++++++++++++++++++++++++++++++++++++++++++++++-
> >   tests/utils.py       |  26 +++++
> >   3 files changed, 295 insertions(+), 1 deletion(-)
> >
> > diff --git a/tests/args_parser.py b/tests/args_parser.py index
> > 446535a..5bc53b0 100644
> > --- a/tests/args_parser.py
> > +++ b/tests/args_parser.py
> > @@ -19,6 +19,10 @@ class ArgsParser(object):
> >           parser.add_argument('--port',
> >                               help='Use port <port> of RDMA device', type=int,
> >                               default=1)
> > +        parser.add_argument('--gpu', nargs='?', type=int, const=0, default=0,
> > +                            help='GPU unit to allocate dmabuf from')
> > +        parser.add_argument('--gtt', action='store_true', default=False,
> > +                            help='Allocate dmabuf from GTT instead of
> > + VRAM')
> 
> Just to be kind to non-GPU people, how about:
> 
> 	s/GTT/GTT (Graphics Translation Table)/
> 
> 

<...>

> > +def check_dmabuf_support(unit=0):
> > +    """
> > +    Check if dma-buf allocation is supported by the system.
> > +    Skip the test on failure.
> > +    """
> > +    device_num = 128 + unit
> > +    try:
> > +        DmaBuf(1, unit=unit)
> 
> unit?? This is a GPU, never anything else! Let's s/unit/gpu/ throughout, yes?
> 

Thanks for the feedback. Will integrate the suggestions in upcoming patch.

Jianxin

^ permalink raw reply	[flat|nested] 42+ messages in thread
* RE: [PATCH rdma-core v7 5/6] tests: Add tests for dma-buf based memory regions
@ 2021-02-01 17:12       ` Xiong, Jianxin
  0 siblings, 0 replies; 42+ messages in thread
From: Xiong, Jianxin @ 2021-02-01 17:12 UTC (permalink / raw)
  To: John Hubbard, linux-rdma, dri-devel
  Cc: Yishai Hadas, Leon Romanovsky, Edward Srouji, Christian Koenig,
	Jason Gunthorpe, Doug Ledford, Vetter, Daniel

> -----Original Message-----
> From: John Hubbard <jhubbard@nvidia.com>
> Sent: Sunday, January 31, 2021 12:45 AM
> To: Xiong, Jianxin <jianxin.xiong@intel.com>; linux-rdma@vger.kernel.org; dri-devel@lists.freedesktop.org
> Cc: Doug Ledford <dledford@redhat.com>; Jason Gunthorpe <jgg@ziepe.ca>; Leon Romanovsky <leon@kernel.org>; Sumit Semwal
> <sumit.semwal@linaro.org>; Christian Koenig <christian.koenig@amd.com>; Vetter, Daniel <daniel.vetter@intel.com>; Edward Srouji
> <edwards@nvidia.com>; Yishai Hadas <yishaih@nvidia.com>
> Subject: Re: [PATCH rdma-core v7 5/6] tests: Add tests for dma-buf based memory regions
> 
> On 1/25/21 11:57 AM, Jianxin Xiong wrote:
> > Define a set of unit tests similar to regular MR tests and a set of
> > tests for send/recv and rdma traffic using dma-buf MRs. Add a utility
> > function to generate access flags for dma-buf based MRs because the
> > set of supported flags is smaller.
> >
> > Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
> 
> Hi Jianxin,
> 
> It's awesome to see a GPU to IB test suite here!
> 
> > ---
> >   tests/args_parser.py |   4 +
> >   tests/test_mr.py     | 266 ++++++++++++++++++++++++++++++++++++++++++++++++++-
> >   tests/utils.py       |  26 +++++
> >   3 files changed, 295 insertions(+), 1 deletion(-)
> >
> > diff --git a/tests/args_parser.py b/tests/args_parser.py index
> > 446535a..5bc53b0 100644
> > --- a/tests/args_parser.py
> > +++ b/tests/args_parser.py
> > @@ -19,6 +19,10 @@ class ArgsParser(object):
> >           parser.add_argument('--port',
> >                               help='Use port <port> of RDMA device', type=int,
> >                               default=1)
> > +        parser.add_argument('--gpu', nargs='?', type=int, const=0, default=0,
> > +                            help='GPU unit to allocate dmabuf from')
> > +        parser.add_argument('--gtt', action='store_true', default=False,
> > +                            help='Allocate dmabuf from GTT instead of
> > + VRAM')
> 
> Just to be kind to non-GPU people, how about:
> 
> 	s/GTT/GTT (Graphics Translation Table)/
> 
> 

<...>

> > +def check_dmabuf_support(unit=0):
> > +    """
> > +    Check if dma-buf allocation is supported by the system.
> > +    Skip the test on failure.
> > +    """
> > +    device_num = 128 + unit
> > +    try:
> > +        DmaBuf(1, unit=unit)
> 
> unit?? This is a GPU, never anything else! Let's s/unit/gpu/ throughout, yes?
> 

Thanks for the feedback. Will integrate the suggestions in upcoming patch.

Jianxin
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

^ permalink raw reply	[flat|nested] 42+ messages in thread
* Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
  2021-02-01 17:03             ` Xiong, Jianxin
@ 2021-02-02 15:24               ` Daniel Vetter
  -1 siblings, 0 replies; 42+ messages in thread
From: Daniel Vetter @ 2021-02-02 15:24 UTC (permalink / raw)
  To: Xiong, Jianxin
  Cc: Jason Gunthorpe, Daniel Vetter, Leon Romanovsky, Gal Pressman,
	Yishai Hadas, linux-rdma, Edward Srouji, dri-devel,
	Christian Koenig, Doug Ledford, Vetter, Daniel

On Mon, Feb 01, 2021 at 05:03:44PM +0000, Xiong, Jianxin wrote:
> > -----Original Message-----
> > From: Jason Gunthorpe <jgg@ziepe.ca>
> > Sent: Monday, February 01, 2021 7:29 AM
> > To: Daniel Vetter <daniel@ffwll.ch>
> > Cc: Leon Romanovsky <leon@kernel.org>; Gal Pressman <galpress@amazon.com>; Xiong, Jianxin <jianxin.xiong@intel.com>; Yishai Hadas
> > <yishaih@nvidia.com>; linux-rdma <linux-rdma@vger.kernel.org>; Edward Srouji <edwards@nvidia.com>; dri-devel <dri-
> > devel@lists.freedesktop.org>; Christian Koenig <christian.koenig@amd.com>; Doug Ledford <dledford@redhat.com>; Vetter, Daniel
> > <daniel.vetter@intel.com>
> > Subject: Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
> > 
> > On Mon, Feb 01, 2021 at 03:10:00PM +0100, Daniel Vetter wrote:
> > > On Mon, Feb 1, 2021 at 7:16 AM Leon Romanovsky <leon@kernel.org> wrote:
> > > >
> > > > On Sun, Jan 31, 2021 at 05:31:16PM +0200, Gal Pressman wrote:
> > > > > On 25/01/2021 21:57, Jianxin Xiong wrote:
> > > > > > Define a new sub-class of 'MR' that uses dma-buf object for the
> > > > > > memory region. Define a new class 'DmaBuf' as a wrapper for
> > > > > > dma-buf allocation mechanism implemented in C.
> > > > > >
> > > > > > Update the cmake function for cython modules to allow building
> > > > > > modules with mixed cython and c source files.
> > > > > >
> > > > > > Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
> > > > > > buildlib/pyverbs_functions.cmake |  78 +++++++----
> > > > > >  pyverbs/CMakeLists.txt           |  11 +-
> > > > > >  pyverbs/dmabuf.pxd               |  15 +++
> > > > > >  pyverbs/dmabuf.pyx               |  73 ++++++++++
> > > > > >  pyverbs/dmabuf_alloc.c           | 278 +++++++++++++++++++++++++++++++++++++++
> > > > > >  pyverbs/dmabuf_alloc.h           |  19 +++
> > > > > >  pyverbs/libibverbs.pxd           |   2 +
> > > > > >  pyverbs/mr.pxd                   |   6 +
> > > > > >  pyverbs/mr.pyx                   | 105 ++++++++++++++-
> > > > > >  9 files changed, 557 insertions(+), 30 deletions(-)  create
> > > > > > mode 100644 pyverbs/dmabuf.pxd  create mode 100644
> > > > > > pyverbs/dmabuf.pyx  create mode 100644 pyverbs/dmabuf_alloc.c
> > > > > > create mode 100644 pyverbs/dmabuf_alloc.h
> > > >
> > > > <...>
> > > >
> > > > > > index 0000000..05eae75
> > > > > > +++ b/pyverbs/dmabuf_alloc.c
> > > > > > @@ -0,0 +1,278 @@
> > > > > > +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> > > > > > +/*
> > > > > > + * Copyright 2020 Intel Corporation. All rights reserved. See
> > > > > > +COPYING file  */
> > > > > > +
> > > > > > +#include <stdio.h>
> > > > > > +#include <stdlib.h>
> > > > > > +#include <stdint.h>
> > > > > > +#include <unistd.h>
> > > > > > +#include <string.h>
> > > > > > +#include <errno.h>
> > > > > > +#include <drm/drm.h>
> > > > > > +#include <drm/i915_drm.h>
> > > > > > +#include <drm/amdgpu_drm.h>
> > > > > > +#include <drm/radeon_drm.h>
> > > > >
> > > > > I assume these should come from the kernel headers package, right?
> > > >
> > > > This is gross, all kernel headers should be placed in
> > > > kernel-headers/* and "update" script needs to be extended to take drm/* files too :(.
> > >
> > > drm kernel headers are in the libdrm package. You need that anyway for
> > > doing the ioctls (if you don't hand-roll the restarting yourself).
> > >
> > > Also our userspace has gone over to just outright copying the driver
> > > headers. Not the generic headers, but for the rendering side of gpus,
> > > which is the topic here, there's really not much generic stuff.
> > >
> > > > Jianxin, are you fixing it?
> > >
> > > So fix is either to depend upon libdrm for building, or have copies of
> > > the headers included in the package for the i915/amdgpu/radeon headers
> > > (drm/drm.h probably not so good idea).
> > 
> > We should have a cmake test to not build the drm parts if it can't be built, and pyverbs should skip the tests.
> > 
> 
> Yes, I will add a test for that. Also, on SLES, the headers could be under /usr/include/libdrm instead of /usr/include/drm. The make test should check that and use proper path. 

Please use pkgconfig for this, libdrm installs a .pc file to make sure you
can find the right headers.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

^ permalink raw reply	[flat|nested] 42+ messages in thread
* Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
@ 2021-02-02 15:24               ` Daniel Vetter
  0 siblings, 0 replies; 42+ messages in thread
From: Daniel Vetter @ 2021-02-02 15:24 UTC (permalink / raw)
  To: Xiong, Jianxin
  Cc: Yishai Hadas, Leon Romanovsky, linux-rdma, Edward Srouji,
	Gal Pressman, dri-devel, Jason Gunthorpe, Doug Ledford, Vetter,
	Daniel, Christian Koenig

On Mon, Feb 01, 2021 at 05:03:44PM +0000, Xiong, Jianxin wrote:
> > -----Original Message-----
> > From: Jason Gunthorpe <jgg@ziepe.ca>
> > Sent: Monday, February 01, 2021 7:29 AM
> > To: Daniel Vetter <daniel@ffwll.ch>
> > Cc: Leon Romanovsky <leon@kernel.org>; Gal Pressman <galpress@amazon.com>; Xiong, Jianxin <jianxin.xiong@intel.com>; Yishai Hadas
> > <yishaih@nvidia.com>; linux-rdma <linux-rdma@vger.kernel.org>; Edward Srouji <edwards@nvidia.com>; dri-devel <dri-
> > devel@lists.freedesktop.org>; Christian Koenig <christian.koenig@amd.com>; Doug Ledford <dledford@redhat.com>; Vetter, Daniel
> > <daniel.vetter@intel.com>
> > Subject: Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
> > 
> > On Mon, Feb 01, 2021 at 03:10:00PM +0100, Daniel Vetter wrote:
> > > On Mon, Feb 1, 2021 at 7:16 AM Leon Romanovsky <leon@kernel.org> wrote:
> > > >
> > > > On Sun, Jan 31, 2021 at 05:31:16PM +0200, Gal Pressman wrote:
> > > > > On 25/01/2021 21:57, Jianxin Xiong wrote:
> > > > > > Define a new sub-class of 'MR' that uses dma-buf object for the
> > > > > > memory region. Define a new class 'DmaBuf' as a wrapper for
> > > > > > dma-buf allocation mechanism implemented in C.
> > > > > >
> > > > > > Update the cmake function for cython modules to allow building
> > > > > > modules with mixed cython and c source files.
> > > > > >
> > > > > > Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
> > > > > > buildlib/pyverbs_functions.cmake |  78 +++++++----
> > > > > >  pyverbs/CMakeLists.txt           |  11 +-
> > > > > >  pyverbs/dmabuf.pxd               |  15 +++
> > > > > >  pyverbs/dmabuf.pyx               |  73 ++++++++++
> > > > > >  pyverbs/dmabuf_alloc.c           | 278 +++++++++++++++++++++++++++++++++++++++
> > > > > >  pyverbs/dmabuf_alloc.h           |  19 +++
> > > > > >  pyverbs/libibverbs.pxd           |   2 +
> > > > > >  pyverbs/mr.pxd                   |   6 +
> > > > > >  pyverbs/mr.pyx                   | 105 ++++++++++++++-
> > > > > >  9 files changed, 557 insertions(+), 30 deletions(-)  create
> > > > > > mode 100644 pyverbs/dmabuf.pxd  create mode 100644
> > > > > > pyverbs/dmabuf.pyx  create mode 100644 pyverbs/dmabuf_alloc.c
> > > > > > create mode 100644 pyverbs/dmabuf_alloc.h
> > > >
> > > > <...>
> > > >
> > > > > > index 0000000..05eae75
> > > > > > +++ b/pyverbs/dmabuf_alloc.c
> > > > > > @@ -0,0 +1,278 @@
> > > > > > +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> > > > > > +/*
> > > > > > + * Copyright 2020 Intel Corporation. All rights reserved. See
> > > > > > +COPYING file  */
> > > > > > +
> > > > > > +#include <stdio.h>
> > > > > > +#include <stdlib.h>
> > > > > > +#include <stdint.h>
> > > > > > +#include <unistd.h>
> > > > > > +#include <string.h>
> > > > > > +#include <errno.h>
> > > > > > +#include <drm/drm.h>
> > > > > > +#include <drm/i915_drm.h>
> > > > > > +#include <drm/amdgpu_drm.h>
> > > > > > +#include <drm/radeon_drm.h>
> > > > >
> > > > > I assume these should come from the kernel headers package, right?
> > > >
> > > > This is gross, all kernel headers should be placed in
> > > > kernel-headers/* and "update" script needs to be extended to take drm/* files too :(.
> > >
> > > drm kernel headers are in the libdrm package. You need that anyway for
> > > doing the ioctls (if you don't hand-roll the restarting yourself).
> > >
> > > Also our userspace has gone over to just outright copying the driver
> > > headers. Not the generic headers, but for the rendering side of gpus,
> > > which is the topic here, there's really not much generic stuff.
> > >
> > > > Jianxin, are you fixing it?
> > >
> > > So fix is either to depend upon libdrm for building, or have copies of
> > > the headers included in the package for the i915/amdgpu/radeon headers
> > > (drm/drm.h probably not so good idea).
> > 
> > We should have a cmake test to not build the drm parts if it can't be built, and pyverbs should skip the tests.
> > 
> 
> Yes, I will add a test for that. Also, on SLES, the headers could be under /usr/include/libdrm instead of /usr/include/drm. The make test should check that and use proper path. 

Please use pkgconfig for this, libdrm installs a .pc file to make sure you
can find the right headers.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

^ permalink raw reply	[flat|nested] 42+ messages in thread
* Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
  2021-02-02 15:24               ` Daniel Vetter
@ 2021-02-03  6:03                 ` Leon Romanovsky
  -1 siblings, 0 replies; 42+ messages in thread
From: Leon Romanovsky @ 2021-02-03  6:03 UTC (permalink / raw)
  To: Daniel Vetter
  Cc: Xiong, Jianxin, Jason Gunthorpe, Gal Pressman, Yishai Hadas,
	linux-rdma, Edward Srouji, dri-devel, Christian Koenig,
	Doug Ledford, Vetter, Daniel

On Tue, Feb 02, 2021 at 04:24:45PM +0100, Daniel Vetter wrote:
> On Mon, Feb 01, 2021 at 05:03:44PM +0000, Xiong, Jianxin wrote:
> > > -----Original Message-----
> > > From: Jason Gunthorpe <jgg@ziepe.ca>
> > > Sent: Monday, February 01, 2021 7:29 AM
> > > To: Daniel Vetter <daniel@ffwll.ch>
> > > Cc: Leon Romanovsky <leon@kernel.org>; Gal Pressman <galpress@amazon.com>; Xiong, Jianxin <jianxin.xiong@intel.com>; Yishai Hadas
> > > <yishaih@nvidia.com>; linux-rdma <linux-rdma@vger.kernel.org>; Edward Srouji <edwards@nvidia.com>; dri-devel <dri-
> > > devel@lists.freedesktop.org>; Christian Koenig <christian.koenig@amd.com>; Doug Ledford <dledford@redhat.com>; Vetter, Daniel
> > > <daniel.vetter@intel.com>
> > > Subject: Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
> > >
> > > On Mon, Feb 01, 2021 at 03:10:00PM +0100, Daniel Vetter wrote:
> > > > On Mon, Feb 1, 2021 at 7:16 AM Leon Romanovsky <leon@kernel.org> wrote:
> > > > >
> > > > > On Sun, Jan 31, 2021 at 05:31:16PM +0200, Gal Pressman wrote:
> > > > > > On 25/01/2021 21:57, Jianxin Xiong wrote:
> > > > > > > Define a new sub-class of 'MR' that uses dma-buf object for the
> > > > > > > memory region. Define a new class 'DmaBuf' as a wrapper for
> > > > > > > dma-buf allocation mechanism implemented in C.
> > > > > > >
> > > > > > > Update the cmake function for cython modules to allow building
> > > > > > > modules with mixed cython and c source files.
> > > > > > >
> > > > > > > Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
> > > > > > > buildlib/pyverbs_functions.cmake |  78 +++++++----
> > > > > > >  pyverbs/CMakeLists.txt           |  11 +-
> > > > > > >  pyverbs/dmabuf.pxd               |  15 +++
> > > > > > >  pyverbs/dmabuf.pyx               |  73 ++++++++++
> > > > > > >  pyverbs/dmabuf_alloc.c           | 278 +++++++++++++++++++++++++++++++++++++++
> > > > > > >  pyverbs/dmabuf_alloc.h           |  19 +++
> > > > > > >  pyverbs/libibverbs.pxd           |   2 +
> > > > > > >  pyverbs/mr.pxd                   |   6 +
> > > > > > >  pyverbs/mr.pyx                   | 105 ++++++++++++++-
> > > > > > >  9 files changed, 557 insertions(+), 30 deletions(-)  create
> > > > > > > mode 100644 pyverbs/dmabuf.pxd  create mode 100644
> > > > > > > pyverbs/dmabuf.pyx  create mode 100644 pyverbs/dmabuf_alloc.c
> > > > > > > create mode 100644 pyverbs/dmabuf_alloc.h
> > > > >
> > > > > <...>
> > > > >
> > > > > > > index 0000000..05eae75
> > > > > > > +++ b/pyverbs/dmabuf_alloc.c
> > > > > > > @@ -0,0 +1,278 @@
> > > > > > > +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> > > > > > > +/*
> > > > > > > + * Copyright 2020 Intel Corporation. All rights reserved. See
> > > > > > > +COPYING file  */
> > > > > > > +
> > > > > > > +#include <stdio.h>
> > > > > > > +#include <stdlib.h>
> > > > > > > +#include <stdint.h>
> > > > > > > +#include <unistd.h>
> > > > > > > +#include <string.h>
> > > > > > > +#include <errno.h>
> > > > > > > +#include <drm/drm.h>
> > > > > > > +#include <drm/i915_drm.h>
> > > > > > > +#include <drm/amdgpu_drm.h>
> > > > > > > +#include <drm/radeon_drm.h>
> > > > > >
> > > > > > I assume these should come from the kernel headers package, right?
> > > > >
> > > > > This is gross, all kernel headers should be placed in
> > > > > kernel-headers/* and "update" script needs to be extended to take drm/* files too :(.
> > > >
> > > > drm kernel headers are in the libdrm package. You need that anyway for
> > > > doing the ioctls (if you don't hand-roll the restarting yourself).
> > > >
> > > > Also our userspace has gone over to just outright copying the driver
> > > > headers. Not the generic headers, but for the rendering side of gpus,
> > > > which is the topic here, there's really not much generic stuff.
> > > >
> > > > > Jianxin, are you fixing it?
> > > >
> > > > So fix is either to depend upon libdrm for building, or have copies of
> > > > the headers included in the package for the i915/amdgpu/radeon headers
> > > > (drm/drm.h probably not so good idea).
> > >
> > > We should have a cmake test to not build the drm parts if it can't be built, and pyverbs should skip the tests.
> > >
> >
> > Yes, I will add a test for that. Also, on SLES, the headers could be under /usr/include/libdrm instead of /usr/include/drm. The make test should check that and use proper path.
>
> Please use pkgconfig for this, libdrm installs a .pc file to make sure you
> can find the right headers.

rdma-core uses cmake build system and in our case cmake find_library()
is preferable over pkgconfig.

Thanks

> -Daniel
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch

^ permalink raw reply	[flat|nested] 42+ messages in thread
* Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
@ 2021-02-03  6:03                 ` Leon Romanovsky
  0 siblings, 0 replies; 42+ messages in thread
From: Leon Romanovsky @ 2021-02-03  6:03 UTC (permalink / raw)
  To: Daniel Vetter
  Cc: Yishai Hadas, linux-rdma, Edward Srouji, Gal Pressman, dri-devel,
	Jason Gunthorpe, Doug Ledford, Vetter, Daniel, Christian Koenig,
	Xiong, Jianxin

On Tue, Feb 02, 2021 at 04:24:45PM +0100, Daniel Vetter wrote:
> On Mon, Feb 01, 2021 at 05:03:44PM +0000, Xiong, Jianxin wrote:
> > > -----Original Message-----
> > > From: Jason Gunthorpe <jgg@ziepe.ca>
> > > Sent: Monday, February 01, 2021 7:29 AM
> > > To: Daniel Vetter <daniel@ffwll.ch>
> > > Cc: Leon Romanovsky <leon@kernel.org>; Gal Pressman <galpress@amazon.com>; Xiong, Jianxin <jianxin.xiong@intel.com>; Yishai Hadas
> > > <yishaih@nvidia.com>; linux-rdma <linux-rdma@vger.kernel.org>; Edward Srouji <edwards@nvidia.com>; dri-devel <dri-
> > > devel@lists.freedesktop.org>; Christian Koenig <christian.koenig@amd.com>; Doug Ledford <dledford@redhat.com>; Vetter, Daniel
> > > <daniel.vetter@intel.com>
> > > Subject: Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
> > >
> > > On Mon, Feb 01, 2021 at 03:10:00PM +0100, Daniel Vetter wrote:
> > > > On Mon, Feb 1, 2021 at 7:16 AM Leon Romanovsky <leon@kernel.org> wrote:
> > > > >
> > > > > On Sun, Jan 31, 2021 at 05:31:16PM +0200, Gal Pressman wrote:
> > > > > > On 25/01/2021 21:57, Jianxin Xiong wrote:
> > > > > > > Define a new sub-class of 'MR' that uses dma-buf object for the
> > > > > > > memory region. Define a new class 'DmaBuf' as a wrapper for
> > > > > > > dma-buf allocation mechanism implemented in C.
> > > > > > >
> > > > > > > Update the cmake function for cython modules to allow building
> > > > > > > modules with mixed cython and c source files.
> > > > > > >
> > > > > > > Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
> > > > > > > buildlib/pyverbs_functions.cmake |  78 +++++++----
> > > > > > >  pyverbs/CMakeLists.txt           |  11 +-
> > > > > > >  pyverbs/dmabuf.pxd               |  15 +++
> > > > > > >  pyverbs/dmabuf.pyx               |  73 ++++++++++
> > > > > > >  pyverbs/dmabuf_alloc.c           | 278 +++++++++++++++++++++++++++++++++++++++
> > > > > > >  pyverbs/dmabuf_alloc.h           |  19 +++
> > > > > > >  pyverbs/libibverbs.pxd           |   2 +
> > > > > > >  pyverbs/mr.pxd                   |   6 +
> > > > > > >  pyverbs/mr.pyx                   | 105 ++++++++++++++-
> > > > > > >  9 files changed, 557 insertions(+), 30 deletions(-)  create
> > > > > > > mode 100644 pyverbs/dmabuf.pxd  create mode 100644
> > > > > > > pyverbs/dmabuf.pyx  create mode 100644 pyverbs/dmabuf_alloc.c
> > > > > > > create mode 100644 pyverbs/dmabuf_alloc.h
> > > > >
> > > > > <...>
> > > > >
> > > > > > > index 0000000..05eae75
> > > > > > > +++ b/pyverbs/dmabuf_alloc.c
> > > > > > > @@ -0,0 +1,278 @@
> > > > > > > +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> > > > > > > +/*
> > > > > > > + * Copyright 2020 Intel Corporation. All rights reserved. See
> > > > > > > +COPYING file  */
> > > > > > > +
> > > > > > > +#include <stdio.h>
> > > > > > > +#include <stdlib.h>
> > > > > > > +#include <stdint.h>
> > > > > > > +#include <unistd.h>
> > > > > > > +#include <string.h>
> > > > > > > +#include <errno.h>
> > > > > > > +#include <drm/drm.h>
> > > > > > > +#include <drm/i915_drm.h>
> > > > > > > +#include <drm/amdgpu_drm.h>
> > > > > > > +#include <drm/radeon_drm.h>
> > > > > >
> > > > > > I assume these should come from the kernel headers package, right?
> > > > >
> > > > > This is gross, all kernel headers should be placed in
> > > > > kernel-headers/* and "update" script needs to be extended to take drm/* files too :(.
> > > >
> > > > drm kernel headers are in the libdrm package. You need that anyway for
> > > > doing the ioctls (if you don't hand-roll the restarting yourself).
> > > >
> > > > Also our userspace has gone over to just outright copying the driver
> > > > headers. Not the generic headers, but for the rendering side of gpus,
> > > > which is the topic here, there's really not much generic stuff.
> > > >
> > > > > Jianxin, are you fixing it?
> > > >
> > > > So fix is either to depend upon libdrm for building, or have copies of
> > > > the headers included in the package for the i915/amdgpu/radeon headers
> > > > (drm/drm.h probably not so good idea).
> > >
> > > We should have a cmake test to not build the drm parts if it can't be built, and pyverbs should skip the tests.
> > >
> >
> > Yes, I will add a test for that. Also, on SLES, the headers could be under /usr/include/libdrm instead of /usr/include/drm. The make test should check that and use proper path.
>
> Please use pkgconfig for this, libdrm installs a .pc file to make sure you
> can find the right headers.

rdma-core uses cmake build system and in our case cmake find_library()
is preferable over pkgconfig.

Thanks

> -Daniel
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

^ permalink raw reply	[flat|nested] 42+ messages in thread
* RE: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
  2021-02-03  6:03                 ` Leon Romanovsky
@ 2021-02-03 16:57                   ` Xiong, Jianxin
  -1 siblings, 0 replies; 42+ messages in thread
From: Xiong, Jianxin @ 2021-02-03 16:57 UTC (permalink / raw)
  To: Leon Romanovsky, Daniel Vetter
  Cc: Jason Gunthorpe, Gal Pressman, Yishai Hadas, linux-rdma,
	Edward Srouji, dri-devel, Christian Koenig, Doug Ledford, Vetter,
	Daniel

> -----Original Message-----
> From: Leon Romanovsky <leon@kernel.org>
> Sent: Tuesday, February 02, 2021 10:03 PM
> To: Daniel Vetter <daniel@ffwll.ch>
> Cc: Xiong, Jianxin <jianxin.xiong@intel.com>; Jason Gunthorpe <jgg@ziepe.ca>; Gal Pressman <galpress@amazon.com>; Yishai Hadas
> <yishaih@nvidia.com>; linux-rdma <linux-rdma@vger.kernel.org>; Edward Srouji <edwards@nvidia.com>; dri-devel <dri-
> devel@lists.freedesktop.org>; Christian Koenig <christian.koenig@amd.com>; Doug Ledford <dledford@redhat.com>; Vetter, Daniel
> <daniel.vetter@intel.com>
> Subject: Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
> 

<...>

> > > > > > > > +#include <drm/drm.h>
> > > > > > > > +#include <drm/i915_drm.h> #include <drm/amdgpu_drm.h>
> > > > > > > > +#include <drm/radeon_drm.h>
> > > > > > >
> > > > > > > I assume these should come from the kernel headers package, right?
> > > > > >
> > > > > > This is gross, all kernel headers should be placed in
> > > > > > kernel-headers/* and "update" script needs to be extended to take drm/* files too :(.
> > > > >
> > > > > drm kernel headers are in the libdrm package. You need that
> > > > > anyway for doing the ioctls (if you don't hand-roll the restarting yourself).
> > > > >
> > > > > Also our userspace has gone over to just outright copying the
> > > > > driver headers. Not the generic headers, but for the rendering
> > > > > side of gpus, which is the topic here, there's really not much generic stuff.
> > > > >
> > > > > > Jianxin, are you fixing it?
> > > > >
> > > > > So fix is either to depend upon libdrm for building, or have
> > > > > copies of the headers included in the package for the
> > > > > i915/amdgpu/radeon headers (drm/drm.h probably not so good idea).
> > > >
> > > > We should have a cmake test to not build the drm parts if it can't be built, and pyverbs should skip the tests.
> > > >
> > >
> > > Yes, I will add a test for that. Also, on SLES, the headers could be under /usr/include/libdrm instead of /usr/include/drm. The make test
> should check that and use proper path.
> >
> > Please use pkgconfig for this, libdrm installs a .pc file to make sure
> > you can find the right headers.
> 
> rdma-core uses cmake build system and in our case cmake find_library() is preferable over pkgconfig.

Only the headers are needed, and they could be installed via either the libdrm-devel package or the kernel-headers package. The cmake find_path() command is more suitable here.
 
> 
> Thanks
> 
> > -Daniel
> > --
> > Daniel Vetter
> > Software Engineer, Intel Corporation
> > http://blog.ffwll.ch

^ permalink raw reply	[flat|nested] 42+ messages in thread
* RE: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
@ 2021-02-03 16:57                   ` Xiong, Jianxin
  0 siblings, 0 replies; 42+ messages in thread
From: Xiong, Jianxin @ 2021-02-03 16:57 UTC (permalink / raw)
  To: Leon Romanovsky, Daniel Vetter
  Cc: Yishai Hadas, linux-rdma, Edward Srouji, Gal Pressman, dri-devel,
	Jason Gunthorpe, Doug Ledford, Vetter, Daniel, Christian Koenig

> -----Original Message-----
> From: Leon Romanovsky <leon@kernel.org>
> Sent: Tuesday, February 02, 2021 10:03 PM
> To: Daniel Vetter <daniel@ffwll.ch>
> Cc: Xiong, Jianxin <jianxin.xiong@intel.com>; Jason Gunthorpe <jgg@ziepe.ca>; Gal Pressman <galpress@amazon.com>; Yishai Hadas
> <yishaih@nvidia.com>; linux-rdma <linux-rdma@vger.kernel.org>; Edward Srouji <edwards@nvidia.com>; dri-devel <dri-
> devel@lists.freedesktop.org>; Christian Koenig <christian.koenig@amd.com>; Doug Ledford <dledford@redhat.com>; Vetter, Daniel
> <daniel.vetter@intel.com>
> Subject: Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
> 

<...>

> > > > > > > > +#include <drm/drm.h>
> > > > > > > > +#include <drm/i915_drm.h> #include <drm/amdgpu_drm.h>
> > > > > > > > +#include <drm/radeon_drm.h>
> > > > > > >
> > > > > > > I assume these should come from the kernel headers package, right?
> > > > > >
> > > > > > This is gross, all kernel headers should be placed in
> > > > > > kernel-headers/* and "update" script needs to be extended to take drm/* files too :(.
> > > > >
> > > > > drm kernel headers are in the libdrm package. You need that
> > > > > anyway for doing the ioctls (if you don't hand-roll the restarting yourself).
> > > > >
> > > > > Also our userspace has gone over to just outright copying the
> > > > > driver headers. Not the generic headers, but for the rendering
> > > > > side of gpus, which is the topic here, there's really not much generic stuff.
> > > > >
> > > > > > Jianxin, are you fixing it?
> > > > >
> > > > > So fix is either to depend upon libdrm for building, or have
> > > > > copies of the headers included in the package for the
> > > > > i915/amdgpu/radeon headers (drm/drm.h probably not so good idea).
> > > >
> > > > We should have a cmake test to not build the drm parts if it can't be built, and pyverbs should skip the tests.
> > > >
> > >
> > > Yes, I will add a test for that. Also, on SLES, the headers could be under /usr/include/libdrm instead of /usr/include/drm. The make test
> should check that and use proper path.
> >
> > Please use pkgconfig for this, libdrm installs a .pc file to make sure
> > you can find the right headers.
> 
> rdma-core uses cmake build system and in our case cmake find_library() is preferable over pkgconfig.

Only the headers are needed, and they could be installed via either the libdrm-devel package or the kernel-headers package. The cmake find_path() command is more suitable here.
 
> 
> Thanks
> 
> > -Daniel
> > --
> > Daniel Vetter
> > Software Engineer, Intel Corporation
> > http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

^ permalink raw reply	[flat|nested] 42+ messages in thread
* Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
  2021-02-03 16:57                   ` Xiong, Jianxin
@ 2021-02-03 17:53                     ` Daniel Vetter
  -1 siblings, 0 replies; 42+ messages in thread
From: Daniel Vetter @ 2021-02-03 17:53 UTC (permalink / raw)
  To: Xiong, Jianxin
  Cc: Leon Romanovsky, Jason Gunthorpe, Gal Pressman, Yishai Hadas,
	linux-rdma, Edward Srouji, dri-devel, Christian Koenig,
	Doug Ledford, Vetter, Daniel

On Wed, Feb 3, 2021 at 5:57 PM Xiong, Jianxin <jianxin.xiong@intel.com> wrote:
>
> > -----Original Message-----
> > From: Leon Romanovsky <leon@kernel.org>
> > Sent: Tuesday, February 02, 2021 10:03 PM
> > To: Daniel Vetter <daniel@ffwll.ch>
> > Cc: Xiong, Jianxin <jianxin.xiong@intel.com>; Jason Gunthorpe <jgg@ziepe.ca>; Gal Pressman <galpress@amazon.com>; Yishai Hadas
> > <yishaih@nvidia.com>; linux-rdma <linux-rdma@vger.kernel.org>; Edward Srouji <edwards@nvidia.com>; dri-devel <dri-
> > devel@lists.freedesktop.org>; Christian Koenig <christian.koenig@amd.com>; Doug Ledford <dledford@redhat.com>; Vetter, Daniel
> > <daniel.vetter@intel.com>
> > Subject: Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
> >
>
> <...>
>
> > > > > > > > > +#include <drm/drm.h>
> > > > > > > > > +#include <drm/i915_drm.h> #include <drm/amdgpu_drm.h>
> > > > > > > > > +#include <drm/radeon_drm.h>
> > > > > > > >
> > > > > > > > I assume these should come from the kernel headers package, right?
> > > > > > >
> > > > > > > This is gross, all kernel headers should be placed in
> > > > > > > kernel-headers/* and "update" script needs to be extended to take drm/* files too :(.
> > > > > >
> > > > > > drm kernel headers are in the libdrm package. You need that
> > > > > > anyway for doing the ioctls (if you don't hand-roll the restarting yourself).
> > > > > >
> > > > > > Also our userspace has gone over to just outright copying the
> > > > > > driver headers. Not the generic headers, but for the rendering
> > > > > > side of gpus, which is the topic here, there's really not much generic stuff.
> > > > > >
> > > > > > > Jianxin, are you fixing it?
> > > > > >
> > > > > > So fix is either to depend upon libdrm for building, or have
> > > > > > copies of the headers included in the package for the
> > > > > > i915/amdgpu/radeon headers (drm/drm.h probably not so good idea).
> > > > >
> > > > > We should have a cmake test to not build the drm parts if it can't be built, and pyverbs should skip the tests.
> > > > >
> > > >
> > > > Yes, I will add a test for that. Also, on SLES, the headers could be under /usr/include/libdrm instead of /usr/include/drm. The make test
> > should check that and use proper path.
> > >
> > > Please use pkgconfig for this, libdrm installs a .pc file to make sure
> > > you can find the right headers.
> >
> > rdma-core uses cmake build system and in our case cmake find_library() is preferable over pkgconfig.
>
> Only the headers are needed, and they could be installed via either the libdrm-devel package or the kernel-headers package. The cmake find_path() command is more suitable here.

Except if your distro is buggy (or doesn't support any gpu drivers at
all) they will never be installed as part of kernel-headers.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

^ permalink raw reply	[flat|nested] 42+ messages in thread
* Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
@ 2021-02-03 17:53                     ` Daniel Vetter
  0 siblings, 0 replies; 42+ messages in thread
From: Daniel Vetter @ 2021-02-03 17:53 UTC (permalink / raw)
  To: Xiong, Jianxin
  Cc: Yishai Hadas, Leon Romanovsky, linux-rdma, Edward Srouji,
	Gal Pressman, dri-devel, Jason Gunthorpe, Doug Ledford, Vetter,
	Daniel, Christian Koenig

On Wed, Feb 3, 2021 at 5:57 PM Xiong, Jianxin <jianxin.xiong@intel.com> wrote:
>
> > -----Original Message-----
> > From: Leon Romanovsky <leon@kernel.org>
> > Sent: Tuesday, February 02, 2021 10:03 PM
> > To: Daniel Vetter <daniel@ffwll.ch>
> > Cc: Xiong, Jianxin <jianxin.xiong@intel.com>; Jason Gunthorpe <jgg@ziepe.ca>; Gal Pressman <galpress@amazon.com>; Yishai Hadas
> > <yishaih@nvidia.com>; linux-rdma <linux-rdma@vger.kernel.org>; Edward Srouji <edwards@nvidia.com>; dri-devel <dri-
> > devel@lists.freedesktop.org>; Christian Koenig <christian.koenig@amd.com>; Doug Ledford <dledford@redhat.com>; Vetter, Daniel
> > <daniel.vetter@intel.com>
> > Subject: Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
> >
>
> <...>
>
> > > > > > > > > +#include <drm/drm.h>
> > > > > > > > > +#include <drm/i915_drm.h> #include <drm/amdgpu_drm.h>
> > > > > > > > > +#include <drm/radeon_drm.h>
> > > > > > > >
> > > > > > > > I assume these should come from the kernel headers package, right?
> > > > > > >
> > > > > > > This is gross, all kernel headers should be placed in
> > > > > > > kernel-headers/* and "update" script needs to be extended to take drm/* files too :(.
> > > > > >
> > > > > > drm kernel headers are in the libdrm package. You need that
> > > > > > anyway for doing the ioctls (if you don't hand-roll the restarting yourself).
> > > > > >
> > > > > > Also our userspace has gone over to just outright copying the
> > > > > > driver headers. Not the generic headers, but for the rendering
> > > > > > side of gpus, which is the topic here, there's really not much generic stuff.
> > > > > >
> > > > > > > Jianxin, are you fixing it?
> > > > > >
> > > > > > So fix is either to depend upon libdrm for building, or have
> > > > > > copies of the headers included in the package for the
> > > > > > i915/amdgpu/radeon headers (drm/drm.h probably not so good idea).
> > > > >
> > > > > We should have a cmake test to not build the drm parts if it can't be built, and pyverbs should skip the tests.
> > > > >
> > > >
> > > > Yes, I will add a test for that. Also, on SLES, the headers could be under /usr/include/libdrm instead of /usr/include/drm. The make test
> > should check that and use proper path.
> > >
> > > Please use pkgconfig for this, libdrm installs a .pc file to make sure
> > > you can find the right headers.
> >
> > rdma-core uses cmake build system and in our case cmake find_library() is preferable over pkgconfig.
>
> Only the headers are needed, and they could be installed via either the libdrm-devel package or the kernel-headers package. The cmake find_path() command is more suitable here.

Except if your distro is buggy (or doesn't support any gpu drivers at
all) they will never be installed as part of kernel-headers.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

^ permalink raw reply	[flat|nested] 42+ messages in thread
* Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
  2021-02-03 17:53                     ` Daniel Vetter
@ 2021-02-03 17:56                       ` Jason Gunthorpe
  -1 siblings, 0 replies; 42+ messages in thread
From: Jason Gunthorpe @ 2021-02-03 17:56 UTC (permalink / raw)
  To: Daniel Vetter
  Cc: Xiong, Jianxin, Leon Romanovsky, Gal Pressman, Yishai Hadas,
	linux-rdma, Edward Srouji, dri-devel, Christian Koenig,
	Doug Ledford, Vetter, Daniel

On Wed, Feb 03, 2021 at 06:53:19PM +0100, Daniel Vetter wrote:
> > > rdma-core uses cmake build system and in our case cmake
> > > find_library() is preferable over pkgconfig.
> >
> > Only the headers are needed, and they could be installed via
> > either the libdrm-devel package or the kernel-headers package. The
> > cmake find_path() command is more suitable here.
> 
> Except if your distro is buggy (or doesn't support any gpu drivers at
> all) they will never be installed as part of kernel-headers.

pkgconfig is OK, we use it for libnl for similar reasons, I just don't
like it to see it used gratuitously as it can cause more problems than
it solves

Jason

^ permalink raw reply	[flat|nested] 42+ messages in thread
* Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
@ 2021-02-03 17:56                       ` Jason Gunthorpe
  0 siblings, 0 replies; 42+ messages in thread
From: Jason Gunthorpe @ 2021-02-03 17:56 UTC (permalink / raw)
  To: Daniel Vetter
  Cc: Yishai Hadas, Leon Romanovsky, linux-rdma, Edward Srouji,
	Gal Pressman, dri-devel, Doug Ledford, Vetter, Daniel,
	Christian Koenig, Xiong, Jianxin

On Wed, Feb 03, 2021 at 06:53:19PM +0100, Daniel Vetter wrote:
> > > rdma-core uses cmake build system and in our case cmake
> > > find_library() is preferable over pkgconfig.
> >
> > Only the headers are needed, and they could be installed via
> > either the libdrm-devel package or the kernel-headers package. The
> > cmake find_path() command is more suitable here.
> 
> Except if your distro is buggy (or doesn't support any gpu drivers at
> all) they will never be installed as part of kernel-headers.

pkgconfig is OK, we use it for libnl for similar reasons, I just don't
like it to see it used gratuitously as it can cause more problems than
it solves

Jason
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

^ permalink raw reply	[flat|nested] 42+ messages in thread
* RE: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
  2021-02-03 17:53                     ` Daniel Vetter
@ 2021-02-03 17:58                       ` Xiong, Jianxin
  -1 siblings, 0 replies; 42+ messages in thread
From: Xiong, Jianxin @ 2021-02-03 17:58 UTC (permalink / raw)
  To: Daniel Vetter
  Cc: Leon Romanovsky, Jason Gunthorpe, Gal Pressman, Yishai Hadas,
	linux-rdma, Edward Srouji, dri-devel, Christian Koenig,
	Doug Ledford, Vetter, Daniel

> -----Original Message-----
> From: Daniel Vetter <daniel@ffwll.ch>
> Sent: Wednesday, February 03, 2021 9:53 AM
> To: Xiong, Jianxin <jianxin.xiong@intel.com>
> Cc: Leon Romanovsky <leon@kernel.org>; Jason Gunthorpe <jgg@ziepe.ca>; Gal Pressman <galpress@amazon.com>; Yishai Hadas
> <yishaih@nvidia.com>; linux-rdma <linux-rdma@vger.kernel.org>; Edward Srouji <edwards@nvidia.com>; dri-devel <dri-
> devel@lists.freedesktop.org>; Christian Koenig <christian.koenig@amd.com>; Doug Ledford <dledford@redhat.com>; Vetter, Daniel
> <daniel.vetter@intel.com>
> Subject: Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
> 
> On Wed, Feb 3, 2021 at 5:57 PM Xiong, Jianxin <jianxin.xiong@intel.com> wrote:
> >
> > > -----Original Message-----
> > > From: Leon Romanovsky <leon@kernel.org>
> > > Sent: Tuesday, February 02, 2021 10:03 PM
> > > To: Daniel Vetter <daniel@ffwll.ch>
> > > Cc: Xiong, Jianxin <jianxin.xiong@intel.com>; Jason Gunthorpe
> > > <jgg@ziepe.ca>; Gal Pressman <galpress@amazon.com>; Yishai Hadas
> > > <yishaih@nvidia.com>; linux-rdma <linux-rdma@vger.kernel.org>;
> > > Edward Srouji <edwards@nvidia.com>; dri-devel <dri-
> > > devel@lists.freedesktop.org>; Christian Koenig
> > > <christian.koenig@amd.com>; Doug Ledford <dledford@redhat.com>;
> > > Vetter, Daniel <daniel.vetter@intel.com>
> > > Subject: Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR
> > > support
> > >
> >
> > <...>
> >
> > > > > > > > > > +#include <drm/drm.h>
> > > > > > > > > > +#include <drm/i915_drm.h> #include <drm/amdgpu_drm.h>
> > > > > > > > > > +#include <drm/radeon_drm.h>
> > > > > > > > >
> > > > > > > > > I assume these should come from the kernel headers package, right?
> > > > > > > >
> > > > > > > > This is gross, all kernel headers should be placed in
> > > > > > > > kernel-headers/* and "update" script needs to be extended to take drm/* files too :(.
> > > > > > >
> > > > > > > drm kernel headers are in the libdrm package. You need that
> > > > > > > anyway for doing the ioctls (if you don't hand-roll the restarting yourself).
> > > > > > >
> > > > > > > Also our userspace has gone over to just outright copying
> > > > > > > the driver headers. Not the generic headers, but for the
> > > > > > > rendering side of gpus, which is the topic here, there's really not much generic stuff.
> > > > > > >
> > > > > > > > Jianxin, are you fixing it?
> > > > > > >
> > > > > > > So fix is either to depend upon libdrm for building, or have
> > > > > > > copies of the headers included in the package for the
> > > > > > > i915/amdgpu/radeon headers (drm/drm.h probably not so good idea).
> > > > > >
> > > > > > We should have a cmake test to not build the drm parts if it can't be built, and pyverbs should skip the tests.
> > > > > >
> > > > >
> > > > > Yes, I will add a test for that. Also, on SLES, the headers
> > > > > could be under /usr/include/libdrm instead of /usr/include/drm.
> > > > > The make test
> > > should check that and use proper path.
> > > >
> > > > Please use pkgconfig for this, libdrm installs a .pc file to make
> > > > sure you can find the right headers.
> > >
> > > rdma-core uses cmake build system and in our case cmake find_library() is preferable over pkgconfig.
> >
> > Only the headers are needed, and they could be installed via either the libdrm-devel package or the kernel-headers package. The cmake
> find_path() command is more suitable here.
> 
> Except if your distro is buggy (or doesn't support any gpu drivers at
> all) they will never be installed as part of kernel-headers.

Right, that's why we want to check for the existence of the header file (find_path() does just that) instead of the existence of the package(s).

> -Daniel
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch

^ permalink raw reply	[flat|nested] 42+ messages in thread
* RE: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
@ 2021-02-03 17:58                       ` Xiong, Jianxin
  0 siblings, 0 replies; 42+ messages in thread
From: Xiong, Jianxin @ 2021-02-03 17:58 UTC (permalink / raw)
  To: Daniel Vetter
  Cc: Yishai Hadas, Leon Romanovsky, linux-rdma, Edward Srouji,
	Gal Pressman, dri-devel, Jason Gunthorpe, Doug Ledford, Vetter,
	 Daniel, Christian Koenig

> -----Original Message-----
> From: Daniel Vetter <daniel@ffwll.ch>
> Sent: Wednesday, February 03, 2021 9:53 AM
> To: Xiong, Jianxin <jianxin.xiong@intel.com>
> Cc: Leon Romanovsky <leon@kernel.org>; Jason Gunthorpe <jgg@ziepe.ca>; Gal Pressman <galpress@amazon.com>; Yishai Hadas
> <yishaih@nvidia.com>; linux-rdma <linux-rdma@vger.kernel.org>; Edward Srouji <edwards@nvidia.com>; dri-devel <dri-
> devel@lists.freedesktop.org>; Christian Koenig <christian.koenig@amd.com>; Doug Ledford <dledford@redhat.com>; Vetter, Daniel
> <daniel.vetter@intel.com>
> Subject: Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
> 
> On Wed, Feb 3, 2021 at 5:57 PM Xiong, Jianxin <jianxin.xiong@intel.com> wrote:
> >
> > > -----Original Message-----
> > > From: Leon Romanovsky <leon@kernel.org>
> > > Sent: Tuesday, February 02, 2021 10:03 PM
> > > To: Daniel Vetter <daniel@ffwll.ch>
> > > Cc: Xiong, Jianxin <jianxin.xiong@intel.com>; Jason Gunthorpe
> > > <jgg@ziepe.ca>; Gal Pressman <galpress@amazon.com>; Yishai Hadas
> > > <yishaih@nvidia.com>; linux-rdma <linux-rdma@vger.kernel.org>;
> > > Edward Srouji <edwards@nvidia.com>; dri-devel <dri-
> > > devel@lists.freedesktop.org>; Christian Koenig
> > > <christian.koenig@amd.com>; Doug Ledford <dledford@redhat.com>;
> > > Vetter, Daniel <daniel.vetter@intel.com>
> > > Subject: Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR
> > > support
> > >
> >
> > <...>
> >
> > > > > > > > > > +#include <drm/drm.h>
> > > > > > > > > > +#include <drm/i915_drm.h> #include <drm/amdgpu_drm.h>
> > > > > > > > > > +#include <drm/radeon_drm.h>
> > > > > > > > >
> > > > > > > > > I assume these should come from the kernel headers package, right?
> > > > > > > >
> > > > > > > > This is gross, all kernel headers should be placed in
> > > > > > > > kernel-headers/* and "update" script needs to be extended to take drm/* files too :(.
> > > > > > >
> > > > > > > drm kernel headers are in the libdrm package. You need that
> > > > > > > anyway for doing the ioctls (if you don't hand-roll the restarting yourself).
> > > > > > >
> > > > > > > Also our userspace has gone over to just outright copying
> > > > > > > the driver headers. Not the generic headers, but for the
> > > > > > > rendering side of gpus, which is the topic here, there's really not much generic stuff.
> > > > > > >
> > > > > > > > Jianxin, are you fixing it?
> > > > > > >
> > > > > > > So fix is either to depend upon libdrm for building, or have
> > > > > > > copies of the headers included in the package for the
> > > > > > > i915/amdgpu/radeon headers (drm/drm.h probably not so good idea).
> > > > > >
> > > > > > We should have a cmake test to not build the drm parts if it can't be built, and pyverbs should skip the tests.
> > > > > >
> > > > >
> > > > > Yes, I will add a test for that. Also, on SLES, the headers
> > > > > could be under /usr/include/libdrm instead of /usr/include/drm.
> > > > > The make test
> > > should check that and use proper path.
> > > >
> > > > Please use pkgconfig for this, libdrm installs a .pc file to make
> > > > sure you can find the right headers.
> > >
> > > rdma-core uses cmake build system and in our case cmake find_library() is preferable over pkgconfig.
> >
> > Only the headers are needed, and they could be installed via either the libdrm-devel package or the kernel-headers package. The cmake
> find_path() command is more suitable here.
> 
> Except if your distro is buggy (or doesn't support any gpu drivers at
> all) they will never be installed as part of kernel-headers.

Right, that's why we want to check for the existence of the header file (find_path() does just that) instead of the existence of the package(s).

> -Daniel
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

^ permalink raw reply	[flat|nested] 42+ messages in thread
* Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
  2021-02-03 17:58                       ` Xiong, Jianxin
@ 2021-02-03 18:14                         ` Emil Velikov
  -1 siblings, 0 replies; 42+ messages in thread
From: Emil Velikov @ 2021-02-03 18:14 UTC (permalink / raw)
  To: Xiong, Jianxin
  Cc: Daniel Vetter, Yishai Hadas, Leon Romanovsky, linux-rdma,
	Edward Srouji, Gal Pressman, dri-devel, Jason Gunthorpe,
	Doug Ledford, Vetter, Daniel, Christian Koenig

On Wed, 3 Feb 2021 at 17:58, Xiong, Jianxin <jianxin.xiong@intel.com> wrote:
>
> > -----Original Message-----
> > From: Daniel Vetter <daniel@ffwll.ch>
> > Sent: Wednesday, February 03, 2021 9:53 AM
> > To: Xiong, Jianxin <jianxin.xiong@intel.com>
> > Cc: Leon Romanovsky <leon@kernel.org>; Jason Gunthorpe <jgg@ziepe.ca>; Gal Pressman <galpress@amazon.com>; Yishai Hadas
> > <yishaih@nvidia.com>; linux-rdma <linux-rdma@vger.kernel.org>; Edward Srouji <edwards@nvidia.com>; dri-devel <dri-
> > devel@lists.freedesktop.org>; Christian Koenig <christian.koenig@amd.com>; Doug Ledford <dledford@redhat.com>; Vetter, Daniel
> > <daniel.vetter@intel.com>
> > Subject: Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
> >
> > On Wed, Feb 3, 2021 at 5:57 PM Xiong, Jianxin <jianxin.xiong@intel.com> wrote:
> > >
> > > > -----Original Message-----
> > > > From: Leon Romanovsky <leon@kernel.org>
> > > > Sent: Tuesday, February 02, 2021 10:03 PM
> > > > To: Daniel Vetter <daniel@ffwll.ch>
> > > > Cc: Xiong, Jianxin <jianxin.xiong@intel.com>; Jason Gunthorpe
> > > > <jgg@ziepe.ca>; Gal Pressman <galpress@amazon.com>; Yishai Hadas
> > > > <yishaih@nvidia.com>; linux-rdma <linux-rdma@vger.kernel.org>;
> > > > Edward Srouji <edwards@nvidia.com>; dri-devel <dri-
> > > > devel@lists.freedesktop.org>; Christian Koenig
> > > > <christian.koenig@amd.com>; Doug Ledford <dledford@redhat.com>;
> > > > Vetter, Daniel <daniel.vetter@intel.com>
> > > > Subject: Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR
> > > > support
> > > >
> > >
> > > <...>
> > >
> > > > > > > > > > > +#include <drm/drm.h>
> > > > > > > > > > > +#include <drm/i915_drm.h> #include <drm/amdgpu_drm.h>
> > > > > > > > > > > +#include <drm/radeon_drm.h>
> > > > > > > > > >
> > > > > > > > > > I assume these should come from the kernel headers package, right?
> > > > > > > > >
> > > > > > > > > This is gross, all kernel headers should be placed in
> > > > > > > > > kernel-headers/* and "update" script needs to be extended to take drm/* files too :(.
> > > > > > > >
> > > > > > > > drm kernel headers are in the libdrm package. You need that
> > > > > > > > anyway for doing the ioctls (if you don't hand-roll the restarting yourself).
> > > > > > > >
> > > > > > > > Also our userspace has gone over to just outright copying
> > > > > > > > the driver headers. Not the generic headers, but for the
> > > > > > > > rendering side of gpus, which is the topic here, there's really not much generic stuff.
> > > > > > > >
> > > > > > > > > Jianxin, are you fixing it?
> > > > > > > >
> > > > > > > > So fix is either to depend upon libdrm for building, or have
> > > > > > > > copies of the headers included in the package for the
> > > > > > > > i915/amdgpu/radeon headers (drm/drm.h probably not so good idea).
> > > > > > >
> > > > > > > We should have a cmake test to not build the drm parts if it can't be built, and pyverbs should skip the tests.
> > > > > > >
> > > > > >
> > > > > > Yes, I will add a test for that. Also, on SLES, the headers
> > > > > > could be under /usr/include/libdrm instead of /usr/include/drm.
> > > > > > The make test
> > > > should check that and use proper path.
> > > > >
> > > > > Please use pkgconfig for this, libdrm installs a .pc file to make
> > > > > sure you can find the right headers.
> > > >
> > > > rdma-core uses cmake build system and in our case cmake find_library() is preferable over pkgconfig.
> > >
> > > Only the headers are needed, and they could be installed via either the libdrm-devel package or the kernel-headers package. The cmake
> > find_path() command is more suitable here.
> >
> > Except if your distro is buggy (or doesn't support any gpu drivers at
> > all) they will never be installed as part of kernel-headers.
>
> Right, that's why we want to check for the existence of the header file (find_path() does just that) instead of the existence of the package(s).
>
Fwiw the .pc file does not list "packages", but the location of the
headers and/or libraries.
Last time I tried find_path() wasn't that great, since it had no way
to detect non-standard (aka outside of /usr/include) installs.

HTH
-Emil

^ permalink raw reply	[flat|nested] 42+ messages in thread
* Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
@ 2021-02-03 18:14                         ` Emil Velikov
  0 siblings, 0 replies; 42+ messages in thread
From: Emil Velikov @ 2021-02-03 18:14 UTC (permalink / raw)
  To: Xiong, Jianxin
  Cc: Yishai Hadas, Leon Romanovsky, linux-rdma, Edward Srouji,
	Gal Pressman, dri-devel, Jason Gunthorpe, Doug Ledford, Vetter,
	Daniel, Christian Koenig

On Wed, 3 Feb 2021 at 17:58, Xiong, Jianxin <jianxin.xiong@intel.com> wrote:
>
> > -----Original Message-----
> > From: Daniel Vetter <daniel@ffwll.ch>
> > Sent: Wednesday, February 03, 2021 9:53 AM
> > To: Xiong, Jianxin <jianxin.xiong@intel.com>
> > Cc: Leon Romanovsky <leon@kernel.org>; Jason Gunthorpe <jgg@ziepe.ca>; Gal Pressman <galpress@amazon.com>; Yishai Hadas
> > <yishaih@nvidia.com>; linux-rdma <linux-rdma@vger.kernel.org>; Edward Srouji <edwards@nvidia.com>; dri-devel <dri-
> > devel@lists.freedesktop.org>; Christian Koenig <christian.koenig@amd.com>; Doug Ledford <dledford@redhat.com>; Vetter, Daniel
> > <daniel.vetter@intel.com>
> > Subject: Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support
> >
> > On Wed, Feb 3, 2021 at 5:57 PM Xiong, Jianxin <jianxin.xiong@intel.com> wrote:
> > >
> > > > -----Original Message-----
> > > > From: Leon Romanovsky <leon@kernel.org>
> > > > Sent: Tuesday, February 02, 2021 10:03 PM
> > > > To: Daniel Vetter <daniel@ffwll.ch>
> > > > Cc: Xiong, Jianxin <jianxin.xiong@intel.com>; Jason Gunthorpe
> > > > <jgg@ziepe.ca>; Gal Pressman <galpress@amazon.com>; Yishai Hadas
> > > > <yishaih@nvidia.com>; linux-rdma <linux-rdma@vger.kernel.org>;
> > > > Edward Srouji <edwards@nvidia.com>; dri-devel <dri-
> > > > devel@lists.freedesktop.org>; Christian Koenig
> > > > <christian.koenig@amd.com>; Doug Ledford <dledford@redhat.com>;
> > > > Vetter, Daniel <daniel.vetter@intel.com>
> > > > Subject: Re: [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR
> > > > support
> > > >
> > >
> > > <...>
> > >
> > > > > > > > > > > +#include <drm/drm.h>
> > > > > > > > > > > +#include <drm/i915_drm.h> #include <drm/amdgpu_drm.h>
> > > > > > > > > > > +#include <drm/radeon_drm.h>
> > > > > > > > > >
> > > > > > > > > > I assume these should come from the kernel headers package, right?
> > > > > > > > >
> > > > > > > > > This is gross, all kernel headers should be placed in
> > > > > > > > > kernel-headers/* and "update" script needs to be extended to take drm/* files too :(.
> > > > > > > >
> > > > > > > > drm kernel headers are in the libdrm package. You need that
> > > > > > > > anyway for doing the ioctls (if you don't hand-roll the restarting yourself).
> > > > > > > >
> > > > > > > > Also our userspace has gone over to just outright copying
> > > > > > > > the driver headers. Not the generic headers, but for the
> > > > > > > > rendering side of gpus, which is the topic here, there's really not much generic stuff.
> > > > > > > >
> > > > > > > > > Jianxin, are you fixing it?
> > > > > > > >
> > > > > > > > So fix is either to depend upon libdrm for building, or have
> > > > > > > > copies of the headers included in the package for the
> > > > > > > > i915/amdgpu/radeon headers (drm/drm.h probably not so good idea).
> > > > > > >
> > > > > > > We should have a cmake test to not build the drm parts if it can't be built, and pyverbs should skip the tests.
> > > > > > >
> > > > > >
> > > > > > Yes, I will add a test for that. Also, on SLES, the headers
> > > > > > could be under /usr/include/libdrm instead of /usr/include/drm.
> > > > > > The make test
> > > > should check that and use proper path.
> > > > >
> > > > > Please use pkgconfig for this, libdrm installs a .pc file to make
> > > > > sure you can find the right headers.
> > > >
> > > > rdma-core uses cmake build system and in our case cmake find_library() is preferable over pkgconfig.
> > >
> > > Only the headers are needed, and they could be installed via either the libdrm-devel package or the kernel-headers package. The cmake
> > find_path() command is more suitable here.
> >
> > Except if your distro is buggy (or doesn't support any gpu drivers at
> > all) they will never be installed as part of kernel-headers.
>
> Right, that's why we want to check for the existence of the header file (find_path() does just that) instead of the existence of the package(s).
>
Fwiw the .pc file does not list "packages", but the location of the
headers and/or libraries.
Last time I tried find_path() wasn't that great, since it had no way
to detect non-standard (aka outside of /usr/include) installs.

HTH
-Emil
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

^ permalink raw reply	[flat|nested] 42+ messages in thread
end of thread, other threads:[~2021-02-04  8:33 UTC | newest]

Thread overview: 42+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-25 19:56 [PATCH rdma-core v7 0/6] Add user space dma-buf support Jianxin Xiong
2021-01-25 19:56 ` Jianxin Xiong
2021-01-25 19:56 ` [PATCH rdma-core v7 1/6] Update kernel headers Jianxin Xiong
2021-01-25 19:56   ` Jianxin Xiong
2021-01-25 19:56 ` [PATCH rdma-core v7 2/6] verbs: Support dma-buf based memory region Jianxin Xiong
2021-01-25 19:56   ` Jianxin Xiong
2021-01-25 19:56 ` [PATCH rdma-core v7 3/6] mlx5: " Jianxin Xiong
2021-01-25 19:56   ` Jianxin Xiong
2021-01-25 19:57 ` [PATCH rdma-core v7 4/6] pyverbs: Add dma-buf based MR support Jianxin Xiong
2021-01-25 19:57   ` Jianxin Xiong
2021-01-31 15:31   ` Gal Pressman
2021-01-31 15:31     ` Gal Pressman
2021-02-01  6:16     ` Leon Romanovsky
2021-02-01  6:16       ` Leon Romanovsky
2021-02-01 14:10       ` Daniel Vetter
2021-02-01 14:10         ` Daniel Vetter
2021-02-01 15:29         ` Jason Gunthorpe
2021-02-01 15:29           ` Jason Gunthorpe
2021-02-01 17:03           ` Xiong, Jianxin
2021-02-01 17:03             ` Xiong, Jianxin
2021-02-02 15:24             ` Daniel Vetter
2021-02-02 15:24               ` Daniel Vetter
2021-02-03  6:03               ` Leon Romanovsky
2021-02-03  6:03                 ` Leon Romanovsky
2021-02-03 16:57                 ` Xiong, Jianxin
2021-02-03 16:57                   ` Xiong, Jianxin
2021-02-03 17:53                   ` Daniel Vetter
2021-02-03 17:53                     ` Daniel Vetter
2021-02-03 17:56                     ` Jason Gunthorpe
2021-02-03 17:56                       ` Jason Gunthorpe
2021-02-03 17:58                     ` Xiong, Jianxin
2021-02-03 17:58                       ` Xiong, Jianxin
2021-02-03 18:14                       ` Emil Velikov
2021-02-03 18:14                         ` Emil Velikov
2021-01-25 19:57 ` [PATCH rdma-core v7 5/6] tests: Add tests for dma-buf based memory regions Jianxin Xiong
2021-01-25 19:57   ` Jianxin Xiong
2021-01-31  8:44   ` John Hubbard
2021-01-31  8:44     ` John Hubbard
2021-02-01 17:12     ` Xiong, Jianxin
2021-02-01 17:12       ` Xiong, Jianxin
2021-01-25 19:57 ` [PATCH rdma-core v7 6/6] tests: Bug fix for get_access_flags() Jianxin Xiong
2021-01-25 19:57   ` Jianxin Xiong

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.