* [PATCH rdma-core v2 0/6] Add user space dma-buf support
@ 2020-11-24 21:38 ` Jianxin Xiong
0 siblings, 0 replies; 20+ messages in thread
From: Jianxin Xiong @ 2020-11-24 21:38 UTC (permalink / raw)
To: linux-rdma, dri-devel
Cc: Jianxin Xiong, Doug Ledford, Jason Gunthorpe, Leon Romanovsky,
Sumit Semwal, Christian Koenig, Daniel Vetter
This is the user space counter-part of the kernel patch set to add
dma-buf support to the RDMA subsystem.
This is the second version of the patch series. Most of the issues found
so far in v1 are addressed except for the use of /dev/dri/card* and
dumb_buf for allocating dma-buf object. The current implementation at
least provides a testing setup that works for most GPUs. Alternatives
are being investigated.
Change log:
v2:
* Put the kernel header updates into a separate commit
* Add comments for the data structure used in python ioctl calls
* Fix issues related to symbol versioning
* Fix styling issues: extra spaces, unncecessary variable, typo
* Fix an inproper error code usage
* Put the new op into ibv_context_ops instead if verbs_context
v1: https://www.spinics.net/lists/linux-rdma/msg97865.html
* Add user space API for registering dma-buf based memory regions
* Update pyverbs with the new API
* Add new tests
This series consists of six patches. The first patch updates the
kernel headers for dma-buf support. Patch 2 adds the new API function
and updates the man pages. Patch 3 implements the new API in the mlx5
provider. Patch 3 adds new class definitions to pyverbs for the new API.
Patch 4 adds a set of new tests for the new API. Patch 5 fixes bug in
the utility code of the tests.
Pull request at github: https://github.com/linux-rdma/rdma-core/pull/895
Jianxin Xiong (6):
Update kernel headers
verbs: Support dma-buf based memory region
mlx5: Support dma-buf based memory region
pyverbs: Add dma-buf based MR support
tests: Add tests for dma-buf based memory regions
tests: Bug fix for get_access_flags()
debian/libibverbs1.symbols | 2 +
kernel-headers/rdma/ib_user_ioctl_cmds.h | 14 ++++
kernel-headers/rdma/ib_user_verbs.h | 14 ----
libibverbs/CMakeLists.txt | 2 +-
libibverbs/cmd_mr.c | 38 +++++++++
libibverbs/driver.h | 7 ++
libibverbs/dummy_ops.c | 11 +++
libibverbs/libibverbs.map.in | 6 ++
libibverbs/man/ibv_reg_mr.3 | 21 ++++-
libibverbs/verbs.c | 17 ++++
libibverbs/verbs.h | 11 +++
providers/mlx5/mlx5.c | 2 +
providers/mlx5/mlx5.h | 3 +
providers/mlx5/verbs.c | 22 ++++++
pyverbs/CMakeLists.txt | 2 +
pyverbs/dmabuf.pxd | 13 ++++
pyverbs/dmabuf.pyx | 85 ++++++++++++++++++++
pyverbs/libibverbs.pxd | 2 +
pyverbs/mr.pxd | 5 ++
pyverbs/mr.pyx | 77 +++++++++++++++++-
tests/test_mr.py | 130 ++++++++++++++++++++++++++++++-
tests/utils.py | 29 ++++++-
22 files changed, 491 insertions(+), 22 deletions(-)
create mode 100644 pyverbs/dmabuf.pxd
create mode 100644 pyverbs/dmabuf.pyx
--
1.8.3.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH rdma-core v2 0/6] Add user space dma-buf support
@ 2020-11-24 21:38 ` Jianxin Xiong
0 siblings, 0 replies; 20+ messages in thread
From: Jianxin Xiong @ 2020-11-24 21:38 UTC (permalink / raw)
To: linux-rdma, dri-devel
Cc: Leon Romanovsky, Jason Gunthorpe, Doug Ledford, Daniel Vetter,
Christian Koenig, Jianxin Xiong
This is the user space counter-part of the kernel patch set to add
dma-buf support to the RDMA subsystem.
This is the second version of the patch series. Most of the issues found
so far in v1 are addressed except for the use of /dev/dri/card* and
dumb_buf for allocating dma-buf object. The current implementation at
least provides a testing setup that works for most GPUs. Alternatives
are being investigated.
Change log:
v2:
* Put the kernel header updates into a separate commit
* Add comments for the data structure used in python ioctl calls
* Fix issues related to symbol versioning
* Fix styling issues: extra spaces, unncecessary variable, typo
* Fix an inproper error code usage
* Put the new op into ibv_context_ops instead if verbs_context
v1: https://www.spinics.net/lists/linux-rdma/msg97865.html
* Add user space API for registering dma-buf based memory regions
* Update pyverbs with the new API
* Add new tests
This series consists of six patches. The first patch updates the
kernel headers for dma-buf support. Patch 2 adds the new API function
and updates the man pages. Patch 3 implements the new API in the mlx5
provider. Patch 3 adds new class definitions to pyverbs for the new API.
Patch 4 adds a set of new tests for the new API. Patch 5 fixes bug in
the utility code of the tests.
Pull request at github: https://github.com/linux-rdma/rdma-core/pull/895
Jianxin Xiong (6):
Update kernel headers
verbs: Support dma-buf based memory region
mlx5: Support dma-buf based memory region
pyverbs: Add dma-buf based MR support
tests: Add tests for dma-buf based memory regions
tests: Bug fix for get_access_flags()
debian/libibverbs1.symbols | 2 +
kernel-headers/rdma/ib_user_ioctl_cmds.h | 14 ++++
kernel-headers/rdma/ib_user_verbs.h | 14 ----
libibverbs/CMakeLists.txt | 2 +-
libibverbs/cmd_mr.c | 38 +++++++++
libibverbs/driver.h | 7 ++
libibverbs/dummy_ops.c | 11 +++
libibverbs/libibverbs.map.in | 6 ++
libibverbs/man/ibv_reg_mr.3 | 21 ++++-
libibverbs/verbs.c | 17 ++++
libibverbs/verbs.h | 11 +++
providers/mlx5/mlx5.c | 2 +
providers/mlx5/mlx5.h | 3 +
providers/mlx5/verbs.c | 22 ++++++
pyverbs/CMakeLists.txt | 2 +
pyverbs/dmabuf.pxd | 13 ++++
pyverbs/dmabuf.pyx | 85 ++++++++++++++++++++
pyverbs/libibverbs.pxd | 2 +
pyverbs/mr.pxd | 5 ++
pyverbs/mr.pyx | 77 +++++++++++++++++-
tests/test_mr.py | 130 ++++++++++++++++++++++++++++++-
tests/utils.py | 29 ++++++-
22 files changed, 491 insertions(+), 22 deletions(-)
create mode 100644 pyverbs/dmabuf.pxd
create mode 100644 pyverbs/dmabuf.pyx
--
1.8.3.1
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH rdma-core v2 1/6] Update kernel headers
2020-11-24 21:38 ` Jianxin Xiong
@ 2020-11-24 21:38 ` Jianxin Xiong
-1 siblings, 0 replies; 20+ messages in thread
From: Jianxin Xiong @ 2020-11-24 21:38 UTC (permalink / raw)
To: linux-rdma, dri-devel
Cc: Jianxin Xiong, Doug Ledford, Jason Gunthorpe, Leon Romanovsky,
Sumit Semwal, Christian Koenig, Daniel Vetter
To commit 2eef437c4669 ("RDMA/uverbs: Add uverbs command for dma-buf based
MR registration").
Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
| 14 ++++++++++++++
| 14 --------------
2 files changed, 14 insertions(+), 14 deletions(-)
--git a/kernel-headers/rdma/ib_user_ioctl_cmds.h b/kernel-headers/rdma/ib_user_ioctl_cmds.h
index 7968a18..dafc7eb 100644
--- a/kernel-headers/rdma/ib_user_ioctl_cmds.h
+++ b/kernel-headers/rdma/ib_user_ioctl_cmds.h
@@ -1,5 +1,6 @@
/*
* Copyright (c) 2018, Mellanox Technologies inc. All rights reserved.
+ * Copyright (c) 2020, Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -251,6 +252,7 @@ enum uverbs_methods_mr {
UVERBS_METHOD_MR_DESTROY,
UVERBS_METHOD_ADVISE_MR,
UVERBS_METHOD_QUERY_MR,
+ UVERBS_METHOD_REG_DMABUF_MR,
};
enum uverbs_attrs_mr_destroy_ids {
@@ -272,6 +274,18 @@ enum uverbs_attrs_query_mr_cmd_attr_ids {
UVERBS_ATTR_QUERY_MR_RESP_IOVA,
};
+enum uverbs_attrs_reg_dmabuf_mr_cmd_attr_ids {
+ UVERBS_ATTR_REG_DMABUF_MR_HANDLE,
+ UVERBS_ATTR_REG_DMABUF_MR_PD_HANDLE,
+ UVERBS_ATTR_REG_DMABUF_MR_OFFSET,
+ UVERBS_ATTR_REG_DMABUF_MR_LENGTH,
+ UVERBS_ATTR_REG_DMABUF_MR_IOVA,
+ UVERBS_ATTR_REG_DMABUF_MR_FD,
+ UVERBS_ATTR_REG_DMABUF_MR_ACCESS_FLAGS,
+ UVERBS_ATTR_REG_DMABUF_MR_RESP_LKEY,
+ UVERBS_ATTR_REG_DMABUF_MR_RESP_RKEY,
+};
+
enum uverbs_attrs_create_counters_cmd_attr_ids {
UVERBS_ATTR_CREATE_COUNTERS_HANDLE,
};
--git a/kernel-headers/rdma/ib_user_verbs.h b/kernel-headers/rdma/ib_user_verbs.h
index 456438c..7ee73a0 100644
--- a/kernel-headers/rdma/ib_user_verbs.h
+++ b/kernel-headers/rdma/ib_user_verbs.h
@@ -596,20 +596,6 @@ enum {
IB_UVERBS_CREATE_QP_SUP_COMP_MASK = IB_UVERBS_CREATE_QP_MASK_IND_TABLE,
};
-enum {
- /*
- * This value is equal to IB_QP_DEST_QPN.
- */
- IB_USER_LEGACY_LAST_QP_ATTR_MASK = 1ULL << 20,
-};
-
-enum {
- /*
- * This value is equal to IB_QP_RATE_LIMIT.
- */
- IB_USER_LAST_QP_ATTR_MASK = 1ULL << 25,
-};
-
struct ib_uverbs_ex_create_qp {
__aligned_u64 user_handle;
__u32 pd_handle;
--
1.8.3.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH rdma-core v2 1/6] Update kernel headers
@ 2020-11-24 21:38 ` Jianxin Xiong
0 siblings, 0 replies; 20+ messages in thread
From: Jianxin Xiong @ 2020-11-24 21:38 UTC (permalink / raw)
To: linux-rdma, dri-devel
Cc: Leon Romanovsky, Jason Gunthorpe, Doug Ledford, Daniel Vetter,
Christian Koenig, Jianxin Xiong
To commit 2eef437c4669 ("RDMA/uverbs: Add uverbs command for dma-buf based
MR registration").
Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
| 14 ++++++++++++++
| 14 --------------
2 files changed, 14 insertions(+), 14 deletions(-)
--git a/kernel-headers/rdma/ib_user_ioctl_cmds.h b/kernel-headers/rdma/ib_user_ioctl_cmds.h
index 7968a18..dafc7eb 100644
--- a/kernel-headers/rdma/ib_user_ioctl_cmds.h
+++ b/kernel-headers/rdma/ib_user_ioctl_cmds.h
@@ -1,5 +1,6 @@
/*
* Copyright (c) 2018, Mellanox Technologies inc. All rights reserved.
+ * Copyright (c) 2020, Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -251,6 +252,7 @@ enum uverbs_methods_mr {
UVERBS_METHOD_MR_DESTROY,
UVERBS_METHOD_ADVISE_MR,
UVERBS_METHOD_QUERY_MR,
+ UVERBS_METHOD_REG_DMABUF_MR,
};
enum uverbs_attrs_mr_destroy_ids {
@@ -272,6 +274,18 @@ enum uverbs_attrs_query_mr_cmd_attr_ids {
UVERBS_ATTR_QUERY_MR_RESP_IOVA,
};
+enum uverbs_attrs_reg_dmabuf_mr_cmd_attr_ids {
+ UVERBS_ATTR_REG_DMABUF_MR_HANDLE,
+ UVERBS_ATTR_REG_DMABUF_MR_PD_HANDLE,
+ UVERBS_ATTR_REG_DMABUF_MR_OFFSET,
+ UVERBS_ATTR_REG_DMABUF_MR_LENGTH,
+ UVERBS_ATTR_REG_DMABUF_MR_IOVA,
+ UVERBS_ATTR_REG_DMABUF_MR_FD,
+ UVERBS_ATTR_REG_DMABUF_MR_ACCESS_FLAGS,
+ UVERBS_ATTR_REG_DMABUF_MR_RESP_LKEY,
+ UVERBS_ATTR_REG_DMABUF_MR_RESP_RKEY,
+};
+
enum uverbs_attrs_create_counters_cmd_attr_ids {
UVERBS_ATTR_CREATE_COUNTERS_HANDLE,
};
--git a/kernel-headers/rdma/ib_user_verbs.h b/kernel-headers/rdma/ib_user_verbs.h
index 456438c..7ee73a0 100644
--- a/kernel-headers/rdma/ib_user_verbs.h
+++ b/kernel-headers/rdma/ib_user_verbs.h
@@ -596,20 +596,6 @@ enum {
IB_UVERBS_CREATE_QP_SUP_COMP_MASK = IB_UVERBS_CREATE_QP_MASK_IND_TABLE,
};
-enum {
- /*
- * This value is equal to IB_QP_DEST_QPN.
- */
- IB_USER_LEGACY_LAST_QP_ATTR_MASK = 1ULL << 20,
-};
-
-enum {
- /*
- * This value is equal to IB_QP_RATE_LIMIT.
- */
- IB_USER_LAST_QP_ATTR_MASK = 1ULL << 25,
-};
-
struct ib_uverbs_ex_create_qp {
__aligned_u64 user_handle;
__u32 pd_handle;
--
1.8.3.1
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH rdma-core v2 2/6] verbs: Support dma-buf based memory region
2020-11-24 21:38 ` Jianxin Xiong
@ 2020-11-24 21:38 ` Jianxin Xiong
-1 siblings, 0 replies; 20+ messages in thread
From: Jianxin Xiong @ 2020-11-24 21:38 UTC (permalink / raw)
To: linux-rdma, dri-devel
Cc: Jianxin Xiong, Doug Ledford, Jason Gunthorpe, Leon Romanovsky,
Sumit Semwal, Christian Koenig, Daniel Vetter
Add new API function and new provider method for registering dma-buf
based memory region. Update the man page and bump the API version.
Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
debian/libibverbs1.symbols | 2 ++
libibverbs/CMakeLists.txt | 2 +-
libibverbs/cmd_mr.c | 38 ++++++++++++++++++++++++++++++++++++++
libibverbs/driver.h | 7 +++++++
libibverbs/dummy_ops.c | 11 +++++++++++
libibverbs/libibverbs.map.in | 6 ++++++
libibverbs/man/ibv_reg_mr.3 | 21 +++++++++++++++++++--
libibverbs/verbs.c | 17 +++++++++++++++++
libibverbs/verbs.h | 11 +++++++++++
9 files changed, 112 insertions(+), 3 deletions(-)
diff --git a/debian/libibverbs1.symbols b/debian/libibverbs1.symbols
index 9130f41..fcf4d87 100644
--- a/debian/libibverbs1.symbols
+++ b/debian/libibverbs1.symbols
@@ -9,6 +9,7 @@ libibverbs.so.1 libibverbs1 #MINVER#
IBVERBS_1.9@IBVERBS_1.9 30
IBVERBS_1.10@IBVERBS_1.10 31
IBVERBS_1.11@IBVERBS_1.11 32
+ IBVERBS_1.12@IBVERBS_1.12 33
(symver)IBVERBS_PRIVATE_33 33
_ibv_query_gid_ex@IBVERBS_1.11 32
_ibv_query_gid_table@IBVERBS_1.11 32
@@ -99,6 +100,7 @@ libibverbs.so.1 libibverbs1 #MINVER#
ibv_rate_to_mbps@IBVERBS_1.1 1.1.8
ibv_rate_to_mult@IBVERBS_1.0 1.1.6
ibv_read_sysfs_file@IBVERBS_1.0 1.1.6
+ ibv_reg_dmabuf_mr@IBVERBS_1.12 33
ibv_reg_mr@IBVERBS_1.0 1.1.6
ibv_reg_mr@IBVERBS_1.1 1.1.6
ibv_reg_mr_iova@IBVERBS_1.7 25
diff --git a/libibverbs/CMakeLists.txt b/libibverbs/CMakeLists.txt
index 0fe4256..d075225 100644
--- a/libibverbs/CMakeLists.txt
+++ b/libibverbs/CMakeLists.txt
@@ -21,7 +21,7 @@ configure_file("libibverbs.map.in"
rdma_library(ibverbs "${CMAKE_CURRENT_BINARY_DIR}/libibverbs.map"
# See Documentation/versioning.md
- 1 1.11.${PACKAGE_VERSION}
+ 1 1.12.${PACKAGE_VERSION}
all_providers.c
cmd.c
cmd_ah.c
diff --git a/libibverbs/cmd_mr.c b/libibverbs/cmd_mr.c
index 42dbe42..95ed2d1 100644
--- a/libibverbs/cmd_mr.c
+++ b/libibverbs/cmd_mr.c
@@ -1,5 +1,6 @@
/*
* Copyright (c) 2018 Mellanox Technologies, Ltd. All rights reserved.
+ * Copyright (c) 2020 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -116,3 +117,40 @@ int ibv_cmd_query_mr(struct ibv_pd *pd, struct verbs_mr *vmr,
return 0;
}
+int ibv_cmd_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+ uint64_t iova, int fd, int access,
+ struct verbs_mr *vmr)
+{
+ DECLARE_COMMAND_BUFFER(cmdb, UVERBS_OBJECT_MR,
+ UVERBS_METHOD_REG_DMABUF_MR,
+ 9);
+ struct ib_uverbs_attr *handle;
+ uint32_t lkey, rkey;
+ int ret;
+
+ handle = fill_attr_out_obj(cmdb, UVERBS_ATTR_REG_DMABUF_MR_HANDLE);
+ fill_attr_out_ptr(cmdb, UVERBS_ATTR_REG_DMABUF_MR_RESP_LKEY, &lkey);
+ fill_attr_out_ptr(cmdb, UVERBS_ATTR_REG_DMABUF_MR_RESP_RKEY, &rkey);
+
+ fill_attr_in_obj(cmdb, UVERBS_ATTR_REG_DMABUF_MR_PD_HANDLE, pd->handle);
+ fill_attr_in_uint64(cmdb, UVERBS_ATTR_REG_DMABUF_MR_OFFSET, offset);
+ fill_attr_in_uint64(cmdb, UVERBS_ATTR_REG_DMABUF_MR_LENGTH, length);
+ fill_attr_in_uint64(cmdb, UVERBS_ATTR_REG_DMABUF_MR_IOVA, iova);
+ fill_attr_in_uint32(cmdb, UVERBS_ATTR_REG_DMABUF_MR_FD, fd);
+ fill_attr_in_uint32(cmdb, UVERBS_ATTR_REG_DMABUF_MR_ACCESS_FLAGS, access);
+
+ ret = execute_ioctl(pd->context, cmdb);
+ if (ret)
+ return errno;
+
+ vmr->ibv_mr.handle = read_attr_obj(UVERBS_ATTR_REG_DMABUF_MR_HANDLE,
+ handle);
+ vmr->ibv_mr.context = pd->context;
+ vmr->ibv_mr.lkey = lkey;
+ vmr->ibv_mr.rkey = rkey;
+ vmr->ibv_mr.pd = pd;
+ vmr->ibv_mr.addr = (void *)offset;
+ vmr->ibv_mr.length = length;
+ vmr->mr_type = IBV_MR_TYPE_MR;
+ return 0;
+}
diff --git a/libibverbs/driver.h b/libibverbs/driver.h
index ab80f4b..d6a9d0a 100644
--- a/libibverbs/driver.h
+++ b/libibverbs/driver.h
@@ -2,6 +2,7 @@
* Copyright (c) 2004, 2005 Topspin Communications. All rights reserved.
* Copyright (c) 2005, 2006 Cisco Systems, Inc. All rights reserved.
* Copyright (c) 2005 PathScale, Inc. All rights reserved.
+ * Copyright (c) 2020 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -373,6 +374,9 @@ struct verbs_context_ops {
struct ibv_mr *(*reg_dm_mr)(struct ibv_pd *pd, struct ibv_dm *dm,
uint64_t dm_offset, size_t length,
unsigned int access);
+ struct ibv_mr *(*reg_dmabuf_mr)(struct ibv_pd *pd, uint64_t offset,
+ size_t length, uint64_t iova,
+ int fd, int access);
struct ibv_mr *(*reg_mr)(struct ibv_pd *pd, void *addr, size_t length,
uint64_t hca_va, int access);
int (*req_notify_cq)(struct ibv_cq *cq, int solicited_only);
@@ -498,6 +502,9 @@ int ibv_cmd_advise_mr(struct ibv_pd *pd,
uint32_t flags,
struct ibv_sge *sg_list,
uint32_t num_sge);
+int ibv_cmd_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+ uint64_t iova, int fd, int access,
+ struct verbs_mr *vmr);
int ibv_cmd_alloc_mw(struct ibv_pd *pd, enum ibv_mw_type type,
struct ibv_mw *mw, struct ibv_alloc_mw *cmd,
size_t cmd_size,
diff --git a/libibverbs/dummy_ops.c b/libibverbs/dummy_ops.c
index e5af9e4..64ef279 100644
--- a/libibverbs/dummy_ops.c
+++ b/libibverbs/dummy_ops.c
@@ -1,5 +1,6 @@
/*
* Copyright (c) 2017 Mellanox Technologies, Inc. All rights reserved.
+ * Copyright (c) 2020 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -452,6 +453,14 @@ static struct ibv_mr *reg_mr(struct ibv_pd *pd, void *addr, size_t length,
return NULL;
}
+static struct ibv_mr *reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset,
+ size_t length, uint64_t iova,
+ int fd, int access)
+{
+ errno = EOPNOTSUPP;
+ return NULL;
+}
+
static int req_notify_cq(struct ibv_cq *cq, int solicited_only)
{
return EOPNOTSUPP;
@@ -560,6 +569,7 @@ const struct verbs_context_ops verbs_dummy_ops = {
query_srq,
read_counters,
reg_dm_mr,
+ reg_dmabuf_mr,
reg_mr,
req_notify_cq,
rereg_mr,
@@ -689,6 +699,7 @@ void verbs_set_ops(struct verbs_context *vctx,
SET_PRIV_OP_IC(vctx, set_ece);
SET_PRIV_OP_IC(vctx, unimport_mr);
SET_PRIV_OP_IC(vctx, unimport_pd);
+ SET_OP(ctx, reg_dmabuf_mr);
#undef SET_OP
#undef SET_OP2
diff --git a/libibverbs/libibverbs.map.in b/libibverbs/libibverbs.map.in
index b5ccaca..f67e1ef 100644
--- a/libibverbs/libibverbs.map.in
+++ b/libibverbs/libibverbs.map.in
@@ -148,6 +148,11 @@ IBVERBS_1.11 {
_ibv_query_gid_table;
} IBVERBS_1.10;
+IBVERBS_1.12 {
+ global:
+ ibv_reg_dmabuf_mr;
+} IBVERBS_1.11;
+
/* If any symbols in this stanza change ABI then the entire staza gets a new symbol
version. See the top level CMakeLists.txt for this setting. */
@@ -211,6 +216,7 @@ IBVERBS_PRIVATE_@IBVERBS_PABI_VERSION@ {
ibv_cmd_query_srq;
ibv_cmd_read_counters;
ibv_cmd_reg_dm_mr;
+ ibv_cmd_reg_dmabuf_mr;
ibv_cmd_reg_mr;
ibv_cmd_req_notify_cq;
ibv_cmd_rereg_mr;
diff --git a/libibverbs/man/ibv_reg_mr.3 b/libibverbs/man/ibv_reg_mr.3
index 2bfc955..4975c79 100644
--- a/libibverbs/man/ibv_reg_mr.3
+++ b/libibverbs/man/ibv_reg_mr.3
@@ -3,7 +3,7 @@
.\"
.TH IBV_REG_MR 3 2006-10-31 libibverbs "Libibverbs Programmer's Manual"
.SH "NAME"
-ibv_reg_mr, ibv_reg_mr_iova, ibv_dereg_mr \- register or deregister a memory region (MR)
+ibv_reg_mr, ibv_reg_mr_iova, ibv_reg_dmabuf_mr, ibv_dereg_mr \- register or deregister a memory region (MR)
.SH "SYNOPSIS"
.nf
.B #include <infiniband/verbs.h>
@@ -15,6 +15,9 @@ ibv_reg_mr, ibv_reg_mr_iova, ibv_dereg_mr \- register or deregister a memory reg
.BI " size_t " "length" ", uint64_t " "hca_va" ,
.BI " int " "access" );
.sp
+.BI "struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd " "*pd" ", uint64_t " "offset" ,
+.BI " size_t " "length" ", int " "access" );
+.sp
.BI "int ibv_dereg_mr(struct ibv_mr " "*mr" );
.fi
.SH "DESCRIPTION"
@@ -71,11 +74,25 @@ a lkey or rkey. The offset in the memory region is computed as 'addr +
(iova - hca_va)'. Specifying 0 for hca_va has the same effect as
IBV_ACCESS_ZERO_BASED.
.PP
+.B ibv_reg_dmabuf_mr()
+registers a dma-buf based memory region (MR) associated with the protection domain
+.I pd\fR.
+The MR starts at
+.I offset
+of the dma-buf and its size is
+.I length\fR.
+The dma-buf is identified by the file descriptor
+.I fd\fR.
+The argument
+.I access
+describes the desired memory protection attributes; it is similar to the ibv_reg_mr case except that only the following flags are supported:
+.B IBV_ACCESS_LOCAL_WRITE, IBV_ACCESS_REMOTE_WRITE, IBV_ACCESS_REMOTE_READ, IBV_ACCESS_REMOTE_ATOMIC, IBV_ACCESS_RELAXED_ORDERING.
+.PP
.B ibv_dereg_mr()
deregisters the MR
.I mr\fR.
.SH "RETURN VALUE"
-.B ibv_reg_mr() / ibv_reg_mr_iova()
+.B ibv_reg_mr() / ibv_reg_mr_iova() / ibv_reg_dmabuf_mr()
returns a pointer to the registered MR, or NULL if the request fails.
The local key (\fBL_Key\fR) field
.B lkey
diff --git a/libibverbs/verbs.c b/libibverbs/verbs.c
index 2b0ede8..b944904 100644
--- a/libibverbs/verbs.c
+++ b/libibverbs/verbs.c
@@ -1,6 +1,7 @@
/*
* Copyright (c) 2005 Topspin Communications. All rights reserved.
* Copyright (c) 2006, 2007 Cisco Systems, Inc. All rights reserved.
+ * Copyright (c) 2020 Intel Corperation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -367,6 +368,22 @@ void ibv_unimport_mr(struct ibv_mr *mr)
get_ops(mr->context)->unimport_mr(mr);
}
+struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset,
+ size_t length, int fd, int access)
+{
+ struct ibv_mr *mr;
+
+ mr = get_ops(pd->context)->reg_dmabuf_mr(pd, offset, length, offset,
+ fd, access);
+ if (mr) {
+ mr->context = pd->context;
+ mr->pd = pd;
+ mr->addr = (void *)offset;
+ mr->length = length;
+ }
+ return mr;
+}
+
LATEST_SYMVER_FUNC(ibv_rereg_mr, 1_1, "IBVERBS_1.1",
int,
struct ibv_mr *mr, int flags,
diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h
index ee57e05..9194ad2 100644
--- a/libibverbs/verbs.h
+++ b/libibverbs/verbs.h
@@ -3,6 +3,7 @@
* Copyright (c) 2004, 2011-2012 Intel Corporation. All rights reserved.
* Copyright (c) 2005, 2006, 2007 Cisco Systems, Inc. All rights reserved.
* Copyright (c) 2005 PathScale, Inc. All rights reserved.
+ * Copyright (c) 2020 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -1961,6 +1962,10 @@ struct ibv_context_ops {
void *(*_compat_attach_mcast)(void);
void *(*_compat_detach_mcast)(void);
void *(*_compat_async_event)(void);
+ struct ibv_mr * (*reg_dmabuf_mr)(struct ibv_pd *pd,
+ uint64_t offset, size_t length,
+ uint64_t iova, int fd,
+ int access);
};
struct ibv_context {
@@ -2535,6 +2540,12 @@ __ibv_reg_mr_iova(struct ibv_pd *pd, void *addr, size_t length, uint64_t iova,
__builtin_constant_p( \
((access) & IBV_ACCESS_OPTIONAL_RANGE) == 0))
+/**
+ * ibv_reg_dmabuf_mr - Register a dambuf-based memory region
+ */
+struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+ int fd, int access);
+
enum ibv_rereg_mr_err_code {
/* Old MR is valid, invalid input */
IBV_REREG_MR_ERR_INPUT = -1,
--
1.8.3.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH rdma-core v2 2/6] verbs: Support dma-buf based memory region
@ 2020-11-24 21:38 ` Jianxin Xiong
0 siblings, 0 replies; 20+ messages in thread
From: Jianxin Xiong @ 2020-11-24 21:38 UTC (permalink / raw)
To: linux-rdma, dri-devel
Cc: Leon Romanovsky, Jason Gunthorpe, Doug Ledford, Daniel Vetter,
Christian Koenig, Jianxin Xiong
Add new API function and new provider method for registering dma-buf
based memory region. Update the man page and bump the API version.
Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
debian/libibverbs1.symbols | 2 ++
libibverbs/CMakeLists.txt | 2 +-
libibverbs/cmd_mr.c | 38 ++++++++++++++++++++++++++++++++++++++
libibverbs/driver.h | 7 +++++++
libibverbs/dummy_ops.c | 11 +++++++++++
libibverbs/libibverbs.map.in | 6 ++++++
libibverbs/man/ibv_reg_mr.3 | 21 +++++++++++++++++++--
libibverbs/verbs.c | 17 +++++++++++++++++
libibverbs/verbs.h | 11 +++++++++++
9 files changed, 112 insertions(+), 3 deletions(-)
diff --git a/debian/libibverbs1.symbols b/debian/libibverbs1.symbols
index 9130f41..fcf4d87 100644
--- a/debian/libibverbs1.symbols
+++ b/debian/libibverbs1.symbols
@@ -9,6 +9,7 @@ libibverbs.so.1 libibverbs1 #MINVER#
IBVERBS_1.9@IBVERBS_1.9 30
IBVERBS_1.10@IBVERBS_1.10 31
IBVERBS_1.11@IBVERBS_1.11 32
+ IBVERBS_1.12@IBVERBS_1.12 33
(symver)IBVERBS_PRIVATE_33 33
_ibv_query_gid_ex@IBVERBS_1.11 32
_ibv_query_gid_table@IBVERBS_1.11 32
@@ -99,6 +100,7 @@ libibverbs.so.1 libibverbs1 #MINVER#
ibv_rate_to_mbps@IBVERBS_1.1 1.1.8
ibv_rate_to_mult@IBVERBS_1.0 1.1.6
ibv_read_sysfs_file@IBVERBS_1.0 1.1.6
+ ibv_reg_dmabuf_mr@IBVERBS_1.12 33
ibv_reg_mr@IBVERBS_1.0 1.1.6
ibv_reg_mr@IBVERBS_1.1 1.1.6
ibv_reg_mr_iova@IBVERBS_1.7 25
diff --git a/libibverbs/CMakeLists.txt b/libibverbs/CMakeLists.txt
index 0fe4256..d075225 100644
--- a/libibverbs/CMakeLists.txt
+++ b/libibverbs/CMakeLists.txt
@@ -21,7 +21,7 @@ configure_file("libibverbs.map.in"
rdma_library(ibverbs "${CMAKE_CURRENT_BINARY_DIR}/libibverbs.map"
# See Documentation/versioning.md
- 1 1.11.${PACKAGE_VERSION}
+ 1 1.12.${PACKAGE_VERSION}
all_providers.c
cmd.c
cmd_ah.c
diff --git a/libibverbs/cmd_mr.c b/libibverbs/cmd_mr.c
index 42dbe42..95ed2d1 100644
--- a/libibverbs/cmd_mr.c
+++ b/libibverbs/cmd_mr.c
@@ -1,5 +1,6 @@
/*
* Copyright (c) 2018 Mellanox Technologies, Ltd. All rights reserved.
+ * Copyright (c) 2020 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -116,3 +117,40 @@ int ibv_cmd_query_mr(struct ibv_pd *pd, struct verbs_mr *vmr,
return 0;
}
+int ibv_cmd_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+ uint64_t iova, int fd, int access,
+ struct verbs_mr *vmr)
+{
+ DECLARE_COMMAND_BUFFER(cmdb, UVERBS_OBJECT_MR,
+ UVERBS_METHOD_REG_DMABUF_MR,
+ 9);
+ struct ib_uverbs_attr *handle;
+ uint32_t lkey, rkey;
+ int ret;
+
+ handle = fill_attr_out_obj(cmdb, UVERBS_ATTR_REG_DMABUF_MR_HANDLE);
+ fill_attr_out_ptr(cmdb, UVERBS_ATTR_REG_DMABUF_MR_RESP_LKEY, &lkey);
+ fill_attr_out_ptr(cmdb, UVERBS_ATTR_REG_DMABUF_MR_RESP_RKEY, &rkey);
+
+ fill_attr_in_obj(cmdb, UVERBS_ATTR_REG_DMABUF_MR_PD_HANDLE, pd->handle);
+ fill_attr_in_uint64(cmdb, UVERBS_ATTR_REG_DMABUF_MR_OFFSET, offset);
+ fill_attr_in_uint64(cmdb, UVERBS_ATTR_REG_DMABUF_MR_LENGTH, length);
+ fill_attr_in_uint64(cmdb, UVERBS_ATTR_REG_DMABUF_MR_IOVA, iova);
+ fill_attr_in_uint32(cmdb, UVERBS_ATTR_REG_DMABUF_MR_FD, fd);
+ fill_attr_in_uint32(cmdb, UVERBS_ATTR_REG_DMABUF_MR_ACCESS_FLAGS, access);
+
+ ret = execute_ioctl(pd->context, cmdb);
+ if (ret)
+ return errno;
+
+ vmr->ibv_mr.handle = read_attr_obj(UVERBS_ATTR_REG_DMABUF_MR_HANDLE,
+ handle);
+ vmr->ibv_mr.context = pd->context;
+ vmr->ibv_mr.lkey = lkey;
+ vmr->ibv_mr.rkey = rkey;
+ vmr->ibv_mr.pd = pd;
+ vmr->ibv_mr.addr = (void *)offset;
+ vmr->ibv_mr.length = length;
+ vmr->mr_type = IBV_MR_TYPE_MR;
+ return 0;
+}
diff --git a/libibverbs/driver.h b/libibverbs/driver.h
index ab80f4b..d6a9d0a 100644
--- a/libibverbs/driver.h
+++ b/libibverbs/driver.h
@@ -2,6 +2,7 @@
* Copyright (c) 2004, 2005 Topspin Communications. All rights reserved.
* Copyright (c) 2005, 2006 Cisco Systems, Inc. All rights reserved.
* Copyright (c) 2005 PathScale, Inc. All rights reserved.
+ * Copyright (c) 2020 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -373,6 +374,9 @@ struct verbs_context_ops {
struct ibv_mr *(*reg_dm_mr)(struct ibv_pd *pd, struct ibv_dm *dm,
uint64_t dm_offset, size_t length,
unsigned int access);
+ struct ibv_mr *(*reg_dmabuf_mr)(struct ibv_pd *pd, uint64_t offset,
+ size_t length, uint64_t iova,
+ int fd, int access);
struct ibv_mr *(*reg_mr)(struct ibv_pd *pd, void *addr, size_t length,
uint64_t hca_va, int access);
int (*req_notify_cq)(struct ibv_cq *cq, int solicited_only);
@@ -498,6 +502,9 @@ int ibv_cmd_advise_mr(struct ibv_pd *pd,
uint32_t flags,
struct ibv_sge *sg_list,
uint32_t num_sge);
+int ibv_cmd_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+ uint64_t iova, int fd, int access,
+ struct verbs_mr *vmr);
int ibv_cmd_alloc_mw(struct ibv_pd *pd, enum ibv_mw_type type,
struct ibv_mw *mw, struct ibv_alloc_mw *cmd,
size_t cmd_size,
diff --git a/libibverbs/dummy_ops.c b/libibverbs/dummy_ops.c
index e5af9e4..64ef279 100644
--- a/libibverbs/dummy_ops.c
+++ b/libibverbs/dummy_ops.c
@@ -1,5 +1,6 @@
/*
* Copyright (c) 2017 Mellanox Technologies, Inc. All rights reserved.
+ * Copyright (c) 2020 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -452,6 +453,14 @@ static struct ibv_mr *reg_mr(struct ibv_pd *pd, void *addr, size_t length,
return NULL;
}
+static struct ibv_mr *reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset,
+ size_t length, uint64_t iova,
+ int fd, int access)
+{
+ errno = EOPNOTSUPP;
+ return NULL;
+}
+
static int req_notify_cq(struct ibv_cq *cq, int solicited_only)
{
return EOPNOTSUPP;
@@ -560,6 +569,7 @@ const struct verbs_context_ops verbs_dummy_ops = {
query_srq,
read_counters,
reg_dm_mr,
+ reg_dmabuf_mr,
reg_mr,
req_notify_cq,
rereg_mr,
@@ -689,6 +699,7 @@ void verbs_set_ops(struct verbs_context *vctx,
SET_PRIV_OP_IC(vctx, set_ece);
SET_PRIV_OP_IC(vctx, unimport_mr);
SET_PRIV_OP_IC(vctx, unimport_pd);
+ SET_OP(ctx, reg_dmabuf_mr);
#undef SET_OP
#undef SET_OP2
diff --git a/libibverbs/libibverbs.map.in b/libibverbs/libibverbs.map.in
index b5ccaca..f67e1ef 100644
--- a/libibverbs/libibverbs.map.in
+++ b/libibverbs/libibverbs.map.in
@@ -148,6 +148,11 @@ IBVERBS_1.11 {
_ibv_query_gid_table;
} IBVERBS_1.10;
+IBVERBS_1.12 {
+ global:
+ ibv_reg_dmabuf_mr;
+} IBVERBS_1.11;
+
/* If any symbols in this stanza change ABI then the entire staza gets a new symbol
version. See the top level CMakeLists.txt for this setting. */
@@ -211,6 +216,7 @@ IBVERBS_PRIVATE_@IBVERBS_PABI_VERSION@ {
ibv_cmd_query_srq;
ibv_cmd_read_counters;
ibv_cmd_reg_dm_mr;
+ ibv_cmd_reg_dmabuf_mr;
ibv_cmd_reg_mr;
ibv_cmd_req_notify_cq;
ibv_cmd_rereg_mr;
diff --git a/libibverbs/man/ibv_reg_mr.3 b/libibverbs/man/ibv_reg_mr.3
index 2bfc955..4975c79 100644
--- a/libibverbs/man/ibv_reg_mr.3
+++ b/libibverbs/man/ibv_reg_mr.3
@@ -3,7 +3,7 @@
.\"
.TH IBV_REG_MR 3 2006-10-31 libibverbs "Libibverbs Programmer's Manual"
.SH "NAME"
-ibv_reg_mr, ibv_reg_mr_iova, ibv_dereg_mr \- register or deregister a memory region (MR)
+ibv_reg_mr, ibv_reg_mr_iova, ibv_reg_dmabuf_mr, ibv_dereg_mr \- register or deregister a memory region (MR)
.SH "SYNOPSIS"
.nf
.B #include <infiniband/verbs.h>
@@ -15,6 +15,9 @@ ibv_reg_mr, ibv_reg_mr_iova, ibv_dereg_mr \- register or deregister a memory reg
.BI " size_t " "length" ", uint64_t " "hca_va" ,
.BI " int " "access" );
.sp
+.BI "struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd " "*pd" ", uint64_t " "offset" ,
+.BI " size_t " "length" ", int " "access" );
+.sp
.BI "int ibv_dereg_mr(struct ibv_mr " "*mr" );
.fi
.SH "DESCRIPTION"
@@ -71,11 +74,25 @@ a lkey or rkey. The offset in the memory region is computed as 'addr +
(iova - hca_va)'. Specifying 0 for hca_va has the same effect as
IBV_ACCESS_ZERO_BASED.
.PP
+.B ibv_reg_dmabuf_mr()
+registers a dma-buf based memory region (MR) associated with the protection domain
+.I pd\fR.
+The MR starts at
+.I offset
+of the dma-buf and its size is
+.I length\fR.
+The dma-buf is identified by the file descriptor
+.I fd\fR.
+The argument
+.I access
+describes the desired memory protection attributes; it is similar to the ibv_reg_mr case except that only the following flags are supported:
+.B IBV_ACCESS_LOCAL_WRITE, IBV_ACCESS_REMOTE_WRITE, IBV_ACCESS_REMOTE_READ, IBV_ACCESS_REMOTE_ATOMIC, IBV_ACCESS_RELAXED_ORDERING.
+.PP
.B ibv_dereg_mr()
deregisters the MR
.I mr\fR.
.SH "RETURN VALUE"
-.B ibv_reg_mr() / ibv_reg_mr_iova()
+.B ibv_reg_mr() / ibv_reg_mr_iova() / ibv_reg_dmabuf_mr()
returns a pointer to the registered MR, or NULL if the request fails.
The local key (\fBL_Key\fR) field
.B lkey
diff --git a/libibverbs/verbs.c b/libibverbs/verbs.c
index 2b0ede8..b944904 100644
--- a/libibverbs/verbs.c
+++ b/libibverbs/verbs.c
@@ -1,6 +1,7 @@
/*
* Copyright (c) 2005 Topspin Communications. All rights reserved.
* Copyright (c) 2006, 2007 Cisco Systems, Inc. All rights reserved.
+ * Copyright (c) 2020 Intel Corperation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -367,6 +368,22 @@ void ibv_unimport_mr(struct ibv_mr *mr)
get_ops(mr->context)->unimport_mr(mr);
}
+struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset,
+ size_t length, int fd, int access)
+{
+ struct ibv_mr *mr;
+
+ mr = get_ops(pd->context)->reg_dmabuf_mr(pd, offset, length, offset,
+ fd, access);
+ if (mr) {
+ mr->context = pd->context;
+ mr->pd = pd;
+ mr->addr = (void *)offset;
+ mr->length = length;
+ }
+ return mr;
+}
+
LATEST_SYMVER_FUNC(ibv_rereg_mr, 1_1, "IBVERBS_1.1",
int,
struct ibv_mr *mr, int flags,
diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h
index ee57e05..9194ad2 100644
--- a/libibverbs/verbs.h
+++ b/libibverbs/verbs.h
@@ -3,6 +3,7 @@
* Copyright (c) 2004, 2011-2012 Intel Corporation. All rights reserved.
* Copyright (c) 2005, 2006, 2007 Cisco Systems, Inc. All rights reserved.
* Copyright (c) 2005 PathScale, Inc. All rights reserved.
+ * Copyright (c) 2020 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -1961,6 +1962,10 @@ struct ibv_context_ops {
void *(*_compat_attach_mcast)(void);
void *(*_compat_detach_mcast)(void);
void *(*_compat_async_event)(void);
+ struct ibv_mr * (*reg_dmabuf_mr)(struct ibv_pd *pd,
+ uint64_t offset, size_t length,
+ uint64_t iova, int fd,
+ int access);
};
struct ibv_context {
@@ -2535,6 +2540,12 @@ __ibv_reg_mr_iova(struct ibv_pd *pd, void *addr, size_t length, uint64_t iova,
__builtin_constant_p( \
((access) & IBV_ACCESS_OPTIONAL_RANGE) == 0))
+/**
+ * ibv_reg_dmabuf_mr - Register a dambuf-based memory region
+ */
+struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+ int fd, int access);
+
enum ibv_rereg_mr_err_code {
/* Old MR is valid, invalid input */
IBV_REREG_MR_ERR_INPUT = -1,
--
1.8.3.1
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH rdma-core v2 3/6] mlx5: Support dma-buf based memory region
2020-11-24 21:38 ` Jianxin Xiong
@ 2020-11-24 21:38 ` Jianxin Xiong
-1 siblings, 0 replies; 20+ messages in thread
From: Jianxin Xiong @ 2020-11-24 21:38 UTC (permalink / raw)
To: linux-rdma, dri-devel
Cc: Jianxin Xiong, Doug Ledford, Jason Gunthorpe, Leon Romanovsky,
Sumit Semwal, Christian Koenig, Daniel Vetter
Implement the new provider method for registering dma-buf based memory
regions.
Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
providers/mlx5/mlx5.c | 2 ++
providers/mlx5/mlx5.h | 3 +++
providers/mlx5/verbs.c | 22 ++++++++++++++++++++++
3 files changed, 27 insertions(+)
diff --git a/providers/mlx5/mlx5.c b/providers/mlx5/mlx5.c
index 1378acf..b3e2d57 100644
--- a/providers/mlx5/mlx5.c
+++ b/providers/mlx5/mlx5.c
@@ -1,5 +1,6 @@
/*
* Copyright (c) 2012 Mellanox Technologies, Inc. All rights reserved.
+ * Copyright (c) 2020 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -96,6 +97,7 @@ static const struct verbs_context_ops mlx5_ctx_common_ops = {
.async_event = mlx5_async_event,
.dealloc_pd = mlx5_free_pd,
.reg_mr = mlx5_reg_mr,
+ .reg_dmabuf_mr = mlx5_reg_dmabuf_mr,
.rereg_mr = mlx5_rereg_mr,
.dereg_mr = mlx5_dereg_mr,
.alloc_mw = mlx5_alloc_mw,
diff --git a/providers/mlx5/mlx5.h b/providers/mlx5/mlx5.h
index 8c94f72..17a2470 100644
--- a/providers/mlx5/mlx5.h
+++ b/providers/mlx5/mlx5.h
@@ -1,5 +1,6 @@
/*
* Copyright (c) 2012 Mellanox Technologies, Inc. All rights reserved.
+ * Copyright (c) 2020 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -903,6 +904,8 @@ void mlx5_async_event(struct ibv_context *context,
struct ibv_mr *mlx5_alloc_null_mr(struct ibv_pd *pd);
struct ibv_mr *mlx5_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
uint64_t hca_va, int access);
+struct ibv_mr *mlx5_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+ uint64_t iova, int fd, int access);
int mlx5_rereg_mr(struct verbs_mr *mr, int flags, struct ibv_pd *pd, void *addr,
size_t length, int access);
int mlx5_dereg_mr(struct verbs_mr *mr);
diff --git a/providers/mlx5/verbs.c b/providers/mlx5/verbs.c
index b956156..a7fc3b0 100644
--- a/providers/mlx5/verbs.c
+++ b/providers/mlx5/verbs.c
@@ -1,5 +1,6 @@
/*
* Copyright (c) 2012 Mellanox Technologies, Inc. All rights reserved.
+ * Copyright (c) 2020 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -647,6 +648,27 @@ struct ibv_mr *mlx5_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
return &mr->vmr.ibv_mr;
}
+struct ibv_mr *mlx5_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+ uint64_t iova, int fd, int acc)
+{
+ struct mlx5_mr *mr;
+ int ret;
+
+ mr = calloc(1, sizeof(*mr));
+ if (!mr)
+ return NULL;
+
+ ret = ibv_cmd_reg_dmabuf_mr(pd, offset, length, iova, fd, acc,
+ &mr->vmr);
+ if (ret) {
+ free(mr);
+ return NULL;
+ }
+ mr->alloc_flags = acc;
+
+ return &mr->vmr.ibv_mr;
+}
+
struct ibv_mr *mlx5_alloc_null_mr(struct ibv_pd *pd)
{
struct mlx5_mr *mr;
--
1.8.3.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH rdma-core v2 3/6] mlx5: Support dma-buf based memory region
@ 2020-11-24 21:38 ` Jianxin Xiong
0 siblings, 0 replies; 20+ messages in thread
From: Jianxin Xiong @ 2020-11-24 21:38 UTC (permalink / raw)
To: linux-rdma, dri-devel
Cc: Leon Romanovsky, Jason Gunthorpe, Doug Ledford, Daniel Vetter,
Christian Koenig, Jianxin Xiong
Implement the new provider method for registering dma-buf based memory
regions.
Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
providers/mlx5/mlx5.c | 2 ++
providers/mlx5/mlx5.h | 3 +++
providers/mlx5/verbs.c | 22 ++++++++++++++++++++++
3 files changed, 27 insertions(+)
diff --git a/providers/mlx5/mlx5.c b/providers/mlx5/mlx5.c
index 1378acf..b3e2d57 100644
--- a/providers/mlx5/mlx5.c
+++ b/providers/mlx5/mlx5.c
@@ -1,5 +1,6 @@
/*
* Copyright (c) 2012 Mellanox Technologies, Inc. All rights reserved.
+ * Copyright (c) 2020 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -96,6 +97,7 @@ static const struct verbs_context_ops mlx5_ctx_common_ops = {
.async_event = mlx5_async_event,
.dealloc_pd = mlx5_free_pd,
.reg_mr = mlx5_reg_mr,
+ .reg_dmabuf_mr = mlx5_reg_dmabuf_mr,
.rereg_mr = mlx5_rereg_mr,
.dereg_mr = mlx5_dereg_mr,
.alloc_mw = mlx5_alloc_mw,
diff --git a/providers/mlx5/mlx5.h b/providers/mlx5/mlx5.h
index 8c94f72..17a2470 100644
--- a/providers/mlx5/mlx5.h
+++ b/providers/mlx5/mlx5.h
@@ -1,5 +1,6 @@
/*
* Copyright (c) 2012 Mellanox Technologies, Inc. All rights reserved.
+ * Copyright (c) 2020 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -903,6 +904,8 @@ void mlx5_async_event(struct ibv_context *context,
struct ibv_mr *mlx5_alloc_null_mr(struct ibv_pd *pd);
struct ibv_mr *mlx5_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
uint64_t hca_va, int access);
+struct ibv_mr *mlx5_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+ uint64_t iova, int fd, int access);
int mlx5_rereg_mr(struct verbs_mr *mr, int flags, struct ibv_pd *pd, void *addr,
size_t length, int access);
int mlx5_dereg_mr(struct verbs_mr *mr);
diff --git a/providers/mlx5/verbs.c b/providers/mlx5/verbs.c
index b956156..a7fc3b0 100644
--- a/providers/mlx5/verbs.c
+++ b/providers/mlx5/verbs.c
@@ -1,5 +1,6 @@
/*
* Copyright (c) 2012 Mellanox Technologies, Inc. All rights reserved.
+ * Copyright (c) 2020 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -647,6 +648,27 @@ struct ibv_mr *mlx5_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
return &mr->vmr.ibv_mr;
}
+struct ibv_mr *mlx5_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+ uint64_t iova, int fd, int acc)
+{
+ struct mlx5_mr *mr;
+ int ret;
+
+ mr = calloc(1, sizeof(*mr));
+ if (!mr)
+ return NULL;
+
+ ret = ibv_cmd_reg_dmabuf_mr(pd, offset, length, iova, fd, acc,
+ &mr->vmr);
+ if (ret) {
+ free(mr);
+ return NULL;
+ }
+ mr->alloc_flags = acc;
+
+ return &mr->vmr.ibv_mr;
+}
+
struct ibv_mr *mlx5_alloc_null_mr(struct ibv_pd *pd)
{
struct mlx5_mr *mr;
--
1.8.3.1
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH rdma-core v2 4/6] pyverbs: Add dma-buf based MR support
2020-11-24 21:38 ` Jianxin Xiong
@ 2020-11-24 21:38 ` Jianxin Xiong
-1 siblings, 0 replies; 20+ messages in thread
From: Jianxin Xiong @ 2020-11-24 21:38 UTC (permalink / raw)
To: linux-rdma, dri-devel
Cc: Jianxin Xiong, Doug Ledford, Jason Gunthorpe, Leon Romanovsky,
Sumit Semwal, Christian Koenig, Daniel Vetter
Define a new sub-class of 'MR' that uses dma-buf object for the memory
region. Define a new class 'DmaBuf' for dma-buf object allocation.
Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
pyverbs/CMakeLists.txt | 2 ++
pyverbs/dmabuf.pxd | 13 ++++++++
pyverbs/dmabuf.pyx | 85 ++++++++++++++++++++++++++++++++++++++++++++++++++
pyverbs/libibverbs.pxd | 2 ++
pyverbs/mr.pxd | 5 +++
pyverbs/mr.pyx | 77 +++++++++++++++++++++++++++++++++++++++++++--
6 files changed, 182 insertions(+), 2 deletions(-)
create mode 100644 pyverbs/dmabuf.pxd
create mode 100644 pyverbs/dmabuf.pyx
diff --git a/pyverbs/CMakeLists.txt b/pyverbs/CMakeLists.txt
index 9542c4b..5aee02b 100644
--- a/pyverbs/CMakeLists.txt
+++ b/pyverbs/CMakeLists.txt
@@ -1,5 +1,6 @@
# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
# Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file
+# Copyright (c) 2020, Intel Corporation. All rights reserved.
rdma_cython_module(pyverbs ""
addr.pyx
@@ -16,6 +17,7 @@ rdma_cython_module(pyverbs ""
wr.pyx
xrcd.pyx
srq.pyx
+ dmabuf.pyx
)
rdma_python_module(pyverbs
diff --git a/pyverbs/dmabuf.pxd b/pyverbs/dmabuf.pxd
new file mode 100644
index 0000000..040db4b
--- /dev/null
+++ b/pyverbs/dmabuf.pxd
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
+# Copyright (c) 2020, Intel Corporation. All rights reserved.
+
+#cython: language_level=3
+
+cdef class DmaBuf:
+ cdef int dri_fd
+ cdef int handle
+ cdef int fd
+ cdef unsigned long size
+ cdef unsigned long map_offset
+ cdef object dmabuf_mrs
+ cdef add_ref(self, obj)
diff --git a/pyverbs/dmabuf.pyx b/pyverbs/dmabuf.pyx
new file mode 100644
index 0000000..a72d808
--- /dev/null
+++ b/pyverbs/dmabuf.pyx
@@ -0,0 +1,85 @@
+# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
+# Copyright (c) 2020, Intel Corporation. All rights reserved.
+
+#cython: language_level=3
+
+import weakref
+
+from os import open, close, O_RDWR
+from fcntl import ioctl
+from struct import pack_into, unpack
+from pyverbs.base cimport close_weakrefs
+from pyverbs.mr cimport DmaBufMR
+
+cdef extern from "drm/drm.h":
+ cdef int DRM_IOCTL_MODE_CREATE_DUMB
+ cdef int DRM_IOCTL_MODE_MAP_DUMB
+ cdef int DRM_IOCTL_MODE_DESTROY_DUMB
+ cdef int DRM_IOCTL_PRIME_HANDLE_TO_FD
+
+cdef class DmaBuf:
+ def __init__(self, size, unit=0):
+ """
+ Allocate DmaBuf object from a GPU device. This is done through the
+ DRI device interface (/dev/dri/card*). Usually this requires the
+ effective user id being root or being a member of the 'video' group.
+ :param size: The size (in number of bytes) of the buffer.
+ :param unit: The unit number of the GPU to allocate the buffer from.
+ :return: The newly created DmaBuf object on success.
+ """
+ self.dmabuf_mrs = weakref.WeakSet()
+ self.dri_fd = open('/dev/dri/card'+str(unit), O_RDWR)
+
+ # create a dumb scanout buffer:
+ # struct drm_mode_create_dumb {
+ # __u32 height;
+ # __u32 width;
+ # __u32 bpp;
+ # __u32 flags;
+ # __u32 handle; /* output */
+ # __u32 pitch; /* output */
+ # __u64 size; /* output */
+ # };
+ create_dumb = bytearray(32)
+ pack_into('=iiiiiiq', create_dumb, 0, 1, size, 8, 0, 0, 0, 0)
+ ioctl(self.dri_fd, DRM_IOCTL_MODE_CREATE_DUMB, create_dumb)
+ a, b, c, d, self.handle, e, self.size = unpack('=iiiiiiq', create_dumb)
+
+ # get dma-buf file descriptor from a PRIME handle:
+ # struct drm_prime_handle {
+ # __u32 handle;
+ # __u32 flags;
+ # __s32 fd; /* output */
+ # };
+ prime_handle = bytearray(12)
+ pack_into('=iii', prime_handle, 0, self.handle, O_RDWR, 0)
+ ioctl(self.dri_fd, DRM_IOCTL_PRIME_HANDLE_TO_FD, prime_handle)
+ a, b, self.fd = unpack('=iii', prime_handle)
+
+ # set up for mmap of a dumb scanout buffer:
+ # struct drm_mode_map_dumb {
+ # __u32 handle;
+ # __u32 pad;
+ # __u64 offset; /* output */
+ # };
+ map_dumb = bytearray(16)
+ pack_into('=iiq', map_dumb, 0, self.handle, 0, 0)
+ ioctl(self.dri_fd, DRM_IOCTL_MODE_MAP_DUMB, map_dumb);
+ a, b, self.map_offset = unpack('=iiq', map_dumb);
+
+ def __dealloc__(self):
+ close_weakrefs([self.dmabuf_mrs])
+
+ # destroy a dumb scanout buffer:
+ # struct drm_mode_destroy_dumb {
+ # __u32 handle;
+ # };
+ destroy_dumb = bytearray(4)
+ pack_into('=i', destroy_dumb, 0, self.handle)
+ ioctl(self.dri_fd, DRM_IOCTL_MODE_DESTROY_DUMB, destroy_dumb)
+ close(self.dri_fd)
+
+ cdef add_ref(self, obj):
+ if isinstance(obj, DmaBufMR):
+ self.dmabuf_mrs.add(obj)
+
diff --git a/pyverbs/libibverbs.pxd b/pyverbs/libibverbs.pxd
index 6fbba54..95e51e1 100644
--- a/pyverbs/libibverbs.pxd
+++ b/pyverbs/libibverbs.pxd
@@ -507,6 +507,8 @@ cdef extern from 'infiniband/verbs.h':
ibv_pd *ibv_alloc_pd(ibv_context *context)
int ibv_dealloc_pd(ibv_pd *pd)
ibv_mr *ibv_reg_mr(ibv_pd *pd, void *addr, size_t length, int access)
+ ibv_mr *ibv_reg_dmabuf_mr(ibv_pd *pd, uint64_t offset, size_t length,
+ int fd, int access)
int ibv_dereg_mr(ibv_mr *mr)
int ibv_advise_mr(ibv_pd *pd, uint32_t advice, uint32_t flags,
ibv_sge *sg_list, uint32_t num_sge)
diff --git a/pyverbs/mr.pxd b/pyverbs/mr.pxd
index ebe8ada..b89cf02 100644
--- a/pyverbs/mr.pxd
+++ b/pyverbs/mr.pxd
@@ -1,5 +1,6 @@
# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
# Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file
+# Copyright (c) 2020, Intel Corporation. All rights reserved.
#cython: language_level=3
@@ -33,3 +34,7 @@ cdef class MW(PyverbsCM):
cdef class DMMR(MR):
cdef object dm
+
+cdef class DmaBufMR(MR):
+ cdef object dmabuf
+ cdef unsigned long offset
diff --git a/pyverbs/mr.pyx b/pyverbs/mr.pyx
index 7011da1..4102d3c 100644
--- a/pyverbs/mr.pyx
+++ b/pyverbs/mr.pyx
@@ -1,11 +1,12 @@
# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
# Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file
+# Copyright (c) 2020, Intel Corporation. All rights reserved.
import resource
import logging
from posix.mman cimport mmap, munmap, MAP_PRIVATE, PROT_READ, PROT_WRITE, \
- MAP_ANONYMOUS, MAP_HUGETLB
+ MAP_ANONYMOUS, MAP_HUGETLB, MAP_SHARED
from pyverbs.pyverbs_error import PyverbsError, PyverbsRDMAError, \
PyverbsUserError
from libc.stdint cimport uintptr_t, SIZE_MAX
@@ -14,9 +15,10 @@ from posix.stdlib cimport posix_memalign
from libc.string cimport memcpy, memset
cimport pyverbs.libibverbs_enums as e
from pyverbs.device cimport DM
-from libc.stdlib cimport free
+from libc.stdlib cimport free, malloc
from .cmid cimport CMID
from .pd cimport PD
+from .dmabuf cimport DmaBuf
cdef extern from 'sys/mman.h':
cdef void* MAP_FAILED
@@ -348,6 +350,77 @@ cdef class DMMR(MR):
cpdef read(self, length, offset):
return self.dm.copy_from_dm(offset, length)
+cdef class DmaBufMR(MR):
+ def __init__(self, PD pd not None, length=0, access=0, DmaBuf dmabuf=None, offset=0):
+ """
+ Initializes a DmaBufMR (DMA-BUF Memory Region) of the given length
+ and access flags using the given PD and DmaBuf objects.
+ :param pd: A PD object
+ :param length: Length in bytes
+ :param access: Access flags, see ibv_access_flags enum
+ :param dmabuf: A DmaBuf object
+ :param offset: Byte offset from the beginning of the dma-buf
+ :return: The newly create DMABUFMR
+ """
+ self.logger = logging.getLogger(self.__class__.__name__)
+ if dmabuf is None:
+ dmabuf = DmaBuf(length + offset)
+ self.mr = v.ibv_reg_dmabuf_mr(pd.pd, offset, length, dmabuf.fd, access)
+ if self.mr == NULL:
+ raise PyverbsRDMAErrno('Failed to register a dma-buf MR. length: {len}, access flags: {flags}'.
+ format(len=length, flags=access,))
+ super().__init__(pd, length, access)
+ self.pd = pd
+ self.dmabuf = dmabuf
+ self.offset = offset
+ pd.add_ref(self)
+ dmabuf.add_ref(self)
+ self.logger.debug('Registered dma-buf ibv_mr. Length: {len}, access flags {flags}'.
+ format(len=length, flags=access))
+
+ def write(self, data, length, offset=0):
+ """
+ Write user data to the dma-buf backing the MR
+ :param data: User data to write
+ :param length: Length of the data to write
+ :param offset: Writing offset
+ :return: None
+ """
+ if isinstance(data, str):
+ data = data.encode()
+ # can't access the attributes if self.dmabuf is used w/o cast
+ dmabuf = <DmaBuf>self.dmabuf
+ cdef int off = offset + self.offset
+ cdef void *buf = mmap(NULL, length + off, PROT_READ | PROT_WRITE,
+ MAP_SHARED, dmabuf.dri_fd, dmabuf.map_offset)
+ if buf == MAP_FAILED:
+ raise PyverbsError('Failed to map dma-buf of size {l}'.
+ format(l=length))
+ memcpy(<char*>(buf + off), <char *>data, length)
+ munmap(buf, length + off)
+
+ cpdef read(self, length, offset):
+ """
+ Reads data from the dma-buf backing the MR
+ :param length: Length of data to read
+ :param offset: Reading offset
+ :return: The data on the buffer in the requested offset
+ """
+ # can't access the attributes if self.dmabuf is used w/o cast
+ dmabuf = <DmaBuf>self.dmabuf
+ cdef int off = offset + self.offset
+ cdef void *buf = mmap(NULL, length + off, PROT_READ | PROT_WRITE,
+ MAP_SHARED, dmabuf.dri_fd, dmabuf.map_offset)
+ if buf == MAP_FAILED:
+ raise PyverbsError('Failed to map dma-buf of size {l}'.
+ format(l=length))
+ cdef char *data =<char*>malloc(length)
+ memset(data, 0, length)
+ memcpy(data, <char*>(buf + off), length)
+ munmap(buf, length + off)
+ res = data[:length]
+ free(data)
+ return res
def mwtype2str(mw_type):
mw_types = {1:'IBV_MW_TYPE_1', 2:'IBV_MW_TYPE_2'}
--
1.8.3.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH rdma-core v2 4/6] pyverbs: Add dma-buf based MR support
@ 2020-11-24 21:38 ` Jianxin Xiong
0 siblings, 0 replies; 20+ messages in thread
From: Jianxin Xiong @ 2020-11-24 21:38 UTC (permalink / raw)
To: linux-rdma, dri-devel
Cc: Leon Romanovsky, Jason Gunthorpe, Doug Ledford, Daniel Vetter,
Christian Koenig, Jianxin Xiong
Define a new sub-class of 'MR' that uses dma-buf object for the memory
region. Define a new class 'DmaBuf' for dma-buf object allocation.
Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
pyverbs/CMakeLists.txt | 2 ++
pyverbs/dmabuf.pxd | 13 ++++++++
pyverbs/dmabuf.pyx | 85 ++++++++++++++++++++++++++++++++++++++++++++++++++
pyverbs/libibverbs.pxd | 2 ++
pyverbs/mr.pxd | 5 +++
pyverbs/mr.pyx | 77 +++++++++++++++++++++++++++++++++++++++++++--
6 files changed, 182 insertions(+), 2 deletions(-)
create mode 100644 pyverbs/dmabuf.pxd
create mode 100644 pyverbs/dmabuf.pyx
diff --git a/pyverbs/CMakeLists.txt b/pyverbs/CMakeLists.txt
index 9542c4b..5aee02b 100644
--- a/pyverbs/CMakeLists.txt
+++ b/pyverbs/CMakeLists.txt
@@ -1,5 +1,6 @@
# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
# Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file
+# Copyright (c) 2020, Intel Corporation. All rights reserved.
rdma_cython_module(pyverbs ""
addr.pyx
@@ -16,6 +17,7 @@ rdma_cython_module(pyverbs ""
wr.pyx
xrcd.pyx
srq.pyx
+ dmabuf.pyx
)
rdma_python_module(pyverbs
diff --git a/pyverbs/dmabuf.pxd b/pyverbs/dmabuf.pxd
new file mode 100644
index 0000000..040db4b
--- /dev/null
+++ b/pyverbs/dmabuf.pxd
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
+# Copyright (c) 2020, Intel Corporation. All rights reserved.
+
+#cython: language_level=3
+
+cdef class DmaBuf:
+ cdef int dri_fd
+ cdef int handle
+ cdef int fd
+ cdef unsigned long size
+ cdef unsigned long map_offset
+ cdef object dmabuf_mrs
+ cdef add_ref(self, obj)
diff --git a/pyverbs/dmabuf.pyx b/pyverbs/dmabuf.pyx
new file mode 100644
index 0000000..a72d808
--- /dev/null
+++ b/pyverbs/dmabuf.pyx
@@ -0,0 +1,85 @@
+# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
+# Copyright (c) 2020, Intel Corporation. All rights reserved.
+
+#cython: language_level=3
+
+import weakref
+
+from os import open, close, O_RDWR
+from fcntl import ioctl
+from struct import pack_into, unpack
+from pyverbs.base cimport close_weakrefs
+from pyverbs.mr cimport DmaBufMR
+
+cdef extern from "drm/drm.h":
+ cdef int DRM_IOCTL_MODE_CREATE_DUMB
+ cdef int DRM_IOCTL_MODE_MAP_DUMB
+ cdef int DRM_IOCTL_MODE_DESTROY_DUMB
+ cdef int DRM_IOCTL_PRIME_HANDLE_TO_FD
+
+cdef class DmaBuf:
+ def __init__(self, size, unit=0):
+ """
+ Allocate DmaBuf object from a GPU device. This is done through the
+ DRI device interface (/dev/dri/card*). Usually this requires the
+ effective user id being root or being a member of the 'video' group.
+ :param size: The size (in number of bytes) of the buffer.
+ :param unit: The unit number of the GPU to allocate the buffer from.
+ :return: The newly created DmaBuf object on success.
+ """
+ self.dmabuf_mrs = weakref.WeakSet()
+ self.dri_fd = open('/dev/dri/card'+str(unit), O_RDWR)
+
+ # create a dumb scanout buffer:
+ # struct drm_mode_create_dumb {
+ # __u32 height;
+ # __u32 width;
+ # __u32 bpp;
+ # __u32 flags;
+ # __u32 handle; /* output */
+ # __u32 pitch; /* output */
+ # __u64 size; /* output */
+ # };
+ create_dumb = bytearray(32)
+ pack_into('=iiiiiiq', create_dumb, 0, 1, size, 8, 0, 0, 0, 0)
+ ioctl(self.dri_fd, DRM_IOCTL_MODE_CREATE_DUMB, create_dumb)
+ a, b, c, d, self.handle, e, self.size = unpack('=iiiiiiq', create_dumb)
+
+ # get dma-buf file descriptor from a PRIME handle:
+ # struct drm_prime_handle {
+ # __u32 handle;
+ # __u32 flags;
+ # __s32 fd; /* output */
+ # };
+ prime_handle = bytearray(12)
+ pack_into('=iii', prime_handle, 0, self.handle, O_RDWR, 0)
+ ioctl(self.dri_fd, DRM_IOCTL_PRIME_HANDLE_TO_FD, prime_handle)
+ a, b, self.fd = unpack('=iii', prime_handle)
+
+ # set up for mmap of a dumb scanout buffer:
+ # struct drm_mode_map_dumb {
+ # __u32 handle;
+ # __u32 pad;
+ # __u64 offset; /* output */
+ # };
+ map_dumb = bytearray(16)
+ pack_into('=iiq', map_dumb, 0, self.handle, 0, 0)
+ ioctl(self.dri_fd, DRM_IOCTL_MODE_MAP_DUMB, map_dumb);
+ a, b, self.map_offset = unpack('=iiq', map_dumb);
+
+ def __dealloc__(self):
+ close_weakrefs([self.dmabuf_mrs])
+
+ # destroy a dumb scanout buffer:
+ # struct drm_mode_destroy_dumb {
+ # __u32 handle;
+ # };
+ destroy_dumb = bytearray(4)
+ pack_into('=i', destroy_dumb, 0, self.handle)
+ ioctl(self.dri_fd, DRM_IOCTL_MODE_DESTROY_DUMB, destroy_dumb)
+ close(self.dri_fd)
+
+ cdef add_ref(self, obj):
+ if isinstance(obj, DmaBufMR):
+ self.dmabuf_mrs.add(obj)
+
diff --git a/pyverbs/libibverbs.pxd b/pyverbs/libibverbs.pxd
index 6fbba54..95e51e1 100644
--- a/pyverbs/libibverbs.pxd
+++ b/pyverbs/libibverbs.pxd
@@ -507,6 +507,8 @@ cdef extern from 'infiniband/verbs.h':
ibv_pd *ibv_alloc_pd(ibv_context *context)
int ibv_dealloc_pd(ibv_pd *pd)
ibv_mr *ibv_reg_mr(ibv_pd *pd, void *addr, size_t length, int access)
+ ibv_mr *ibv_reg_dmabuf_mr(ibv_pd *pd, uint64_t offset, size_t length,
+ int fd, int access)
int ibv_dereg_mr(ibv_mr *mr)
int ibv_advise_mr(ibv_pd *pd, uint32_t advice, uint32_t flags,
ibv_sge *sg_list, uint32_t num_sge)
diff --git a/pyverbs/mr.pxd b/pyverbs/mr.pxd
index ebe8ada..b89cf02 100644
--- a/pyverbs/mr.pxd
+++ b/pyverbs/mr.pxd
@@ -1,5 +1,6 @@
# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
# Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file
+# Copyright (c) 2020, Intel Corporation. All rights reserved.
#cython: language_level=3
@@ -33,3 +34,7 @@ cdef class MW(PyverbsCM):
cdef class DMMR(MR):
cdef object dm
+
+cdef class DmaBufMR(MR):
+ cdef object dmabuf
+ cdef unsigned long offset
diff --git a/pyverbs/mr.pyx b/pyverbs/mr.pyx
index 7011da1..4102d3c 100644
--- a/pyverbs/mr.pyx
+++ b/pyverbs/mr.pyx
@@ -1,11 +1,12 @@
# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
# Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file
+# Copyright (c) 2020, Intel Corporation. All rights reserved.
import resource
import logging
from posix.mman cimport mmap, munmap, MAP_PRIVATE, PROT_READ, PROT_WRITE, \
- MAP_ANONYMOUS, MAP_HUGETLB
+ MAP_ANONYMOUS, MAP_HUGETLB, MAP_SHARED
from pyverbs.pyverbs_error import PyverbsError, PyverbsRDMAError, \
PyverbsUserError
from libc.stdint cimport uintptr_t, SIZE_MAX
@@ -14,9 +15,10 @@ from posix.stdlib cimport posix_memalign
from libc.string cimport memcpy, memset
cimport pyverbs.libibverbs_enums as e
from pyverbs.device cimport DM
-from libc.stdlib cimport free
+from libc.stdlib cimport free, malloc
from .cmid cimport CMID
from .pd cimport PD
+from .dmabuf cimport DmaBuf
cdef extern from 'sys/mman.h':
cdef void* MAP_FAILED
@@ -348,6 +350,77 @@ cdef class DMMR(MR):
cpdef read(self, length, offset):
return self.dm.copy_from_dm(offset, length)
+cdef class DmaBufMR(MR):
+ def __init__(self, PD pd not None, length=0, access=0, DmaBuf dmabuf=None, offset=0):
+ """
+ Initializes a DmaBufMR (DMA-BUF Memory Region) of the given length
+ and access flags using the given PD and DmaBuf objects.
+ :param pd: A PD object
+ :param length: Length in bytes
+ :param access: Access flags, see ibv_access_flags enum
+ :param dmabuf: A DmaBuf object
+ :param offset: Byte offset from the beginning of the dma-buf
+ :return: The newly create DMABUFMR
+ """
+ self.logger = logging.getLogger(self.__class__.__name__)
+ if dmabuf is None:
+ dmabuf = DmaBuf(length + offset)
+ self.mr = v.ibv_reg_dmabuf_mr(pd.pd, offset, length, dmabuf.fd, access)
+ if self.mr == NULL:
+ raise PyverbsRDMAErrno('Failed to register a dma-buf MR. length: {len}, access flags: {flags}'.
+ format(len=length, flags=access,))
+ super().__init__(pd, length, access)
+ self.pd = pd
+ self.dmabuf = dmabuf
+ self.offset = offset
+ pd.add_ref(self)
+ dmabuf.add_ref(self)
+ self.logger.debug('Registered dma-buf ibv_mr. Length: {len}, access flags {flags}'.
+ format(len=length, flags=access))
+
+ def write(self, data, length, offset=0):
+ """
+ Write user data to the dma-buf backing the MR
+ :param data: User data to write
+ :param length: Length of the data to write
+ :param offset: Writing offset
+ :return: None
+ """
+ if isinstance(data, str):
+ data = data.encode()
+ # can't access the attributes if self.dmabuf is used w/o cast
+ dmabuf = <DmaBuf>self.dmabuf
+ cdef int off = offset + self.offset
+ cdef void *buf = mmap(NULL, length + off, PROT_READ | PROT_WRITE,
+ MAP_SHARED, dmabuf.dri_fd, dmabuf.map_offset)
+ if buf == MAP_FAILED:
+ raise PyverbsError('Failed to map dma-buf of size {l}'.
+ format(l=length))
+ memcpy(<char*>(buf + off), <char *>data, length)
+ munmap(buf, length + off)
+
+ cpdef read(self, length, offset):
+ """
+ Reads data from the dma-buf backing the MR
+ :param length: Length of data to read
+ :param offset: Reading offset
+ :return: The data on the buffer in the requested offset
+ """
+ # can't access the attributes if self.dmabuf is used w/o cast
+ dmabuf = <DmaBuf>self.dmabuf
+ cdef int off = offset + self.offset
+ cdef void *buf = mmap(NULL, length + off, PROT_READ | PROT_WRITE,
+ MAP_SHARED, dmabuf.dri_fd, dmabuf.map_offset)
+ if buf == MAP_FAILED:
+ raise PyverbsError('Failed to map dma-buf of size {l}'.
+ format(l=length))
+ cdef char *data =<char*>malloc(length)
+ memset(data, 0, length)
+ memcpy(data, <char*>(buf + off), length)
+ munmap(buf, length + off)
+ res = data[:length]
+ free(data)
+ return res
def mwtype2str(mw_type):
mw_types = {1:'IBV_MW_TYPE_1', 2:'IBV_MW_TYPE_2'}
--
1.8.3.1
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH rdma-core v2 5/6] tests: Add tests for dma-buf based memory regions
2020-11-24 21:38 ` Jianxin Xiong
@ 2020-11-24 21:38 ` Jianxin Xiong
-1 siblings, 0 replies; 20+ messages in thread
From: Jianxin Xiong @ 2020-11-24 21:38 UTC (permalink / raw)
To: linux-rdma, dri-devel
Cc: Jianxin Xiong, Doug Ledford, Jason Gunthorpe, Leon Romanovsky,
Sumit Semwal, Christian Koenig, Daniel Vetter
Define a full set of tests similar to regular MR tests. Add a utility
function to generate access flags for dma-buf based MRs because the
set of supported flags is smaller.
Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
tests/test_mr.py | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
tests/utils.py | 25 +++++++++++
2 files changed, 154 insertions(+), 1 deletion(-)
diff --git a/tests/test_mr.py b/tests/test_mr.py
index adc649c..8d7f002 100644
--- a/tests/test_mr.py
+++ b/tests/test_mr.py
@@ -9,9 +9,10 @@ import errno
from tests.base import PyverbsAPITestCase, RCResources, RDMATestCase
from pyverbs.pyverbs_error import PyverbsRDMAError, PyverbsError
-from pyverbs.mr import MR, MW, DMMR, MWBindInfo, MWBind
+from pyverbs.mr import MR, MW, DMMR, DmaBufMR, MWBindInfo, MWBind
from pyverbs.qp import QPCap, QPInitAttr, QPAttr, QP
from pyverbs.wr import SendWR
+from pyverbs.dmabuf import DmaBuf
import pyverbs.device as d
from pyverbs.pd import PD
import pyverbs.enums as e
@@ -366,3 +367,130 @@ class DMMRTest(PyverbsAPITestCase):
dm_mr = DMMR(pd, dm_mr_len, e.IBV_ACCESS_ZERO_BASED,
dm=dm, offset=dm_mr_offset)
dm_mr.close()
+
+class DmaBufMRTest(PyverbsAPITestCase):
+ """
+ Test various functionalities of the DmaBufMR class.
+ """
+ def test_dmabuf_reg_mr(self):
+ """
+ Test ibv_reg_dmabuf_mr()
+ """
+ for ctx, attr, attr_ex in self.devices:
+ with PD(ctx) as pd:
+ flags = u.get_dmabuf_access_flags(ctx)
+ for f in flags:
+ len = u.get_mr_length()
+ off = random.randint(0, len//2)
+ with DmaBufMR(pd, len, f, offset=off) as mr:
+ pass
+
+ def test_dmabuf_dereg_mr(self):
+ """
+ Test ibv_dereg_mr() with DmaBufMR
+ """
+ for ctx, attr, attr_ex in self.devices:
+ with PD(ctx) as pd:
+ flags = u.get_dmabuf_access_flags(ctx)
+ for f in flags:
+ len = u.get_mr_length()
+ off = random.randint(0, len//2)
+ with DmaBufMR(pd, len, f, offset=off) as mr:
+ mr.close()
+
+ def test_dmabuf_dereg_mr_twice(self):
+ """
+ Verify that explicit call to DmaBufMR's close() doesn't fail
+ """
+ for ctx, attr, attr_ex in self.devices:
+ with PD(ctx) as pd:
+ flags = u.get_dmabuf_access_flags(ctx)
+ for f in flags:
+ len = u.get_mr_length()
+ off = random.randint(0, len//2)
+ with DmaBufMR(pd, len, f, offset=off) as mr:
+ # Pyverbs supports multiple destruction of objects,
+ # we are not expecting an exception here.
+ mr.close()
+ mr.close()
+
+ def test_dmabuf_reg_mr_bad_flags(self):
+ """
+ Verify that illegal flags combination fails as expected
+ """
+ for ctx, attr, attr_ex in self.devices:
+ with PD(ctx) as pd:
+ for i in range(5):
+ flags = random.sample([e.IBV_ACCESS_REMOTE_WRITE,
+ e.IBV_ACCESS_REMOTE_ATOMIC],
+ random.randint(1, 2))
+ mr_flags = 0
+ for i in flags:
+ mr_flags += i.value
+ try:
+ DmaBufMR(pd, u.get_mr_length(), mr_flags)
+ except PyverbsRDMAError as err:
+ assert 'Failed to register a dma-buf MR' in err.args[0]
+ else:
+ raise PyverbsRDMAError('Registered a dma-buf MR with illegal falgs')
+
+ def test_dmabuf_write(self):
+ """
+ Test writing to DmaBufMR's buffer
+ """
+ for ctx, attr, attr_ex in self.devices:
+ with PD(ctx) as pd:
+ for i in range(10):
+ mr_len = u.get_mr_length()
+ mr_off = random.randint(0, mr_len//2)
+ flags = u.get_dmabuf_access_flags(ctx)
+ for f in flags:
+ with DmaBufMR(pd, mr_len, f, offset=mr_off) as mr:
+ write_len = min(random.randint(1, MAX_IO_LEN),
+ mr_len)
+ mr.write('a' * write_len, write_len)
+
+ def test_dmabuf_read(self):
+ """
+ Test reading from DmaBufMR's buffer
+ """
+ for ctx, attr, attr_ex in self.devices:
+ with PD(ctx) as pd:
+ for i in range(10):
+ mr_len = u.get_mr_length()
+ mr_off = random.randint(0, mr_len//2)
+ flags = u.get_dmabuf_access_flags(ctx)
+ for f in flags:
+ with DmaBufMR(pd, mr_len, f, offset=mr_off) as mr:
+ write_len = min(random.randint(1, MAX_IO_LEN),
+ mr_len)
+ write_str = 'a' * write_len
+ mr.write(write_str, write_len)
+ read_len = random.randint(1, write_len)
+ offset = random.randint(0, write_len-read_len)
+ read_str = mr.read(read_len, offset).decode()
+ assert read_str in write_str
+
+ def test_dmabuf_lkey(self):
+ """
+ Test reading lkey property
+ """
+ for ctx, attr, attr_ex in self.devices:
+ with PD(ctx) as pd:
+ length = u.get_mr_length()
+ flags = u.get_dmabuf_access_flags(ctx)
+ for f in flags:
+ with DmaBufMR(pd, length, f) as mr:
+ mr.lkey
+
+ def test_dmabuf_rkey(self):
+ """
+ Test reading rkey property
+ """
+ for ctx, attr, attr_ex in self.devices:
+ with PD(ctx) as pd:
+ length = u.get_mr_length()
+ flags = u.get_dmabuf_access_flags(ctx)
+ for f in flags:
+ with DmaBufMR(pd, length, f) as mr:
+ mr.rkey
diff --git a/tests/utils.py b/tests/utils.py
index 7039f41..0ad7110 100644
--- a/tests/utils.py
+++ b/tests/utils.py
@@ -94,6 +94,31 @@ def get_access_flags(ctx):
return arr
+def get_dmabuf_access_flags(ctx):
+ """
+ Similar to get_access_flags, except that dma-buf MR only support
+ a subset of the flags.
+ :param ctx: Device Context to check capabilities
+ :return: A random legal value for MR flags
+ """
+ attr = ctx.query_device()
+ vals = [e.IBV_ACCESS_LOCAL_WRITE, e.IBV_ACCESS_REMOTE_WRITE,
+ e.IBV_ACCESS_REMOTE_READ, e.IBV_ACCESS_REMOTE_ATOMIC,
+ e.IBV_ACCESS_RELAXED_ORDERING]
+ if not attr.atomic_caps & e.IBV_ATOMIC_HCA:
+ vals.remove(e.IBV_ACCESS_REMOTE_ATOMIC)
+ arr = []
+ for i in range(1, len(vals)):
+ tmp = list(com(vals, i))
+ tmp = filter(filter_illegal_access_flags, tmp)
+ for t in tmp: # Iterate legal combinations and bitwise OR them
+ val = 0
+ for flag in t:
+ val += flag.value
+ arr.append(val)
+ return arr
+
+
def get_dm_attrs(dm_len):
"""
Initializes an AllocDmAttr member with the given length and random
--
1.8.3.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH rdma-core v2 5/6] tests: Add tests for dma-buf based memory regions
@ 2020-11-24 21:38 ` Jianxin Xiong
0 siblings, 0 replies; 20+ messages in thread
From: Jianxin Xiong @ 2020-11-24 21:38 UTC (permalink / raw)
To: linux-rdma, dri-devel
Cc: Leon Romanovsky, Jason Gunthorpe, Doug Ledford, Daniel Vetter,
Christian Koenig, Jianxin Xiong
Define a full set of tests similar to regular MR tests. Add a utility
function to generate access flags for dma-buf based MRs because the
set of supported flags is smaller.
Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
tests/test_mr.py | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
tests/utils.py | 25 +++++++++++
2 files changed, 154 insertions(+), 1 deletion(-)
diff --git a/tests/test_mr.py b/tests/test_mr.py
index adc649c..8d7f002 100644
--- a/tests/test_mr.py
+++ b/tests/test_mr.py
@@ -9,9 +9,10 @@ import errno
from tests.base import PyverbsAPITestCase, RCResources, RDMATestCase
from pyverbs.pyverbs_error import PyverbsRDMAError, PyverbsError
-from pyverbs.mr import MR, MW, DMMR, MWBindInfo, MWBind
+from pyverbs.mr import MR, MW, DMMR, DmaBufMR, MWBindInfo, MWBind
from pyverbs.qp import QPCap, QPInitAttr, QPAttr, QP
from pyverbs.wr import SendWR
+from pyverbs.dmabuf import DmaBuf
import pyverbs.device as d
from pyverbs.pd import PD
import pyverbs.enums as e
@@ -366,3 +367,130 @@ class DMMRTest(PyverbsAPITestCase):
dm_mr = DMMR(pd, dm_mr_len, e.IBV_ACCESS_ZERO_BASED,
dm=dm, offset=dm_mr_offset)
dm_mr.close()
+
+class DmaBufMRTest(PyverbsAPITestCase):
+ """
+ Test various functionalities of the DmaBufMR class.
+ """
+ def test_dmabuf_reg_mr(self):
+ """
+ Test ibv_reg_dmabuf_mr()
+ """
+ for ctx, attr, attr_ex in self.devices:
+ with PD(ctx) as pd:
+ flags = u.get_dmabuf_access_flags(ctx)
+ for f in flags:
+ len = u.get_mr_length()
+ off = random.randint(0, len//2)
+ with DmaBufMR(pd, len, f, offset=off) as mr:
+ pass
+
+ def test_dmabuf_dereg_mr(self):
+ """
+ Test ibv_dereg_mr() with DmaBufMR
+ """
+ for ctx, attr, attr_ex in self.devices:
+ with PD(ctx) as pd:
+ flags = u.get_dmabuf_access_flags(ctx)
+ for f in flags:
+ len = u.get_mr_length()
+ off = random.randint(0, len//2)
+ with DmaBufMR(pd, len, f, offset=off) as mr:
+ mr.close()
+
+ def test_dmabuf_dereg_mr_twice(self):
+ """
+ Verify that explicit call to DmaBufMR's close() doesn't fail
+ """
+ for ctx, attr, attr_ex in self.devices:
+ with PD(ctx) as pd:
+ flags = u.get_dmabuf_access_flags(ctx)
+ for f in flags:
+ len = u.get_mr_length()
+ off = random.randint(0, len//2)
+ with DmaBufMR(pd, len, f, offset=off) as mr:
+ # Pyverbs supports multiple destruction of objects,
+ # we are not expecting an exception here.
+ mr.close()
+ mr.close()
+
+ def test_dmabuf_reg_mr_bad_flags(self):
+ """
+ Verify that illegal flags combination fails as expected
+ """
+ for ctx, attr, attr_ex in self.devices:
+ with PD(ctx) as pd:
+ for i in range(5):
+ flags = random.sample([e.IBV_ACCESS_REMOTE_WRITE,
+ e.IBV_ACCESS_REMOTE_ATOMIC],
+ random.randint(1, 2))
+ mr_flags = 0
+ for i in flags:
+ mr_flags += i.value
+ try:
+ DmaBufMR(pd, u.get_mr_length(), mr_flags)
+ except PyverbsRDMAError as err:
+ assert 'Failed to register a dma-buf MR' in err.args[0]
+ else:
+ raise PyverbsRDMAError('Registered a dma-buf MR with illegal falgs')
+
+ def test_dmabuf_write(self):
+ """
+ Test writing to DmaBufMR's buffer
+ """
+ for ctx, attr, attr_ex in self.devices:
+ with PD(ctx) as pd:
+ for i in range(10):
+ mr_len = u.get_mr_length()
+ mr_off = random.randint(0, mr_len//2)
+ flags = u.get_dmabuf_access_flags(ctx)
+ for f in flags:
+ with DmaBufMR(pd, mr_len, f, offset=mr_off) as mr:
+ write_len = min(random.randint(1, MAX_IO_LEN),
+ mr_len)
+ mr.write('a' * write_len, write_len)
+
+ def test_dmabuf_read(self):
+ """
+ Test reading from DmaBufMR's buffer
+ """
+ for ctx, attr, attr_ex in self.devices:
+ with PD(ctx) as pd:
+ for i in range(10):
+ mr_len = u.get_mr_length()
+ mr_off = random.randint(0, mr_len//2)
+ flags = u.get_dmabuf_access_flags(ctx)
+ for f in flags:
+ with DmaBufMR(pd, mr_len, f, offset=mr_off) as mr:
+ write_len = min(random.randint(1, MAX_IO_LEN),
+ mr_len)
+ write_str = 'a' * write_len
+ mr.write(write_str, write_len)
+ read_len = random.randint(1, write_len)
+ offset = random.randint(0, write_len-read_len)
+ read_str = mr.read(read_len, offset).decode()
+ assert read_str in write_str
+
+ def test_dmabuf_lkey(self):
+ """
+ Test reading lkey property
+ """
+ for ctx, attr, attr_ex in self.devices:
+ with PD(ctx) as pd:
+ length = u.get_mr_length()
+ flags = u.get_dmabuf_access_flags(ctx)
+ for f in flags:
+ with DmaBufMR(pd, length, f) as mr:
+ mr.lkey
+
+ def test_dmabuf_rkey(self):
+ """
+ Test reading rkey property
+ """
+ for ctx, attr, attr_ex in self.devices:
+ with PD(ctx) as pd:
+ length = u.get_mr_length()
+ flags = u.get_dmabuf_access_flags(ctx)
+ for f in flags:
+ with DmaBufMR(pd, length, f) as mr:
+ mr.rkey
diff --git a/tests/utils.py b/tests/utils.py
index 7039f41..0ad7110 100644
--- a/tests/utils.py
+++ b/tests/utils.py
@@ -94,6 +94,31 @@ def get_access_flags(ctx):
return arr
+def get_dmabuf_access_flags(ctx):
+ """
+ Similar to get_access_flags, except that dma-buf MR only support
+ a subset of the flags.
+ :param ctx: Device Context to check capabilities
+ :return: A random legal value for MR flags
+ """
+ attr = ctx.query_device()
+ vals = [e.IBV_ACCESS_LOCAL_WRITE, e.IBV_ACCESS_REMOTE_WRITE,
+ e.IBV_ACCESS_REMOTE_READ, e.IBV_ACCESS_REMOTE_ATOMIC,
+ e.IBV_ACCESS_RELAXED_ORDERING]
+ if not attr.atomic_caps & e.IBV_ATOMIC_HCA:
+ vals.remove(e.IBV_ACCESS_REMOTE_ATOMIC)
+ arr = []
+ for i in range(1, len(vals)):
+ tmp = list(com(vals, i))
+ tmp = filter(filter_illegal_access_flags, tmp)
+ for t in tmp: # Iterate legal combinations and bitwise OR them
+ val = 0
+ for flag in t:
+ val += flag.value
+ arr.append(val)
+ return arr
+
+
def get_dm_attrs(dm_len):
"""
Initializes an AllocDmAttr member with the given length and random
--
1.8.3.1
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH rdma-core v2 6/6] tests: Bug fix for get_access_flags()
2020-11-24 21:38 ` Jianxin Xiong
@ 2020-11-24 21:38 ` Jianxin Xiong
-1 siblings, 0 replies; 20+ messages in thread
From: Jianxin Xiong @ 2020-11-24 21:38 UTC (permalink / raw)
To: linux-rdma, dri-devel
Cc: Jianxin Xiong, Doug Ledford, Jason Gunthorpe, Leon Romanovsky,
Sumit Semwal, Christian Koenig, Daniel Vetter
The filter definition is wrong and causes get_access_flags() always
returning empty list. As the result the MR tests using this function
are effectively skipped (but report success).
Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
tests/utils.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tests/utils.py b/tests/utils.py
index 0ad7110..9ff2603 100644
--- a/tests/utils.py
+++ b/tests/utils.py
@@ -55,8 +55,8 @@ def filter_illegal_access_flags(element):
:param element: A list of access flags to check
:return: True if this list is legal, else False
"""
- if e.IBV_ACCESS_REMOTE_ATOMIC in element or e.IBV_ACCESS_REMOTE_WRITE:
- if e.IBV_ACCESS_LOCAL_WRITE:
+ if e.IBV_ACCESS_REMOTE_ATOMIC in element or e.IBV_ACCESS_REMOTE_WRITE in element:
+ if not e.IBV_ACCESS_LOCAL_WRITE in element:
return False
return True
--
1.8.3.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH rdma-core v2 6/6] tests: Bug fix for get_access_flags()
@ 2020-11-24 21:38 ` Jianxin Xiong
0 siblings, 0 replies; 20+ messages in thread
From: Jianxin Xiong @ 2020-11-24 21:38 UTC (permalink / raw)
To: linux-rdma, dri-devel
Cc: Leon Romanovsky, Jason Gunthorpe, Doug Ledford, Daniel Vetter,
Christian Koenig, Jianxin Xiong
The filter definition is wrong and causes get_access_flags() always
returning empty list. As the result the MR tests using this function
are effectively skipped (but report success).
Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
tests/utils.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tests/utils.py b/tests/utils.py
index 0ad7110..9ff2603 100644
--- a/tests/utils.py
+++ b/tests/utils.py
@@ -55,8 +55,8 @@ def filter_illegal_access_flags(element):
:param element: A list of access flags to check
:return: True if this list is legal, else False
"""
- if e.IBV_ACCESS_REMOTE_ATOMIC in element or e.IBV_ACCESS_REMOTE_WRITE:
- if e.IBV_ACCESS_LOCAL_WRITE:
+ if e.IBV_ACCESS_REMOTE_ATOMIC in element or e.IBV_ACCESS_REMOTE_WRITE in element:
+ if not e.IBV_ACCESS_LOCAL_WRITE in element:
return False
return True
--
1.8.3.1
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: [PATCH rdma-core v2 2/6] verbs: Support dma-buf based memory region
2020-11-24 21:38 ` Jianxin Xiong
@ 2020-11-25 8:03 ` Yishai Hadas
-1 siblings, 0 replies; 20+ messages in thread
From: Yishai Hadas @ 2020-11-25 8:03 UTC (permalink / raw)
To: Jianxin Xiong
Cc: linux-rdma, dri-devel, Doug Ledford, Jason Gunthorpe,
Leon Romanovsky, Sumit Semwal, Christian Koenig, Daniel Vetter,
Yishai Hadas
On 11/24/2020 11:38 PM, Jianxin Xiong wrote:
> Add new API function and new provider method for registering dma-buf
> based memory region. Update the man page and bump the API version.
>
> Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
I don't see that this V2 fixes the notes that I published on V1 (fork
handling , man page fix).
> ---
> debian/libibverbs1.symbols | 2 ++
> libibverbs/CMakeLists.txt | 2 +-
> libibverbs/cmd_mr.c | 38 ++++++++++++++++++++++++++++++++++++++
> libibverbs/driver.h | 7 +++++++
> libibverbs/dummy_ops.c | 11 +++++++++++
> libibverbs/libibverbs.map.in | 6 ++++++
> libibverbs/man/ibv_reg_mr.3 | 21 +++++++++++++++++++--
> libibverbs/verbs.c | 17 +++++++++++++++++
> libibverbs/verbs.h | 11 +++++++++++
> 9 files changed, 112 insertions(+), 3 deletions(-)
>
> diff --git a/debian/libibverbs1.symbols b/debian/libibverbs1.symbols
> index 9130f41..fcf4d87 100644
> --- a/debian/libibverbs1.symbols
> +++ b/debian/libibverbs1.symbols
> @@ -9,6 +9,7 @@ libibverbs.so.1 libibverbs1 #MINVER#
> IBVERBS_1.9@IBVERBS_1.9 30
> IBVERBS_1.10@IBVERBS_1.10 31
> IBVERBS_1.11@IBVERBS_1.11 32
> + IBVERBS_1.12@IBVERBS_1.12 33
> (symver)IBVERBS_PRIVATE_33 33
> _ibv_query_gid_ex@IBVERBS_1.11 32
> _ibv_query_gid_table@IBVERBS_1.11 32
> @@ -99,6 +100,7 @@ libibverbs.so.1 libibverbs1 #MINVER#
> ibv_rate_to_mbps@IBVERBS_1.1 1.1.8
> ibv_rate_to_mult@IBVERBS_1.0 1.1.6
> ibv_read_sysfs_file@IBVERBS_1.0 1.1.6
> + ibv_reg_dmabuf_mr@IBVERBS_1.12 33
> ibv_reg_mr@IBVERBS_1.0 1.1.6
> ibv_reg_mr@IBVERBS_1.1 1.1.6
> ibv_reg_mr_iova@IBVERBS_1.7 25
> diff --git a/libibverbs/CMakeLists.txt b/libibverbs/CMakeLists.txt
> index 0fe4256..d075225 100644
> --- a/libibverbs/CMakeLists.txt
> +++ b/libibverbs/CMakeLists.txt
> @@ -21,7 +21,7 @@ configure_file("libibverbs.map.in"
>
> rdma_library(ibverbs "${CMAKE_CURRENT_BINARY_DIR}/libibverbs.map"
> # See Documentation/versioning.md
> - 1 1.11.${PACKAGE_VERSION}
> + 1 1.12.${PACKAGE_VERSION}
> all_providers.c
> cmd.c
> cmd_ah.c
> diff --git a/libibverbs/cmd_mr.c b/libibverbs/cmd_mr.c
> index 42dbe42..95ed2d1 100644
> --- a/libibverbs/cmd_mr.c
> +++ b/libibverbs/cmd_mr.c
> @@ -1,5 +1,6 @@
> /*
> * Copyright (c) 2018 Mellanox Technologies, Ltd. All rights reserved.
> + * Copyright (c) 2020 Intel Corporation. All rights reserved.
> *
> * This software is available to you under a choice of one of two
> * licenses. You may choose to be licensed under the terms of the GNU
> @@ -116,3 +117,40 @@ int ibv_cmd_query_mr(struct ibv_pd *pd, struct verbs_mr *vmr,
> return 0;
> }
>
> +int ibv_cmd_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
> + uint64_t iova, int fd, int access,
> + struct verbs_mr *vmr)
> +{
> + DECLARE_COMMAND_BUFFER(cmdb, UVERBS_OBJECT_MR,
> + UVERBS_METHOD_REG_DMABUF_MR,
> + 9);
> + struct ib_uverbs_attr *handle;
> + uint32_t lkey, rkey;
> + int ret;
> +
> + handle = fill_attr_out_obj(cmdb, UVERBS_ATTR_REG_DMABUF_MR_HANDLE);
> + fill_attr_out_ptr(cmdb, UVERBS_ATTR_REG_DMABUF_MR_RESP_LKEY, &lkey);
> + fill_attr_out_ptr(cmdb, UVERBS_ATTR_REG_DMABUF_MR_RESP_RKEY, &rkey);
> +
> + fill_attr_in_obj(cmdb, UVERBS_ATTR_REG_DMABUF_MR_PD_HANDLE, pd->handle);
> + fill_attr_in_uint64(cmdb, UVERBS_ATTR_REG_DMABUF_MR_OFFSET, offset);
> + fill_attr_in_uint64(cmdb, UVERBS_ATTR_REG_DMABUF_MR_LENGTH, length);
> + fill_attr_in_uint64(cmdb, UVERBS_ATTR_REG_DMABUF_MR_IOVA, iova);
> + fill_attr_in_uint32(cmdb, UVERBS_ATTR_REG_DMABUF_MR_FD, fd);
> + fill_attr_in_uint32(cmdb, UVERBS_ATTR_REG_DMABUF_MR_ACCESS_FLAGS, access);
> +
> + ret = execute_ioctl(pd->context, cmdb);
> + if (ret)
> + return errno;
> +
> + vmr->ibv_mr.handle = read_attr_obj(UVERBS_ATTR_REG_DMABUF_MR_HANDLE,
> + handle);
> + vmr->ibv_mr.context = pd->context;
> + vmr->ibv_mr.lkey = lkey;
> + vmr->ibv_mr.rkey = rkey;
> + vmr->ibv_mr.pd = pd;
> + vmr->ibv_mr.addr = (void *)offset;
> + vmr->ibv_mr.length = length;
> + vmr->mr_type = IBV_MR_TYPE_MR;
> + return 0;
> +}
> diff --git a/libibverbs/driver.h b/libibverbs/driver.h
> index ab80f4b..d6a9d0a 100644
> --- a/libibverbs/driver.h
> +++ b/libibverbs/driver.h
> @@ -2,6 +2,7 @@
> * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved.
> * Copyright (c) 2005, 2006 Cisco Systems, Inc. All rights reserved.
> * Copyright (c) 2005 PathScale, Inc. All rights reserved.
> + * Copyright (c) 2020 Intel Corporation. All rights reserved.
> *
> * This software is available to you under a choice of one of two
> * licenses. You may choose to be licensed under the terms of the GNU
> @@ -373,6 +374,9 @@ struct verbs_context_ops {
> struct ibv_mr *(*reg_dm_mr)(struct ibv_pd *pd, struct ibv_dm *dm,
> uint64_t dm_offset, size_t length,
> unsigned int access);
> + struct ibv_mr *(*reg_dmabuf_mr)(struct ibv_pd *pd, uint64_t offset,
> + size_t length, uint64_t iova,
> + int fd, int access);
> struct ibv_mr *(*reg_mr)(struct ibv_pd *pd, void *addr, size_t length,
> uint64_t hca_va, int access);
> int (*req_notify_cq)(struct ibv_cq *cq, int solicited_only);
> @@ -498,6 +502,9 @@ int ibv_cmd_advise_mr(struct ibv_pd *pd,
> uint32_t flags,
> struct ibv_sge *sg_list,
> uint32_t num_sge);
> +int ibv_cmd_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
> + uint64_t iova, int fd, int access,
> + struct verbs_mr *vmr);
> int ibv_cmd_alloc_mw(struct ibv_pd *pd, enum ibv_mw_type type,
> struct ibv_mw *mw, struct ibv_alloc_mw *cmd,
> size_t cmd_size,
> diff --git a/libibverbs/dummy_ops.c b/libibverbs/dummy_ops.c
> index e5af9e4..64ef279 100644
> --- a/libibverbs/dummy_ops.c
> +++ b/libibverbs/dummy_ops.c
> @@ -1,5 +1,6 @@
> /*
> * Copyright (c) 2017 Mellanox Technologies, Inc. All rights reserved.
> + * Copyright (c) 2020 Intel Corporation. All rights reserved.
> *
> * This software is available to you under a choice of one of two
> * licenses. You may choose to be licensed under the terms of the GNU
> @@ -452,6 +453,14 @@ static struct ibv_mr *reg_mr(struct ibv_pd *pd, void *addr, size_t length,
> return NULL;
> }
>
> +static struct ibv_mr *reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset,
> + size_t length, uint64_t iova,
> + int fd, int access)
> +{
> + errno = EOPNOTSUPP;
> + return NULL;
> +}
> +
> static int req_notify_cq(struct ibv_cq *cq, int solicited_only)
> {
> return EOPNOTSUPP;
> @@ -560,6 +569,7 @@ const struct verbs_context_ops verbs_dummy_ops = {
> query_srq,
> read_counters,
> reg_dm_mr,
> + reg_dmabuf_mr,
> reg_mr,
> req_notify_cq,
> rereg_mr,
> @@ -689,6 +699,7 @@ void verbs_set_ops(struct verbs_context *vctx,
> SET_PRIV_OP_IC(vctx, set_ece);
> SET_PRIV_OP_IC(vctx, unimport_mr);
> SET_PRIV_OP_IC(vctx, unimport_pd);
> + SET_OP(ctx, reg_dmabuf_mr);
>
> #undef SET_OP
> #undef SET_OP2
> diff --git a/libibverbs/libibverbs.map.in b/libibverbs/libibverbs.map.in
> index b5ccaca..f67e1ef 100644
> --- a/libibverbs/libibverbs.map.in
> +++ b/libibverbs/libibverbs.map.in
> @@ -148,6 +148,11 @@ IBVERBS_1.11 {
> _ibv_query_gid_table;
> } IBVERBS_1.10;
>
> +IBVERBS_1.12 {
> + global:
> + ibv_reg_dmabuf_mr;
> +} IBVERBS_1.11;
> +
> /* If any symbols in this stanza change ABI then the entire staza gets a new symbol
> version. See the top level CMakeLists.txt for this setting. */
>
> @@ -211,6 +216,7 @@ IBVERBS_PRIVATE_@IBVERBS_PABI_VERSION@ {
> ibv_cmd_query_srq;
> ibv_cmd_read_counters;
> ibv_cmd_reg_dm_mr;
> + ibv_cmd_reg_dmabuf_mr;
> ibv_cmd_reg_mr;
> ibv_cmd_req_notify_cq;
> ibv_cmd_rereg_mr;
> diff --git a/libibverbs/man/ibv_reg_mr.3 b/libibverbs/man/ibv_reg_mr.3
> index 2bfc955..4975c79 100644
> --- a/libibverbs/man/ibv_reg_mr.3
> +++ b/libibverbs/man/ibv_reg_mr.3
> @@ -3,7 +3,7 @@
> .\"
> .TH IBV_REG_MR 3 2006-10-31 libibverbs "Libibverbs Programmer's Manual"
> .SH "NAME"
> -ibv_reg_mr, ibv_reg_mr_iova, ibv_dereg_mr \- register or deregister a memory region (MR)
> +ibv_reg_mr, ibv_reg_mr_iova, ibv_reg_dmabuf_mr, ibv_dereg_mr \- register or deregister a memory region (MR)
> .SH "SYNOPSIS"
> .nf
> .B #include <infiniband/verbs.h>
> @@ -15,6 +15,9 @@ ibv_reg_mr, ibv_reg_mr_iova, ibv_dereg_mr \- register or deregister a memory reg
> .BI " size_t " "length" ", uint64_t " "hca_va" ,
> .BI " int " "access" );
> .sp
> +.BI "struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd " "*pd" ", uint64_t " "offset" ,
> +.BI " size_t " "length" ", int " "access" );
> +.sp
> .BI "int ibv_dereg_mr(struct ibv_mr " "*mr" );
> .fi
> .SH "DESCRIPTION"
> @@ -71,11 +74,25 @@ a lkey or rkey. The offset in the memory region is computed as 'addr +
> (iova - hca_va)'. Specifying 0 for hca_va has the same effect as
> IBV_ACCESS_ZERO_BASED.
> .PP
> +.B ibv_reg_dmabuf_mr()
> +registers a dma-buf based memory region (MR) associated with the protection domain
> +.I pd\fR.
> +The MR starts at
> +.I offset
> +of the dma-buf and its size is
> +.I length\fR.
> +The dma-buf is identified by the file descriptor
> +.I fd\fR.
> +The argument
> +.I access
> +describes the desired memory protection attributes; it is similar to the ibv_reg_mr case except that only the following flags are supported:
> +.B IBV_ACCESS_LOCAL_WRITE, IBV_ACCESS_REMOTE_WRITE, IBV_ACCESS_REMOTE_READ, IBV_ACCESS_REMOTE_ATOMIC, IBV_ACCESS_RELAXED_ORDERING.
> +.PP
> .B ibv_dereg_mr()
> deregisters the MR
> .I mr\fR.
> .SH "RETURN VALUE"
> -.B ibv_reg_mr() / ibv_reg_mr_iova()
> +.B ibv_reg_mr() / ibv_reg_mr_iova() / ibv_reg_dmabuf_mr()
> returns a pointer to the registered MR, or NULL if the request fails.
> The local key (\fBL_Key\fR) field
> .B lkey
> diff --git a/libibverbs/verbs.c b/libibverbs/verbs.c
> index 2b0ede8..b944904 100644
> --- a/libibverbs/verbs.c
> +++ b/libibverbs/verbs.c
> @@ -1,6 +1,7 @@
> /*
> * Copyright (c) 2005 Topspin Communications. All rights reserved.
> * Copyright (c) 2006, 2007 Cisco Systems, Inc. All rights reserved.
> + * Copyright (c) 2020 Intel Corperation. All rights reserved.
> *
> * This software is available to you under a choice of one of two
> * licenses. You may choose to be licensed under the terms of the GNU
> @@ -367,6 +368,22 @@ void ibv_unimport_mr(struct ibv_mr *mr)
> get_ops(mr->context)->unimport_mr(mr);
> }
>
> +struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset,
> + size_t length, int fd, int access)
> +{
> + struct ibv_mr *mr;
> +
> + mr = get_ops(pd->context)->reg_dmabuf_mr(pd, offset, length, offset,
> + fd, access);
> + if (mr) {
> + mr->context = pd->context;
> + mr->pd = pd;
> + mr->addr = (void *)offset;
> + mr->length = length;
> + }
> + return mr;
> +}
> +
> LATEST_SYMVER_FUNC(ibv_rereg_mr, 1_1, "IBVERBS_1.1",
> int,
> struct ibv_mr *mr, int flags,
> diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h
> index ee57e05..9194ad2 100644
> --- a/libibverbs/verbs.h
> +++ b/libibverbs/verbs.h
> @@ -3,6 +3,7 @@
> * Copyright (c) 2004, 2011-2012 Intel Corporation. All rights reserved.
> * Copyright (c) 2005, 2006, 2007 Cisco Systems, Inc. All rights reserved.
> * Copyright (c) 2005 PathScale, Inc. All rights reserved.
> + * Copyright (c) 2020 Intel Corporation. All rights reserved.
> *
> * This software is available to you under a choice of one of two
> * licenses. You may choose to be licensed under the terms of the GNU
> @@ -1961,6 +1962,10 @@ struct ibv_context_ops {
> void *(*_compat_attach_mcast)(void);
> void *(*_compat_detach_mcast)(void);
> void *(*_compat_async_event)(void);
> + struct ibv_mr * (*reg_dmabuf_mr)(struct ibv_pd *pd,
> + uint64_t offset, size_t length,
> + uint64_t iova, int fd,
> + int access);
> };
>
> struct ibv_context {
> @@ -2535,6 +2540,12 @@ __ibv_reg_mr_iova(struct ibv_pd *pd, void *addr, size_t length, uint64_t iova,
> __builtin_constant_p( \
> ((access) & IBV_ACCESS_OPTIONAL_RANGE) == 0))
>
> +/**
> + * ibv_reg_dmabuf_mr - Register a dambuf-based memory region
> + */
> +struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
> + int fd, int access);
> +
> enum ibv_rereg_mr_err_code {
> /* Old MR is valid, invalid input */
> IBV_REREG_MR_ERR_INPUT = -1,
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH rdma-core v2 2/6] verbs: Support dma-buf based memory region
@ 2020-11-25 8:03 ` Yishai Hadas
0 siblings, 0 replies; 20+ messages in thread
From: Yishai Hadas @ 2020-11-25 8:03 UTC (permalink / raw)
To: Jianxin Xiong
Cc: Yishai Hadas, Leon Romanovsky, linux-rdma, dri-devel,
Christian Koenig, Jason Gunthorpe, Doug Ledford, Daniel Vetter
On 11/24/2020 11:38 PM, Jianxin Xiong wrote:
> Add new API function and new provider method for registering dma-buf
> based memory region. Update the man page and bump the API version.
>
> Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
I don't see that this V2 fixes the notes that I published on V1 (fork
handling , man page fix).
> ---
> debian/libibverbs1.symbols | 2 ++
> libibverbs/CMakeLists.txt | 2 +-
> libibverbs/cmd_mr.c | 38 ++++++++++++++++++++++++++++++++++++++
> libibverbs/driver.h | 7 +++++++
> libibverbs/dummy_ops.c | 11 +++++++++++
> libibverbs/libibverbs.map.in | 6 ++++++
> libibverbs/man/ibv_reg_mr.3 | 21 +++++++++++++++++++--
> libibverbs/verbs.c | 17 +++++++++++++++++
> libibverbs/verbs.h | 11 +++++++++++
> 9 files changed, 112 insertions(+), 3 deletions(-)
>
> diff --git a/debian/libibverbs1.symbols b/debian/libibverbs1.symbols
> index 9130f41..fcf4d87 100644
> --- a/debian/libibverbs1.symbols
> +++ b/debian/libibverbs1.symbols
> @@ -9,6 +9,7 @@ libibverbs.so.1 libibverbs1 #MINVER#
> IBVERBS_1.9@IBVERBS_1.9 30
> IBVERBS_1.10@IBVERBS_1.10 31
> IBVERBS_1.11@IBVERBS_1.11 32
> + IBVERBS_1.12@IBVERBS_1.12 33
> (symver)IBVERBS_PRIVATE_33 33
> _ibv_query_gid_ex@IBVERBS_1.11 32
> _ibv_query_gid_table@IBVERBS_1.11 32
> @@ -99,6 +100,7 @@ libibverbs.so.1 libibverbs1 #MINVER#
> ibv_rate_to_mbps@IBVERBS_1.1 1.1.8
> ibv_rate_to_mult@IBVERBS_1.0 1.1.6
> ibv_read_sysfs_file@IBVERBS_1.0 1.1.6
> + ibv_reg_dmabuf_mr@IBVERBS_1.12 33
> ibv_reg_mr@IBVERBS_1.0 1.1.6
> ibv_reg_mr@IBVERBS_1.1 1.1.6
> ibv_reg_mr_iova@IBVERBS_1.7 25
> diff --git a/libibverbs/CMakeLists.txt b/libibverbs/CMakeLists.txt
> index 0fe4256..d075225 100644
> --- a/libibverbs/CMakeLists.txt
> +++ b/libibverbs/CMakeLists.txt
> @@ -21,7 +21,7 @@ configure_file("libibverbs.map.in"
>
> rdma_library(ibverbs "${CMAKE_CURRENT_BINARY_DIR}/libibverbs.map"
> # See Documentation/versioning.md
> - 1 1.11.${PACKAGE_VERSION}
> + 1 1.12.${PACKAGE_VERSION}
> all_providers.c
> cmd.c
> cmd_ah.c
> diff --git a/libibverbs/cmd_mr.c b/libibverbs/cmd_mr.c
> index 42dbe42..95ed2d1 100644
> --- a/libibverbs/cmd_mr.c
> +++ b/libibverbs/cmd_mr.c
> @@ -1,5 +1,6 @@
> /*
> * Copyright (c) 2018 Mellanox Technologies, Ltd. All rights reserved.
> + * Copyright (c) 2020 Intel Corporation. All rights reserved.
> *
> * This software is available to you under a choice of one of two
> * licenses. You may choose to be licensed under the terms of the GNU
> @@ -116,3 +117,40 @@ int ibv_cmd_query_mr(struct ibv_pd *pd, struct verbs_mr *vmr,
> return 0;
> }
>
> +int ibv_cmd_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
> + uint64_t iova, int fd, int access,
> + struct verbs_mr *vmr)
> +{
> + DECLARE_COMMAND_BUFFER(cmdb, UVERBS_OBJECT_MR,
> + UVERBS_METHOD_REG_DMABUF_MR,
> + 9);
> + struct ib_uverbs_attr *handle;
> + uint32_t lkey, rkey;
> + int ret;
> +
> + handle = fill_attr_out_obj(cmdb, UVERBS_ATTR_REG_DMABUF_MR_HANDLE);
> + fill_attr_out_ptr(cmdb, UVERBS_ATTR_REG_DMABUF_MR_RESP_LKEY, &lkey);
> + fill_attr_out_ptr(cmdb, UVERBS_ATTR_REG_DMABUF_MR_RESP_RKEY, &rkey);
> +
> + fill_attr_in_obj(cmdb, UVERBS_ATTR_REG_DMABUF_MR_PD_HANDLE, pd->handle);
> + fill_attr_in_uint64(cmdb, UVERBS_ATTR_REG_DMABUF_MR_OFFSET, offset);
> + fill_attr_in_uint64(cmdb, UVERBS_ATTR_REG_DMABUF_MR_LENGTH, length);
> + fill_attr_in_uint64(cmdb, UVERBS_ATTR_REG_DMABUF_MR_IOVA, iova);
> + fill_attr_in_uint32(cmdb, UVERBS_ATTR_REG_DMABUF_MR_FD, fd);
> + fill_attr_in_uint32(cmdb, UVERBS_ATTR_REG_DMABUF_MR_ACCESS_FLAGS, access);
> +
> + ret = execute_ioctl(pd->context, cmdb);
> + if (ret)
> + return errno;
> +
> + vmr->ibv_mr.handle = read_attr_obj(UVERBS_ATTR_REG_DMABUF_MR_HANDLE,
> + handle);
> + vmr->ibv_mr.context = pd->context;
> + vmr->ibv_mr.lkey = lkey;
> + vmr->ibv_mr.rkey = rkey;
> + vmr->ibv_mr.pd = pd;
> + vmr->ibv_mr.addr = (void *)offset;
> + vmr->ibv_mr.length = length;
> + vmr->mr_type = IBV_MR_TYPE_MR;
> + return 0;
> +}
> diff --git a/libibverbs/driver.h b/libibverbs/driver.h
> index ab80f4b..d6a9d0a 100644
> --- a/libibverbs/driver.h
> +++ b/libibverbs/driver.h
> @@ -2,6 +2,7 @@
> * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved.
> * Copyright (c) 2005, 2006 Cisco Systems, Inc. All rights reserved.
> * Copyright (c) 2005 PathScale, Inc. All rights reserved.
> + * Copyright (c) 2020 Intel Corporation. All rights reserved.
> *
> * This software is available to you under a choice of one of two
> * licenses. You may choose to be licensed under the terms of the GNU
> @@ -373,6 +374,9 @@ struct verbs_context_ops {
> struct ibv_mr *(*reg_dm_mr)(struct ibv_pd *pd, struct ibv_dm *dm,
> uint64_t dm_offset, size_t length,
> unsigned int access);
> + struct ibv_mr *(*reg_dmabuf_mr)(struct ibv_pd *pd, uint64_t offset,
> + size_t length, uint64_t iova,
> + int fd, int access);
> struct ibv_mr *(*reg_mr)(struct ibv_pd *pd, void *addr, size_t length,
> uint64_t hca_va, int access);
> int (*req_notify_cq)(struct ibv_cq *cq, int solicited_only);
> @@ -498,6 +502,9 @@ int ibv_cmd_advise_mr(struct ibv_pd *pd,
> uint32_t flags,
> struct ibv_sge *sg_list,
> uint32_t num_sge);
> +int ibv_cmd_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
> + uint64_t iova, int fd, int access,
> + struct verbs_mr *vmr);
> int ibv_cmd_alloc_mw(struct ibv_pd *pd, enum ibv_mw_type type,
> struct ibv_mw *mw, struct ibv_alloc_mw *cmd,
> size_t cmd_size,
> diff --git a/libibverbs/dummy_ops.c b/libibverbs/dummy_ops.c
> index e5af9e4..64ef279 100644
> --- a/libibverbs/dummy_ops.c
> +++ b/libibverbs/dummy_ops.c
> @@ -1,5 +1,6 @@
> /*
> * Copyright (c) 2017 Mellanox Technologies, Inc. All rights reserved.
> + * Copyright (c) 2020 Intel Corporation. All rights reserved.
> *
> * This software is available to you under a choice of one of two
> * licenses. You may choose to be licensed under the terms of the GNU
> @@ -452,6 +453,14 @@ static struct ibv_mr *reg_mr(struct ibv_pd *pd, void *addr, size_t length,
> return NULL;
> }
>
> +static struct ibv_mr *reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset,
> + size_t length, uint64_t iova,
> + int fd, int access)
> +{
> + errno = EOPNOTSUPP;
> + return NULL;
> +}
> +
> static int req_notify_cq(struct ibv_cq *cq, int solicited_only)
> {
> return EOPNOTSUPP;
> @@ -560,6 +569,7 @@ const struct verbs_context_ops verbs_dummy_ops = {
> query_srq,
> read_counters,
> reg_dm_mr,
> + reg_dmabuf_mr,
> reg_mr,
> req_notify_cq,
> rereg_mr,
> @@ -689,6 +699,7 @@ void verbs_set_ops(struct verbs_context *vctx,
> SET_PRIV_OP_IC(vctx, set_ece);
> SET_PRIV_OP_IC(vctx, unimport_mr);
> SET_PRIV_OP_IC(vctx, unimport_pd);
> + SET_OP(ctx, reg_dmabuf_mr);
>
> #undef SET_OP
> #undef SET_OP2
> diff --git a/libibverbs/libibverbs.map.in b/libibverbs/libibverbs.map.in
> index b5ccaca..f67e1ef 100644
> --- a/libibverbs/libibverbs.map.in
> +++ b/libibverbs/libibverbs.map.in
> @@ -148,6 +148,11 @@ IBVERBS_1.11 {
> _ibv_query_gid_table;
> } IBVERBS_1.10;
>
> +IBVERBS_1.12 {
> + global:
> + ibv_reg_dmabuf_mr;
> +} IBVERBS_1.11;
> +
> /* If any symbols in this stanza change ABI then the entire staza gets a new symbol
> version. See the top level CMakeLists.txt for this setting. */
>
> @@ -211,6 +216,7 @@ IBVERBS_PRIVATE_@IBVERBS_PABI_VERSION@ {
> ibv_cmd_query_srq;
> ibv_cmd_read_counters;
> ibv_cmd_reg_dm_mr;
> + ibv_cmd_reg_dmabuf_mr;
> ibv_cmd_reg_mr;
> ibv_cmd_req_notify_cq;
> ibv_cmd_rereg_mr;
> diff --git a/libibverbs/man/ibv_reg_mr.3 b/libibverbs/man/ibv_reg_mr.3
> index 2bfc955..4975c79 100644
> --- a/libibverbs/man/ibv_reg_mr.3
> +++ b/libibverbs/man/ibv_reg_mr.3
> @@ -3,7 +3,7 @@
> .\"
> .TH IBV_REG_MR 3 2006-10-31 libibverbs "Libibverbs Programmer's Manual"
> .SH "NAME"
> -ibv_reg_mr, ibv_reg_mr_iova, ibv_dereg_mr \- register or deregister a memory region (MR)
> +ibv_reg_mr, ibv_reg_mr_iova, ibv_reg_dmabuf_mr, ibv_dereg_mr \- register or deregister a memory region (MR)
> .SH "SYNOPSIS"
> .nf
> .B #include <infiniband/verbs.h>
> @@ -15,6 +15,9 @@ ibv_reg_mr, ibv_reg_mr_iova, ibv_dereg_mr \- register or deregister a memory reg
> .BI " size_t " "length" ", uint64_t " "hca_va" ,
> .BI " int " "access" );
> .sp
> +.BI "struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd " "*pd" ", uint64_t " "offset" ,
> +.BI " size_t " "length" ", int " "access" );
> +.sp
> .BI "int ibv_dereg_mr(struct ibv_mr " "*mr" );
> .fi
> .SH "DESCRIPTION"
> @@ -71,11 +74,25 @@ a lkey or rkey. The offset in the memory region is computed as 'addr +
> (iova - hca_va)'. Specifying 0 for hca_va has the same effect as
> IBV_ACCESS_ZERO_BASED.
> .PP
> +.B ibv_reg_dmabuf_mr()
> +registers a dma-buf based memory region (MR) associated with the protection domain
> +.I pd\fR.
> +The MR starts at
> +.I offset
> +of the dma-buf and its size is
> +.I length\fR.
> +The dma-buf is identified by the file descriptor
> +.I fd\fR.
> +The argument
> +.I access
> +describes the desired memory protection attributes; it is similar to the ibv_reg_mr case except that only the following flags are supported:
> +.B IBV_ACCESS_LOCAL_WRITE, IBV_ACCESS_REMOTE_WRITE, IBV_ACCESS_REMOTE_READ, IBV_ACCESS_REMOTE_ATOMIC, IBV_ACCESS_RELAXED_ORDERING.
> +.PP
> .B ibv_dereg_mr()
> deregisters the MR
> .I mr\fR.
> .SH "RETURN VALUE"
> -.B ibv_reg_mr() / ibv_reg_mr_iova()
> +.B ibv_reg_mr() / ibv_reg_mr_iova() / ibv_reg_dmabuf_mr()
> returns a pointer to the registered MR, or NULL if the request fails.
> The local key (\fBL_Key\fR) field
> .B lkey
> diff --git a/libibverbs/verbs.c b/libibverbs/verbs.c
> index 2b0ede8..b944904 100644
> --- a/libibverbs/verbs.c
> +++ b/libibverbs/verbs.c
> @@ -1,6 +1,7 @@
> /*
> * Copyright (c) 2005 Topspin Communications. All rights reserved.
> * Copyright (c) 2006, 2007 Cisco Systems, Inc. All rights reserved.
> + * Copyright (c) 2020 Intel Corperation. All rights reserved.
> *
> * This software is available to you under a choice of one of two
> * licenses. You may choose to be licensed under the terms of the GNU
> @@ -367,6 +368,22 @@ void ibv_unimport_mr(struct ibv_mr *mr)
> get_ops(mr->context)->unimport_mr(mr);
> }
>
> +struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset,
> + size_t length, int fd, int access)
> +{
> + struct ibv_mr *mr;
> +
> + mr = get_ops(pd->context)->reg_dmabuf_mr(pd, offset, length, offset,
> + fd, access);
> + if (mr) {
> + mr->context = pd->context;
> + mr->pd = pd;
> + mr->addr = (void *)offset;
> + mr->length = length;
> + }
> + return mr;
> +}
> +
> LATEST_SYMVER_FUNC(ibv_rereg_mr, 1_1, "IBVERBS_1.1",
> int,
> struct ibv_mr *mr, int flags,
> diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h
> index ee57e05..9194ad2 100644
> --- a/libibverbs/verbs.h
> +++ b/libibverbs/verbs.h
> @@ -3,6 +3,7 @@
> * Copyright (c) 2004, 2011-2012 Intel Corporation. All rights reserved.
> * Copyright (c) 2005, 2006, 2007 Cisco Systems, Inc. All rights reserved.
> * Copyright (c) 2005 PathScale, Inc. All rights reserved.
> + * Copyright (c) 2020 Intel Corporation. All rights reserved.
> *
> * This software is available to you under a choice of one of two
> * licenses. You may choose to be licensed under the terms of the GNU
> @@ -1961,6 +1962,10 @@ struct ibv_context_ops {
> void *(*_compat_attach_mcast)(void);
> void *(*_compat_detach_mcast)(void);
> void *(*_compat_async_event)(void);
> + struct ibv_mr * (*reg_dmabuf_mr)(struct ibv_pd *pd,
> + uint64_t offset, size_t length,
> + uint64_t iova, int fd,
> + int access);
> };
>
> struct ibv_context {
> @@ -2535,6 +2540,12 @@ __ibv_reg_mr_iova(struct ibv_pd *pd, void *addr, size_t length, uint64_t iova,
> __builtin_constant_p( \
> ((access) & IBV_ACCESS_OPTIONAL_RANGE) == 0))
>
> +/**
> + * ibv_reg_dmabuf_mr - Register a dambuf-based memory region
> + */
> +struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
> + int fd, int access);
> +
> enum ibv_rereg_mr_err_code {
> /* Old MR is valid, invalid input */
> IBV_REREG_MR_ERR_INPUT = -1,
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH rdma-core v2 2/6] verbs: Support dma-buf based memory region
2020-11-24 21:38 ` Jianxin Xiong
@ 2020-11-25 14:12 ` Jason Gunthorpe
-1 siblings, 0 replies; 20+ messages in thread
From: Jason Gunthorpe @ 2020-11-25 14:12 UTC (permalink / raw)
To: Jianxin Xiong
Cc: linux-rdma, dri-devel, Doug Ledford, Leon Romanovsky,
Sumit Semwal, Christian Koenig, Daniel Vetter
On Tue, Nov 24, 2020 at 01:38:50PM -0800, Jianxin Xiong wrote:
> +/**
> + * ibv_reg_dmabuf_mr - Register a dambuf-based memory region
> + */
> +struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
> + int fd, int access);
Please include the iova in addition to the offset for this API
Similar to ibv_reg_mr_iova
Jason
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH rdma-core v2 2/6] verbs: Support dma-buf based memory region
@ 2020-11-25 14:12 ` Jason Gunthorpe
0 siblings, 0 replies; 20+ messages in thread
From: Jason Gunthorpe @ 2020-11-25 14:12 UTC (permalink / raw)
To: Jianxin Xiong
Cc: Leon Romanovsky, linux-rdma, dri-devel, Doug Ledford,
Daniel Vetter, Christian Koenig
On Tue, Nov 24, 2020 at 01:38:50PM -0800, Jianxin Xiong wrote:
> +/**
> + * ibv_reg_dmabuf_mr - Register a dambuf-based memory region
> + */
> +struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
> + int fd, int access);
Please include the iova in addition to the offset for this API
Similar to ibv_reg_mr_iova
Jason
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH rdma-core v2 5/6] tests: Add tests for dma-buf based memory regions
2020-11-24 21:38 ` Jianxin Xiong
@ 2020-11-25 14:41 ` Yishai Hadas
-1 siblings, 0 replies; 20+ messages in thread
From: Yishai Hadas @ 2020-11-25 14:41 UTC (permalink / raw)
To: Jianxin Xiong
Cc: linux-rdma, dri-devel, Doug Ledford, Jason Gunthorpe,
Leon Romanovsky, Sumit Semwal, Christian Koenig, Daniel Vetter,
Yishai Hadas
On 11/24/2020 11:38 PM, Jianxin Xiong wrote:
> Define a full set of tests similar to regular MR tests. Add a utility
> function to generate access flags for dma-buf based MRs because the
> set of supported flags is smaller.
Looking on the tests, It doesn't seem that the registered dma-buf lkey
is used with some RDMA traffic, right ?
Let's please use it in some test to cover a real use case of RDMA.
>
> Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
> ---
> tests/test_mr.py | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
> tests/utils.py | 25 +++++++++++
> 2 files changed, 154 insertions(+), 1 deletion(-)
>
> diff --git a/tests/test_mr.py b/tests/test_mr.py
> index adc649c..8d7f002 100644
> --- a/tests/test_mr.py
> +++ b/tests/test_mr.py
> @@ -9,9 +9,10 @@ import errno
>
> from tests.base import PyverbsAPITestCase, RCResources, RDMATestCase
> from pyverbs.pyverbs_error import PyverbsRDMAError, PyverbsError
> -from pyverbs.mr import MR, MW, DMMR, MWBindInfo, MWBind
> +from pyverbs.mr import MR, MW, DMMR, DmaBufMR, MWBindInfo, MWBind
> from pyverbs.qp import QPCap, QPInitAttr, QPAttr, QP
> from pyverbs.wr import SendWR
> +from pyverbs.dmabuf import DmaBuf
> import pyverbs.device as d
> from pyverbs.pd import PD
> import pyverbs.enums as e
> @@ -366,3 +367,130 @@ class DMMRTest(PyverbsAPITestCase):
> dm_mr = DMMR(pd, dm_mr_len, e.IBV_ACCESS_ZERO_BASED,
> dm=dm, offset=dm_mr_offset)
> dm_mr.close()
> +
> +class DmaBufMRTest(PyverbsAPITestCase):
> + """
> + Test various functionalities of the DmaBufMR class.
> + """
> + def test_dmabuf_reg_mr(self):
> + """
> + Test ibv_reg_dmabuf_mr()
> + """
> + for ctx, attr, attr_ex in self.devices:
> + with PD(ctx) as pd:
> + flags = u.get_dmabuf_access_flags(ctx)
> + for f in flags:
> + len = u.get_mr_length()
> + off = random.randint(0, len//2)
> + with DmaBufMR(pd, len, f, offset=off) as mr:
> + pass
> +
> + def test_dmabuf_dereg_mr(self):
> + """
> + Test ibv_dereg_mr() with DmaBufMR
> + """
> + for ctx, attr, attr_ex in self.devices:
> + with PD(ctx) as pd:
> + flags = u.get_dmabuf_access_flags(ctx)
> + for f in flags:
> + len = u.get_mr_length()
> + off = random.randint(0, len//2)
> + with DmaBufMR(pd, len, f, offset=off) as mr:
> + mr.close()
> +
> + def test_dmabuf_dereg_mr_twice(self):
> + """
> + Verify that explicit call to DmaBufMR's close() doesn't fail
> + """
> + for ctx, attr, attr_ex in self.devices:
> + with PD(ctx) as pd:
> + flags = u.get_dmabuf_access_flags(ctx)
> + for f in flags:
> + len = u.get_mr_length()
> + off = random.randint(0, len//2)
> + with DmaBufMR(pd, len, f, offset=off) as mr:
> + # Pyverbs supports multiple destruction of objects,
> + # we are not expecting an exception here.
> + mr.close()
> + mr.close()
> +
> + def test_dmabuf_reg_mr_bad_flags(self):
> + """
> + Verify that illegal flags combination fails as expected
> + """
> + for ctx, attr, attr_ex in self.devices:
> + with PD(ctx) as pd:
> + for i in range(5):
> + flags = random.sample([e.IBV_ACCESS_REMOTE_WRITE,
> + e.IBV_ACCESS_REMOTE_ATOMIC],
> + random.randint(1, 2))
> + mr_flags = 0
> + for i in flags:
> + mr_flags += i.value
> + try:
> + DmaBufMR(pd, u.get_mr_length(), mr_flags)
> + except PyverbsRDMAError as err:
> + assert 'Failed to register a dma-buf MR' in err.args[0]
> + else:
> + raise PyverbsRDMAError('Registered a dma-buf MR with illegal falgs')
> +
> + def test_dmabuf_write(self):
> + """
> + Test writing to DmaBufMR's buffer
> + """
> + for ctx, attr, attr_ex in self.devices:
> + with PD(ctx) as pd:
> + for i in range(10):
> + mr_len = u.get_mr_length()
> + mr_off = random.randint(0, mr_len//2)
> + flags = u.get_dmabuf_access_flags(ctx)
> + for f in flags:
> + with DmaBufMR(pd, mr_len, f, offset=mr_off) as mr:
> + write_len = min(random.randint(1, MAX_IO_LEN),
> + mr_len)
> + mr.write('a' * write_len, write_len)
> +
> + def test_dmabuf_read(self):
> + """
> + Test reading from DmaBufMR's buffer
> + """
> + for ctx, attr, attr_ex in self.devices:
> + with PD(ctx) as pd:
> + for i in range(10):
> + mr_len = u.get_mr_length()
> + mr_off = random.randint(0, mr_len//2)
> + flags = u.get_dmabuf_access_flags(ctx)
> + for f in flags:
> + with DmaBufMR(pd, mr_len, f, offset=mr_off) as mr:
> + write_len = min(random.randint(1, MAX_IO_LEN),
> + mr_len)
> + write_str = 'a' * write_len
> + mr.write(write_str, write_len)
> + read_len = random.randint(1, write_len)
> + offset = random.randint(0, write_len-read_len)
> + read_str = mr.read(read_len, offset).decode()
> + assert read_str in write_str
> +
> + def test_dmabuf_lkey(self):
> + """
> + Test reading lkey property
> + """
> + for ctx, attr, attr_ex in self.devices:
> + with PD(ctx) as pd:
> + length = u.get_mr_length()
> + flags = u.get_dmabuf_access_flags(ctx)
> + for f in flags:
> + with DmaBufMR(pd, length, f) as mr:
> + mr.lkey
> +
> + def test_dmabuf_rkey(self):
> + """
> + Test reading rkey property
> + """
> + for ctx, attr, attr_ex in self.devices:
> + with PD(ctx) as pd:
> + length = u.get_mr_length()
> + flags = u.get_dmabuf_access_flags(ctx)
> + for f in flags:
> + with DmaBufMR(pd, length, f) as mr:
> + mr.rkey
> diff --git a/tests/utils.py b/tests/utils.py
> index 7039f41..0ad7110 100644
> --- a/tests/utils.py
> +++ b/tests/utils.py
> @@ -94,6 +94,31 @@ def get_access_flags(ctx):
> return arr
>
>
> +def get_dmabuf_access_flags(ctx):
> + """
> + Similar to get_access_flags, except that dma-buf MR only support
> + a subset of the flags.
> + :param ctx: Device Context to check capabilities
> + :return: A random legal value for MR flags
> + """
> + attr = ctx.query_device()
> + vals = [e.IBV_ACCESS_LOCAL_WRITE, e.IBV_ACCESS_REMOTE_WRITE,
> + e.IBV_ACCESS_REMOTE_READ, e.IBV_ACCESS_REMOTE_ATOMIC,
> + e.IBV_ACCESS_RELAXED_ORDERING]
> + if not attr.atomic_caps & e.IBV_ATOMIC_HCA:
> + vals.remove(e.IBV_ACCESS_REMOTE_ATOMIC)
> + arr = []
> + for i in range(1, len(vals)):
> + tmp = list(com(vals, i))
> + tmp = filter(filter_illegal_access_flags, tmp)
> + for t in tmp: # Iterate legal combinations and bitwise OR them
> + val = 0
> + for flag in t:
> + val += flag.value
> + arr.append(val)
> + return arr
> +
> +
> def get_dm_attrs(dm_len):
> """
> Initializes an AllocDmAttr member with the given length and random
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH rdma-core v2 5/6] tests: Add tests for dma-buf based memory regions
@ 2020-11-25 14:41 ` Yishai Hadas
0 siblings, 0 replies; 20+ messages in thread
From: Yishai Hadas @ 2020-11-25 14:41 UTC (permalink / raw)
To: Jianxin Xiong
Cc: Yishai Hadas, Leon Romanovsky, linux-rdma, dri-devel,
Christian Koenig, Jason Gunthorpe, Doug Ledford, Daniel Vetter
On 11/24/2020 11:38 PM, Jianxin Xiong wrote:
> Define a full set of tests similar to regular MR tests. Add a utility
> function to generate access flags for dma-buf based MRs because the
> set of supported flags is smaller.
Looking on the tests, It doesn't seem that the registered dma-buf lkey
is used with some RDMA traffic, right ?
Let's please use it in some test to cover a real use case of RDMA.
>
> Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
> ---
> tests/test_mr.py | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
> tests/utils.py | 25 +++++++++++
> 2 files changed, 154 insertions(+), 1 deletion(-)
>
> diff --git a/tests/test_mr.py b/tests/test_mr.py
> index adc649c..8d7f002 100644
> --- a/tests/test_mr.py
> +++ b/tests/test_mr.py
> @@ -9,9 +9,10 @@ import errno
>
> from tests.base import PyverbsAPITestCase, RCResources, RDMATestCase
> from pyverbs.pyverbs_error import PyverbsRDMAError, PyverbsError
> -from pyverbs.mr import MR, MW, DMMR, MWBindInfo, MWBind
> +from pyverbs.mr import MR, MW, DMMR, DmaBufMR, MWBindInfo, MWBind
> from pyverbs.qp import QPCap, QPInitAttr, QPAttr, QP
> from pyverbs.wr import SendWR
> +from pyverbs.dmabuf import DmaBuf
> import pyverbs.device as d
> from pyverbs.pd import PD
> import pyverbs.enums as e
> @@ -366,3 +367,130 @@ class DMMRTest(PyverbsAPITestCase):
> dm_mr = DMMR(pd, dm_mr_len, e.IBV_ACCESS_ZERO_BASED,
> dm=dm, offset=dm_mr_offset)
> dm_mr.close()
> +
> +class DmaBufMRTest(PyverbsAPITestCase):
> + """
> + Test various functionalities of the DmaBufMR class.
> + """
> + def test_dmabuf_reg_mr(self):
> + """
> + Test ibv_reg_dmabuf_mr()
> + """
> + for ctx, attr, attr_ex in self.devices:
> + with PD(ctx) as pd:
> + flags = u.get_dmabuf_access_flags(ctx)
> + for f in flags:
> + len = u.get_mr_length()
> + off = random.randint(0, len//2)
> + with DmaBufMR(pd, len, f, offset=off) as mr:
> + pass
> +
> + def test_dmabuf_dereg_mr(self):
> + """
> + Test ibv_dereg_mr() with DmaBufMR
> + """
> + for ctx, attr, attr_ex in self.devices:
> + with PD(ctx) as pd:
> + flags = u.get_dmabuf_access_flags(ctx)
> + for f in flags:
> + len = u.get_mr_length()
> + off = random.randint(0, len//2)
> + with DmaBufMR(pd, len, f, offset=off) as mr:
> + mr.close()
> +
> + def test_dmabuf_dereg_mr_twice(self):
> + """
> + Verify that explicit call to DmaBufMR's close() doesn't fail
> + """
> + for ctx, attr, attr_ex in self.devices:
> + with PD(ctx) as pd:
> + flags = u.get_dmabuf_access_flags(ctx)
> + for f in flags:
> + len = u.get_mr_length()
> + off = random.randint(0, len//2)
> + with DmaBufMR(pd, len, f, offset=off) as mr:
> + # Pyverbs supports multiple destruction of objects,
> + # we are not expecting an exception here.
> + mr.close()
> + mr.close()
> +
> + def test_dmabuf_reg_mr_bad_flags(self):
> + """
> + Verify that illegal flags combination fails as expected
> + """
> + for ctx, attr, attr_ex in self.devices:
> + with PD(ctx) as pd:
> + for i in range(5):
> + flags = random.sample([e.IBV_ACCESS_REMOTE_WRITE,
> + e.IBV_ACCESS_REMOTE_ATOMIC],
> + random.randint(1, 2))
> + mr_flags = 0
> + for i in flags:
> + mr_flags += i.value
> + try:
> + DmaBufMR(pd, u.get_mr_length(), mr_flags)
> + except PyverbsRDMAError as err:
> + assert 'Failed to register a dma-buf MR' in err.args[0]
> + else:
> + raise PyverbsRDMAError('Registered a dma-buf MR with illegal falgs')
> +
> + def test_dmabuf_write(self):
> + """
> + Test writing to DmaBufMR's buffer
> + """
> + for ctx, attr, attr_ex in self.devices:
> + with PD(ctx) as pd:
> + for i in range(10):
> + mr_len = u.get_mr_length()
> + mr_off = random.randint(0, mr_len//2)
> + flags = u.get_dmabuf_access_flags(ctx)
> + for f in flags:
> + with DmaBufMR(pd, mr_len, f, offset=mr_off) as mr:
> + write_len = min(random.randint(1, MAX_IO_LEN),
> + mr_len)
> + mr.write('a' * write_len, write_len)
> +
> + def test_dmabuf_read(self):
> + """
> + Test reading from DmaBufMR's buffer
> + """
> + for ctx, attr, attr_ex in self.devices:
> + with PD(ctx) as pd:
> + for i in range(10):
> + mr_len = u.get_mr_length()
> + mr_off = random.randint(0, mr_len//2)
> + flags = u.get_dmabuf_access_flags(ctx)
> + for f in flags:
> + with DmaBufMR(pd, mr_len, f, offset=mr_off) as mr:
> + write_len = min(random.randint(1, MAX_IO_LEN),
> + mr_len)
> + write_str = 'a' * write_len
> + mr.write(write_str, write_len)
> + read_len = random.randint(1, write_len)
> + offset = random.randint(0, write_len-read_len)
> + read_str = mr.read(read_len, offset).decode()
> + assert read_str in write_str
> +
> + def test_dmabuf_lkey(self):
> + """
> + Test reading lkey property
> + """
> + for ctx, attr, attr_ex in self.devices:
> + with PD(ctx) as pd:
> + length = u.get_mr_length()
> + flags = u.get_dmabuf_access_flags(ctx)
> + for f in flags:
> + with DmaBufMR(pd, length, f) as mr:
> + mr.lkey
> +
> + def test_dmabuf_rkey(self):
> + """
> + Test reading rkey property
> + """
> + for ctx, attr, attr_ex in self.devices:
> + with PD(ctx) as pd:
> + length = u.get_mr_length()
> + flags = u.get_dmabuf_access_flags(ctx)
> + for f in flags:
> + with DmaBufMR(pd, length, f) as mr:
> + mr.rkey
> diff --git a/tests/utils.py b/tests/utils.py
> index 7039f41..0ad7110 100644
> --- a/tests/utils.py
> +++ b/tests/utils.py
> @@ -94,6 +94,31 @@ def get_access_flags(ctx):
> return arr
>
>
> +def get_dmabuf_access_flags(ctx):
> + """
> + Similar to get_access_flags, except that dma-buf MR only support
> + a subset of the flags.
> + :param ctx: Device Context to check capabilities
> + :return: A random legal value for MR flags
> + """
> + attr = ctx.query_device()
> + vals = [e.IBV_ACCESS_LOCAL_WRITE, e.IBV_ACCESS_REMOTE_WRITE,
> + e.IBV_ACCESS_REMOTE_READ, e.IBV_ACCESS_REMOTE_ATOMIC,
> + e.IBV_ACCESS_RELAXED_ORDERING]
> + if not attr.atomic_caps & e.IBV_ATOMIC_HCA:
> + vals.remove(e.IBV_ACCESS_REMOTE_ATOMIC)
> + arr = []
> + for i in range(1, len(vals)):
> + tmp = list(com(vals, i))
> + tmp = filter(filter_illegal_access_flags, tmp)
> + for t in tmp: # Iterate legal combinations and bitwise OR them
> + val = 0
> + for flag in t:
> + val += flag.value
> + arr.append(val)
> + return arr
> +
> +
> def get_dm_attrs(dm_len):
> """
> Initializes an AllocDmAttr member with the given length and random
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2020-11-26 8:24 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-11-24 21:38 [PATCH rdma-core v2 0/6] Add user space dma-buf support Jianxin Xiong
2020-11-24 21:38 ` Jianxin Xiong
2020-11-24 21:38 ` [PATCH rdma-core v2 1/6] Update kernel headers Jianxin Xiong
2020-11-24 21:38 ` Jianxin Xiong
2020-11-24 21:38 ` [PATCH rdma-core v2 2/6] verbs: Support dma-buf based memory region Jianxin Xiong
2020-11-24 21:38 ` Jianxin Xiong
2020-11-25 8:03 ` Yishai Hadas
2020-11-25 8:03 ` Yishai Hadas
2020-11-25 14:12 ` Jason Gunthorpe
2020-11-25 14:12 ` Jason Gunthorpe
2020-11-24 21:38 ` [PATCH rdma-core v2 3/6] mlx5: " Jianxin Xiong
2020-11-24 21:38 ` Jianxin Xiong
2020-11-24 21:38 ` [PATCH rdma-core v2 4/6] pyverbs: Add dma-buf based MR support Jianxin Xiong
2020-11-24 21:38 ` Jianxin Xiong
2020-11-24 21:38 ` [PATCH rdma-core v2 5/6] tests: Add tests for dma-buf based memory regions Jianxin Xiong
2020-11-24 21:38 ` Jianxin Xiong
2020-11-25 14:41 ` Yishai Hadas
2020-11-25 14:41 ` Yishai Hadas
2020-11-24 21:38 ` [PATCH rdma-core v2 6/6] tests: Bug fix for get_access_flags() Jianxin Xiong
2020-11-24 21:38 ` Jianxin Xiong
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.