* [PATCH rdma-core v6 0/6] Add user space dma-buf support
@ 2021-01-21 0:14 Jianxin Xiong
2021-01-21 0:14 ` [PATCH rdma-core v6 1/6] Update kernel headers Jianxin Xiong
` (5 more replies)
0 siblings, 6 replies; 7+ messages in thread
From: Jianxin Xiong @ 2021-01-21 0:14 UTC (permalink / raw)
To: linux-rdma, dri-devel
Cc: Yishai Hadas, Leon Romanovsky, Edward Srouji, Jason Gunthorpe,
Doug Ledford, Daniel Vetter, Christian Koenig, Jianxin Xiong
This is the sixth version of the patch series. Change log:
v6:
* Rebase to the latest rdma-core master (commit 14006f2f841b0c)
* Update the ABI symbol version to match new package version; also bump
the private ABI version because new function has been added to the
provider interface
* Avoid changing 'struct ibv_context_ops' by replacing SET_OP() with
SET_PRIV_OP_IC()
* Replace sprintf() with snprintf()
* Keep the ops in verbs_set_ops() sorted
* Fix some styling issues: extra spaces, struct 0-initialization, error
checking control flow
v5: https://www.spinics.net/lists/linux-rdma/msg99015.html
* Use a different mr_type for dmabuf so that ibv_dofork_range() is not
called inside ibv_dereg_mr() for dmabuf based mr
v4: https://www.spinics.net/lists/linux-rdma/msg98135.html
* Rework the cmake funciton rdma_cython_module to support both single
source (.pyx) and multiple source (.pyx + [.c]*) scenarios instead
of using two separate functions
* Rename 'dri_*' to 'drm_*' for the dmabuf allocation interface
* Add option to dmabuf allocation routine to allow allocation from GTT
instead of VRAM
* Add proper CPU access flags when allocating dmabufs
* Remove 'radeon' driver support from the dmabuf allocation routines
* Add comand line arguments to the tests for selecting GPU unit and
setting the option for allocating from GTT
v3: https://www.spinics.net/lists/linux-rdma/msg98059.html
* Add parameter 'iova' to the new ibv_reg_dmabuf_mr() API
* Change the way of allocating dma-buf object - use /dev/dri/renderD*
instead of /dev/dri/card* and use GEM object instead of dumb buffer
* Add cmake function to allow building modules with mixed cython and C
source files
* Add new tests that use dma-buf MRs for send/recv and rdma traffic
* Skip dma-buf tests on unsupported systems
* Remove some use of random values in the new tests
* Add dealloc() and close() methods to the new classes
* Replace string.format with f-string in python code
* Fix some coding style issues: spacing, indentation, typo, comments
v2: https://www.spinics.net/lists/linux-rdma/msg97936.html
* Put the kernel header updates into a separate commit
* Add comments for the data structure used in python ioctl calls
* Fix issues related to symbol versioning
* Fix styling issues: extra spaces, unncecessary variable, typo
* Fix an inproper error code usage
* Put the new op into ibv_context_ops instead if verbs_context
v1: https://www.spinics.net/lists/linux-rdma/msg97865.html
* Add user space API for registering dma-buf based memory regions
* Update pyverbs with the new API
* Add new tests
This is the user space counter-part of the kernel patch set to add
dma-buf support to the RDMA subsystem.
This series consists of six patches. The first patch updates the
kernel headers for dma-buf support. Patch 2 adds the new API function
and updates the man pages. Patch 3 implements the new API in the mlx5
provider. Patch 4 adds new class definitions to pyverbs for the new API.
Patch 5 adds a set of new tests for the new API. Patch 6 fixes bug in
the utility code of the tests.
Pull request at github: https://github.com/linux-rdma/rdma-core/pull/895
Jianxin Xiong (6):
Update kernel headers
verbs: Support dma-buf based memory region
mlx5: Support dma-buf based memory region
pyverbs: Add dma-buf based MR support
tests: Add tests for dma-buf based memory regions
tests: Bug fix for get_access_flags()
CMakeLists.txt | 2 +-
buildlib/pyverbs_functions.cmake | 78 ++++++---
debian/control | 2 +-
debian/libibverbs1.symbols | 4 +-
kernel-headers/rdma/ib_user_ioctl_cmds.h | 14 ++
libibverbs/CMakeLists.txt | 2 +-
libibverbs/cmd_mr.c | 38 +++++
libibverbs/driver.h | 8 +
libibverbs/dummy_ops.c | 11 ++
libibverbs/libibverbs.map.in | 6 +
libibverbs/man/ibv_reg_mr.3 | 27 ++-
libibverbs/verbs.c | 19 +++
libibverbs/verbs.h | 7 +
providers/mlx5/mlx5.c | 2 +
providers/mlx5/mlx5.h | 3 +
providers/mlx5/verbs.c | 22 +++
pyverbs/CMakeLists.txt | 11 +-
pyverbs/dmabuf.pxd | 15 ++
pyverbs/dmabuf.pyx | 73 ++++++++
pyverbs/dmabuf_alloc.c | 278 +++++++++++++++++++++++++++++++
pyverbs/dmabuf_alloc.h | 19 +++
pyverbs/libibverbs.pxd | 2 +
pyverbs/mr.pxd | 6 +
pyverbs/mr.pyx | 105 +++++++++++-
tests/args_parser.py | 4 +
tests/test_mr.py | 264 ++++++++++++++++++++++++++++-
tests/utils.py | 30 +++-
27 files changed, 1013 insertions(+), 39 deletions(-)
create mode 100644 pyverbs/dmabuf.pxd
create mode 100644 pyverbs/dmabuf.pyx
create mode 100644 pyverbs/dmabuf_alloc.c
create mode 100644 pyverbs/dmabuf_alloc.h
--
1.8.3.1
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH rdma-core v6 1/6] Update kernel headers
2021-01-21 0:14 [PATCH rdma-core v6 0/6] Add user space dma-buf support Jianxin Xiong
@ 2021-01-21 0:14 ` Jianxin Xiong
2021-01-21 0:14 ` [PATCH rdma-core v6 2/6] verbs: Support dma-buf based memory region Jianxin Xiong
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Jianxin Xiong @ 2021-01-21 0:14 UTC (permalink / raw)
To: linux-rdma, dri-devel
Cc: Yishai Hadas, Leon Romanovsky, Edward Srouji, Jason Gunthorpe,
Doug Ledford, Daniel Vetter, Christian Koenig, Jianxin Xiong
To commit 2eef437c4669 ("RDMA/uverbs: Add uverbs command for dma-buf based
MR registration").
Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
| 14 ++++++++++++++
1 file changed, 14 insertions(+)
--git a/kernel-headers/rdma/ib_user_ioctl_cmds.h b/kernel-headers/rdma/ib_user_ioctl_cmds.h
index 7968a18..dafc7eb 100644
--- a/kernel-headers/rdma/ib_user_ioctl_cmds.h
+++ b/kernel-headers/rdma/ib_user_ioctl_cmds.h
@@ -1,5 +1,6 @@
/*
* Copyright (c) 2018, Mellanox Technologies inc. All rights reserved.
+ * Copyright (c) 2020, Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -251,6 +252,7 @@ enum uverbs_methods_mr {
UVERBS_METHOD_MR_DESTROY,
UVERBS_METHOD_ADVISE_MR,
UVERBS_METHOD_QUERY_MR,
+ UVERBS_METHOD_REG_DMABUF_MR,
};
enum uverbs_attrs_mr_destroy_ids {
@@ -272,6 +274,18 @@ enum uverbs_attrs_query_mr_cmd_attr_ids {
UVERBS_ATTR_QUERY_MR_RESP_IOVA,
};
+enum uverbs_attrs_reg_dmabuf_mr_cmd_attr_ids {
+ UVERBS_ATTR_REG_DMABUF_MR_HANDLE,
+ UVERBS_ATTR_REG_DMABUF_MR_PD_HANDLE,
+ UVERBS_ATTR_REG_DMABUF_MR_OFFSET,
+ UVERBS_ATTR_REG_DMABUF_MR_LENGTH,
+ UVERBS_ATTR_REG_DMABUF_MR_IOVA,
+ UVERBS_ATTR_REG_DMABUF_MR_FD,
+ UVERBS_ATTR_REG_DMABUF_MR_ACCESS_FLAGS,
+ UVERBS_ATTR_REG_DMABUF_MR_RESP_LKEY,
+ UVERBS_ATTR_REG_DMABUF_MR_RESP_RKEY,
+};
+
enum uverbs_attrs_create_counters_cmd_attr_ids {
UVERBS_ATTR_CREATE_COUNTERS_HANDLE,
};
--
1.8.3.1
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH rdma-core v6 2/6] verbs: Support dma-buf based memory region
2021-01-21 0:14 [PATCH rdma-core v6 0/6] Add user space dma-buf support Jianxin Xiong
2021-01-21 0:14 ` [PATCH rdma-core v6 1/6] Update kernel headers Jianxin Xiong
@ 2021-01-21 0:14 ` Jianxin Xiong
2021-01-21 0:14 ` [PATCH rdma-core v6 3/6] mlx5: " Jianxin Xiong
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Jianxin Xiong @ 2021-01-21 0:14 UTC (permalink / raw)
To: linux-rdma, dri-devel
Cc: Yishai Hadas, Leon Romanovsky, Edward Srouji, Jason Gunthorpe,
Doug Ledford, Daniel Vetter, Christian Koenig, Jianxin Xiong
Add new API function and new provider method for registering dma-buf
based memory region. Update the man page and bump the API version.
Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
CMakeLists.txt | 2 +-
debian/control | 2 +-
debian/libibverbs1.symbols | 4 +++-
libibverbs/CMakeLists.txt | 2 +-
libibverbs/cmd_mr.c | 38 ++++++++++++++++++++++++++++++++++++++
libibverbs/driver.h | 8 ++++++++
libibverbs/dummy_ops.c | 11 +++++++++++
libibverbs/libibverbs.map.in | 6 ++++++
libibverbs/man/ibv_reg_mr.3 | 27 +++++++++++++++++++++++++--
libibverbs/verbs.c | 19 +++++++++++++++++++
libibverbs/verbs.h | 7 +++++++
11 files changed, 120 insertions(+), 6 deletions(-)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 30ae0cc..4113423 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -76,7 +76,7 @@ set(PACKAGE_VERSION "34.0")
# When this is changed the values in these files need changing too:
# debian/control
# debian/libibverbs1.symbols
-set(IBVERBS_PABI_VERSION "33")
+set(IBVERBS_PABI_VERSION "34")
set(IBVERBS_PROVIDER_SUFFIX "-rdmav${IBVERBS_PABI_VERSION}.so")
#-------------------------
diff --git a/debian/control b/debian/control
index a423e4f..9022644 100644
--- a/debian/control
+++ b/debian/control
@@ -152,7 +152,7 @@ Section: libs
Pre-Depends: ${misc:Pre-Depends}
Depends: adduser, ${misc:Depends}, ${shlibs:Depends}
Recommends: ibverbs-providers
-Breaks: ibverbs-providers (<< 33~)
+Breaks: ibverbs-providers (<< 34~)
Description: Library for direct userspace use of RDMA (InfiniBand/iWARP)
libibverbs is a library that allows userspace processes to use RDMA
"verbs" as described in the InfiniBand Architecture Specification and
diff --git a/debian/libibverbs1.symbols b/debian/libibverbs1.symbols
index 9130f41..6cf5c2f 100644
--- a/debian/libibverbs1.symbols
+++ b/debian/libibverbs1.symbols
@@ -9,7 +9,8 @@ libibverbs.so.1 libibverbs1 #MINVER#
IBVERBS_1.9@IBVERBS_1.9 30
IBVERBS_1.10@IBVERBS_1.10 31
IBVERBS_1.11@IBVERBS_1.11 32
- (symver)IBVERBS_PRIVATE_33 33
+ IBVERBS_1.12@IBVERBS_1.12 34
+ (symver)IBVERBS_PRIVATE_34 34
_ibv_query_gid_ex@IBVERBS_1.11 32
_ibv_query_gid_table@IBVERBS_1.11 32
ibv_ack_async_event@IBVERBS_1.0 1.1.6
@@ -99,6 +100,7 @@ libibverbs.so.1 libibverbs1 #MINVER#
ibv_rate_to_mbps@IBVERBS_1.1 1.1.8
ibv_rate_to_mult@IBVERBS_1.0 1.1.6
ibv_read_sysfs_file@IBVERBS_1.0 1.1.6
+ ibv_reg_dmabuf_mr@IBVERBS_1.12 34
ibv_reg_mr@IBVERBS_1.0 1.1.6
ibv_reg_mr@IBVERBS_1.1 1.1.6
ibv_reg_mr_iova@IBVERBS_1.7 25
diff --git a/libibverbs/CMakeLists.txt b/libibverbs/CMakeLists.txt
index 0fe4256..d075225 100644
--- a/libibverbs/CMakeLists.txt
+++ b/libibverbs/CMakeLists.txt
@@ -21,7 +21,7 @@ configure_file("libibverbs.map.in"
rdma_library(ibverbs "${CMAKE_CURRENT_BINARY_DIR}/libibverbs.map"
# See Documentation/versioning.md
- 1 1.11.${PACKAGE_VERSION}
+ 1 1.12.${PACKAGE_VERSION}
all_providers.c
cmd.c
cmd_ah.c
diff --git a/libibverbs/cmd_mr.c b/libibverbs/cmd_mr.c
index 42dbe42..af0fad7 100644
--- a/libibverbs/cmd_mr.c
+++ b/libibverbs/cmd_mr.c
@@ -1,5 +1,6 @@
/*
* Copyright (c) 2018 Mellanox Technologies, Ltd. All rights reserved.
+ * Copyright (c) 2020 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -116,3 +117,40 @@ int ibv_cmd_query_mr(struct ibv_pd *pd, struct verbs_mr *vmr,
return 0;
}
+int ibv_cmd_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+ uint64_t iova, int fd, int access,
+ struct verbs_mr *vmr)
+{
+ DECLARE_COMMAND_BUFFER(cmdb, UVERBS_OBJECT_MR,
+ UVERBS_METHOD_REG_DMABUF_MR,
+ 9);
+ struct ib_uverbs_attr *handle;
+ uint32_t lkey, rkey;
+ int ret;
+
+ handle = fill_attr_out_obj(cmdb, UVERBS_ATTR_REG_DMABUF_MR_HANDLE);
+ fill_attr_out_ptr(cmdb, UVERBS_ATTR_REG_DMABUF_MR_RESP_LKEY, &lkey);
+ fill_attr_out_ptr(cmdb, UVERBS_ATTR_REG_DMABUF_MR_RESP_RKEY, &rkey);
+
+ fill_attr_in_obj(cmdb, UVERBS_ATTR_REG_DMABUF_MR_PD_HANDLE, pd->handle);
+ fill_attr_in_uint64(cmdb, UVERBS_ATTR_REG_DMABUF_MR_OFFSET, offset);
+ fill_attr_in_uint64(cmdb, UVERBS_ATTR_REG_DMABUF_MR_LENGTH, length);
+ fill_attr_in_uint64(cmdb, UVERBS_ATTR_REG_DMABUF_MR_IOVA, iova);
+ fill_attr_in_uint32(cmdb, UVERBS_ATTR_REG_DMABUF_MR_FD, fd);
+ fill_attr_in_uint32(cmdb, UVERBS_ATTR_REG_DMABUF_MR_ACCESS_FLAGS, access);
+
+ ret = execute_ioctl(pd->context, cmdb);
+ if (ret)
+ return errno;
+
+ vmr->ibv_mr.handle = read_attr_obj(UVERBS_ATTR_REG_DMABUF_MR_HANDLE,
+ handle);
+ vmr->ibv_mr.context = pd->context;
+ vmr->ibv_mr.lkey = lkey;
+ vmr->ibv_mr.rkey = rkey;
+ vmr->ibv_mr.pd = pd;
+ vmr->ibv_mr.addr = (void *)offset;
+ vmr->ibv_mr.length = length;
+ vmr->mr_type = IBV_MR_TYPE_DMABUF_MR;
+ return 0;
+}
diff --git a/libibverbs/driver.h b/libibverbs/driver.h
index 427c225..0798152 100644
--- a/libibverbs/driver.h
+++ b/libibverbs/driver.h
@@ -2,6 +2,7 @@
* Copyright (c) 2004, 2005 Topspin Communications. All rights reserved.
* Copyright (c) 2005, 2006 Cisco Systems, Inc. All rights reserved.
* Copyright (c) 2005 PathScale, Inc. All rights reserved.
+ * Copyright (c) 2020 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -87,6 +88,7 @@ enum ibv_mr_type {
IBV_MR_TYPE_MR,
IBV_MR_TYPE_NULL_MR,
IBV_MR_TYPE_IMPORTED_MR,
+ IBV_MR_TYPE_DMABUF_MR,
};
struct verbs_mr {
@@ -371,6 +373,9 @@ struct verbs_context_ops {
struct ibv_mr *(*reg_dm_mr)(struct ibv_pd *pd, struct ibv_dm *dm,
uint64_t dm_offset, size_t length,
unsigned int access);
+ struct ibv_mr *(*reg_dmabuf_mr)(struct ibv_pd *pd, uint64_t offset,
+ size_t length, uint64_t iova,
+ int fd, int access);
struct ibv_mr *(*reg_mr)(struct ibv_pd *pd, void *addr, size_t length,
uint64_t hca_va, int access);
int (*req_notify_cq)(struct ibv_cq *cq, int solicited_only);
@@ -489,6 +494,9 @@ int ibv_cmd_advise_mr(struct ibv_pd *pd,
uint32_t flags,
struct ibv_sge *sg_list,
uint32_t num_sge);
+int ibv_cmd_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+ uint64_t iova, int fd, int access,
+ struct verbs_mr *vmr);
int ibv_cmd_alloc_mw(struct ibv_pd *pd, enum ibv_mw_type type,
struct ibv_mw *mw, struct ibv_alloc_mw *cmd,
size_t cmd_size,
diff --git a/libibverbs/dummy_ops.c b/libibverbs/dummy_ops.c
index b6f272d..a4769ad 100644
--- a/libibverbs/dummy_ops.c
+++ b/libibverbs/dummy_ops.c
@@ -1,5 +1,6 @@
/*
* Copyright (c) 2017 Mellanox Technologies, Inc. All rights reserved.
+ * Copyright (c) 2020 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -435,6 +436,14 @@ static struct ibv_mr *reg_mr(struct ibv_pd *pd, void *addr, size_t length,
return NULL;
}
+static struct ibv_mr *reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset,
+ size_t length, uint64_t iova,
+ int fd, int access)
+{
+ errno = EOPNOTSUPP;
+ return NULL;
+}
+
static int req_notify_cq(struct ibv_cq *cq, int solicited_only)
{
return EOPNOTSUPP;
@@ -542,6 +551,7 @@ const struct verbs_context_ops verbs_dummy_ops = {
query_srq,
read_counters,
reg_dm_mr,
+ reg_dmabuf_mr,
reg_mr,
req_notify_cq,
rereg_mr,
@@ -663,6 +673,7 @@ void verbs_set_ops(struct verbs_context *vctx,
SET_OP(vctx, read_counters);
SET_PRIV_OP(ctx, query_srq);
SET_OP(vctx, reg_dm_mr);
+ SET_PRIV_OP_IC(vctx, reg_dmabuf_mr);
SET_PRIV_OP(ctx, reg_mr);
SET_OP(ctx, req_notify_cq);
SET_PRIV_OP(ctx, rereg_mr);
diff --git a/libibverbs/libibverbs.map.in b/libibverbs/libibverbs.map.in
index d124004..745146d 100644
--- a/libibverbs/libibverbs.map.in
+++ b/libibverbs/libibverbs.map.in
@@ -148,6 +148,11 @@ IBVERBS_1.11 {
_ibv_query_gid_table;
} IBVERBS_1.10;
+IBVERBS_1.12 {
+ global:
+ ibv_reg_dmabuf_mr;
+} IBVERBS_1.11;
+
/* If any symbols in this stanza change ABI then the entire staza gets a new symbol
version. See the top level CMakeLists.txt for this setting. */
@@ -211,6 +216,7 @@ IBVERBS_PRIVATE_@IBVERBS_PABI_VERSION@ {
ibv_cmd_query_srq;
ibv_cmd_read_counters;
ibv_cmd_reg_dm_mr;
+ ibv_cmd_reg_dmabuf_mr;
ibv_cmd_reg_mr;
ibv_cmd_req_notify_cq;
ibv_cmd_rereg_mr;
diff --git a/libibverbs/man/ibv_reg_mr.3 b/libibverbs/man/ibv_reg_mr.3
index 2bfc955..a522527 100644
--- a/libibverbs/man/ibv_reg_mr.3
+++ b/libibverbs/man/ibv_reg_mr.3
@@ -3,7 +3,7 @@
.\"
.TH IBV_REG_MR 3 2006-10-31 libibverbs "Libibverbs Programmer's Manual"
.SH "NAME"
-ibv_reg_mr, ibv_reg_mr_iova, ibv_dereg_mr \- register or deregister a memory region (MR)
+ibv_reg_mr, ibv_reg_mr_iova, ibv_reg_dmabuf_mr, ibv_dereg_mr \- register or deregister a memory region (MR)
.SH "SYNOPSIS"
.nf
.B #include <infiniband/verbs.h>
@@ -15,6 +15,10 @@ ibv_reg_mr, ibv_reg_mr_iova, ibv_dereg_mr \- register or deregister a memory reg
.BI " size_t " "length" ", uint64_t " "hca_va" ,
.BI " int " "access" );
.sp
+.BI "struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd " "*pd" ", uint64_t " "offset" ,
+.BI " size_t " "length" ", uint64_t " "iova" ,
+.BI " int " "fd" ", int " "access" );
+.sp
.BI "int ibv_dereg_mr(struct ibv_mr " "*mr" );
.fi
.SH "DESCRIPTION"
@@ -71,11 +75,30 @@ a lkey or rkey. The offset in the memory region is computed as 'addr +
(iova - hca_va)'. Specifying 0 for hca_va has the same effect as
IBV_ACCESS_ZERO_BASED.
.PP
+.B ibv_reg_dmabuf_mr()
+registers a dma-buf based memory region (MR) associated with the protection domain
+.I pd\fR.
+The MR starts at
+.I offset
+of the dma-buf and its size is
+.I length\fR.
+The dma-buf is identified by the file descriptor
+.I fd\fR.
+The argument
+.I iova
+specifies the virtual base address of the MR when accessed through a lkey or rkey.
+It must have the same page offset as
+.I offset\fR.
+The argument
+.I access
+describes the desired memory protection attributes; it is similar to the ibv_reg_mr case except that only the following flags are supported:
+.B IBV_ACCESS_LOCAL_WRITE, IBV_ACCESS_REMOTE_WRITE, IBV_ACCESS_REMOTE_READ, IBV_ACCESS_REMOTE_ATOMIC, IBV_ACCESS_RELAXED_ORDERING.
+.PP
.B ibv_dereg_mr()
deregisters the MR
.I mr\fR.
.SH "RETURN VALUE"
-.B ibv_reg_mr() / ibv_reg_mr_iova()
+.B ibv_reg_mr() / ibv_reg_mr_iova() / ibv_reg_dmabuf_mr()
returns a pointer to the registered MR, or NULL if the request fails.
The local key (\fBL_Key\fR) field
.B lkey
diff --git a/libibverbs/verbs.c b/libibverbs/verbs.c
index 4fb3893..b93046a 100644
--- a/libibverbs/verbs.c
+++ b/libibverbs/verbs.c
@@ -1,6 +1,7 @@
/*
* Copyright (c) 2005 Topspin Communications. All rights reserved.
* Copyright (c) 2006, 2007 Cisco Systems, Inc. All rights reserved.
+ * Copyright (c) 2020 Intel Corperation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -370,6 +371,24 @@ void ibv_unimport_mr(struct ibv_mr *mr)
get_ops(mr->context)->unimport_mr(mr);
}
+struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset,
+ size_t length, uint64_t iova, int fd,
+ int access)
+{
+ struct ibv_mr *mr;
+
+ mr = get_ops(pd->context)->reg_dmabuf_mr(pd, offset, length, iova,
+ fd, access);
+ if (!mr)
+ return NULL;
+
+ mr->context = pd->context;
+ mr->pd = pd;
+ mr->addr = (void *)offset;
+ mr->length = length;
+ return mr;
+}
+
LATEST_SYMVER_FUNC(ibv_rereg_mr, 1_1, "IBVERBS_1.1",
int,
struct ibv_mr *mr, int flags,
diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h
index da782f2..656b0f9 100644
--- a/libibverbs/verbs.h
+++ b/libibverbs/verbs.h
@@ -3,6 +3,7 @@
* Copyright (c) 2004, 2011-2012 Intel Corporation. All rights reserved.
* Copyright (c) 2005, 2006, 2007 Cisco Systems, Inc. All rights reserved.
* Copyright (c) 2005 PathScale, Inc. All rights reserved.
+ * Copyright (c) 2020 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -2536,6 +2537,12 @@ __ibv_reg_mr_iova(struct ibv_pd *pd, void *addr, size_t length, uint64_t iova,
__builtin_constant_p( \
((access) & IBV_ACCESS_OPTIONAL_RANGE) == 0))
+/**
+ * ibv_reg_dmabuf_mr - Register a dambuf-based memory region
+ */
+struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+ uint64_t iova, int fd, int access);
+
enum ibv_rereg_mr_err_code {
/* Old MR is valid, invalid input */
IBV_REREG_MR_ERR_INPUT = -1,
--
1.8.3.1
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH rdma-core v6 3/6] mlx5: Support dma-buf based memory region
2021-01-21 0:14 [PATCH rdma-core v6 0/6] Add user space dma-buf support Jianxin Xiong
2021-01-21 0:14 ` [PATCH rdma-core v6 1/6] Update kernel headers Jianxin Xiong
2021-01-21 0:14 ` [PATCH rdma-core v6 2/6] verbs: Support dma-buf based memory region Jianxin Xiong
@ 2021-01-21 0:14 ` Jianxin Xiong
2021-01-21 0:14 ` [PATCH rdma-core v6 4/6] pyverbs: Add dma-buf based MR support Jianxin Xiong
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Jianxin Xiong @ 2021-01-21 0:14 UTC (permalink / raw)
To: linux-rdma, dri-devel
Cc: Yishai Hadas, Leon Romanovsky, Edward Srouji, Jason Gunthorpe,
Doug Ledford, Daniel Vetter, Christian Koenig, Jianxin Xiong
Implement the new provider method for registering dma-buf based memory
regions.
Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
providers/mlx5/mlx5.c | 2 ++
providers/mlx5/mlx5.h | 3 +++
providers/mlx5/verbs.c | 22 ++++++++++++++++++++++
3 files changed, 27 insertions(+)
diff --git a/providers/mlx5/mlx5.c b/providers/mlx5/mlx5.c
index a2a1696..b3c49af 100644
--- a/providers/mlx5/mlx5.c
+++ b/providers/mlx5/mlx5.c
@@ -1,5 +1,6 @@
/*
* Copyright (c) 2012 Mellanox Technologies, Inc. All rights reserved.
+ * Copyright (c) 2020 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -95,6 +96,7 @@ static const struct verbs_context_ops mlx5_ctx_common_ops = {
.async_event = mlx5_async_event,
.dealloc_pd = mlx5_free_pd,
.reg_mr = mlx5_reg_mr,
+ .reg_dmabuf_mr = mlx5_reg_dmabuf_mr,
.rereg_mr = mlx5_rereg_mr,
.dereg_mr = mlx5_dereg_mr,
.alloc_mw = mlx5_alloc_mw,
diff --git a/providers/mlx5/mlx5.h b/providers/mlx5/mlx5.h
index bafe077..decc7f8 100644
--- a/providers/mlx5/mlx5.h
+++ b/providers/mlx5/mlx5.h
@@ -1,5 +1,6 @@
/*
* Copyright (c) 2012 Mellanox Technologies, Inc. All rights reserved.
+ * Copyright (c) 2020 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -941,6 +942,8 @@ void mlx5_async_event(struct ibv_context *context,
struct ibv_mr *mlx5_alloc_null_mr(struct ibv_pd *pd);
struct ibv_mr *mlx5_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
uint64_t hca_va, int access);
+struct ibv_mr *mlx5_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+ uint64_t iova, int fd, int access);
int mlx5_rereg_mr(struct verbs_mr *mr, int flags, struct ibv_pd *pd, void *addr,
size_t length, int access);
int mlx5_dereg_mr(struct verbs_mr *mr);
diff --git a/providers/mlx5/verbs.c b/providers/mlx5/verbs.c
index b2391e8..cc0bc42 100644
--- a/providers/mlx5/verbs.c
+++ b/providers/mlx5/verbs.c
@@ -1,5 +1,6 @@
/*
* Copyright (c) 2012 Mellanox Technologies, Inc. All rights reserved.
+ * Copyright (c) 2020 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -626,6 +627,27 @@ struct ibv_mr *mlx5_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
return &mr->vmr.ibv_mr;
}
+struct ibv_mr *mlx5_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset, size_t length,
+ uint64_t iova, int fd, int acc)
+{
+ struct mlx5_mr *mr;
+ int ret;
+
+ mr = calloc(1, sizeof(*mr));
+ if (!mr)
+ return NULL;
+
+ ret = ibv_cmd_reg_dmabuf_mr(pd, offset, length, iova, fd, acc,
+ &mr->vmr);
+ if (ret) {
+ free(mr);
+ return NULL;
+ }
+ mr->alloc_flags = acc;
+
+ return &mr->vmr.ibv_mr;
+}
+
struct ibv_mr *mlx5_alloc_null_mr(struct ibv_pd *pd)
{
struct mlx5_mr *mr;
--
1.8.3.1
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH rdma-core v6 4/6] pyverbs: Add dma-buf based MR support
2021-01-21 0:14 [PATCH rdma-core v6 0/6] Add user space dma-buf support Jianxin Xiong
` (2 preceding siblings ...)
2021-01-21 0:14 ` [PATCH rdma-core v6 3/6] mlx5: " Jianxin Xiong
@ 2021-01-21 0:14 ` Jianxin Xiong
2021-01-21 0:14 ` [PATCH rdma-core v6 5/6] tests: Add tests for dma-buf based memory regions Jianxin Xiong
2021-01-21 0:14 ` [PATCH rdma-core v6 6/6] tests: Bug fix for get_access_flags() Jianxin Xiong
5 siblings, 0 replies; 7+ messages in thread
From: Jianxin Xiong @ 2021-01-21 0:14 UTC (permalink / raw)
To: linux-rdma, dri-devel
Cc: Yishai Hadas, Leon Romanovsky, Edward Srouji, Jason Gunthorpe,
Doug Ledford, Daniel Vetter, Christian Koenig, Jianxin Xiong
Define a new sub-class of 'MR' that uses dma-buf object for the memory
region. Define a new class 'DmaBuf' as a wrapper for dma-buf allocation
mechanism implemented in C.
Update the cmake function for cython modules to allow building modules
with mixed cython and c source files.
Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
buildlib/pyverbs_functions.cmake | 78 +++++++----
pyverbs/CMakeLists.txt | 11 +-
pyverbs/dmabuf.pxd | 15 +++
pyverbs/dmabuf.pyx | 73 ++++++++++
pyverbs/dmabuf_alloc.c | 278 +++++++++++++++++++++++++++++++++++++++
pyverbs/dmabuf_alloc.h | 19 +++
pyverbs/libibverbs.pxd | 2 +
pyverbs/mr.pxd | 6 +
pyverbs/mr.pyx | 105 ++++++++++++++-
9 files changed, 557 insertions(+), 30 deletions(-)
create mode 100644 pyverbs/dmabuf.pxd
create mode 100644 pyverbs/dmabuf.pyx
create mode 100644 pyverbs/dmabuf_alloc.c
create mode 100644 pyverbs/dmabuf_alloc.h
diff --git a/buildlib/pyverbs_functions.cmake b/buildlib/pyverbs_functions.cmake
index 953cec2..0792410 100644
--- a/buildlib/pyverbs_functions.cmake
+++ b/buildlib/pyverbs_functions.cmake
@@ -1,35 +1,61 @@
# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
# Copyright (c) 2018, Mellanox Technologies. All rights reserved. See COPYING file
+# Copyright (c) 2020, Intel Corporation. All rights reserved. See COPYING file
+
+function(build_module_from_cfiles PY_MODULE MODULE_NAME ALL_CFILES LINKER_FLAGS)
+ string(REGEX REPLACE "\\.so$" "" SONAME "${MODULE_NAME}${CMAKE_PYTHON_SO_SUFFIX}")
+ add_library(${SONAME} SHARED ${ALL_CFILES})
+ set_target_properties(${SONAME} PROPERTIES
+ COMPILE_FLAGS "${CMAKE_C_FLAGS} -fPIC -fno-strict-aliasing -Wno-unused-function -Wno-redundant-decls -Wno-shadow -Wno-cast-function-type -Wno-implicit-fallthrough -Wno-unknown-warning -Wno-unknown-warning-option -Wno-deprecated-declarations ${NO_VAR_TRACKING_FLAGS}"
+ LIBRARY_OUTPUT_DIRECTORY "${BUILD_PYTHON}/${PY_MODULE}"
+ PREFIX "")
+ target_link_libraries(${SONAME} LINK_PRIVATE ${PYTHON_LIBRARIES} ibverbs rdmacm ${LINKER_FLAGS})
+ install(TARGETS ${SONAME}
+ DESTINATION ${CMAKE_INSTALL_PYTHON_ARCH_LIB}/${PY_MODULE})
+endfunction()
function(rdma_cython_module PY_MODULE LINKER_FLAGS)
- foreach(PYX_FILE ${ARGN})
- get_filename_component(FILENAME ${PYX_FILE} NAME_WE)
- get_filename_component(DIR ${PYX_FILE} DIRECTORY)
- if (DIR)
- set(PYX "${CMAKE_CURRENT_SOURCE_DIR}/${DIR}/${FILENAME}.pyx")
- else()
- set(PYX "${CMAKE_CURRENT_SOURCE_DIR}/${FILENAME}.pyx")
- endif()
- set(CFILE "${CMAKE_CURRENT_BINARY_DIR}/${FILENAME}.c")
- include_directories(${PYTHON_INCLUDE_DIRS})
- add_custom_command(
- OUTPUT "${CFILE}"
- MAIN_DEPENDENCY "${PYX}"
- COMMAND ${CYTHON_EXECUTABLE} "${PYX}" -o "${CFILE}"
- "-I${PYTHON_INCLUDE_DIRS}"
- COMMENT "Cythonizing ${PYX}"
+ set(ALL_CFILES "")
+ set(MODULE_NAME "")
+ foreach(SRC_FILE ${ARGN})
+ get_filename_component(FILENAME ${SRC_FILE} NAME_WE)
+ get_filename_component(DIR ${SRC_FILE} DIRECTORY)
+ get_filename_component(EXT ${SRC_FILE} EXT)
+ if (DIR)
+ set(SRC_PATH "${CMAKE_CURRENT_SOURCE_DIR}/${DIR}")
+ else()
+ set(SRC_PATH "${CMAKE_CURRENT_SOURCE_DIR}")
+ endif()
+ if (${EXT} STREQUAL ".pyx")
+ # each .pyx file starts a new module, finish the previous module first
+ if (ALL_CFILES AND MODULE_NAME)
+ build_module_from_cfiles(${PY_MODULE} ${MODULE_NAME} "${ALL_CFILES}" "${LINKER_FLAGS}")
+ endif()
+ set(PYX "${SRC_PATH}/${FILENAME}.pyx")
+ set(CFILE "${CMAKE_CURRENT_BINARY_DIR}/${FILENAME}.c")
+ include_directories(${PYTHON_INCLUDE_DIRS})
+ add_custom_command(
+ OUTPUT "${CFILE}"
+ MAIN_DEPENDENCY "${PYX}"
+ COMMAND ${CYTHON_EXECUTABLE} "${PYX}" -o "${CFILE}"
+ "-I${PYTHON_INCLUDE_DIRS}"
+ COMMENT "Cythonizing ${PYX}"
)
-
- string(REGEX REPLACE "\\.so$" "" SONAME "${FILENAME}${CMAKE_PYTHON_SO_SUFFIX}")
- add_library(${SONAME} SHARED ${CFILE})
- set_target_properties(${SONAME} PROPERTIES
- COMPILE_FLAGS "${CMAKE_C_FLAGS} -fPIC -fno-strict-aliasing -Wno-unused-function -Wno-redundant-decls -Wno-shadow -Wno-cast-function-type -Wno-implicit-fallthrough -Wno-unknown-warning -Wno-unknown-warning-option -Wno-deprecated-declarations ${NO_VAR_TRACKING_FLAGS}"
- LIBRARY_OUTPUT_DIRECTORY "${BUILD_PYTHON}/${PY_MODULE}"
- PREFIX "")
- target_link_libraries(${SONAME} LINK_PRIVATE ${PYTHON_LIBRARIES} ibverbs rdmacm ${LINKER_FLAGS})
- install(TARGETS ${SONAME}
- DESTINATION ${CMAKE_INSTALL_PYTHON_ARCH_LIB}/${PY_MODULE})
+ set(MODULE_NAME ${FILENAME})
+ set(ALL_CFILES "${CFILE}")
+ elseif(${EXT} STREQUAL ".c")
+ # .c files belong to the same module as the most recent .pyx file,
+ # ignored if appearing before all .pyx files
+ set(CFILE "${SRC_PATH}/${FILENAME}.c")
+ set(ALL_CFILES "${ALL_CFILES};${CFILE}")
+ else()
+ continue()
+ endif()
endforeach()
+ # finish the last module
+ if (ALL_CFILES AND MODULE_NAME)
+ build_module_from_cfiles(${PY_MODULE} ${MODULE_NAME} "${ALL_CFILES}" "${LINKER_FLAGS}")
+ endif()
endfunction()
function(rdma_python_module PY_MODULE)
diff --git a/pyverbs/CMakeLists.txt b/pyverbs/CMakeLists.txt
index 9542c4b..6fd7625 100644
--- a/pyverbs/CMakeLists.txt
+++ b/pyverbs/CMakeLists.txt
@@ -1,5 +1,10 @@
# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
# Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file
+# Copyright (c) 2020, Intel Corporation. All rights reserved. See COPYING file
+
+publish_internal_headers(""
+ dmabuf_alloc.h
+)
rdma_cython_module(pyverbs ""
addr.pyx
@@ -8,15 +13,17 @@ rdma_cython_module(pyverbs ""
cmid.pyx
cq.pyx
device.pyx
+ dmabuf.pyx
+ dmabuf_alloc.c
enums.pyx
mem_alloc.pyx
mr.pyx
pd.pyx
qp.pyx
+ srq.pyx
wr.pyx
xrcd.pyx
- srq.pyx
- )
+)
rdma_python_module(pyverbs
__init__.py
diff --git a/pyverbs/dmabuf.pxd b/pyverbs/dmabuf.pxd
new file mode 100644
index 0000000..a063acb
--- /dev/null
+++ b/pyverbs/dmabuf.pxd
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
+# Copyright (c) 2020, Intel Corporation. All rights reserved. See COPYING file
+
+#cython: language_level=3
+
+cdef class DmaBuf:
+ cdef int drm_fd
+ cdef int handle
+ cdef int fd
+ cdef unsigned long size
+ cdef unsigned long map_offset
+ cdef void *dmabuf
+ cdef object dmabuf_mrs
+ cdef add_ref(self, obj)
+ cpdef close(self)
diff --git a/pyverbs/dmabuf.pyx b/pyverbs/dmabuf.pyx
new file mode 100644
index 0000000..b9406bd
--- /dev/null
+++ b/pyverbs/dmabuf.pyx
@@ -0,0 +1,73 @@
+# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
+# Copyright (c) 2020, Intel Corporation. All rights reserved. See COPYING file
+
+#cython: language_level=3
+
+import weakref
+
+from pyverbs.base cimport close_weakrefs
+from pyverbs.base import PyverbsRDMAErrno
+from pyverbs.mr cimport DmaBufMR
+
+cdef extern from "dmabuf_alloc.h":
+ cdef struct dmabuf:
+ pass
+ dmabuf *dmabuf_alloc(unsigned long size, int unit, int gtt)
+ void dmabuf_free(dmabuf *dmabuf)
+ int dmabuf_get_drm_fd(dmabuf *dmabuf)
+ int dmabuf_get_fd(dmabuf *dmabuf)
+ unsigned long dmabuf_get_offset(dmabuf *dmabuf)
+
+
+cdef class DmaBuf:
+ def __init__(self, size, unit=0, gtt=0):
+ """
+ Allocate DmaBuf object from a GPU device. This is done through the
+ DRI device interface. Usually this requires the effective user id
+ being a member of the 'render' group.
+ :param size: The size (in number of bytes) of the buffer.
+ :param unit: The unit number of the GPU to allocate the buffer from.
+ :param gtt: Allocate from GTT instead of VRAM.
+ :return: The newly created DmaBuf object on success.
+ """
+ self.dmabuf_mrs = weakref.WeakSet()
+ self.dmabuf = dmabuf_alloc(size, unit, gtt)
+ if self.dmabuf == NULL:
+ raise PyverbsRDMAErrno(f'Failed to allocate dmabuf of size {size} on unit {unit}')
+ self.drm_fd = dmabuf_get_drm_fd(<dmabuf *>self.dmabuf)
+ self.fd = dmabuf_get_fd(<dmabuf *>self.dmabuf)
+ self.map_offset = dmabuf_get_offset(<dmabuf *>self.dmabuf)
+
+ def __dealloc__(self):
+ self.close()
+
+ cpdef close(self):
+ if self.dmabuf == NULL:
+ return None
+ close_weakrefs([self.dmabuf_mrs])
+ dmabuf_free(<dmabuf *>self.dmabuf)
+ self.dmabuf = NULL
+
+ cdef add_ref(self, obj):
+ if isinstance(obj, DmaBufMR):
+ self.dmabuf_mrs.add(obj)
+
+ @property
+ def drm_fd(self):
+ return self.drm_fd
+
+ @property
+ def handle(self):
+ return self.handle
+
+ @property
+ def fd(self):
+ return self.fd
+
+ @property
+ def size(self):
+ return self.size
+
+ @property
+ def map_offset(self):
+ return self.map_offset
diff --git a/pyverbs/dmabuf_alloc.c b/pyverbs/dmabuf_alloc.c
new file mode 100644
index 0000000..05eae75
--- /dev/null
+++ b/pyverbs/dmabuf_alloc.c
@@ -0,0 +1,278 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/*
+ * Copyright 2020 Intel Corporation. All rights reserved. See COPYING file
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <unistd.h>
+#include <string.h>
+#include <errno.h>
+#include <drm/drm.h>
+#include <drm/i915_drm.h>
+#include <drm/amdgpu_drm.h>
+#include <drm/radeon_drm.h>
+#include <fcntl.h>
+#include <sys/ioctl.h>
+#include "dmabuf_alloc.h"
+
+/*
+ * Abstraction of the buffer allocation mechanism using the DRM interface.
+ * The interface is accessed by ioctl() calls over the '/dev/dri/renderD*'
+ * device. Successful access usually requires the effective user id being
+ * in the 'render' group.
+ */
+
+struct drm {
+ int fd;
+ int (*alloc)(struct drm *drm, uint64_t size, uint32_t *handle, int gtt);
+ int (*mmap_offset)(struct drm *drm, uint32_t handle, uint64_t *offset);
+};
+
+static int i915_alloc(struct drm *drm, uint64_t size, uint32_t *handle, int gtt)
+{
+ struct drm_i915_gem_create gem_create = {};
+ int err;
+
+ gem_create.size = size;
+ err = ioctl(drm->fd, DRM_IOCTL_I915_GEM_CREATE, &gem_create);
+ if (err)
+ return err;
+
+ *handle = gem_create.handle;
+ return 0;
+}
+
+static int amdgpu_alloc(struct drm *drm, size_t size, uint32_t *handle, int gtt)
+{
+ union drm_amdgpu_gem_create gem_create = {{}};
+ int err;
+
+ gem_create.in.bo_size = size;
+ if (gtt) {
+ gem_create.in.domains = AMDGPU_GEM_DOMAIN_GTT;
+ gem_create.in.domain_flags = AMDGPU_GEM_CREATE_CPU_GTT_USWC;
+ } else {
+ gem_create.in.domains = AMDGPU_GEM_DOMAIN_VRAM;
+ gem_create.in.domain_flags =
+ AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED;
+ }
+ err = ioctl(drm->fd, DRM_IOCTL_AMDGPU_GEM_CREATE, &gem_create);
+ if (err)
+ return err;
+
+ *handle = gem_create.out.handle;
+ return 0;
+}
+
+static int i915_mmap_offset(struct drm *drm, uint32_t handle, uint64_t *offset)
+{
+ struct drm_i915_gem_mmap_gtt gem_mmap = {};
+ int err;
+
+ gem_mmap.handle = handle;
+ err = ioctl(drm->fd, DRM_IOCTL_I915_GEM_MMAP_GTT, &gem_mmap);
+ if (err)
+ return err;
+
+ *offset = gem_mmap.offset;
+ return 0;
+}
+
+static int amdgpu_mmap_offset(struct drm *drm, uint32_t handle,
+ uint64_t *offset)
+{
+ union drm_amdgpu_gem_mmap gem_mmap = {{}};
+ int err;
+
+ gem_mmap.in.handle = handle;
+ err = ioctl(drm->fd, DRM_IOCTL_AMDGPU_GEM_MMAP, &gem_mmap);
+ if (err)
+ return err;
+
+ *offset = gem_mmap.out.addr_ptr;
+ return 0;
+}
+
+static struct drm *drm_open(int unit)
+{
+ char path[32];
+ struct drm_version version = {};
+ char name[16] = {};
+ int err;
+ struct drm *drm;
+
+ drm = malloc(sizeof(*drm));
+ if (!drm)
+ return NULL;
+
+ snprintf(path, sizeof(path), "/dev/dri/renderD%d", unit + 128);
+
+ drm->fd = open(path, O_RDWR);
+ if (drm->fd < 0)
+ goto out_free;
+
+ version.name = name;
+ version.name_len = 16;
+ err = ioctl(drm->fd, DRM_IOCTL_VERSION, &version);
+ if (err)
+ goto out_close;
+
+ if (!strcmp(name, "amdgpu")) {
+ drm->alloc = amdgpu_alloc;
+ drm->mmap_offset = amdgpu_mmap_offset;
+ } else if (!strcmp(name, "i915")) {
+ drm->alloc = i915_alloc;
+ drm->mmap_offset = i915_mmap_offset;
+ } else {
+ errno = EOPNOTSUPP;
+ goto out_close;
+ }
+ return drm;
+
+out_close:
+ close(drm->fd);
+
+out_free:
+ free(drm);
+ return NULL;
+}
+
+static void drm_close(struct drm *drm)
+{
+ if (!drm || drm->fd < 0)
+ return;
+
+ close(drm->fd);
+ free(drm);
+}
+
+static void drm_free_buf(struct drm *drm, uint32_t handle)
+{
+ struct drm_gem_close close = {};
+
+ close.handle = handle;
+ ioctl(drm->fd, DRM_IOCTL_GEM_CLOSE, &close);
+}
+
+static int drm_alloc_buf(struct drm *drm, size_t size, uint32_t *handle,
+ int *fd, int gtt)
+{
+ struct drm_prime_handle prime_handle = {};
+ int err;
+
+ if (!drm || drm->fd < 0)
+ return -EINVAL;
+
+ err = drm->alloc(drm, size, handle, gtt);
+ if (err)
+ return err;
+
+ prime_handle.handle = *handle;
+ prime_handle.flags = O_RDWR;
+ err = ioctl(drm->fd, DRM_IOCTL_PRIME_HANDLE_TO_FD, &prime_handle);
+ if (err) {
+ drm_free_buf(drm, *handle);
+ return err;
+ }
+
+ *fd = prime_handle.fd;
+ return 0;
+}
+
+static int drm_map_buf(struct drm *drm, uint32_t handle, uint64_t *offset)
+{
+ if (!drm || drm->fd < 0)
+ return -EINVAL;
+
+ return drm->mmap_offset(drm, handle, offset);
+}
+
+/*
+ * Abstraction of dmabuf object, allocated using the DRI abstraction defined
+ * above.
+ */
+
+struct dmabuf {
+ struct drm *drm;
+ int fd;
+ uint32_t handle;
+ uint64_t map_offset;
+};
+
+/*
+ * dmabuf_alloc - allocate a dmabuf from GPU
+ * @size - byte size of the buffer to allocate
+ * @unit - the GPU unit to use
+ * @gtt - if true, allocate from GTT instead of VRAM
+ */
+struct dmabuf *dmabuf_alloc(uint64_t size, int unit, int gtt)
+{
+ struct dmabuf *dmabuf;
+ int err;
+
+ dmabuf = malloc(sizeof(*dmabuf));
+ if (!dmabuf)
+ return NULL;
+
+ dmabuf->drm = drm_open(unit);
+ if (!dmabuf->drm)
+ goto out_free;
+
+ err = drm_alloc_buf(dmabuf->drm, size, &dmabuf->handle, &dmabuf->fd, gtt);
+ if (err)
+ goto out_close;
+
+ err = drm_map_buf(dmabuf->drm, dmabuf->handle, &dmabuf->map_offset);
+ if (err)
+ goto out_free_buf;
+
+ return dmabuf;
+
+out_free_buf:
+ drm_free_buf(dmabuf->drm, dmabuf->handle);
+
+out_close:
+ drm_close(dmabuf->drm);
+
+out_free:
+ free(dmabuf);
+ return NULL;
+}
+
+void dmabuf_free(struct dmabuf *dmabuf)
+{
+ if (!dmabuf)
+ return;
+
+ close(dmabuf->fd);
+ drm_free_buf(dmabuf->drm, dmabuf->handle);
+ drm_close(dmabuf->drm);
+ free(dmabuf);
+}
+
+int dmabuf_get_drm_fd(struct dmabuf *dmabuf)
+{
+ if (!dmabuf || !dmabuf->drm)
+ return -1;
+
+ return dmabuf->drm->fd;
+}
+
+int dmabuf_get_fd(struct dmabuf *dmabuf)
+{
+ if (!dmabuf)
+ return -1;
+
+ return dmabuf->fd;
+}
+
+uint64_t dmabuf_get_offset(struct dmabuf *dmabuf)
+{
+ if (!dmabuf)
+ return -1;
+
+ return dmabuf->map_offset;
+}
+
diff --git a/pyverbs/dmabuf_alloc.h b/pyverbs/dmabuf_alloc.h
new file mode 100644
index 0000000..f1b03c5
--- /dev/null
+++ b/pyverbs/dmabuf_alloc.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/*
+ * Copyright 2020 Intel Corporation. All rights reserved. See COPYING file
+ */
+
+#ifndef _DMABUF_ALLOC_H_
+#define _DMABUF_ALLOC_H_
+
+#include <stdint.h>
+
+struct dmabuf;
+
+struct dmabuf *dmabuf_alloc(uint64_t size, int unit, int gtt);
+void dmabuf_free(struct dmabuf *dmabuf);
+int dmabuf_get_drm_fd(struct dmabuf *dmabuf);
+int dmabuf_get_fd(struct dmabuf *dmabuf);
+uint64_t dmabuf_get_offset(struct dmabuf *dmabuf);
+
+#endif /* _DMABUF_ALLOC_H_ */
diff --git a/pyverbs/libibverbs.pxd b/pyverbs/libibverbs.pxd
index 7c41c57..516ec98 100644
--- a/pyverbs/libibverbs.pxd
+++ b/pyverbs/libibverbs.pxd
@@ -507,6 +507,8 @@ cdef extern from 'infiniband/verbs.h':
ibv_pd *ibv_alloc_pd(ibv_context *context)
int ibv_dealloc_pd(ibv_pd *pd)
ibv_mr *ibv_reg_mr(ibv_pd *pd, void *addr, size_t length, int access)
+ ibv_mr *ibv_reg_dmabuf_mr(ibv_pd *pd, uint64_t offset, size_t length,
+ uint64_t iova, int fd, int access)
int ibv_dereg_mr(ibv_mr *mr)
int ibv_advise_mr(ibv_pd *pd, uint32_t advice, uint32_t flags,
ibv_sge *sg_list, uint32_t num_sge)
diff --git a/pyverbs/mr.pxd b/pyverbs/mr.pxd
index ebe8ada..d9a79ff 100644
--- a/pyverbs/mr.pxd
+++ b/pyverbs/mr.pxd
@@ -1,5 +1,6 @@
# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
# Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file
+# Copyright (c) 2020, Intel Corporation. All rights reserved. See COPYING file
#cython: language_level=3
@@ -33,3 +34,8 @@ cdef class MW(PyverbsCM):
cdef class DMMR(MR):
cdef object dm
+
+cdef class DmaBufMR(MR):
+ cdef object dmabuf
+ cdef unsigned long offset
+ cdef object is_dmabuf_internal
diff --git a/pyverbs/mr.pyx b/pyverbs/mr.pyx
index 7011da1..151c47d 100644
--- a/pyverbs/mr.pyx
+++ b/pyverbs/mr.pyx
@@ -1,11 +1,12 @@
# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
# Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file
+# Copyright (c) 2020, Intel Corporation. All rights reserved. See COPYING file
import resource
import logging
from posix.mman cimport mmap, munmap, MAP_PRIVATE, PROT_READ, PROT_WRITE, \
- MAP_ANONYMOUS, MAP_HUGETLB
+ MAP_ANONYMOUS, MAP_HUGETLB, MAP_SHARED
from pyverbs.pyverbs_error import PyverbsError, PyverbsRDMAError, \
PyverbsUserError
from libc.stdint cimport uintptr_t, SIZE_MAX
@@ -14,9 +15,10 @@ from posix.stdlib cimport posix_memalign
from libc.string cimport memcpy, memset
cimport pyverbs.libibverbs_enums as e
from pyverbs.device cimport DM
-from libc.stdlib cimport free
+from libc.stdlib cimport free, malloc
from .cmid cimport CMID
from .pd cimport PD
+from .dmabuf cimport DmaBuf
cdef extern from 'sys/mman.h':
cdef void* MAP_FAILED
@@ -348,6 +350,105 @@ cdef class DMMR(MR):
cpdef read(self, length, offset):
return self.dm.copy_from_dm(offset, length)
+cdef class DmaBufMR(MR):
+ def __init__(self, PD pd not None, length, access, DmaBuf dmabuf=None,
+ offset=0, unit=0, gtt=0):
+ """
+ Initializes a DmaBufMR (DMA-BUF Memory Region) of the given length
+ and access flags using the given PD and DmaBuf objects.
+ :param pd: A PD object
+ :param length: Length in bytes
+ :param access: Access flags, see ibv_access_flags enum
+ :param dmabuf: A DmaBuf object. One will be allocated if absent
+ :param offset: Byte offset from the beginning of the dma-buf
+ :param unit: GPU unit for internal dmabuf allocation
+ :param gtt: If true allocate internal dmabuf from GTT instead of VRAM
+ :return: The newly created DMABUFMR
+ """
+ self.logger = logging.getLogger(self.__class__.__name__)
+ if dmabuf is None:
+ self.is_dmabuf_internal = True
+ dmabuf = DmaBuf(length + offset, unit, gtt)
+ self.mr = v.ibv_reg_dmabuf_mr(pd.pd, offset, length, offset, dmabuf.fd, access)
+ if self.mr == NULL:
+ raise PyverbsRDMAErrno(f'Failed to register a dma-buf MR. length: {length}, access flags: {access}')
+ super().__init__(pd, length, access)
+ self.pd = pd
+ self.dmabuf = dmabuf
+ self.offset = offset
+ pd.add_ref(self)
+ dmabuf.add_ref(self)
+ self.logger.debug(f'Registered dma-buf ibv_mr. Length: {length}, access flags {access}')
+
+ def __dealloc__(self):
+ self.close()
+
+ cpdef close(self):
+ """
+ Closes the underlying C object of the MR and frees the memory allocated.
+ :return: None
+ """
+ if self.mr != NULL:
+ self.logger.debug('Closing dma-buf MR')
+ rc = v.ibv_dereg_mr(self.mr)
+ if rc != 0:
+ raise PyverbsRDMAError('Failed to dereg dma-buf MR', rc)
+ self.pd = None
+ self.mr = NULL
+ # Set self.mr to NULL before closing dmabuf because this method is
+ # re-entered when close_weakrefs() is called inside dmabuf.close().
+ if self.is_dmabuf_internal:
+ self.dmabuf.close()
+ self.dmabuf = None
+
+ @property
+ def offset(self):
+ return self.offset
+
+ @property
+ def dmabuf(self):
+ return self.dmabuf
+
+ def write(self, data, length, offset=0):
+ """
+ Write user data to the dma-buf backing the MR
+ :param data: User data to write
+ :param length: Length of the data to write
+ :param offset: Writing offset
+ :return: None
+ """
+ if isinstance(data, str):
+ data = data.encode()
+ cdef int off = offset + self.offset
+ cdef void *buf = mmap(NULL, length + off, PROT_READ | PROT_WRITE,
+ MAP_SHARED, self.dmabuf.drm_fd,
+ self.dmabuf.map_offset)
+ if buf == MAP_FAILED:
+ raise PyverbsError(f'Failed to map dma-buf of size {length}')
+ memcpy(<char*>(buf + off), <char *>data, length)
+ munmap(buf, length + off)
+
+ cpdef read(self, length, offset):
+ """
+ Reads data from the dma-buf backing the MR
+ :param length: Length of data to read
+ :param offset: Reading offset
+ :return: The data on the buffer in the requested offset
+ """
+ cdef int off = offset + self.offset
+ cdef void *buf = mmap(NULL, length + off, PROT_READ | PROT_WRITE,
+ MAP_SHARED, self.dmabuf.drm_fd,
+ self.dmabuf.map_offset)
+ if buf == MAP_FAILED:
+ raise PyverbsError(f'Failed to map dma-buf of size {length}')
+ cdef char *data =<char*>malloc(length)
+ memset(data, 0, length)
+ memcpy(data, <char*>(buf + off), length)
+ munmap(buf, length + off)
+ res = data[:length]
+ free(data)
+ return res
+
def mwtype2str(mw_type):
mw_types = {1:'IBV_MW_TYPE_1', 2:'IBV_MW_TYPE_2'}
--
1.8.3.1
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH rdma-core v6 5/6] tests: Add tests for dma-buf based memory regions
2021-01-21 0:14 [PATCH rdma-core v6 0/6] Add user space dma-buf support Jianxin Xiong
` (3 preceding siblings ...)
2021-01-21 0:14 ` [PATCH rdma-core v6 4/6] pyverbs: Add dma-buf based MR support Jianxin Xiong
@ 2021-01-21 0:14 ` Jianxin Xiong
2021-01-21 0:14 ` [PATCH rdma-core v6 6/6] tests: Bug fix for get_access_flags() Jianxin Xiong
5 siblings, 0 replies; 7+ messages in thread
From: Jianxin Xiong @ 2021-01-21 0:14 UTC (permalink / raw)
To: linux-rdma, dri-devel
Cc: Yishai Hadas, Leon Romanovsky, Edward Srouji, Jason Gunthorpe,
Doug Ledford, Daniel Vetter, Christian Koenig, Jianxin Xiong
Define a set of unit tests similar to regular MR tests and a set of
tests for send/recv and rdma traffic using dma-buf MRs. Add a utility
function to generate access flags for dma-buf based MRs because the
set of supported flags is smaller.
Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
tests/args_parser.py | 4 +
tests/test_mr.py | 264 ++++++++++++++++++++++++++++++++++++++++++++++++++-
tests/utils.py | 26 +++++
3 files changed, 293 insertions(+), 1 deletion(-)
diff --git a/tests/args_parser.py b/tests/args_parser.py
index 446535a..5bc53b0 100644
--- a/tests/args_parser.py
+++ b/tests/args_parser.py
@@ -19,6 +19,10 @@ class ArgsParser(object):
parser.add_argument('--port',
help='Use port <port> of RDMA device', type=int,
default=1)
+ parser.add_argument('--gpu', nargs='?', type=int, const=0, default=0,
+ help='GPU unit to allocate dmabuf from')
+ parser.add_argument('--gtt', action='store_true', default=False,
+ help='Allocate dmabuf from GTT instead of VRAM')
parser.add_argument('-v', '--verbose', dest='verbosity',
action='store_const',
const=2, help='Verbose output')
diff --git a/tests/test_mr.py b/tests/test_mr.py
index adc649c..6915853 100644
--- a/tests/test_mr.py
+++ b/tests/test_mr.py
@@ -1,5 +1,6 @@
# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
# Copyright (c) 2019 Mellanox Technologies, Inc. All rights reserved. See COPYING file
+# Copyright (c) 2020 Intel Corporation. All rights reserved. See COPYING file
"""
Test module for pyverbs' mr module.
"""
@@ -9,9 +10,10 @@ import errno
from tests.base import PyverbsAPITestCase, RCResources, RDMATestCase
from pyverbs.pyverbs_error import PyverbsRDMAError, PyverbsError
-from pyverbs.mr import MR, MW, DMMR, MWBindInfo, MWBind
+from pyverbs.mr import MR, MW, DMMR, DmaBufMR, MWBindInfo, MWBind
from pyverbs.qp import QPCap, QPInitAttr, QPAttr, QP
from pyverbs.wr import SendWR
+from pyverbs.dmabuf import DmaBuf
import pyverbs.device as d
from pyverbs.pd import PD
import pyverbs.enums as e
@@ -366,3 +368,263 @@ class DMMRTest(PyverbsAPITestCase):
dm_mr = DMMR(pd, dm_mr_len, e.IBV_ACCESS_ZERO_BASED,
dm=dm, offset=dm_mr_offset)
dm_mr.close()
+
+
+def check_dmabuf_support(unit=0):
+ """
+ Check if dma-buf allocation is supported by the system.
+ Skip the test on failure.
+ """
+ device_num = 128 + unit
+ try:
+ DmaBuf(1, unit=unit)
+ except PyverbsRDMAError as ex:
+ if ex.error_code == errno.ENOENT:
+ raise unittest.SkipTest(f'Device /dev/dri/renderD{device_num} is not present')
+ if ex.error_code == errno.EACCES:
+ raise unittest.SkipTest(f'Lack of permission to access /dev/dri/renderD{device_num}')
+ if ex.error_code == errno.EOPNOTSUPP:
+ raise unittest.SkipTest(f'Allocating dmabuf is not supported by /dev/dri/renderD{device_num}')
+
+
+def check_dmabuf_mr_support(pd, unit=0):
+ """
+ Check if dma-buf MR registration is supported by the driver.
+ Skip the test on failure
+ """
+ try:
+ DmaBufMR(pd, 1, 0, unit=unit)
+ except PyverbsRDMAError as ex:
+ if ex.error_code == errno.EOPNOTSUPP:
+ raise unittest.SkipTest('Reg dma-buf MR is not supported by the RDMA driver')
+
+
+class DmaBufMRTest(PyverbsAPITestCase):
+ """
+ Test various functionalities of the DmaBufMR class.
+ """
+ def setUp(self):
+ super().setUp()
+ self.unit = self.config['gpu']
+ self.gtt = self.config['gtt']
+
+ def test_dmabuf_reg_mr(self):
+ """
+ Test ibv_reg_dmabuf_mr()
+ """
+ check_dmabuf_support(self.unit)
+ for ctx, attr, attr_ex in self.devices:
+ with PD(ctx) as pd:
+ check_dmabuf_mr_support(pd, self.unit)
+ flags = u.get_dmabuf_access_flags(ctx)
+ for f in flags:
+ len = u.get_mr_length()
+ for off in [0, len//2]:
+ with DmaBufMR(pd, len, f, offset=off, unit=self.unit,
+ gtt=self.gtt) as mr:
+ pass
+
+ def test_dmabuf_dereg_mr(self):
+ """
+ Test ibv_dereg_mr() with DmaBufMR
+ """
+ check_dmabuf_support(self.unit)
+ for ctx, attr, attr_ex in self.devices:
+ with PD(ctx) as pd:
+ check_dmabuf_mr_support(pd, self.unit)
+ flags = u.get_dmabuf_access_flags(ctx)
+ for f in flags:
+ len = u.get_mr_length()
+ for off in [0, len//2]:
+ with DmaBufMR(pd, len, f, offset=off, unit=self.unit,
+ gtt=self.gtt) as mr:
+ mr.close()
+
+ def test_dmabuf_dereg_mr_twice(self):
+ """
+ Verify that explicit call to DmaBufMR's close() doesn't fail
+ """
+ check_dmabuf_support(self.unit)
+ for ctx, attr, attr_ex in self.devices:
+ with PD(ctx) as pd:
+ check_dmabuf_mr_support(pd, self.unit)
+ flags = u.get_dmabuf_access_flags(ctx)
+ for f in flags:
+ len = u.get_mr_length()
+ for off in [0, len//2]:
+ with DmaBufMR(pd, len, f, offset=off, unit=self.unit,
+ gtt=self.gtt) as mr:
+ # Pyverbs supports multiple destruction of objects,
+ # we are not expecting an exception here.
+ mr.close()
+ mr.close()
+
+ def test_dmabuf_reg_mr_bad_flags(self):
+ """
+ Verify that illegal flags combination fails as expected
+ """
+ check_dmabuf_support(self.unit)
+ for ctx, attr, attr_ex in self.devices:
+ with PD(ctx) as pd:
+ check_dmabuf_mr_support(pd, self.unit)
+ for i in range(5):
+ flags = random.sample([e.IBV_ACCESS_REMOTE_WRITE,
+ e.IBV_ACCESS_REMOTE_ATOMIC],
+ random.randint(1, 2))
+ mr_flags = 0
+ for i in flags:
+ mr_flags += i.value
+ try:
+ DmaBufMR(pd, u.get_mr_length(), mr_flags,
+ unit=self.unit, gtt=self.gtt)
+ except PyverbsRDMAError as err:
+ assert 'Failed to register a dma-buf MR' in err.args[0]
+ else:
+ raise PyverbsRDMAError('Registered a dma-buf MR with illegal falgs')
+
+ def test_dmabuf_write(self):
+ """
+ Test writing to DmaBufMR's buffer
+ """
+ check_dmabuf_support(self.unit)
+ for ctx, attr, attr_ex in self.devices:
+ with PD(ctx) as pd:
+ check_dmabuf_mr_support(pd, self.unit)
+ for i in range(10):
+ mr_len = u.get_mr_length()
+ flags = u.get_dmabuf_access_flags(ctx)
+ for f in flags:
+ for mr_off in [0, mr_len//2]:
+ with DmaBufMR(pd, mr_len, f, offset=mr_off,
+ unit=self.unit, gtt=self.gtt) as mr:
+ write_len = min(random.randint(1, MAX_IO_LEN),
+ mr_len)
+ mr.write('a' * write_len, write_len)
+
+ def test_dmabuf_read(self):
+ """
+ Test reading from DmaBufMR's buffer
+ """
+ check_dmabuf_support(self.unit)
+ for ctx, attr, attr_ex in self.devices:
+ with PD(ctx) as pd:
+ check_dmabuf_mr_support(pd, self.unit)
+ for i in range(10):
+ mr_len = u.get_mr_length()
+ flags = u.get_dmabuf_access_flags(ctx)
+ for f in flags:
+ for mr_off in [0, mr_len//2]:
+ with DmaBufMR(pd, mr_len, f, offset=mr_off,
+ unit=self.unit, gtt=self.gtt) as mr:
+ write_len = min(random.randint(1, MAX_IO_LEN),
+ mr_len)
+ write_str = 'a' * write_len
+ mr.write(write_str, write_len)
+ read_len = random.randint(1, write_len)
+ offset = random.randint(0, write_len-read_len)
+ read_str = mr.read(read_len, offset).decode()
+ assert read_str in write_str
+
+ def test_dmabuf_lkey(self):
+ """
+ Test reading lkey property
+ """
+ check_dmabuf_support(self.unit)
+ for ctx, attr, attr_ex in self.devices:
+ with PD(ctx) as pd:
+ check_dmabuf_mr_support(pd, self.unit)
+ length = u.get_mr_length()
+ flags = u.get_dmabuf_access_flags(ctx)
+ for f in flags:
+ with DmaBufMR(pd, length, f, unit=self.unit,
+ gtt=self.gtt) as mr:
+ mr.lkey
+
+ def test_dmabuf_rkey(self):
+ """
+ Test reading rkey property
+ """
+ check_dmabuf_support(self.unit)
+ for ctx, attr, attr_ex in self.devices:
+ with PD(ctx) as pd:
+ check_dmabuf_mr_support(pd, self.unit)
+ length = u.get_mr_length()
+ flags = u.get_dmabuf_access_flags(ctx)
+ for f in flags:
+ with DmaBufMR(pd, length, f, unit=self.unit,
+ gtt=self.gtt) as mr:
+ mr.rkey
+
+
+class DmaBufRC(RCResources):
+ def __init__(self, dev_name, ib_port, gid_index, unit, gtt):
+ """
+ Initialize an DmaBufRC object.
+ :param dev_name: Device name to be used
+ :param ib_port: IB port of the device to use
+ :param gid_index: Which GID index to use
+ :param unit: GPU unit to allocate dmabuf from
+ :gtt: Allocate dmabuf from GTT instead og VRAM
+ """
+ self.unit = unit
+ self.gtt = gtt
+ super(DmaBufRC, self).__init__(dev_name=dev_name, ib_port=ib_port,
+ gid_index=gid_index)
+
+ def create_mr(self):
+ check_dmabuf_support(self.unit)
+ check_dmabuf_mr_support(self.pd, self.unit)
+ access = e.IBV_ACCESS_LOCAL_WRITE | e.IBV_ACCESS_REMOTE_WRITE
+ mr = DmaBufMR(self.pd, self.msg_size, access, unit=self.unit,
+ gtt=self.gtt)
+ self.mr = mr
+
+ def create_qp_attr(self):
+ qp_attr = QPAttr(port_num=self.ib_port)
+ qp_access = e.IBV_ACCESS_LOCAL_WRITE | e.IBV_ACCESS_REMOTE_WRITE
+ qp_attr.qp_access_flags = qp_access
+ return qp_attr
+
+
+class DmaBufTestCase(RDMATestCase):
+ def setUp(self):
+ super(DmaBufTestCase, self).setUp()
+ self.iters = 100
+ self.unit = self.config['gpu']
+ self.gtt = self.config['gtt']
+
+ def create_players(self, resource, **resource_arg):
+ """
+ Init dma-buf tests resources.
+ :param resource: The RDMA resources to use. A class of type
+ BaseResources.
+ :param resource_arg: Dict of args that specify the resource specific
+ attributes.
+ :return: The (client, server) resources.
+ """
+ client = resource(**self.dev_info, **resource_arg)
+ server = resource(**self.dev_info, **resource_arg)
+ client.pre_run(server.psns, server.qps_num)
+ server.pre_run(client.psns, client.qps_num)
+ return client, server
+
+ def test_dmabuf_rc_traffic(self):
+ """
+ Test send/recv using dma-buf MR over RC
+ """
+ client, server = self.create_players(DmaBufRC, unit=self.unit,
+ gtt=self.gtt)
+ u.traffic(client, server, self.iters, self.gid_index, self.ib_port)
+
+ def test_dmabuf_rdma_traffic(self):
+ """
+ Test rdma write using dma-buf MR
+ """
+ client, server = self.create_players(DmaBufRC, unit=self.unit,
+ gtt=self.gtt)
+ server.rkey = client.mr.rkey
+ server.remote_addr = client.mr.offset
+ client.rkey = server.mr.rkey
+ client.remote_addr = server.mr.offset
+ u.rdma_traffic(client, server, self.iters, self.gid_index, self.ib_port,
+ send_op=e.IBV_WR_RDMA_WRITE)
diff --git a/tests/utils.py b/tests/utils.py
index e2ab669..8546329 100644
--- a/tests/utils.py
+++ b/tests/utils.py
@@ -1,5 +1,6 @@
# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
# Copyright (c) 2019 Mellanox Technologies, Inc. All rights reserved. See COPYING file
+# Copyright (c) 2020 Intel Corporation. All rights reserved. See COPYING file
"""
Provide some useful helper function for pyverbs' tests.
"""
@@ -96,6 +97,31 @@ def get_access_flags(ctx):
return arr
+def get_dmabuf_access_flags(ctx):
+ """
+ Similar to get_access_flags, except that dma-buf MR only support
+ a subset of the flags.
+ :param ctx: Device Context to check capabilities
+ :return: A random legal value for MR flags
+ """
+ attr = ctx.query_device()
+ vals = [e.IBV_ACCESS_LOCAL_WRITE, e.IBV_ACCESS_REMOTE_WRITE,
+ e.IBV_ACCESS_REMOTE_READ, e.IBV_ACCESS_REMOTE_ATOMIC,
+ e.IBV_ACCESS_RELAXED_ORDERING]
+ if not attr.atomic_caps & e.IBV_ATOMIC_HCA:
+ vals.remove(e.IBV_ACCESS_REMOTE_ATOMIC)
+ arr = []
+ for i in range(1, len(vals)):
+ tmp = list(com(vals, i))
+ tmp = filter(filter_illegal_access_flags, tmp)
+ for t in tmp: # Iterate legal combinations and bitwise OR them
+ val = 0
+ for flag in t:
+ val += flag.value
+ arr.append(val)
+ return arr
+
+
def get_dm_attrs(dm_len):
"""
Initializes an AllocDmAttr member with the given length and random
--
1.8.3.1
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH rdma-core v6 6/6] tests: Bug fix for get_access_flags()
2021-01-21 0:14 [PATCH rdma-core v6 0/6] Add user space dma-buf support Jianxin Xiong
` (4 preceding siblings ...)
2021-01-21 0:14 ` [PATCH rdma-core v6 5/6] tests: Add tests for dma-buf based memory regions Jianxin Xiong
@ 2021-01-21 0:14 ` Jianxin Xiong
5 siblings, 0 replies; 7+ messages in thread
From: Jianxin Xiong @ 2021-01-21 0:14 UTC (permalink / raw)
To: linux-rdma, dri-devel
Cc: Yishai Hadas, Leon Romanovsky, Edward Srouji, Jason Gunthorpe,
Doug Ledford, Daniel Vetter, Christian Koenig, Jianxin Xiong
The filter definition is wrong and causes get_access_flags() always
returning empty list. As the result the MR tests using this function
are effectively skipped (but report success).
Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
---
tests/utils.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tests/utils.py b/tests/utils.py
index 8546329..c41cb3b 100644
--- a/tests/utils.py
+++ b/tests/utils.py
@@ -58,8 +58,8 @@ def filter_illegal_access_flags(element):
:param element: A list of access flags to check
:return: True if this list is legal, else False
"""
- if e.IBV_ACCESS_REMOTE_ATOMIC in element or e.IBV_ACCESS_REMOTE_WRITE:
- if e.IBV_ACCESS_LOCAL_WRITE:
+ if e.IBV_ACCESS_REMOTE_ATOMIC in element or e.IBV_ACCESS_REMOTE_WRITE in element:
+ if not e.IBV_ACCESS_LOCAL_WRITE in element:
return False
return True
--
1.8.3.1
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
^ permalink raw reply related [flat|nested] 7+ messages in thread
end of thread, other threads:[~2021-01-21 0:00 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-21 0:14 [PATCH rdma-core v6 0/6] Add user space dma-buf support Jianxin Xiong
2021-01-21 0:14 ` [PATCH rdma-core v6 1/6] Update kernel headers Jianxin Xiong
2021-01-21 0:14 ` [PATCH rdma-core v6 2/6] verbs: Support dma-buf based memory region Jianxin Xiong
2021-01-21 0:14 ` [PATCH rdma-core v6 3/6] mlx5: " Jianxin Xiong
2021-01-21 0:14 ` [PATCH rdma-core v6 4/6] pyverbs: Add dma-buf based MR support Jianxin Xiong
2021-01-21 0:14 ` [PATCH rdma-core v6 5/6] tests: Add tests for dma-buf based memory regions Jianxin Xiong
2021-01-21 0:14 ` [PATCH rdma-core v6 6/6] tests: Bug fix for get_access_flags() Jianxin Xiong
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).