* [PATCH rdma-core 0/4] pyverbs: Introducing mlx5 DV support
@ 2019-10-24 6:00 Noa Osherovich
2019-10-24 6:00 ` [PATCH rdma-core 1/4] pyverbs: Add support for providers' context Noa Osherovich
` (3 more replies)
0 siblings, 4 replies; 5+ messages in thread
From: Noa Osherovich @ 2019-10-24 6:00 UTC (permalink / raw)
To: dledford, Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, Noa Osherovich
Direct verbs (DV) provide fast data path execution. This series
introduces the DV infrastructure to pyverbs:
Allow providers to open contexts rather than use ibv_open_device.
Add Mlx5Context and expose the mlx5dv_query_device.
Noa Osherovich (4):
pyverbs: Add support for providers' context
pyverbs/mlx5: Add support for driver-specific context
pyverbs: Add providers to cmake build
pyverbs/mlx5: Add query device capability
buildlib/pyverbs_functions.cmake | 11 +-
pyverbs/CMakeLists.txt | 7 +-
pyverbs/device.pxd | 1 +
pyverbs/device.pyx | 17 +-
pyverbs/providers/__init__.pxd | 0
pyverbs/providers/__init__.py | 0
pyverbs/providers/mlx5/CMakeLists.txt | 7 +
pyverbs/providers/mlx5/__init__.pxd | 0
pyverbs/providers/mlx5/__init__.py | 0
pyverbs/providers/mlx5/libmlx5.pxd | 46 +++++
pyverbs/providers/mlx5/mlx5_enums.pyx | 1 +
pyverbs/providers/mlx5/mlx5dv.pxd | 17 ++
pyverbs/providers/mlx5/mlx5dv.pyx | 253 ++++++++++++++++++++++++
pyverbs/providers/mlx5/mlx5dv_enums.pxd | 47 +++++
pyverbs/qp.pyx | 2 +-
15 files changed, 400 insertions(+), 9 deletions(-)
create mode 100644 pyverbs/providers/__init__.pxd
create mode 100644 pyverbs/providers/__init__.py
create mode 100644 pyverbs/providers/mlx5/CMakeLists.txt
create mode 100644 pyverbs/providers/mlx5/__init__.pxd
create mode 100644 pyverbs/providers/mlx5/__init__.py
create mode 100644 pyverbs/providers/mlx5/libmlx5.pxd
create mode 120000 pyverbs/providers/mlx5/mlx5_enums.pyx
create mode 100644 pyverbs/providers/mlx5/mlx5dv.pxd
create mode 100644 pyverbs/providers/mlx5/mlx5dv.pyx
create mode 100644 pyverbs/providers/mlx5/mlx5dv_enums.pxd
--
2.21.0
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH rdma-core 1/4] pyverbs: Add support for providers' context
2019-10-24 6:00 [PATCH rdma-core 0/4] pyverbs: Introducing mlx5 DV support Noa Osherovich
@ 2019-10-24 6:00 ` Noa Osherovich
2019-10-24 6:00 ` [PATCH rdma-core 2/4] pyverbs/mlx5: Add support for driver-specific context Noa Osherovich
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Noa Osherovich @ 2019-10-24 6:00 UTC (permalink / raw)
To: dledford, Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, Noa Osherovich
Providers can supply an API for a driver-specific context, extending
ibv_context.
To allow the provider to open the device given provider-specific
attributes, Context class will only locate the right ibv_device but
will not open it, leaving the provider to open it in any way it finds
fit.
Signed-off-by: Noa Osherovich <noaos@mellanox.com>
---
pyverbs/device.pxd | 1 +
pyverbs/device.pyx | 17 +++++++++++++----
pyverbs/qp.pyx | 2 +-
3 files changed, 15 insertions(+), 5 deletions(-)
diff --git a/pyverbs/device.pxd b/pyverbs/device.pxd
index 5cc13acb4331..a701a3deeabd 100755
--- a/pyverbs/device.pxd
+++ b/pyverbs/device.pxd
@@ -9,6 +9,7 @@ cimport pyverbs.libibverbs as v
cdef class Context(PyverbsCM):
cdef v.ibv_context *context
+ cdef v.ibv_device *device
cdef object name
cdef add_ref(self, obj)
cdef object pds
diff --git a/pyverbs/device.pyx b/pyverbs/device.pyx
index 084086bdbc69..58a2aca27fcc 100755
--- a/pyverbs/device.pyx
+++ b/pyverbs/device.pyx
@@ -78,9 +78,13 @@ cdef class Context(PyverbsCM):
"""
Initializes a Context object. The function searches the IB devices list
for a device with the name provided by the user. If such a device is
- found, it is opened.
- :param kwargs: Currently supports 'name' argument only, the IB device's
- name.
+ found, it is opened (unless provider attributes were given).
+ :param kwargs: Arguments:
+ * *name* (str)
+ The RDMA device's name
+ * *attr* (object)
+ Device-specific attributes, meaning that the device is to be
+ opened by the provider
:return: None
"""
cdef int count
@@ -94,7 +98,7 @@ cdef class Context(PyverbsCM):
self.xrcds = weakref.WeakSet()
dev_name = kwargs.get('name')
-
+ provider_attr = kwargs.get('attr')
if dev_name is not None:
self.name = dev_name
else:
@@ -106,6 +110,11 @@ cdef class Context(PyverbsCM):
try:
for i in range(count):
if dev_list[i].name.decode() == self.name:
+ if provider_attr is not None:
+ # A provider opens its own context, we're just
+ # setting its IB device
+ self.device = dev_list[i]
+ return
self.context = v.ibv_open_device(dev_list[i])
if self.context == NULL:
raise PyverbsRDMAErrno('Failed to open device {dev}'.
diff --git a/pyverbs/qp.pyx b/pyverbs/qp.pyx
index 961e0d59c589..9167e5f2bd01 100755
--- a/pyverbs/qp.pyx
+++ b/pyverbs/qp.pyx
@@ -812,7 +812,7 @@ cdef class QP(PyverbsCM):
self.update_cqs(init_attr)
# In order to use cdef'd methods, a proper casting must be done, let's
# infer the type.
- if type(creator) == Context:
+ if issubclass(type(creator), Context):
self._create_qp_ex(creator, init_attr)
ctx = <Context>creator
self.context = ctx
--
2.21.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH rdma-core 2/4] pyverbs/mlx5: Add support for driver-specific context
2019-10-24 6:00 [PATCH rdma-core 0/4] pyverbs: Introducing mlx5 DV support Noa Osherovich
2019-10-24 6:00 ` [PATCH rdma-core 1/4] pyverbs: Add support for providers' context Noa Osherovich
@ 2019-10-24 6:00 ` Noa Osherovich
2019-10-24 6:00 ` [PATCH rdma-core 3/4] pyverbs: Add providers to cmake build Noa Osherovich
2019-10-24 6:00 ` [PATCH rdma-core 4/4] pyverbs/mlx5: Add query device capability Noa Osherovich
3 siblings, 0 replies; 5+ messages in thread
From: Noa Osherovich @ 2019-10-24 6:00 UTC (permalink / raw)
To: dledford, Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, Noa Osherovich
Allow users to open a device using mlx5dv_open_device. For this, two
new classes are introduced:
- Mlx5Context which inherits from Context.
- Mlx5DVContextAttr, which represents mlx5dv_context_attr struct,
used by users to define driver-specific options for the device
opening.
Signed-off-by: Noa Osherovich <noaos@mellanox.com>
---
pyverbs/providers/__init__.pxd | 0
pyverbs/providers/__init__.py | 0
pyverbs/providers/mlx5/__init__.pxd | 0
pyverbs/providers/mlx5/__init__.py | 0
pyverbs/providers/mlx5/libmlx5.pxd | 17 +++++++++
pyverbs/providers/mlx5/mlx5dv.pxd | 14 +++++++
pyverbs/providers/mlx5/mlx5dv.pyx | 57 +++++++++++++++++++++++++++++
7 files changed, 88 insertions(+)
create mode 100644 pyverbs/providers/__init__.pxd
create mode 100644 pyverbs/providers/__init__.py
create mode 100644 pyverbs/providers/mlx5/__init__.pxd
create mode 100644 pyverbs/providers/mlx5/__init__.py
create mode 100644 pyverbs/providers/mlx5/libmlx5.pxd
create mode 100644 pyverbs/providers/mlx5/mlx5dv.pxd
create mode 100644 pyverbs/providers/mlx5/mlx5dv.pyx
diff --git a/pyverbs/providers/__init__.pxd b/pyverbs/providers/__init__.pxd
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/pyverbs/providers/__init__.py b/pyverbs/providers/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/pyverbs/providers/mlx5/__init__.pxd b/pyverbs/providers/mlx5/__init__.pxd
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/pyverbs/providers/mlx5/__init__.py b/pyverbs/providers/mlx5/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/pyverbs/providers/mlx5/libmlx5.pxd b/pyverbs/providers/mlx5/libmlx5.pxd
new file mode 100644
index 000000000000..54d91e288590
--- /dev/null
+++ b/pyverbs/providers/mlx5/libmlx5.pxd
@@ -0,0 +1,17 @@
+# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
+# Copyright (c) 2019 Mellanox Technologies, Inc. All rights reserved. See COPYING file
+
+from libcpp cimport bool
+
+cimport pyverbs.libibverbs as v
+
+
+cdef extern from 'infiniband/mlx5dv.h':
+
+ cdef struct mlx5dv_context_attr:
+ unsigned int flags
+ unsigned long comp_mask
+
+ bool mlx5dv_is_supported(v.ibv_device *device)
+ v.ibv_context* mlx5dv_open_device(v.ibv_device *device,
+ mlx5dv_context_attr *attr)
diff --git a/pyverbs/providers/mlx5/mlx5dv.pxd b/pyverbs/providers/mlx5/mlx5dv.pxd
new file mode 100644
index 000000000000..6ab94b6484b0
--- /dev/null
+++ b/pyverbs/providers/mlx5/mlx5dv.pxd
@@ -0,0 +1,14 @@
+# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
+# Copyright (c) 2019 Mellanox Technologies, Inc. All rights reserved. See COPYING file
+
+#cython: language_level=3
+
+cimport pyverbs.providers.mlx5.libmlx5 as dv
+from pyverbs.base cimport PyverbsObject
+from pyverbs.device cimport Context
+
+cdef class Mlx5Context(Context):
+ pass
+
+cdef class Mlx5DVContextAttr(PyverbsObject):
+ cdef dv.mlx5dv_context_attr attr
diff --git a/pyverbs/providers/mlx5/mlx5dv.pyx b/pyverbs/providers/mlx5/mlx5dv.pyx
new file mode 100644
index 000000000000..0c6b28be1d5a
--- /dev/null
+++ b/pyverbs/providers/mlx5/mlx5dv.pyx
@@ -0,0 +1,57 @@
+# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
+# Copyright (c) 2019 Mellanox Technologies, Inc. All rights reserved. See COPYING file
+
+from pyverbs.pyverbs_error import PyverbsUserError
+cimport pyverbs.providers.mlx5.libmlx5 as dv
+
+
+cdef class Mlx5DVContextAttr(PyverbsObject):
+ """
+ Represent mlx5dv_context_attr struct. This class is used to open an mlx5
+ device.
+ """
+ def __cinit__(self, flags=0, comp_mask=0):
+ self.attr.flags = flags
+ self.attr.comp_mask = comp_mask
+
+ def __str__(self):
+ print_format = '{:20}: {:<20}\n'
+ return print_format.format('flags', self.attr.flags) +\
+ print_format.format('comp_mask', self.attr.comp_mask)
+
+ @property
+ def flags(self):
+ return self.attr.flags
+ @flags.setter
+ def flags(self, val):
+ self.attr.flags = val
+
+ @property
+ def comp_mask(self):
+ return self.attr.comp_mask
+ @comp_mask.setter
+ def comp_mask(self, val):
+ self.attr.comp_mask = val
+
+
+cdef class Mlx5Context(Context):
+ """
+ Represent mlx5 context, which extends Context.
+ """
+ def __cinit__(self, **kwargs):
+ """
+ Open an mlx5 device using the given attributes
+ :param kwargs: Arguments:
+ * *name* (str)
+ The RDMA device's name (used by parent class)
+ * *attr* (Mlx5DVContextAttr)
+ mlx5-specific device attributes
+ :return: None
+ """
+ cdef Mlx5DVContextAttr attr
+ attr = kwargs.get('attr')
+ if not attr or not isinstance(attr, Mlx5DVContextAttr):
+ raise PyverbsUserError('Missing provider attributes')
+ if not dv.mlx5dv_is_supported(self.device):
+ raise PyverbsUserError('This is not an MLX5 device')
+ self.context = dv.mlx5dv_open_device(self.device, &attr.attr)
--
2.21.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH rdma-core 3/4] pyverbs: Add providers to cmake build
2019-10-24 6:00 [PATCH rdma-core 0/4] pyverbs: Introducing mlx5 DV support Noa Osherovich
2019-10-24 6:00 ` [PATCH rdma-core 1/4] pyverbs: Add support for providers' context Noa Osherovich
2019-10-24 6:00 ` [PATCH rdma-core 2/4] pyverbs/mlx5: Add support for driver-specific context Noa Osherovich
@ 2019-10-24 6:00 ` Noa Osherovich
2019-10-24 6:00 ` [PATCH rdma-core 4/4] pyverbs/mlx5: Add query device capability Noa Osherovich
3 siblings, 0 replies; 5+ messages in thread
From: Noa Osherovich @ 2019-10-24 6:00 UTC (permalink / raw)
To: dledford, Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, Noa Osherovich
Adapt the building function to support subdirectories under pyverbs.
Add the needed CMakeLists file to pyverbs/providers.
Update pyverbs' CMakeLists to include the providers sub directory.
Signed-off-by: Noa Osherovich <noaos@mellanox.com>
---
buildlib/pyverbs_functions.cmake | 11 ++++++++---
pyverbs/CMakeLists.txt | 7 ++++++-
pyverbs/providers/mlx5/CMakeLists.txt | 6 ++++++
3 files changed, 20 insertions(+), 4 deletions(-)
create mode 100644 pyverbs/providers/mlx5/CMakeLists.txt
diff --git a/buildlib/pyverbs_functions.cmake b/buildlib/pyverbs_functions.cmake
index 8ea5dc0df7de..4c255054fe94 100644
--- a/buildlib/pyverbs_functions.cmake
+++ b/buildlib/pyverbs_functions.cmake
@@ -1,10 +1,15 @@
# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
# Copyright (c) 2018, Mellanox Technologies. All rights reserved. See COPYING file
-function(rdma_cython_module PY_MODULE)
+function(rdma_cython_module PY_MODULE LINKER_FLAGS)
foreach(PYX_FILE ${ARGN})
get_filename_component(FILENAME ${PYX_FILE} NAME_WE)
- set(PYX "${CMAKE_CURRENT_SOURCE_DIR}/${FILENAME}.pyx")
+ get_filename_component(DIR ${PYX_FILE} DIRECTORY)
+ if (DIR)
+ set(PYX "${CMAKE_CURRENT_SOURCE_DIR}/${DIR}/${FILENAME}.pyx")
+ else()
+ set(PYX "${CMAKE_CURRENT_SOURCE_DIR}/${FILENAME}.pyx")
+ endif()
set(CFILE "${CMAKE_CURRENT_BINARY_DIR}/${FILENAME}.c")
include_directories(${PYTHON_INCLUDE_DIRS})
add_custom_command(
@@ -20,7 +25,7 @@ function(rdma_cython_module PY_MODULE)
COMPILE_FLAGS "${CMAKE_C_FLAGS} -fPIC -fno-strict-aliasing -Wno-unused-function -Wno-redundant-decls -Wno-shadow -Wno-cast-function-type -Wno-implicit-fallthrough -Wno-unknown-warning -Wno-unknown-warning-option ${NO_VAR_TRACKING_FLAGS}"
LIBRARY_OUTPUT_DIRECTORY "${BUILD_PYTHON}/${PY_MODULE}"
PREFIX "")
- target_link_libraries(${SONAME} LINK_PRIVATE ${PYTHON_LIBRARIES} ibverbs)
+ target_link_libraries(${SONAME} LINK_PRIVATE ${PYTHON_LIBRARIES} ibverbs ${LINKER_FLAGS})
install(TARGETS ${SONAME}
DESTINATION ${CMAKE_INSTALL_PYTHON_ARCH_LIB}/${PY_MODULE})
endforeach()
diff --git a/pyverbs/CMakeLists.txt b/pyverbs/CMakeLists.txt
index 90293982b280..7bbb5fc841c0 100755
--- a/pyverbs/CMakeLists.txt
+++ b/pyverbs/CMakeLists.txt
@@ -1,7 +1,7 @@
# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
# Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file
-rdma_cython_module(pyverbs
+rdma_cython_module(pyverbs ""
addr.pyx
base.pyx
cq.pyx
@@ -20,3 +20,8 @@ rdma_python_module(pyverbs
pyverbs_error.py
utils.py
)
+
+# mlx5 provider is not built without coherent DMA, e.g. ARM32 build.
+if (HAVE_COHERENT_DMA)
+add_subdirectory(providers/mlx5)
+endif()
diff --git a/pyverbs/providers/mlx5/CMakeLists.txt b/pyverbs/providers/mlx5/CMakeLists.txt
new file mode 100644
index 000000000000..f6536de8a932
--- /dev/null
+++ b/pyverbs/providers/mlx5/CMakeLists.txt
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
+# Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file
+
+rdma_cython_module(pyverbs/providers/mlx5 mlx5
+ mlx5dv.pyx
+)
--
2.21.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH rdma-core 4/4] pyverbs/mlx5: Add query device capability
2019-10-24 6:00 [PATCH rdma-core 0/4] pyverbs: Introducing mlx5 DV support Noa Osherovich
` (2 preceding siblings ...)
2019-10-24 6:00 ` [PATCH rdma-core 3/4] pyverbs: Add providers to cmake build Noa Osherovich
@ 2019-10-24 6:00 ` Noa Osherovich
3 siblings, 0 replies; 5+ messages in thread
From: Noa Osherovich @ 2019-10-24 6:00 UTC (permalink / raw)
To: dledford, Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, Noa Osherovich
Allow users to use the mlx5-specific query_device() function via
Mlx5Context's query_mlx5_device method. This device querying exposes
device properties that are not available via legacy query such as
software parsing and CQE compression capabilities.
This method returns an Mlx5DVContext object which supports a
user-friendly print format.
Signed-off-by: Noa Osherovich <noaos@mellanox.com>
---
pyverbs/providers/mlx5/CMakeLists.txt | 1 +
pyverbs/providers/mlx5/libmlx5.pxd | 29 ++++
pyverbs/providers/mlx5/mlx5_enums.pyx | 1 +
pyverbs/providers/mlx5/mlx5dv.pxd | 3 +
pyverbs/providers/mlx5/mlx5dv.pyx | 196 ++++++++++++++++++++++++
pyverbs/providers/mlx5/mlx5dv_enums.pxd | 47 ++++++
6 files changed, 277 insertions(+)
create mode 120000 pyverbs/providers/mlx5/mlx5_enums.pyx
create mode 100644 pyverbs/providers/mlx5/mlx5dv_enums.pxd
diff --git a/pyverbs/providers/mlx5/CMakeLists.txt b/pyverbs/providers/mlx5/CMakeLists.txt
index f6536de8a932..d9b0849f02dc 100644
--- a/pyverbs/providers/mlx5/CMakeLists.txt
+++ b/pyverbs/providers/mlx5/CMakeLists.txt
@@ -3,4 +3,5 @@
rdma_cython_module(pyverbs/providers/mlx5 mlx5
mlx5dv.pyx
+ mlx5_enums.pyx
)
diff --git a/pyverbs/providers/mlx5/libmlx5.pxd b/pyverbs/providers/mlx5/libmlx5.pxd
index 54d91e288590..aaf75620a9bb 100644
--- a/pyverbs/providers/mlx5/libmlx5.pxd
+++ b/pyverbs/providers/mlx5/libmlx5.pxd
@@ -12,6 +12,35 @@ cdef extern from 'infiniband/mlx5dv.h':
unsigned int flags
unsigned long comp_mask
+ cdef struct mlx5dv_cqe_comp_caps:
+ unsigned int max_num
+ unsigned int supported_format
+
+ cdef struct mlx5dv_sw_parsing_caps:
+ unsigned int sw_parsing_offloads
+ unsigned int supported_qpts
+
+ cdef struct mlx5dv_striding_rq_caps:
+ unsigned int min_single_stride_log_num_of_bytes
+ unsigned int max_single_stride_log_num_of_bytes
+ unsigned int min_single_wqe_log_num_of_strides
+ unsigned int max_single_wqe_log_num_of_strides
+ unsigned int supported_qpts
+
+ cdef struct mlx5dv_context:
+ unsigned char version
+ unsigned long flags
+ unsigned long comp_mask
+ mlx5dv_cqe_comp_caps cqe_comp_caps
+ mlx5dv_sw_parsing_caps sw_parsing_caps
+ mlx5dv_striding_rq_caps striding_rq_caps
+ unsigned int tunnel_offloads_caps
+ unsigned int max_dynamic_bfregs
+ unsigned long max_clock_info_update_nsec
+ unsigned int flow_action_flags
+ unsigned int dc_odp_caps
+
bool mlx5dv_is_supported(v.ibv_device *device)
v.ibv_context* mlx5dv_open_device(v.ibv_device *device,
mlx5dv_context_attr *attr)
+ int mlx5dv_query_device(v.ibv_context *ctx, mlx5dv_context *attrs_out)
diff --git a/pyverbs/providers/mlx5/mlx5_enums.pyx b/pyverbs/providers/mlx5/mlx5_enums.pyx
new file mode 120000
index 000000000000..ba0e916f9e62
--- /dev/null
+++ b/pyverbs/providers/mlx5/mlx5_enums.pyx
@@ -0,0 +1 @@
+mlx5dv_enums.pxd
\ No newline at end of file
diff --git a/pyverbs/providers/mlx5/mlx5dv.pxd b/pyverbs/providers/mlx5/mlx5dv.pxd
index 6ab94b6484b0..d9fea82af894 100644
--- a/pyverbs/providers/mlx5/mlx5dv.pxd
+++ b/pyverbs/providers/mlx5/mlx5dv.pxd
@@ -12,3 +12,6 @@ cdef class Mlx5Context(Context):
cdef class Mlx5DVContextAttr(PyverbsObject):
cdef dv.mlx5dv_context_attr attr
+
+cdef class Mlx5DVContext(PyverbsObject):
+ cdef dv.mlx5dv_context dv
diff --git a/pyverbs/providers/mlx5/mlx5dv.pyx b/pyverbs/providers/mlx5/mlx5dv.pyx
index 0c6b28be1d5a..dadc9cdcceee 100644
--- a/pyverbs/providers/mlx5/mlx5dv.pyx
+++ b/pyverbs/providers/mlx5/mlx5dv.pyx
@@ -2,7 +2,10 @@
# Copyright (c) 2019 Mellanox Technologies, Inc. All rights reserved. See COPYING file
from pyverbs.pyverbs_error import PyverbsUserError
+cimport pyverbs.providers.mlx5.mlx5dv_enums as dve
cimport pyverbs.providers.mlx5.libmlx5 as dv
+from pyverbs.base import PyverbsRDMAErrno
+cimport pyverbs.libibverbs_enums as e
cdef class Mlx5DVContextAttr(PyverbsObject):
@@ -55,3 +58,196 @@ cdef class Mlx5Context(Context):
if not dv.mlx5dv_is_supported(self.device):
raise PyverbsUserError('This is not an MLX5 device')
self.context = dv.mlx5dv_open_device(self.device, &attr.attr)
+
+ def query_mlx5_device(self, comp_mask=-1):
+ """
+ Queries the provider for device-specific attributes.
+ :param comp_mask: Which attributes to query. Default value is -1. If
+ not changed by user, pyverbs will pass a bitwise OR
+ of all available enum entries.
+ :return: A Mlx5DVContext containing the attributes.
+ """
+ dv_attr = Mlx5DVContext()
+ if comp_mask == -1:
+ dv_attr.comp_mask = \
+ dve.MLX5DV_CONTEXT_MASK_CQE_COMPRESION |\
+ dve.MLX5DV_CONTEXT_MASK_SWP |\
+ dve.MLX5DV_CONTEXT_MASK_STRIDING_RQ |\
+ dve.MLX5DV_CONTEXT_MASK_TUNNEL_OFFLOADS |\
+ dve.MLX5DV_CONTEXT_MASK_DYN_BFREGS |\
+ dve.MLX5DV_CONTEXT_MASK_CLOCK_INFO_UPDATE |\
+ dve.MLX5DV_CONTEXT_MASK_FLOW_ACTION_FLAGS
+ else:
+ dv_attr.comp_mask = comp_mask
+ rc = dv.mlx5dv_query_device(self.context, &dv_attr.dv)
+ if rc != 0:
+ raise PyverbsRDMAErrno('Failed to query mlx5 device {name}, got {rc}'.
+ format(name=self.name, rc=rc))
+ return dv_attr
+
+
+cdef class Mlx5DVContext(PyverbsObject):
+ """
+ Represents mlx5dv_context struct, which exposes mlx5-specific capabilities,
+ reported by mlx5dv_query_device.
+ """
+ @property
+ def version(self):
+ return self.dv.version
+
+ @property
+ def flags(self):
+ return self.dv.flags
+
+ @property
+ def comp_mask(self):
+ return self.dv.comp_mask
+ @comp_mask.setter
+ def comp_mask(self, val):
+ self.dv.comp_mask = val
+
+ @property
+ def cqe_comp_caps(self):
+ return self.dv.cqe_comp_caps
+
+ @property
+ def sw_parsing_caps(self):
+ return self.dv.sw_parsing_caps
+
+ @property
+ def striding_rq_caps(self):
+ return self.dv.striding_rq_caps
+
+ @property
+ def tunnel_offload_caps(self):
+ return self.dv.tunnel_offloads_caps
+
+ @property
+ def max_dynamic_bfregs(self):
+ return self.dv.max_dynamic_bfregs
+
+ @property
+ def max_clock_info_update_nsec(self):
+ return self.dv.max_clock_info_update_nsec
+
+ @property
+ def flow_action_flags(self):
+ return self.dv.flow_action_flags
+
+ @property
+ def dc_odp_caps(self):
+ return self.dv.dc_odp_caps
+
+ def __str__(self):
+ print_format = '{:20}: {:<20}\n'
+ ident_format = ' {:20}: {:<20}\n'
+ cqe = 'CQE compression caps:\n' +\
+ ident_format.format('max num',
+ self.dv.cqe_comp_caps.max_num) +\
+ ident_format.format('supported formats',
+ cqe_comp_to_str(self.dv.cqe_comp_caps.supported_format))
+ swp = 'SW parsing caps:\n' +\
+ ident_format.format('SW parsing offloads',
+ swp_to_str(self.dv.sw_parsing_caps.sw_parsing_offloads)) +\
+ ident_format.format('supported QP types',
+ qpts_to_str(self.dv.sw_parsing_caps.supported_qpts))
+ strd = 'Striding RQ caps:\n' +\
+ ident_format.format('min single stride log num of bytes',
+ self.dv.striding_rq_caps.min_single_stride_log_num_of_bytes) +\
+ ident_format.format('max single stride log num of bytes',
+ self.dv.striding_rq_caps.max_single_stride_log_num_of_bytes) +\
+ ident_format.format('min single wqe log num of strides',
+ self.dv.striding_rq_caps.min_single_wqe_log_num_of_strides) +\
+ ident_format.format('max single wqe log num of strides',
+ self.dv.striding_rq_caps.max_single_wqe_log_num_of_strides) +\
+ ident_format.format('supported QP types',
+ qpts_to_str(self.dv.striding_rq_caps.supported_qpts))
+ return print_format.format('Version', self.dv.version) +\
+ print_format.format('Flags',
+ context_flags_to_str(self.dv.flags)) +\
+ print_format.format('comp mask',
+ context_comp_mask_to_str(self.dv.comp_mask)) +\
+ cqe + swp + strd +\
+ print_format.format('Tunnel offloads caps',
+ tunnel_offloads_to_str(self.dv.tunnel_offloads_caps)) +\
+ print_format.format('Max dynamic BF registers',
+ self.dv.max_dynamic_bfregs) +\
+ print_format.format('Max clock info update [nsec]',
+ self.dv.max_clock_info_update_nsec) +\
+ print_format.format('Flow action flags',
+ self.dv.flow_action_flags) +\
+ print_format.format('DC ODP caps', self.dv.dc_odp_caps)
+
+
+def qpts_to_str(qp_types):
+ numberic_types = qp_types
+ qpts_str = ''
+ qpts = {e.IBV_QPT_RC: 'RC', e.IBV_QPT_UC: 'UC', e.IBV_QPT_UD: 'UD',
+ e.IBV_QPT_RAW_PACKET: 'Raw Packet', e.IBV_QPT_XRC_SEND: 'XRC Send',
+ e.IBV_QPT_XRC_RECV: 'XRC Recv', e.IBV_QPT_DRIVER: 'Driver QPT'}
+ for t in qpts.keys():
+ if (1 << t) & qp_types:
+ qpts_str += qpts[t] + ', '
+ qp_types -= t
+ if qp_types == 0:
+ break
+ return qpts_str[:-2] + ' ({})'.format(numberic_types)
+
+
+def bitmask_to_str(bits, values):
+ numeric_bits = bits
+ res = ''
+ for t in values.keys():
+ if t & bits:
+ res += values[t] + ', '
+ bits -= t
+ if bits == 0:
+ break
+ return res[:-2] + ' ({})'.format(numeric_bits) # Remove last comma and space
+
+
+def context_comp_mask_to_str(mask):
+ l = {dve.MLX5DV_CONTEXT_MASK_CQE_COMPRESION: 'CQE compression',
+ dve.MLX5DV_CONTEXT_MASK_SWP: 'SW parsing',
+ dve.MLX5DV_CONTEXT_MASK_STRIDING_RQ: 'Striding RQ',
+ dve.MLX5DV_CONTEXT_MASK_TUNNEL_OFFLOADS: 'Tunnel offloads',
+ dve.MLX5DV_CONTEXT_MASK_DYN_BFREGS: 'Dynamic BF regs',
+ dve.MLX5DV_CONTEXT_MASK_CLOCK_INFO_UPDATE: 'Clock info update',
+ dve.MLX5DV_CONTEXT_MASK_FLOW_ACTION_FLAGS: 'Flow action flags'}
+ return bitmask_to_str(mask, l)
+
+
+def context_flags_to_str(flags):
+ l = {dve.MLX5DV_CONTEXT_FLAGS_CQE_V1: 'CQE v1',
+ dve.MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED: 'Multi packet WQE allowed',
+ dve.MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW: 'Enhanced multi packet WQE',
+ dve.MLX5DV_CONTEXT_FLAGS_CQE_128B_COMP: 'Support CQE 128B compression',
+ dve.MLX5DV_CONTEXT_FLAGS_CQE_128B_PAD: 'Support CQE 128B padding',
+ dve.MLX5DV_CONTEXT_FLAGS_PACKET_BASED_CREDIT_MODE:
+ 'Support packet based credit mode (in RC QP)'}
+ return bitmask_to_str(flags, l)
+
+
+def swp_to_str(swps):
+ l = {dve.MLX5DV_SW_PARSING: 'SW Parsing',
+ dve.MLX5DV_SW_PARSING_CSUM: 'SW Parsing CSUM',
+ dve.MLX5DV_SW_PARSING_LSO: 'SW Parsing LSO'}
+ return bitmask_to_str(swps, l)
+
+
+def cqe_comp_to_str(cqe):
+ l = {dve.MLX5DV_CQE_RES_FORMAT_HASH: 'with hash',
+ dve.MLX5DV_CQE_RES_FORMAT_CSUM: 'with RX checksum CSUM',
+ dve.MLX5DV_CQE_RES_FORMAT_CSUM_STRIDX: 'with stride index'}
+ return bitmask_to_str(cqe, l)
+
+
+def tunnel_offloads_to_str(tun):
+ l = {dve.MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_VXLAN: 'VXLAN',
+ dve.MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_GRE: 'GRE',
+ dve.MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_GENEVE: 'Geneve',
+ dve.MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_CW_MPLS_OVER_GRE:\
+ 'Ctrl word + MPLS over GRE',
+ dve.MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_CW_MPLS_OVER_UDP:\
+ 'Ctrl word + MPLS over UDP'}
+ return bitmask_to_str(tun, l)
diff --git a/pyverbs/providers/mlx5/mlx5dv_enums.pxd b/pyverbs/providers/mlx5/mlx5dv_enums.pxd
new file mode 100644
index 000000000000..038a49111a3b
--- /dev/null
+++ b/pyverbs/providers/mlx5/mlx5dv_enums.pxd
@@ -0,0 +1,47 @@
+# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
+# Copyright (c) 2019 Mellanox Technologies, Inc. All rights reserved. See COPYING file
+
+#cython: language_level=3
+
+cdef extern from 'infiniband/mlx5dv.h':
+
+ cpdef enum mlx5dv_context_comp_mask:
+ MLX5DV_CONTEXT_MASK_CQE_COMPRESION = 1 << 0
+ MLX5DV_CONTEXT_MASK_SWP = 1 << 1
+ MLX5DV_CONTEXT_MASK_STRIDING_RQ = 1 << 2
+ MLX5DV_CONTEXT_MASK_TUNNEL_OFFLOADS = 1 << 3
+ MLX5DV_CONTEXT_MASK_DYN_BFREGS = 1 << 4
+ MLX5DV_CONTEXT_MASK_CLOCK_INFO_UPDATE = 1 << 5
+ MLX5DV_CONTEXT_MASK_FLOW_ACTION_FLAGS = 1 << 6
+
+ cpdef enum mlx5dv_context_flags:
+ MLX5DV_CONTEXT_FLAGS_CQE_V1 = 1 << 0
+ MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED = 1 << 2
+ MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW = 1 << 3
+ MLX5DV_CONTEXT_FLAGS_CQE_128B_COMP = 1 << 4
+ MLX5DV_CONTEXT_FLAGS_CQE_128B_PAD = 1 << 5
+ MLX5DV_CONTEXT_FLAGS_PACKET_BASED_CREDIT_MODE = 1 << 6
+
+ cpdef enum mlx5dv_sw_parsing_offloads:
+ MLX5DV_SW_PARSING = 1 << 0
+ MLX5DV_SW_PARSING_CSUM = 1 << 1
+ MLX5DV_SW_PARSING_LSO = 1 << 2
+
+ cpdef enum mlx5dv_cqe_comp_res_format:
+ MLX5DV_CQE_RES_FORMAT_HASH = 1 << 0
+ MLX5DV_CQE_RES_FORMAT_CSUM = 1 << 1
+ MLX5DV_CQE_RES_FORMAT_CSUM_STRIDX = 1 << 2
+
+ cpdef enum mlx5dv_tunnel_offloads:
+ MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_VXLAN = 1 << 0
+ MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_GRE = 1 << 1
+ MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_GENEVE = 1 << 2
+ MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_CW_MPLS_OVER_GRE = 1 << 3
+ MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_CW_MPLS_OVER_UDP = 1 << 4
+
+ cpdef enum mlx5dv_flow_action_cap_flags:
+ MLX5DV_FLOW_ACTION_FLAGS_ESP_AES_GCM = 1 << 0
+ MLX5DV_FLOW_ACTION_FLAGS_ESP_AES_GCM_REQ_METADATA = 1 << 1
+ MLX5DV_FLOW_ACTION_FLAGS_ESP_AES_GCM_SPI_STEERING = 1 << 2
+ MLX5DV_FLOW_ACTION_FLAGS_ESP_AES_GCM_FULL_OFFLOAD = 1 << 3
+ MLX5DV_FLOW_ACTION_FLAGS_ESP_AES_GCM_TX_IV_IS_ESN = 1 << 4
--
2.21.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
end of thread, other threads:[~2019-10-24 6:01 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-24 6:00 [PATCH rdma-core 0/4] pyverbs: Introducing mlx5 DV support Noa Osherovich
2019-10-24 6:00 ` [PATCH rdma-core 1/4] pyverbs: Add support for providers' context Noa Osherovich
2019-10-24 6:00 ` [PATCH rdma-core 2/4] pyverbs/mlx5: Add support for driver-specific context Noa Osherovich
2019-10-24 6:00 ` [PATCH rdma-core 3/4] pyverbs: Add providers to cmake build Noa Osherovich
2019-10-24 6:00 ` [PATCH rdma-core 4/4] pyverbs/mlx5: Add query device capability Noa Osherovich
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.