* [PATCH rdma-core 0/7] pyverbs/mlx5: Support mlx5 CQ and QP
@ 2019-11-17 13:30 Noa Osherovich
2019-11-17 13:30 ` [PATCH rdma-core 1/7] pyverbs: Allow QP creation by provider Noa Osherovich
` (6 more replies)
0 siblings, 7 replies; 8+ messages in thread
From: Noa Osherovich @ 2019-11-17 13:30 UTC (permalink / raw)
To: dledford, Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, Noa Osherovich
This series adds the needed infrastructure to pyverbs to allow users
to create mlx5 CQ and QP.
Tests using the extended CQ were added as well.
Noa Osherovich (7):
pyverbs: Allow QP creation by provider
pyverbs/mlx5: Add support for mlx5 QP
pyverbs: Add default values for CQ creation
pyverbs: Add support for provider extended CQ
pyverbs/mlx5: Add support for mlx5 CQ
Documentation: Add mlx5 provider to documentation
tests: Add traffic tests using extended CQ
Documentation/pyverbs.md | 105 +++++++++
pyverbs/cq.pyx | 14 +-
pyverbs/libibverbs_enums.pxd | 3 +
pyverbs/providers/mlx5/libmlx5.pxd | 24 ++
pyverbs/providers/mlx5/mlx5dv.pxd | 18 ++
pyverbs/providers/mlx5/mlx5dv.pyx | 295 +++++++++++++++++++++++-
pyverbs/providers/mlx5/mlx5dv_enums.pxd | 30 +++
pyverbs/qp.pyx | 7 +-
tests/CMakeLists.txt | 1 +
tests/test_cqex.py | 75 ++++++
tests/utils.py | 56 ++++-
11 files changed, 609 insertions(+), 19 deletions(-)
create mode 100644 tests/test_cqex.py
--
2.21.0
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH rdma-core 1/7] pyverbs: Allow QP creation by provider
2019-11-17 13:30 [PATCH rdma-core 0/7] pyverbs/mlx5: Support mlx5 CQ and QP Noa Osherovich
@ 2019-11-17 13:30 ` Noa Osherovich
2019-11-17 13:30 ` [PATCH rdma-core 2/7] pyverbs/mlx5: Add support for mlx5 QP Noa Osherovich
` (5 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Noa Osherovich @ 2019-11-17 13:30 UTC (permalink / raw)
To: dledford, Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, Noa Osherovich
A QP can be created using either the generic interface or a provider
specific interface.
Provider QPs extend the legacy QP, which in Cython means that the
legacy QP's __cinit__ will be called automatically. To avoid QP being
initialized via legacy interface, add **kwargs to QP's __cinit__(). If
kwargs is not empty, the QP's __cinit__ will return without creating
the QP, leaving the initialization to the provider.
Signed-off-by: Noa Osherovich <noaos@mellanox.com>
---
pyverbs/qp.pyx | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/pyverbs/qp.pyx b/pyverbs/qp.pyx
index c526f0ad1bb6..74e2ce76665e 100755
--- a/pyverbs/qp.pyx
+++ b/pyverbs/qp.pyx
@@ -783,7 +783,7 @@ cdef class QPAttr(PyverbsObject):
cdef class QP(PyverbsCM):
def __cinit__(self, object creator not None, object init_attr not None,
- QPAttr qp_attr=None):
+ QPAttr qp_attr=None, **kwargs):
"""
Initializes a QP object and performs state transitions according to
user request.
@@ -803,11 +803,16 @@ cdef class QP(PyverbsCM):
using Context).
:param qp_attr: Optional QPAttr object. Will be used for QP state
transitions after creation.
+ :param kwargs: Provider-specific QP creation attributes, meaning that
+ the QP will be created by the provider.
:return: An initialized QP object
"""
cdef PD pd
cdef Context ctx
self.update_cqs(init_attr)
+ if len(kwargs) > 0:
+ # Leave QP initialization to the provider
+ return
# In order to use cdef'd methods, a proper casting must be done, let's
# infer the type.
if issubclass(type(creator), Context):
--
2.21.0
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH rdma-core 2/7] pyverbs/mlx5: Add support for mlx5 QP
2019-11-17 13:30 [PATCH rdma-core 0/7] pyverbs/mlx5: Support mlx5 CQ and QP Noa Osherovich
2019-11-17 13:30 ` [PATCH rdma-core 1/7] pyverbs: Allow QP creation by provider Noa Osherovich
@ 2019-11-17 13:30 ` Noa Osherovich
2019-11-17 13:30 ` [PATCH rdma-core 4/7] pyverbs: Add support for provider extended CQ Noa Osherovich
` (4 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Noa Osherovich @ 2019-11-17 13:30 UTC (permalink / raw)
To: dledford, Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, Noa Osherovich
Add the needed declarations and structs to allow users to create a QP
via the mlx5dv interface.
Creation process is similar to that of a legacy QP but the user shall
use an Mlx5Context and provide an Mlx5DVQPInitAttr object as well as
a QPInitAttrEx object.
Signed-off-by: Noa Osherovich <noaos@mellanox.com>
---
pyverbs/providers/mlx5/libmlx5.pxd | 15 ++
pyverbs/providers/mlx5/mlx5dv.pxd | 11 ++
pyverbs/providers/mlx5/mlx5dv.pyx | 192 +++++++++++++++++++++++-
pyverbs/providers/mlx5/mlx5dv_enums.pxd | 21 +++
4 files changed, 236 insertions(+), 3 deletions(-)
diff --git a/pyverbs/providers/mlx5/libmlx5.pxd b/pyverbs/providers/mlx5/libmlx5.pxd
index aaf75620a9bb..2c5be241a3ec 100644
--- a/pyverbs/providers/mlx5/libmlx5.pxd
+++ b/pyverbs/providers/mlx5/libmlx5.pxd
@@ -1,6 +1,8 @@
# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
# Copyright (c) 2019 Mellanox Technologies, Inc. All rights reserved. See COPYING file
+include 'mlx5dv_enums.pxd'
+
from libcpp cimport bool
cimport pyverbs.libibverbs as v
@@ -40,7 +42,20 @@ cdef extern from 'infiniband/mlx5dv.h':
unsigned int flow_action_flags
unsigned int dc_odp_caps
+ cdef struct mlx5dv_dc_init_attr:
+ mlx5dv_dc_type dc_type
+ unsigned long dct_access_key
+
+ cdef struct mlx5dv_qp_init_attr:
+ unsigned long comp_mask
+ unsigned int create_flags
+ mlx5dv_dc_init_attr dc_init_attr
+ unsigned long send_ops_flags
+
bool mlx5dv_is_supported(v.ibv_device *device)
v.ibv_context* mlx5dv_open_device(v.ibv_device *device,
mlx5dv_context_attr *attr)
int mlx5dv_query_device(v.ibv_context *ctx, mlx5dv_context *attrs_out)
+ v.ibv_qp *mlx5dv_create_qp(v.ibv_context *context,
+ v.ibv_qp_init_attr_ex *qp_attr,
+ mlx5dv_qp_init_attr *mlx5_qp_attr)
diff --git a/pyverbs/providers/mlx5/mlx5dv.pxd b/pyverbs/providers/mlx5/mlx5dv.pxd
index d9fea82af894..918065012d30 100644
--- a/pyverbs/providers/mlx5/mlx5dv.pxd
+++ b/pyverbs/providers/mlx5/mlx5dv.pxd
@@ -6,6 +6,8 @@
cimport pyverbs.providers.mlx5.libmlx5 as dv
from pyverbs.base cimport PyverbsObject
from pyverbs.device cimport Context
+from pyverbs.qp cimport QP
+
cdef class Mlx5Context(Context):
pass
@@ -15,3 +17,12 @@ cdef class Mlx5DVContextAttr(PyverbsObject):
cdef class Mlx5DVContext(PyverbsObject):
cdef dv.mlx5dv_context dv
+
+cdef class Mlx5DVDCInitAttr(PyverbsObject):
+ cdef dv.mlx5dv_dc_init_attr attr
+
+cdef class Mlx5DVQPInitAttr(PyverbsObject):
+ cdef dv.mlx5dv_qp_init_attr attr
+
+cdef class Mlx5QP(QP):
+ cdef object dc_type
diff --git a/pyverbs/providers/mlx5/mlx5dv.pyx b/pyverbs/providers/mlx5/mlx5dv.pyx
index dadc9cdcceee..b4a971a5b935 100644
--- a/pyverbs/providers/mlx5/mlx5dv.pyx
+++ b/pyverbs/providers/mlx5/mlx5dv.pyx
@@ -6,7 +6,8 @@ cimport pyverbs.providers.mlx5.mlx5dv_enums as dve
cimport pyverbs.providers.mlx5.libmlx5 as dv
from pyverbs.base import PyverbsRDMAErrno
cimport pyverbs.libibverbs_enums as e
-
+from pyverbs.qp cimport QPInitAttrEx
+from pyverbs.pd cimport PD
cdef class Mlx5DVContextAttr(PyverbsObject):
"""
@@ -179,8 +180,159 @@ cdef class Mlx5DVContext(PyverbsObject):
print_format.format('DC ODP caps', self.dv.dc_odp_caps)
+cdef class Mlx5DVDCInitAttr(PyverbsObject):
+ """
+ Represents mlx5dv_dc_init_attr struct, which defines initial attributes
+ for DC QP creation.
+ """
+ def __cinit__(self, dc_type=dve.MLX5DV_DCTYPE_DCI, dct_access_key=0):
+ """
+ Initializes an Mlx5DVDCInitAttr object with the given DC type and DCT
+ access key.
+ :param dc_type: Which DC QP to create (DCI/DCT).
+ :param dct_access_key: Access key to be used by the DCT
+ :return: An initializes object
+ """
+ self.attr.dc_type = dc_type
+ self.attr.dct_access_key = dct_access_key
+
+ def __str__(self):
+ print_format = '{:20}: {:<20}\n'
+ return print_format.format('DC type', dc_type_to_str(self.attr.dc_type)) +\
+ print_format.format('DCT access key', self.attr.dct_access_key)
+
+ @property
+ def dc_type(self):
+ return self.attr.dc_type
+ @dc_type.setter
+ def dc_type(self, val):
+ self.attr.dc_type = val
+
+ @property
+ def dct_access_key(self):
+ return self.attr.dct_access_key
+ @dct_access_key.setter
+ def dct_access_key(self, val):
+ self.attr.dct_access_key = val
+
+
+cdef class Mlx5DVQPInitAttr(PyverbsObject):
+ """
+ Represents mlx5dv_qp_init_attr struct, initial attributes used for mlx5 QP
+ creation.
+ """
+ def __cinit__(self, comp_mask=0, create_flags=0,
+ Mlx5DVDCInitAttr dc_init_attr=None, send_ops_flags=0):
+ """
+ Initializes an Mlx5DVQPInitAttr object with the given user data.
+ :param comp_mask: A bitmask specifying which fields are valid
+ :param create_flags: A bitwise OR of mlx5dv_qp_create_flags
+ :param dc_init_attr: Mlx5DVDCInitAttr object
+ :param send_ops_flags: A bitwise OR of mlx5dv_qp_create_send_ops_flags
+ :return: An initialized Mlx5DVQPInitAttr object
+ """
+ self.attr.comp_mask = comp_mask
+ self.attr.create_flags = create_flags
+ self.attr.send_ops_flags = send_ops_flags
+ if dc_init_attr is not None:
+ self.attr.dc_init_attr.dc_type = dc_init_attr.dc_type
+ self.attr.dc_init_attr.dct_access_key = dc_init_attr.dct_access_key
+
+ def __str__(self):
+ print_format = '{:20}: {:<20}\n'
+ return print_format.format('Comp mask',
+ qp_comp_mask_to_str(self.attr.comp_mask)) +\
+ print_format.format('Create flags',
+ qp_create_flags_to_str(self.attr.create_flags)) +\
+ 'DC init attr:\n' +\
+ print_format.format(' DC type',
+ dc_type_to_str(self.attr.dc_init_attr.dc_type)) +\
+ print_format.format(' DCT access key',
+ self.attr.dc_init_attr.dct_access_key) +\
+ print_format.format('Send ops flags',
+ send_ops_flags_to_str(self.attr.send_ops_flags))
+
+ @property
+ def comp_mask(self):
+ return self.attr.comp_mask
+ @comp_mask.setter
+ def comp_mask(self, val):
+ self.attr.comp_mask = val
+
+ @property
+ def create_flags(self):
+ return self.attr.create_flags
+ @create_flags.setter
+ def create_flags(self, val):
+ self.attr.create_flags = val
+
+ @property
+ def send_ops_flags(self):
+ return self.attr.send_ops_flags
+ @send_ops_flags.setter
+ def send_ops_flags(self, val):
+ self.attr.send_ops_flags = val
+
+ @property
+ def dc_type(self):
+ return self.attr.dc_init_attr.dc_type
+ @dc_type.setter
+ def dc_type(self, val):
+ self.attr.dc_init_attr.dc_type = val
+
+ @property
+ def dct_access_key(self):
+ return self.attr.dc_init_attr.dct_access_key
+ @dct_access_key.setter
+ def dct_access_key(self, val):
+ self.attr.dc_init_attr.dct_access_key = val
+
+
+cdef class Mlx5QP(QP):
+ def __cinit__(self, Mlx5Context context, QPInitAttrEx init_attr,
+ Mlx5DVQPInitAttr dv_init_attr):
+ """
+ Initializes an mlx5 QP according to the user-provided data.
+ :param context: mlx5 Context object
+ :param init_attr: QPInitAttrEx object
+ :param dv_init_attr: Mlx5DVQPInitAttr object
+ :return: An initialized Mlx5QP
+ """
+ cdef PD pd
+ self.dc_type = dv_init_attr.dc_type if dv_init_attr else 0
+ if init_attr.pd is not None:
+ pd = <PD>init_attr.pd
+ pd.add_ref(self)
+ self.qp = \
+ dv.mlx5dv_create_qp(context.context,
+ &init_attr.attr,
+ &dv_init_attr.attr if dv_init_attr is not None
+ else NULL)
+ if self.qp == NULL:
+ raise PyverbsRDMAErrno('Failed to create MLX5 QP.\nQPInitAttrEx '
+ 'attributes:\n{}\nMLX5DVQPInitAttr:\n{}'.
+ format(init_attr, dv_init_attr))
+
+ def _get_comp_mask(self, dst):
+ masks = {dve.MLX5DV_DCTYPE_DCT: {'INIT': e.IBV_QP_PKEY_INDEX |
+ e.IBV_QP_PORT | e.IBV_QP_ACCESS_FLAGS,
+ 'RTR': e.IBV_QP_AV |\
+ e.IBV_QP_PATH_MTU |\
+ e.IBV_QP_MIN_RNR_TIMER},
+ dve.MLX5DV_DCTYPE_DCI: {'INIT': e.IBV_QP_PKEY_INDEX |\
+ e.IBV_QP_PORT,
+ 'RTR': e.IBV_QP_PATH_MTU,
+ 'RTS': e.IBV_QP_TIMEOUT |\
+ e.IBV_QP_RETRY_CNT |\
+ e.IBV_QP_RNR_RETRY | e.IBV_QP_SQ_PSN |\
+ e.IBV_QP_MAX_QP_RD_ATOMIC}}
+ if self.dc_type == 0:
+ return super()._get_comp_mask(dst)
+ return masks[self.dc_type][dst] | e.IBV_QP_STATE
+
+
def qpts_to_str(qp_types):
- numberic_types = qp_types
+ numeric_types = qp_types
qpts_str = ''
qpts = {e.IBV_QPT_RC: 'RC', e.IBV_QPT_UC: 'UC', e.IBV_QPT_UD: 'UD',
e.IBV_QPT_RAW_PACKET: 'Raw Packet', e.IBV_QPT_XRC_SEND: 'XRC Send',
@@ -191,7 +343,7 @@ def qpts_to_str(qp_types):
qp_types -= t
if qp_types == 0:
break
- return qpts_str[:-2] + ' ({})'.format(numberic_types)
+ return qpts_str[:-2] + ' ({})'.format(numeric_types)
def bitmask_to_str(bits, values):
@@ -251,3 +403,37 @@ def tunnel_offloads_to_str(tun):
dve.MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_CW_MPLS_OVER_UDP:\
'Ctrl word + MPLS over UDP'}
return bitmask_to_str(tun, l)
+
+
+def dc_type_to_str(dctype):
+ l = {dve.MLX5DV_DCTYPE_DCT: 'DCT', dve.MLX5DV_DCTYPE_DCI: 'DCI'}
+ try:
+ return l[dctype]
+ except KeyError:
+ return 'Unknown DC type ({dc})'.format(dc=dctype)
+
+
+def qp_comp_mask_to_str(flags):
+ l = {dve.MLX5DV_QP_INIT_ATTR_MASK_QP_CREATE_FLAGS: 'Create flags',
+ dve.MLX5DV_QP_INIT_ATTR_MASK_DC: 'DC',
+ dve.MLX5DV_QP_INIT_ATTR_MASK_SEND_OPS_FLAGS: 'Send ops flags'}
+ return bitmask_to_str(flags, l)
+
+
+def qp_create_flags_to_str(flags):
+ l = {dve.MLX5DV_QP_CREATE_TUNNEL_OFFLOADS: 'Tunnel offloads',
+ dve.MLX5DV_QP_CREATE_TIR_ALLOW_SELF_LOOPBACK_UC:
+ 'Allow UC self loopback',
+ dve.MLX5DV_QP_CREATE_TIR_ALLOW_SELF_LOOPBACK_MC:
+ 'Allow MC self loopback',
+ dve.MLX5DV_QP_CREATE_DISABLE_SCATTER_TO_CQE: 'Disable scatter to CQE',
+ dve.MLX5DV_QP_CREATE_ALLOW_SCATTER_TO_CQE: 'Allow scatter to CQE',
+ dve.MLX5DV_QP_CREATE_PACKET_BASED_CREDIT_MODE:
+ 'Packet based credit mode'}
+ return bitmask_to_str(flags, l)
+
+
+def send_ops_flags_to_str(flags):
+ l = {dve.MLX5DV_QP_EX_WITH_MR_INTERLEAVED: 'With MR interleaved',
+ dve.MLX5DV_QP_EX_WITH_MR_LIST: 'With MR list'}
+ return bitmask_to_str(flags, l)
diff --git a/pyverbs/providers/mlx5/mlx5dv_enums.pxd b/pyverbs/providers/mlx5/mlx5dv_enums.pxd
index 038a49111a3b..3f7f591e23a5 100644
--- a/pyverbs/providers/mlx5/mlx5dv_enums.pxd
+++ b/pyverbs/providers/mlx5/mlx5dv_enums.pxd
@@ -45,3 +45,24 @@ cdef extern from 'infiniband/mlx5dv.h':
MLX5DV_FLOW_ACTION_FLAGS_ESP_AES_GCM_SPI_STEERING = 1 << 2
MLX5DV_FLOW_ACTION_FLAGS_ESP_AES_GCM_FULL_OFFLOAD = 1 << 3
MLX5DV_FLOW_ACTION_FLAGS_ESP_AES_GCM_TX_IV_IS_ESN = 1 << 4
+
+ cpdef enum mlx5dv_qp_init_attr_mask:
+ MLX5DV_QP_INIT_ATTR_MASK_QP_CREATE_FLAGS = 1 << 0
+ MLX5DV_QP_INIT_ATTR_MASK_DC = 1 << 1
+ MLX5DV_QP_INIT_ATTR_MASK_SEND_OPS_FLAGS = 1 << 2
+
+ cpdef enum mlx5dv_qp_create_flags:
+ MLX5DV_QP_CREATE_TUNNEL_OFFLOADS = 1 << 0
+ MLX5DV_QP_CREATE_TIR_ALLOW_SELF_LOOPBACK_UC = 1 << 1
+ MLX5DV_QP_CREATE_TIR_ALLOW_SELF_LOOPBACK_MC = 1 << 2
+ MLX5DV_QP_CREATE_DISABLE_SCATTER_TO_CQE = 1 << 3
+ MLX5DV_QP_CREATE_ALLOW_SCATTER_TO_CQE = 1 << 4
+ MLX5DV_QP_CREATE_PACKET_BASED_CREDIT_MODE = 1 << 5
+
+ cpdef enum mlx5dv_dc_type:
+ MLX5DV_DCTYPE_DCT = 1
+ MLX5DV_DCTYPE_DCI = 2
+
+ cpdef enum mlx5dv_qp_create_send_ops_flags:
+ MLX5DV_QP_EX_WITH_MR_INTERLEAVED = 1 << 0
+ MLX5DV_QP_EX_WITH_MR_LIST = 1 << 1
--
2.21.0
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH rdma-core 3/7] pyverbs: Add default values for CQ creation
2019-11-17 13:30 [PATCH rdma-core 0/7] pyverbs/mlx5: Support mlx5 CQ and QP Noa Osherovich
` (2 preceding siblings ...)
2019-11-17 13:30 ` [PATCH rdma-core 4/7] pyverbs: Add support for provider extended CQ Noa Osherovich
@ 2019-11-17 13:30 ` Noa Osherovich
2019-11-17 13:30 ` [PATCH rdma-core 5/7] pyverbs/mlx5: Add support for mlx5 CQ Noa Osherovich
` (2 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Noa Osherovich @ 2019-11-17 13:30 UTC (permalink / raw)
To: dledford, Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, Noa Osherovich
When creating a CQ, set CQ context and completion channel to None and
completion vector to 0 by default, to make the simple CQ creation
shorter.
Signed-off-by: Noa Osherovich <noaos@mellanox.com>
---
pyverbs/cq.pyx | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/pyverbs/cq.pyx b/pyverbs/cq.pyx
index 6a96347ee979..ecfef6d7fdb9 100755
--- a/pyverbs/cq.pyx
+++ b/pyverbs/cq.pyx
@@ -67,8 +67,8 @@ cdef class CQ(PyverbsCM):
A Completion Queue is the notification mechanism for work request
completions. A CQ can have 0 or more associated QPs.
"""
- def __cinit__(self, Context context not None, cqe, cq_context,
- CompChannel channel, comp_vector):
+ def __cinit__(self, Context context not None, cqe, cq_context=None,
+ CompChannel channel=None, comp_vector=0):
"""
Initializes a CQ object with the given parameters.
:param context: The device's context on which to open the CQ
--
2.21.0
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH rdma-core 4/7] pyverbs: Add support for provider extended CQ
2019-11-17 13:30 [PATCH rdma-core 0/7] pyverbs/mlx5: Support mlx5 CQ and QP Noa Osherovich
2019-11-17 13:30 ` [PATCH rdma-core 1/7] pyverbs: Allow QP creation by provider Noa Osherovich
2019-11-17 13:30 ` [PATCH rdma-core 2/7] pyverbs/mlx5: Add support for mlx5 QP Noa Osherovich
@ 2019-11-17 13:30 ` Noa Osherovich
2019-11-17 13:30 ` [PATCH rdma-core 3/7] pyverbs: Add default values for CQ creation Noa Osherovich
` (3 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Noa Osherovich @ 2019-11-17 13:30 UTC (permalink / raw)
To: dledford, Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, Noa Osherovich
An extended CQ can be created using either the generic interface or a
provider specific interface.
Provider CQs extend the legacy extended CQ, which in Cython means that
the legacy CQEX's __cinit__ will be called automatically. To avoid
initialization via legacy interface, add **kwargs to CQEX's
__cinit__(). If kwargs is not empty, the CQEX's __cinit__ will return
without creating the CQ, leaving the initialization to the provider.
Signed-off-by: Noa Osherovich <noaos@mellanox.com>
---
pyverbs/cq.pyx | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/pyverbs/cq.pyx b/pyverbs/cq.pyx
index ecfef6d7fdb9..ac6baa618c3d 100755
--- a/pyverbs/cq.pyx
+++ b/pyverbs/cq.pyx
@@ -254,7 +254,8 @@ cdef class CqInitAttrEx(PyverbsObject):
cdef class CQEX(PyverbsCM):
- def __cinit__(self, Context context not None, CqInitAttrEx init_attr):
+ def __cinit__(self, Context context not None, CqInitAttrEx init_attr,
+ **kwargs):
"""
Initializes a CQEX object on the given device's context with the given
attributes.
@@ -262,6 +263,11 @@ cdef class CQEX(PyverbsCM):
:param init_attr: Initial attributes that describe the CQ
:return: The newly created CQEX on success
"""
+ self.qps = weakref.WeakSet()
+ self.srqs = weakref.WeakSet()
+ if len(kwargs) > 0:
+ # Leave CQ initialization to the provider
+ return
if init_attr is None:
init_attr = CqInitAttrEx()
self.cq = v.ibv_create_cq_ex(context.context, &init_attr.attr)
@@ -272,8 +278,6 @@ cdef class CQEX(PyverbsCM):
self.ibv_cq = v.ibv_cq_ex_to_cq(self.cq)
self.context = context
context.add_ref(self)
- self.qps = weakref.WeakSet()
- self.srqs = weakref.WeakSet()
cdef add_ref(self, obj):
if isinstance(obj, QP):
--
2.21.0
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH rdma-core 5/7] pyverbs/mlx5: Add support for mlx5 CQ
2019-11-17 13:30 [PATCH rdma-core 0/7] pyverbs/mlx5: Support mlx5 CQ and QP Noa Osherovich
` (3 preceding siblings ...)
2019-11-17 13:30 ` [PATCH rdma-core 3/7] pyverbs: Add default values for CQ creation Noa Osherovich
@ 2019-11-17 13:30 ` Noa Osherovich
2019-11-17 13:30 ` [PATCH rdma-core 6/7] Documentation: Add mlx5 provider to documentation Noa Osherovich
2019-11-17 13:30 ` [PATCH rdma-core 7/7] tests: Add traffic tests using extended CQ Noa Osherovich
6 siblings, 0 replies; 8+ messages in thread
From: Noa Osherovich @ 2019-11-17 13:30 UTC (permalink / raw)
To: dledford, Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, Noa Osherovich
Add the needed infrastructure to allow users to create a CQ via the
mlx5dv interface.
Creation process requires CQInitAttrEx, similarly to the creation of
an extended CQ, but requires an Mlx5DVCQInitAttr object as well
(provided as kwargs, i.e. dv_init_attr=<your object>).
Signed-off-by: Noa Osherovich <noaos@mellanox.com>
---
pyverbs/providers/mlx5/libmlx5.pxd | 9 +++
pyverbs/providers/mlx5/mlx5dv.pxd | 7 ++
pyverbs/providers/mlx5/mlx5dv.pyx | 103 ++++++++++++++++++++++++
pyverbs/providers/mlx5/mlx5dv_enums.pxd | 9 +++
4 files changed, 128 insertions(+)
diff --git a/pyverbs/providers/mlx5/libmlx5.pxd b/pyverbs/providers/mlx5/libmlx5.pxd
index 2c5be241a3ec..f471f2453e38 100644
--- a/pyverbs/providers/mlx5/libmlx5.pxd
+++ b/pyverbs/providers/mlx5/libmlx5.pxd
@@ -52,6 +52,12 @@ cdef extern from 'infiniband/mlx5dv.h':
mlx5dv_dc_init_attr dc_init_attr
unsigned long send_ops_flags
+ cdef struct mlx5dv_cq_init_attr:
+ unsigned long comp_mask
+ unsigned char cqe_comp_res_format
+ unsigned int flags
+ unsigned short cqe_size
+
bool mlx5dv_is_supported(v.ibv_device *device)
v.ibv_context* mlx5dv_open_device(v.ibv_device *device,
mlx5dv_context_attr *attr)
@@ -59,3 +65,6 @@ cdef extern from 'infiniband/mlx5dv.h':
v.ibv_qp *mlx5dv_create_qp(v.ibv_context *context,
v.ibv_qp_init_attr_ex *qp_attr,
mlx5dv_qp_init_attr *mlx5_qp_attr)
+ v.ibv_cq_ex *mlx5dv_create_cq(v.ibv_context *context,
+ v.ibv_cq_init_attr_ex *cq_attr,
+ mlx5dv_cq_init_attr *mlx5_cq_attr)
diff --git a/pyverbs/providers/mlx5/mlx5dv.pxd b/pyverbs/providers/mlx5/mlx5dv.pxd
index 918065012d30..cc9c89d82fa6 100644
--- a/pyverbs/providers/mlx5/mlx5dv.pxd
+++ b/pyverbs/providers/mlx5/mlx5dv.pxd
@@ -6,6 +6,7 @@
cimport pyverbs.providers.mlx5.libmlx5 as dv
from pyverbs.base cimport PyverbsObject
from pyverbs.device cimport Context
+from pyverbs.cq cimport CQEX
from pyverbs.qp cimport QP
@@ -26,3 +27,9 @@ cdef class Mlx5DVQPInitAttr(PyverbsObject):
cdef class Mlx5QP(QP):
cdef object dc_type
+
+cdef class Mlx5DVCQInitAttr(PyverbsObject):
+ cdef dv.mlx5dv_cq_init_attr attr
+
+cdef class Mlx5CQ(CQEX):
+ pass
diff --git a/pyverbs/providers/mlx5/mlx5dv.pyx b/pyverbs/providers/mlx5/mlx5dv.pyx
index b4a971a5b935..e4500567ba23 100644
--- a/pyverbs/providers/mlx5/mlx5dv.pyx
+++ b/pyverbs/providers/mlx5/mlx5dv.pyx
@@ -7,6 +7,8 @@ cimport pyverbs.providers.mlx5.libmlx5 as dv
from pyverbs.base import PyverbsRDMAErrno
cimport pyverbs.libibverbs_enums as e
from pyverbs.qp cimport QPInitAttrEx
+from pyverbs.cq cimport CqInitAttrEx
+cimport pyverbs.libibverbs as v
from pyverbs.pd cimport PD
cdef class Mlx5DVContextAttr(PyverbsObject):
@@ -331,6 +333,107 @@ cdef class Mlx5QP(QP):
return masks[self.dc_type][dst] | e.IBV_QP_STATE
+cdef class Mlx5DVCQInitAttr(PyverbsObject):
+ """
+ Represents mlx5dv_cq_init_attr struct, initial attributes used for mlx5 CQ
+ creation.
+ """
+ def __cinit__(self, comp_mask=0, cqe_comp_res_format=0, flags=0, cqe_size=0):
+ """
+ Initializes an Mlx5CQInitAttr object with zeroes as default values.
+ :param comp_mask: Marks which of the following fields should be
+ considered. Use mlx5dv_cq_init_attr_mask enum.
+ :param cqe_comp_res_format: The various CQE response formats of the
+ responder side. Use
+ mlx5dv_cqe_comp_res_format enum.
+ :param flags: A bitwise OR of the various values described in
+ mlx5dv_cq_init_attr_flags.
+ :param cqe_size: Configure the CQE size to be 64 or 128 bytes, other
+ values will cause the CQ creation process to fail.
+ Valid when MLX5DV_CQ_INIT_ATTR_MASK_CQE_SIZE is set.
+ :return: None
+ """
+ self.attr.comp_mask = comp_mask
+ self.attr.cqe_comp_res_format = cqe_comp_res_format
+ self.attr.flags = flags
+ self.attr.cqe_size = cqe_size
+
+ @property
+ def comp_mask(self):
+ return self.attr.comp_mask
+ @comp_mask.setter
+ def comp_mask(self, val):
+ self.attr.comp_mask = val
+
+ @property
+ def cqe_comp_res_format(self):
+ return self.attr.cqe_comp_res_format
+ @cqe_comp_res_format.setter
+ def cqe_comp_res_format(self, val):
+ self.attr.cqe_comp_res_format = val
+
+ @property
+ def flags(self):
+ return self.attr.flags
+ @flags.setter
+ def flags(self, val):
+ self.attr.flags = val
+
+ @property
+ def cqe_size(self):
+ return self.attr.cqe_size
+ @cqe_size.setter
+ def cqe_size(self, val):
+ self.attr.cqe_size = val
+
+ def __str__(self):
+ print_format = '{:22}: {:<20}\n'
+ flags = {dve.MLX5DV_CQ_INIT_ATTR_FLAGS_CQE_PAD:
+ "MLX5DV_CQ_INIT_ATTR_FLAGS_CQE_PAD}"}
+ mask = {dve.MLX5DV_CQ_INIT_ATTR_MASK_COMPRESSED_CQE:
+ "MLX5DV_CQ_INIT_ATTR_MASK_COMPRESSED_CQE",
+ dve.MLX5DV_CQ_INIT_ATTR_MASK_FLAGS:
+ "MLX5DV_CQ_INIT_ATTR_MASK_FLAGS",
+ dve.MLX5DV_CQ_INIT_ATTR_MASK_CQE_SIZE:
+ "MLX5DV_CQ_INIT_ATTR_MASK_CQE_SIZE"}
+ fmt = {dve.MLX5DV_CQE_RES_FORMAT_HASH: "MLX5DV_CQE_RES_FORMAT_HASH",
+ dve.MLX5DV_CQE_RES_FORMAT_CSUM: "MLX5DV_CQE_RES_FORMAT_CSUM",
+ dve.MLX5DV_CQE_RES_FORMAT_CSUM_STRIDX:
+ "MLX5DV_CQE_RES_FORMAT_CSUM_STRIDX"}
+
+ return 'Mlx5DVCQInitAttr:\n' +\
+ print_format.format('comp_mask', bitmask_to_str(self.comp_mask,
+ mask)) +\
+ print_format.format('CQE compression format',
+ bitmask_to_str(self.cqe_comp_res_format,
+ fmt)) +\
+ print_format.format('flags', bitmask_to_str(self.flags,
+ flags)) + \
+ print_format.format('CQE size', self.cqe_size)
+
+
+cdef class Mlx5CQ(CQEX):
+ def __cinit__(self, Mlx5Context context, CqInitAttrEx init_attr,
+ Mlx5DVCQInitAttr dv_init_attr):
+ self.cq = \
+ dv.mlx5dv_create_cq(context.context, &init_attr.attr,
+ &dv_init_attr.attr if dv_init_attr is not None
+ else NULL)
+ if self.cq == NULL:
+ raise PyverbsRDMAErrno('Failed to create MLX5 CQ.\nCQInitAttrEx:\n'
+ '{}\nMLX5DVCQInitAttr:\n{}'.
+ format(init_attr, dv_init_attr))
+ self.ibv_cq = v.ibv_cq_ex_to_cq(self.cq)
+ self.context = context
+ context.add_ref(self)
+
+ def __str__(self):
+ print_format = '{:<22}: {:<20}\n'
+ return 'Mlx5 CQ:\n' +\
+ print_format.format('Handle', self.cq.handle) +\
+ print_format.format('CQEs', self.cq.cqe)
+
+
def qpts_to_str(qp_types):
numeric_types = qp_types
qpts_str = ''
diff --git a/pyverbs/providers/mlx5/mlx5dv_enums.pxd b/pyverbs/providers/mlx5/mlx5dv_enums.pxd
index 3f7f591e23a5..3859dff81ad1 100644
--- a/pyverbs/providers/mlx5/mlx5dv_enums.pxd
+++ b/pyverbs/providers/mlx5/mlx5dv_enums.pxd
@@ -66,3 +66,12 @@ cdef extern from 'infiniband/mlx5dv.h':
cpdef enum mlx5dv_qp_create_send_ops_flags:
MLX5DV_QP_EX_WITH_MR_INTERLEAVED = 1 << 0
MLX5DV_QP_EX_WITH_MR_LIST = 1 << 1
+
+ cpdef enum mlx5dv_cq_init_attr_mask:
+ MLX5DV_CQ_INIT_ATTR_MASK_COMPRESSED_CQE = 1 << 0
+ MLX5DV_CQ_INIT_ATTR_MASK_FLAGS = 1 << 1
+ MLX5DV_CQ_INIT_ATTR_MASK_CQE_SIZE = 1 << 2
+
+ cpdef enum mlx5dv_cq_init_attr_flags:
+ MLX5DV_CQ_INIT_ATTR_FLAGS_CQE_PAD = 1 << 0
+ MLX5DV_CQ_INIT_ATTR_FLAGS_RESERVED = 1 << 1
--
2.21.0
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH rdma-core 6/7] Documentation: Add mlx5 provider to documentation
2019-11-17 13:30 [PATCH rdma-core 0/7] pyverbs/mlx5: Support mlx5 CQ and QP Noa Osherovich
` (4 preceding siblings ...)
2019-11-17 13:30 ` [PATCH rdma-core 5/7] pyverbs/mlx5: Add support for mlx5 CQ Noa Osherovich
@ 2019-11-17 13:30 ` Noa Osherovich
2019-11-17 13:30 ` [PATCH rdma-core 7/7] tests: Add traffic tests using extended CQ Noa Osherovich
6 siblings, 0 replies; 8+ messages in thread
From: Noa Osherovich @ 2019-11-17 13:30 UTC (permalink / raw)
To: dledford, Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, Noa Osherovich
Add code snippets to show:
- Opening an mlx5 device
- Querying an mlx5 device
- Creating an mlx5 QP (DCI and Raw Packet QP with mlx5-specific
capabilties)
- Creating an mlx5 CQ
Signed-off-by: Noa Osherovich <noaos@mellanox.com>
---
Documentation/pyverbs.md | 105 +++++++++++++++++++++++++++++++++++++++
1 file changed, 105 insertions(+)
diff --git a/Documentation/pyverbs.md b/Documentation/pyverbs.md
index 7e81690a064a..e405ab53b423 100755
--- a/Documentation/pyverbs.md
+++ b/Documentation/pyverbs.md
@@ -389,6 +389,111 @@ srq_attr.cq = cq
srq_attr.comp_mask = e.IBV_SRQ_INIT_ATTR_TYPE | e.IBV_SRQ_INIT_ATTR_PD | \
e.IBV_SRQ_INIT_ATTR_CQ | e.IBV_SRQ_INIT_ATTR_XRCD
srq = SRQ(ctx, srq_attr)
+
+
+##### Open an mlx5 provider
+A provider is essentially a Context with driver-specific extra features. As
+such, it inherits from Context. In legcay flow Context iterates over the IB
+devices and opens the one matches the name given by the user (name= argument).
+When provider attributes are also given (attr=), the Context will assign the
+relevant ib_device to its device member, so that the provider will be able to
+open the device in its specific way as demonstated below:
+
+```python
+import pyverbs.providers.mlx5.mlx5dv as m
+from pyverbs.pd import PD
+attr = m.Mlx5DVContextAttr() # Default values are fine
+ctx = m.Mlx5Context(attr=attr, name='rocep0s8f0')
+# The provider context can be used as a regular Context, e.g.:
+pd = PD(ctx) # Success
+```
+
+##### Query an mlx5 provider
+After opening an mlx5 provider, users can use the device-specific query for
+non-legacy attributes. The following snippet demonstrates how to do that.
+```python
+import pyverbs.providers.mlx5.mlx5dv as m
+ctx = m.Mlx5Context(attr=m.Mlx5DVContextAttr(), name='ibp0s8f0')
+mlx5_attrs = ctx.query_mlx5_device()
+print(mlx5_attrs)
+Version : 0
+Flags : CQE v1, Support CQE 128B compression, Support CQE 128B padding, Support packet based credit mode (in RC QP)
+comp mask : CQE compression, SW parsing, Striding RQ, Tunnel offloads, Dynamic BF regs, Clock info update, Flow action flags
+CQE compression caps:
+ max num : 64
+ supported formats : with hash, with RX checksum CSUM, with stride index
+SW parsing caps:
+ SW parsing offloads :
+ supported QP types :
+Striding RQ caps:
+ min single stride log num of bytes: 6
+ max single stride log num of bytes: 13
+ min single wqe log num of strides: 9
+ max single wqe log num of strides: 16
+ supported QP types : Raw Packet
+Tunnel offloads caps:
+Max dynamic BF registers: 1024
+Max clock info update [nsec]: 1099511
+Flow action flags : 0
+```
+
+##### Create an mlx5 QP
+Using an Mlx5Context object, one can create either a legacy QP (creation
+process is the same) or an mlx5 QP. An mlx5 QP is a QP by inheritance but its
+constructor receives a keyword argument named `dv_init_attr`. If the user
+provides it, the QP will be created using `mlx5dv_create_qp` rather than
+`ibv_create_qp_ex`. The following snippet demonstrates how to create both a DC
+(dynamically connected) QP and a Raw Packet QP which uses mlx5-specific
+capabilities, unavailable using the legacy interface. Currently, pyverbs
+supports only creation of a DCI. DCT support will be added in one of the
+following PRs.
+```python
+from pyverbs.providers.mlx5.mlx5dv import Mlx5Context, Mlx5DVContextAttr
+from pyverbs.providers.mlx5.mlx5dv import Mlx5DVQPInitAttr, Mlx5QP
+import pyverbs.providers.mlx5.mlx5_enums as me
+from pyverbs.qp import QPInitAttrEx, QPCap
+import pyverbs.enums as e
+from pyverbs.cq import CQ
+from pyverbs.pd import PD
+
+with Mlx5Context(name='rocep0s8f0', attr=Mlx5DVContextAttr()) as ctx:
+ with PD(ctx) as pd:
+ with CQ(ctx, 100) as cq:
+ cap = QPCap(100, 0, 1, 0)
+ # Create a DC QP of type DCI
+ qia = QPInitAttrEx(cap=cap, pd=pd, scq=cq, qp_type=e.IBV_QPT_DRIVER,
+ comp_mask=e.IBV_QP_INIT_ATTR_PD, rcq=cq)
+ attr = Mlx5DVQPInitAttr(comp_mask=me.MLX5DV_QP_INIT_ATTR_MASK_DC)
+ attr.dc_type = me.MLX5DV_DCTYPE_DCI
+
+ dci = Mlx5QP(ctx, qia, dv_init_attr=attr)
+
+ # Create a Raw Packet QP using mlx5-specific capabilities
+ qia.qp_type = e.IBV_QPT_RAW_PACKET
+ attr.comp_mask = me.MLX5DV_QP_INIT_ATTR_MASK_QP_CREATE_FLAGS
+ attr.create_flags = me.MLX5DV_QP_CREATE_ALLOW_SCATTER_TO_CQE |\
+ me.MLX5DV_QP_CREATE_TIR_ALLOW_SELF_LOOPBACK_UC |\
+ me.MLX5DV_QP_CREATE_TUNNEL_OFFLOADS
+ qp = Mlx5QP(ctx, qia, dv_init_attr=attr)
+```
+
+##### Create an mlx5 CQ
+Mlx5Context also allows users to create an mlx5 specific CQ. The Mlx5CQ inherits
+from CQEX, but its constructor receives 3 parameters instead of 2. The 3rd
+parameter is a keyword argument named `dv_init_attr`. If provided by the user,
+the CQ will be created using `mlx5dv_create_cq`.
+The following snippet shows this simple creation process.
+```python
+from pyverbs.providers.mlx5.mlx5dv import Mlx5Context, Mlx5DVContextAttr
+from pyverbs.providers.mlx5.mlx5dv import Mlx5DVCQInitAttr, Mlx5CQ
+import pyverbs.providers.mlx5.mlx5_enums as me
+from pyverbs.cq import CqInitAttrEx
+
+with Mlx5Context(name='rocep0s8f0', attr=Mlx5DVContextAttr()) as ctx:
+ cqia = CqInitAttrEx()
+ mlx5_cqia = Mlx5DVCQInitAttr(comp_mask=me.MLX5DV_CQ_INIT_ATTR_MASK_COMPRESSED_CQE,
+ cqe_comp_res_format=me.MLX5DV_CQE_RES_FORMAT_CSUM)
+ cq = Mlx5CQ(ctx, cqia, dv_init_attr=mlx5_cqia)
```
##### CMID
--
2.21.0
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH rdma-core 7/7] tests: Add traffic tests using extended CQ
2019-11-17 13:30 [PATCH rdma-core 0/7] pyverbs/mlx5: Support mlx5 CQ and QP Noa Osherovich
` (5 preceding siblings ...)
2019-11-17 13:30 ` [PATCH rdma-core 6/7] Documentation: Add mlx5 provider to documentation Noa Osherovich
@ 2019-11-17 13:30 ` Noa Osherovich
6 siblings, 0 replies; 8+ messages in thread
From: Noa Osherovich @ 2019-11-17 13:30 UTC (permalink / raw)
To: dledford, Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, Noa Osherovich
Add UD/RC/XRC traffic tests which use an extended CQ instead of the
legacy one.
Signed-off-by: Noa Osherovich <noaos@mellanox.com>
---
pyverbs/libibverbs_enums.pxd | 3 ++
tests/CMakeLists.txt | 1 +
tests/test_cqex.py | 75 ++++++++++++++++++++++++++++++++++++
tests/utils.py | 56 ++++++++++++++++++++++-----
4 files changed, 125 insertions(+), 10 deletions(-)
create mode 100644 tests/test_cqex.py
diff --git a/pyverbs/libibverbs_enums.pxd b/pyverbs/libibverbs_enums.pxd
index 114915d0a751..4bf77d3f14f7 100755
--- a/pyverbs/libibverbs_enums.pxd
+++ b/pyverbs/libibverbs_enums.pxd
@@ -398,6 +398,9 @@ cdef extern from '<infiniband/verbs.h>':
IBV_XRCD_INIT_ATTR_OFLAGS
IBV_XRCD_INIT_ATTR_RESERVED
+ cpdef enum:
+ IBV_WC_STANDARD_FLAGS
+
cdef unsigned long long IBV_DEVICE_RAW_SCATTER_FCS
cdef unsigned long long IBV_DEVICE_PCI_WRITE_END_PADDING
diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt
index 0d81d1a98fb7..7c6be35dc58a 100755
--- a/tests/CMakeLists.txt
+++ b/tests/CMakeLists.txt
@@ -7,6 +7,7 @@ rdma_python_test(tests
test_addr.py
base.py
test_cq.py
+ test_cqex.py
test_device.py
test_mr.py
test_pd.py
diff --git a/tests/test_cqex.py b/tests/test_cqex.py
new file mode 100644
index 000000000000..df135a991c15
--- /dev/null
+++ b/tests/test_cqex.py
@@ -0,0 +1,75 @@
+from pyverbs.cq import CqInitAttrEx, CQEX
+import pyverbs.enums as e
+from pyverbs.mr import MR
+
+from tests.base import RCResources, UDResources, XRCResources, RDMATestCase
+import tests.utils as u
+
+
+def create_ex_cq(res):
+ """
+ Create an Extended CQ using res's context and assign it to res's cq member.
+ IBV_WC_STANDARD_FLAGS is used for WC flags to avoid support differences
+ between devices.
+ :param res: An instance of TrafficResources
+ """
+ wc_flags = e.IBV_WC_STANDARD_FLAGS
+ cia = CqInitAttrEx(cqe=2000, wc_flags=wc_flags)
+ res.cq = CQEX(res.ctx, cia)
+
+
+class CqExUD(UDResources):
+ def create_cq(self):
+ create_ex_cq(self)
+
+ def create_mr(self):
+ self.mr = MR(self.pd, self.msg_size + self.GRH_SIZE,
+ e.IBV_ACCESS_LOCAL_WRITE)
+
+
+class CqExRC(RCResources):
+ def create_cq(self):
+ create_ex_cq(self)
+
+
+class CqExXRC(XRCResources):
+ def create_cq(self):
+ create_ex_cq(self)
+
+
+class CqExTestCase(RDMATestCase):
+ """
+ Run traffic over the existing UD, RC and XRC infrastructure, but use
+ ibv_cq_ex instead of legacy ibv_cq
+ """
+ def setUp(self):
+ super().setUp()
+ self.iters = 100
+ self.qp_dict = {'ud': CqExUD, 'rc': CqExRC, 'xrc': CqExXRC}
+
+ def create_players(self, qp_type):
+ client = self.qp_dict[qp_type](self.dev_name, self.ib_port,
+ self.gid_index)
+ server = self.qp_dict[qp_type](self.dev_name, self.ib_port,
+ self.gid_index)
+ if qp_type == 'xrc':
+ client.pre_run(server.psns, server.qps_num)
+ server.pre_run(client.psns, client.qps_num)
+ else:
+ client.pre_run(server.psn, server.qpn)
+ server.pre_run(client.psn, client.qpn)
+ return client, server
+
+ def test_ud_traffic_cq_ex(self):
+ client, server = self.create_players('ud')
+ u.traffic(client, server, self.iters, self.gid_index, self.ib_port,
+ is_cq_ex=True)
+
+ def test_rc_traffic_cq_ex(self):
+ client, server = self.create_players('rc')
+ u.traffic(client, server, self.iters, self.gid_index, self.ib_port,
+ is_cq_ex=True)
+
+ def test_xrc_traffic_cq_ex(self):
+ client, server = self.create_players('xrc')
+ u.xrc_traffic(client, server, is_cq_ex=True)
diff --git a/tests/utils.py b/tests/utils.py
index 785309552e25..04a988a531e4 100755
--- a/tests/utils.py
+++ b/tests/utils.py
@@ -13,6 +13,7 @@ from pyverbs.addr import AHAttr, AH, GlobalRoute
from pyverbs.wr import SGE, SendWR, RecvWR
from pyverbs.qp import QPCap, QPInitAttrEx
from tests.base import XRCResources
+from pyverbs.cq import PollCqAttr
import pyverbs.device as d
import pyverbs.enums as e
@@ -342,6 +343,37 @@ def poll_cq(cq, count=1):
return wcs
+def poll_cq_ex(cqex, count=1):
+ """
+ Poll <count> completions from the extended CQ.
+ :param cq: CQEX to poll from
+ :param count: How many completions to poll
+ :return: None
+ """
+ poll_attr = PollCqAttr()
+ ret = cqex.start_poll(poll_attr)
+ while ret == 2: # ENOENT
+ ret = cqex.start_poll(poll_attr)
+ if ret != 0:
+ raise PyverbsRDMAErrno('Failed to poll CQ')
+ count -= 1
+ if cqex.status != e.IBV_WC_SUCCESS:
+ raise PyverbsRDMAErrno('Completion status is {s}'.
+ format(s=cqex.status))
+ # Now poll the rest of the packets
+ while count > 0:
+ ret = cqex.poll_next()
+ while ret == 2:
+ ret = cqex.poll_next()
+ if ret != 0:
+ raise PyverbsRDMAErrno('Failed to poll CQ')
+ if cqex.status != e.IBV_WC_SUCCESS:
+ raise PyverbsRDMAErrno('Completion status is {s}'.
+ format(s=cqex.status))
+ count -= 1
+ cqex.end_poll()
+
+
def validate(received_str, is_server, msg_size):
"""
Validates the received buffer against the expected result.
@@ -366,7 +398,7 @@ def validate(received_str, is_server, msg_size):
format(exp=expected_str, rcv=received_str))
-def traffic(client, server, iters, gid_idx, port):
+def traffic(client, server, iters, gid_idx, port, is_cq_ex=False):
"""
Runs basic traffic between two sides
:param client: client side, clients base class is BaseTraffic
@@ -374,8 +406,10 @@ def traffic(client, server, iters, gid_idx, port):
:param iters: number of traffic iterations
:param gid_idx: local gid index
:param port: IB port
+ :param is_cq_ex: If True, use poll_cq_ex() rather than poll_cq()
:return:
"""
+ poll = poll_cq_ex if is_cq_ex else poll_cq
s_recv_wr = get_recv_wr(server)
c_recv_wr = get_recv_wr(client)
post_recv(client.qp, c_recv_wr, client.num_msgs)
@@ -383,21 +417,21 @@ def traffic(client, server, iters, gid_idx, port):
for _ in range(iters):
c_send_wr = get_send_wr(client, False)
post_send(client, c_send_wr, gid_idx, port)
- poll_cq(client.cq)
- poll_cq(server.cq)
+ poll(client.cq)
+ poll(server.cq)
post_recv(client.qp, c_recv_wr)
msg_received = server.mr.read(server.msg_size, 0)
validate(msg_received, True, server.msg_size)
s_send_wr = get_send_wr(server, True)
post_send(server, s_send_wr, gid_idx, port)
- poll_cq(server.cq)
- poll_cq(client.cq)
+ poll(server.cq)
+ poll(client.cq)
post_recv(server.qp, s_recv_wr)
msg_received = client.mr.read(client.msg_size, 0)
validate(msg_received, False, client.msg_size)
-def xrc_traffic(client, server):
+def xrc_traffic(client, server, is_cq_ex=False):
"""
Runs basic xrc traffic, this function assumes that number of QPs, which
server and client have are equal, server.send_qp[i] is connected to
@@ -408,8 +442,10 @@ def xrc_traffic(client, server):
of XRCResources class
:param server: Aggregation object of the passive side, should be an instance
of XRCResources class
+ :param is_cq_ex: If True, use poll_cq_ex() rather than poll_cq()
:return: None
"""
+ poll = poll_cq_ex if is_cq_ex else poll_cq
client_srqn = client.srq.get_srq_num()
server_srqn = server.srq.get_srq_num()
s_recv_wr = get_recv_wr(server)
@@ -421,15 +457,15 @@ def xrc_traffic(client, server):
c_send_wr = get_send_wr(client, False)
c_send_wr.set_qp_type_xrc(server_srqn)
client.sqp_lst[i].post_send(c_send_wr)
- poll_cq(client.cq)
- poll_cq(server.cq)
+ poll(client.cq)
+ poll(server.cq)
msg_received = server.mr.read(server.msg_size, 0)
validate(msg_received, True, server.msg_size)
s_send_wr = get_send_wr(server, True)
s_send_wr.set_qp_type_xrc(client_srqn)
server.sqp_lst[i].post_send(s_send_wr)
- poll_cq(server.cq)
- poll_cq(client.cq)
+ poll(server.cq)
+ poll(client.cq)
msg_received = client.mr.read(client.msg_size, 0)
validate(msg_received, False, client.msg_size)
--
2.21.0
^ permalink raw reply related [flat|nested] 8+ messages in thread
end of thread, other threads:[~2019-11-17 13:30 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-11-17 13:30 [PATCH rdma-core 0/7] pyverbs/mlx5: Support mlx5 CQ and QP Noa Osherovich
2019-11-17 13:30 ` [PATCH rdma-core 1/7] pyverbs: Allow QP creation by provider Noa Osherovich
2019-11-17 13:30 ` [PATCH rdma-core 2/7] pyverbs/mlx5: Add support for mlx5 QP Noa Osherovich
2019-11-17 13:30 ` [PATCH rdma-core 4/7] pyverbs: Add support for provider extended CQ Noa Osherovich
2019-11-17 13:30 ` [PATCH rdma-core 3/7] pyverbs: Add default values for CQ creation Noa Osherovich
2019-11-17 13:30 ` [PATCH rdma-core 5/7] pyverbs/mlx5: Add support for mlx5 CQ Noa Osherovich
2019-11-17 13:30 ` [PATCH rdma-core 6/7] Documentation: Add mlx5 provider to documentation Noa Osherovich
2019-11-17 13:30 ` [PATCH rdma-core 7/7] tests: Add traffic tests using extended CQ Noa Osherovich
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).