[PATCH V1 rdma-core 0/7] Add mlx5 direct verbs

[PATCH V1 rdma-core 0/7] Add mlx5 direct verbs

[Yishai Hadas]

This patchset from Leon adds direct access to mlx5 devices.
Sending V1 to mainly handle Jason's comments, details below.

The libibverbs API is an abstract API. It is agnostic to any underlying
provider specific implementation. While this abstraction has the advantage of
user applications portability it has a performance penalty. For some
applications optimizing performance is more important than portability.

The mlx5 direct verbs API introduced in this patchset is intended for such
applications. It exposes mlx5 specific low level data path
(send/receive/completion) operations, allowing the application to bypass the
libibverbs data path API.

The proposed interface consists from one hardware specific header with relevant
inline functions and conversion logic from ibverbs structures to mlx5 related
structures.

New header is going to be installed in below path location:
/usr/inlcude/infiniband/mlx5dv.h

The direct include of mlx5dv.h together with linkage to mlx5 library will
allow usage of this new interface.

Note: all Mellanox NIC devices starting from Connect-IB (Connect-IB,
ConnectX-4, ConnectX-4Lx, ConnectX-5 and more to come in the future) implement
the mlx5 API, thus using the mlx5 direct verbs does not limit the applications
to a single NIC HW device thus keeping some level of portability.

Pull request was sent:
https://github.com/linux-rdma/rdma-core/pull/63

Yishai

Changes from V0:
     * Removed mlx5 and include sub-directories from installed
       path and source code.
     * Put all DV code into one header file.
     * Renamed mlx5/dv.h to be mlx5dv.h file.
     * Added providers/mlx5/libmlx5.map file with exported functions.
     * Allowed mlx5 provider to be shared library.
     * Added manual pages for direct verbs.
     * Rebased to latest rdma-core.
     * Create symlink for libmlx5-rdmav2.so to point to libmlx5.so.
     * Dropped patch which changed INLINE_SCATTER defines.
     * Replaced ntohll functions to be64toh, followup of
       https://github.com/linux-rdma/rdma-core/commit/b21ff60e0e111e295a18cc8e3397347893c524bc
     * Added Jason's SOB tag for his debian/redhat/Cmake work.

Leon Romanovsky (7):
  mlx5: Add mlx5 direct verbs
  mlx5: Add CQE fields
  mlx5: Add WQE segments implementation
  mlx5: Use macro for CQE version 1
  mlx5: Convert to use predefined get CQE opcodes
  mlx5: Add direct verbs man pages
  mlx5: Export mlx5 direct verbs interface

 CMakeLists.txt                           |   1 +
 buildlib/rdma_functions.cmake            |  51 +++
 buildlib/relpath                         |   7 +
 debian/ibverbs-providers.install         |   4 +
 debian/ibverbs-providers.symbols         |   4 +
 providers/mlx5/CMakeLists.txt            |   7 +-
 providers/mlx5/cq.c                      |  96 +----
 providers/mlx5/libmlx5.map               |   8 +
 providers/mlx5/man/CMakeLists.txt        |   5 +
 providers/mlx5/man/mlx5dv.7              |  44 +++
 providers/mlx5/man/mlx5dv_init_obj.3     | 129 +++++++
 providers/mlx5/man/mlx5dv_query_device.3 |  48 +++
 providers/mlx5/mlx5.c                    | 111 +++++-
 providers/mlx5/mlx5.h                    |  62 +---
 providers/mlx5/mlx5dv.h                  | 588 +++++++++++++++++++++++++++++++
 providers/mlx5/qp.c                      |   2 +-
 providers/mlx5/wqe.h                     |  97 +----
 redhat/rdma-core.spec                    |   3 +
 18 files changed, 1024 insertions(+), 243 deletions(-)
 create mode 100644 buildlib/relpath
 create mode 100644 debian/ibverbs-providers.symbols
 create mode 100644 providers/mlx5/libmlx5.map
 create mode 100644 providers/mlx5/man/CMakeLists.txt
 create mode 100644 providers/mlx5/man/mlx5dv.7
 create mode 100644 providers/mlx5/man/mlx5dv_init_obj.3
 create mode 100644 providers/mlx5/man/mlx5dv_query_device.3
 create mode 100644 providers/mlx5/mlx5dv.h

-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V1 rdma-core 1/7] mlx5: Add mlx5 direct verbs

[Yishai Hadas]

From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

Direct verbs feature provides fast data path execution by
implementing verbs directly over hardware specification.

It consists from the two parts:
 * Exported functions to translate internal mlx5 structures to be
   accessible by user applications.
 * Inline functions, enums and defines to simplify programmer's life.

This patch adds one header file (mlx5dv.h) with general, CQE and WQE data.

Signed-off-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 providers/mlx5/cq.c     |  84 +----------
 providers/mlx5/mlx5.c   | 109 ++++++++++++++
 providers/mlx5/mlx5.h   |  62 +-------
 providers/mlx5/mlx5dv.h | 371 ++++++++++++++++++++++++++++++++++++++++++++++++
 providers/mlx5/qp.c     |   2 +-
 providers/mlx5/wqe.h    |  97 +------------
 6 files changed, 490 insertions(+), 235 deletions(-)
 create mode 100644 providers/mlx5/mlx5dv.h

diff --git a/providers/mlx5/cq.c b/providers/mlx5/cq.c
index b53fa66..88bca75 100644
--- a/providers/mlx5/cq.c
+++ b/providers/mlx5/cq.c
@@ -48,97 +48,17 @@
 #include "doorbell.h"
 
 enum {
-	MLX5_CQ_DOORBELL			= 0x20
-};
-
-enum {
 	CQ_OK					=  0,
 	CQ_EMPTY				= -1,
 	CQ_POLL_ERR				= -2
 };
 
-#define MLX5_CQ_DB_REQ_NOT_SOL			(1 << 24)
-#define MLX5_CQ_DB_REQ_NOT			(0 << 24)
-
-enum {
-	MLX5_CQE_SYNDROME_LOCAL_LENGTH_ERR		= 0x01,
-	MLX5_CQE_SYNDROME_LOCAL_QP_OP_ERR		= 0x02,
-	MLX5_CQE_SYNDROME_LOCAL_PROT_ERR		= 0x04,
-	MLX5_CQE_SYNDROME_WR_FLUSH_ERR			= 0x05,
-	MLX5_CQE_SYNDROME_MW_BIND_ERR			= 0x06,
-	MLX5_CQE_SYNDROME_BAD_RESP_ERR			= 0x10,
-	MLX5_CQE_SYNDROME_LOCAL_ACCESS_ERR		= 0x11,
-	MLX5_CQE_SYNDROME_REMOTE_INVAL_REQ_ERR		= 0x12,
-	MLX5_CQE_SYNDROME_REMOTE_ACCESS_ERR		= 0x13,
-	MLX5_CQE_SYNDROME_REMOTE_OP_ERR			= 0x14,
-	MLX5_CQE_SYNDROME_TRANSPORT_RETRY_EXC_ERR	= 0x15,
-	MLX5_CQE_SYNDROME_RNR_RETRY_EXC_ERR		= 0x16,
-	MLX5_CQE_SYNDROME_REMOTE_ABORTED_ERR		= 0x22,
-};
-
-enum {
-	MLX5_CQE_OWNER_MASK	= 1,
-	MLX5_CQE_REQ		= 0,
-	MLX5_CQE_RESP_WR_IMM	= 1,
-	MLX5_CQE_RESP_SEND	= 2,
-	MLX5_CQE_RESP_SEND_IMM	= 3,
-	MLX5_CQE_RESP_SEND_INV	= 4,
-	MLX5_CQE_RESIZE_CQ	= 5,
-	MLX5_CQE_REQ_ERR	= 13,
-	MLX5_CQE_RESP_ERR	= 14,
-	MLX5_CQE_INVALID	= 15,
-};
-
 enum {
 	MLX5_CQ_MODIFY_RESEIZE = 0,
 	MLX5_CQ_MODIFY_MODER = 1,
 	MLX5_CQ_MODIFY_MAPPING = 2,
 };
 
-enum {
-	MLX5_CQE_L2_OK = 1 << 0,
-	MLX5_CQE_L3_OK = 1 << 1,
-	MLX5_CQE_L4_OK = 1 << 2,
-};
-
-enum {
-	MLX5_CQE_L3_HDR_TYPE_NONE = 0x0,
-	MLX5_CQE_L3_HDR_TYPE_IPV6 = 0x1,
-	MLX5_CQE_L3_HDR_TYPE_IPV4 = 0x2,
-};
-
-struct mlx5_err_cqe {
-	uint8_t		rsvd0[32];
-	uint32_t	srqn;
-	uint8_t		rsvd1[18];
-	uint8_t		vendor_err_synd;
-	uint8_t		syndrome;
-	uint32_t	s_wqe_opcode_qpn;
-	uint16_t	wqe_counter;
-	uint8_t		signature;
-	uint8_t		op_own;
-};
-
-struct mlx5_cqe64 {
-	uint8_t		rsvd0[17];
-	uint8_t		ml_path;
-	uint8_t		rsvd20[4];
-	uint16_t	slid;
-	uint32_t	flags_rqpn;
-	uint8_t		hds_ip_ext;
-	uint8_t		l4_hdr_type_etc;
-	uint16_t	vlan_info;
-	uint32_t	srqn_uidx;
-	uint32_t	imm_inval_pkey;
-	uint8_t		rsvd40[4];
-	uint32_t	byte_cnt;
-	__be64		timestamp;
-	uint32_t	sop_drop_qpn;
-	uint16_t	wqe_counter;
-	uint8_t		signature;
-	uint8_t		op_own;
-};
-
 int mlx5_stall_num_loop = 60;
 int mlx5_stall_cq_poll_min = 60;
 int mlx5_stall_cq_poll_max = 100000;
@@ -258,7 +178,7 @@ static inline int handle_responder_lazy(struct mlx5_cq *cq, struct mlx5_cqe64 *c
 		else if (cqe->op_own & MLX5_INLINE_SCATTER_64)
 			err = mlx5_copy_to_recv_wqe(qp, wqe_ctr, cqe - 1,
 						    ntohl(cqe->byte_cnt));
-	}
+}
 
 	return err;
 }
@@ -1435,7 +1355,7 @@ void __mlx5_cq_clean(struct mlx5_cq *cq, uint32_t rsn, struct mlx5_srq *srq)
 	uint8_t owner_bit;
 	int cqe_version;
 
-	if (!cq)
+	if (!cq || cq->flags & MLX5_CQ_FLAGS_DV_OWNED)
 		return;
 
 	/*
diff --git a/providers/mlx5/mlx5.c b/providers/mlx5/mlx5.c
index 747e242..1cf5a5f 100644
--- a/providers/mlx5/mlx5.c
+++ b/providers/mlx5/mlx5.c
@@ -578,6 +578,115 @@ static int mlx5_map_internal_clock(struct mlx5_device *mdev,
 	return 0;
 }
 
+int mlx5dv_query_device(struct ibv_context *ctx_in,
+			 struct mlx5dv_context *attrs_out)
+{
+	attrs_out->comp_mask = 0;
+	attrs_out->version   = 0;
+	attrs_out->flags     = 0;
+
+	if (to_mctx(ctx_in)->cqe_version == MLX5_CQE_VERSION_V1)
+		attrs_out->flags |= MLX5DV_CONTEXT_FLAGS_CQE_V1;
+
+	return 0;
+}
+
+static int mlx5dv_get_qp(struct ibv_qp *qp_in,
+			 struct mlx5dv_qp *qp_out)
+{
+	struct mlx5_qp *mqp = to_mqp(qp_in);
+
+	qp_out->comp_mask = 0;
+	qp_out->dbrec     = mqp->db;
+
+	if (mqp->sq_buf_size)
+		/* IBV_QPT_RAW_PACKET */
+		qp_out->sq.buf = (void *)((uintptr_t)mqp->sq_buf.buf);
+	else
+		qp_out->sq.buf = (void *)((uintptr_t)mqp->buf.buf + mqp->sq.offset);
+	qp_out->sq.wqe_cnt = mqp->sq.wqe_cnt;
+	qp_out->sq.stride  = 1 << mqp->sq.wqe_shift;
+
+	qp_out->rq.buf     = (void *)((uintptr_t)mqp->buf.buf + mqp->rq.offset);
+	qp_out->rq.wqe_cnt = mqp->rq.wqe_cnt;
+	qp_out->rq.stride  = 1 << mqp->rq.wqe_shift;
+
+	qp_out->bf.reg    = mqp->bf->reg;
+
+	if (mqp->bf->uuarn > 0)
+		qp_out->bf.size = mqp->bf->buf_size;
+	else
+		qp_out->bf.size = 0;
+
+	return 0;
+}
+
+static int mlx5dv_get_cq(struct ibv_cq *cq_in,
+			 struct mlx5dv_cq *cq_out)
+{
+	struct mlx5_cq *mcq = to_mcq(cq_in);
+	struct mlx5_context *mctx = to_mctx(cq_in->context);
+
+	cq_out->comp_mask = 0;
+	cq_out->cqn       = mcq->cqn;
+	cq_out->cqe_cnt   = mcq->ibv_cq.cqe + 1;
+	cq_out->cqe_size  = mcq->cqe_sz;
+	cq_out->buf       = mcq->active_buf->buf;
+	cq_out->dbrec     = mcq->dbrec;
+	cq_out->uar	  = mctx->uar;
+
+	mcq->flags	 |= MLX5_CQ_FLAGS_DV_OWNED;
+
+	return 0;
+}
+
+static int mlx5dv_get_rwq(struct ibv_wq *wq_in,
+			  struct mlx5dv_rwq *rwq_out)
+{
+	struct mlx5_rwq *mrwq = to_mrwq(wq_in);
+
+	rwq_out->comp_mask = 0;
+	rwq_out->buf       = mrwq->pbuff;
+	rwq_out->dbrec     = mrwq->recv_db;
+	rwq_out->wqe_cnt   = mrwq->rq.wqe_cnt;
+	rwq_out->stride    = 1 << mrwq->rq.wqe_shift;
+
+	return 0;
+}
+
+static int mlx5dv_get_srq(struct ibv_srq *srq_in,
+			  struct mlx5dv_srq *srq_out)
+{
+	struct mlx5_srq *msrq;
+
+	msrq = container_of(srq_in, struct mlx5_srq, vsrq.srq);
+
+	srq_out->comp_mask = 0;
+	srq_out->buf       = msrq->buf.buf;
+	srq_out->dbrec     = msrq->db;
+	srq_out->stride    = 1 << msrq->wqe_shift;
+	srq_out->head      = msrq->head;
+	srq_out->tail      = msrq->tail;
+
+	return 0;
+}
+
+int mlx5dv_init_obj(struct mlx5dv_obj *obj, uint64_t obj_type)
+{
+	int ret = 0;
+
+	if (obj_type & MLX5DV_OBJ_QP)
+		ret = mlx5dv_get_qp(obj->qp.in, obj->qp.out);
+	if (!ret && (obj_type & MLX5DV_OBJ_CQ))
+		ret = mlx5dv_get_cq(obj->cq.in, obj->cq.out);
+	if (!ret && (obj_type & MLX5DV_OBJ_SRQ))
+		ret = mlx5dv_get_srq(obj->srq.in, obj->srq.out);
+	if (!ret && (obj_type & MLX5DV_OBJ_RWQ))
+		ret = mlx5dv_get_rwq(obj->rwq.in, obj->rwq.out);
+
+	return ret;
+}
+
 static int mlx5_init_context(struct verbs_device *vdev,
 			     struct ibv_context *ctx, int cmd_fd)
 {
diff --git a/providers/mlx5/mlx5.h b/providers/mlx5/mlx5.h
index 3f89f4b..f75bbf1 100644
--- a/providers/mlx5/mlx5.h
+++ b/providers/mlx5/mlx5.h
@@ -43,6 +43,7 @@
 #include <ccan/list.h>
 #include "bitmap.h"
 #include <ccan/minmax.h>
+#include "mlx5dv.h"
 
 #include <valgrind/memcheck.h>
 
@@ -100,11 +101,6 @@ static inline void mlx5_dbg(FILE *fp, uint32_t mask, const char *fmt, ...)
 #endif
 
 enum {
-	MLX5_RCV_DBR	= 0,
-	MLX5_SND_DBR	= 1,
-};
-
-enum {
 	MLX5_STAT_RATE_OFFSET		= 5
 };
 
@@ -127,36 +123,10 @@ enum {
 };
 
 enum {
-	MLX5_SEND_WQE_BB	= 64,
-	MLX5_SEND_WQE_SHIFT	= 6,
-};
-
-enum {
 	MLX5_BF_OFFSET	= 0x800
 };
 
 enum {
-	MLX5_INLINE_SCATTER_32	= 0x4,
-	MLX5_INLINE_SCATTER_64	= 0x8,
-};
-
-enum {
-	MLX5_OPCODE_NOP			= 0x00,
-	MLX5_OPCODE_SEND_INVAL		= 0x01,
-	MLX5_OPCODE_RDMA_WRITE		= 0x08,
-	MLX5_OPCODE_RDMA_WRITE_IMM	= 0x09,
-	MLX5_OPCODE_SEND		= 0x0a,
-	MLX5_OPCODE_SEND_IMM		= 0x0b,
-	MLX5_OPCODE_TSO			= 0x0e,
-	MLX5_OPCODE_RDMA_READ		= 0x10,
-	MLX5_OPCODE_ATOMIC_CS		= 0x11,
-	MLX5_OPCODE_ATOMIC_FA		= 0x12,
-	MLX5_OPCODE_ATOMIC_MASKED_CS	= 0x14,
-	MLX5_OPCODE_ATOMIC_MASKED_FA	= 0x15,
-	MLX5_OPCODE_FMR			= 0x19,
-	MLX5_OPCODE_LOCAL_INVAL		= 0x1b,
-	MLX5_OPCODE_CONFIG_CMD		= 0x1f,
-
 	MLX5_RECV_OPCODE_RDMA_WRITE_IMM	= 0x00,
 	MLX5_RECV_OPCODE_SEND		= 0x01,
 	MLX5_RECV_OPCODE_SEND_IMM	= 0x02,
@@ -164,7 +134,6 @@ enum {
 
 	MLX5_CQE_OPCODE_ERROR		= 0x1e,
 	MLX5_CQE_OPCODE_RESIZE		= 0x16,
-	MLX5_OPCODE_UMR			= 0x25,
 };
 
 enum {
@@ -172,10 +141,6 @@ enum {
 };
 
 enum {
-	MLX5_INLINE_SEG	= 0x80000000,
-};
-
-enum {
 	MLX5_MAX_PORTS_NUM = 2,
 };
 
@@ -239,7 +204,7 @@ struct mlx5_context {
 	int				prefer_bf;
 	int				shut_up_bf;
 	struct {
-		struct mlx5_qp	      **table;
+		struct mlx5_qp        **table;
 		int			refcnt;
 	}				qp_table[MLX5_QP_TABLE_SIZE];
 	pthread_mutex_t			qp_table_mutex;
@@ -328,6 +293,7 @@ enum {
 	MLX5_CQ_FLAGS_FOUND_CQES = 1 << 2,
 	MLX5_CQ_FLAGS_EXTENDED = 1 << 3,
 	MLX5_CQ_FLAGS_SINGLE_THREADED = 1 << 4,
+	MLX5_CQ_FLAGS_DV_OWNED = 1 << 5,
 };
 
 struct mlx5_cq {
@@ -438,29 +404,9 @@ struct mlx5_qp {
 	int                             rss_qp;
 };
 
-struct mlx5_av {
-	union {
-		struct {
-			uint32_t	qkey;
-			uint32_t	reserved;
-		} qkey;
-		uint64_t	dc_key;
-	} key;
-	uint32_t	dqp_dct;
-	uint8_t		stat_rate_sl;
-	uint8_t		fl_mlid;
-	uint16_t	rlid;
-	uint8_t		reserved0[4];
-	uint8_t		rmac[6];
-	uint8_t		tclass;
-	uint8_t		hop_limit;
-	uint32_t	grh_gid_fl;
-	uint8_t		rgid[16];
-};
-
 struct mlx5_ah {
 	struct ibv_ah			ibv_ah;
-	struct mlx5_av			av;
+	struct mlx5_wqe_av		av;
 	bool				kern_ah;
 };
 
diff --git a/providers/mlx5/mlx5dv.h b/providers/mlx5/mlx5dv.h
new file mode 100644
index 0000000..34502a8
--- /dev/null
+++ b/providers/mlx5/mlx5dv.h
@@ -0,0 +1,371 @@
+/*
+ * Copyright (c) 2017 Mellanox Technologies, Inc.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#ifndef _MLX5DV_H_
+#define _MLX5DV_H_
+
+/* For __be64 type */
+#include <linux/types.h>
+#include <arpa/inet.h>
+
+enum {
+	MLX5_RCV_DBR	= 0,
+	MLX5_SND_DBR	= 1,
+};
+
+/*
+ * Direct verbs device-specific attributes
+ */
+struct mlx5dv_context {
+	uint8_t		version;
+	uint64_t	flags;
+	uint64_t	comp_mask;
+};
+
+enum mlx5dv_context_flags {
+	/*
+	 * This flag indicates if CQE version 0 or 1 is needed.
+	 */
+	MLX5DV_CONTEXT_FLAGS_CQE_V1 = (1 << 0),
+};
+
+/*
+ * Most device capabilities are exported by ibv_query_device(...),
+ * but there is HW device-specific information which is important
+ * for data-path, but isn't provided.
+ *
+ * Return 0 on success.
+ */
+int mlx5dv_query_device(struct ibv_context *ctx_in,
+			struct mlx5dv_context *attrs_out);
+
+struct mlx5dv_qp {
+	uint32_t		*dbrec;
+	struct {
+		void		*buf;
+		uint32_t	wqe_cnt;
+		uint32_t	stride;
+	} sq;
+	struct {
+		void		*buf;
+		uint32_t	wqe_cnt;
+		uint32_t	stride;
+	} rq;
+	struct {
+		void		*reg;
+		uint32_t	size;
+	} bf;
+	uint64_t		comp_mask;
+};
+
+struct mlx5dv_cq {
+	void			*buf;
+	uint32_t		*dbrec;
+	uint32_t		cqe_cnt;
+	uint32_t		cqe_size;
+	void			*uar;
+	uint32_t		cqn;
+	uint64_t		comp_mask;
+};
+
+struct mlx5dv_srq {
+	void			*buf;
+	uint32_t		*dbrec;
+	uint32_t		stride;
+	uint32_t		head;
+	uint32_t		tail;
+	uint64_t		comp_mask;
+};
+
+struct mlx5dv_rwq {
+	void		*buf;
+	uint32_t	*dbrec;
+	uint32_t	wqe_cnt;
+	uint32_t	stride;
+	uint64_t	comp_mask;
+};
+
+struct mlx5dv_obj {
+	struct {
+		struct ibv_qp		*in;
+		struct mlx5dv_qp	*out;
+	} qp;
+	struct {
+		struct ibv_cq		*in;
+		struct mlx5dv_cq	*out;
+	} cq;
+	struct {
+		struct ibv_srq		*in;
+		struct mlx5dv_srq	*out;
+	} srq;
+	struct {
+		struct ibv_wq		*in;
+		struct mlx5dv_rwq	*out;
+	} rwq;
+};
+
+enum mlx5dv_obj_type {
+	MLX5DV_OBJ_QP	= 1 << 0,
+	MLX5DV_OBJ_CQ	= 1 << 1,
+	MLX5DV_OBJ_SRQ	= 1 << 2,
+	MLX5DV_OBJ_RWQ	= 1 << 3,
+};
+
+/*
+ * This function will initialize mlx5dv_xxx structs based on supplied type.
+ * The information for initialization is taken from ibv_xx structs supplied
+ * as part of input.
+ *
+ * Request information of CQ marks its owned by DV for all consumer index
+ * related actions.
+ *
+ * The initialization type can be combination of several types together.
+ *
+ * Return: 0 in case of success.
+ */
+int mlx5dv_init_obj(struct mlx5dv_obj *obj, uint64_t obj_type);
+
+enum {
+	MLX5_OPCODE_NOP			= 0x00,
+	MLX5_OPCODE_SEND_INVAL		= 0x01,
+	MLX5_OPCODE_RDMA_WRITE		= 0x08,
+	MLX5_OPCODE_RDMA_WRITE_IMM	= 0x09,
+	MLX5_OPCODE_SEND		= 0x0a,
+	MLX5_OPCODE_SEND_IMM		= 0x0b,
+	MLX5_OPCODE_TSO			= 0x0e,
+	MLX5_OPCODE_RDMA_READ		= 0x10,
+	MLX5_OPCODE_ATOMIC_CS		= 0x11,
+	MLX5_OPCODE_ATOMIC_FA		= 0x12,
+	MLX5_OPCODE_ATOMIC_MASKED_CS	= 0x14,
+	MLX5_OPCODE_ATOMIC_MASKED_FA	= 0x15,
+	MLX5_OPCODE_FMR			= 0x19,
+	MLX5_OPCODE_LOCAL_INVAL		= 0x1b,
+	MLX5_OPCODE_CONFIG_CMD		= 0x1f,
+	MLX5_OPCODE_UMR			= 0x25,
+};
+
+/*
+ * CQE related part
+ */
+
+enum {
+	MLX5_INLINE_SCATTER_32	= 0x4,
+	MLX5_INLINE_SCATTER_64	= 0x8,
+};
+
+enum {
+	MLX5_CQE_SYNDROME_LOCAL_LENGTH_ERR		= 0x01,
+	MLX5_CQE_SYNDROME_LOCAL_QP_OP_ERR		= 0x02,
+	MLX5_CQE_SYNDROME_LOCAL_PROT_ERR		= 0x04,
+	MLX5_CQE_SYNDROME_WR_FLUSH_ERR			= 0x05,
+	MLX5_CQE_SYNDROME_MW_BIND_ERR			= 0x06,
+	MLX5_CQE_SYNDROME_BAD_RESP_ERR			= 0x10,
+	MLX5_CQE_SYNDROME_LOCAL_ACCESS_ERR		= 0x11,
+	MLX5_CQE_SYNDROME_REMOTE_INVAL_REQ_ERR		= 0x12,
+	MLX5_CQE_SYNDROME_REMOTE_ACCESS_ERR		= 0x13,
+	MLX5_CQE_SYNDROME_REMOTE_OP_ERR			= 0x14,
+	MLX5_CQE_SYNDROME_TRANSPORT_RETRY_EXC_ERR	= 0x15,
+	MLX5_CQE_SYNDROME_RNR_RETRY_EXC_ERR		= 0x16,
+	MLX5_CQE_SYNDROME_REMOTE_ABORTED_ERR		= 0x22,
+};
+
+enum {
+	MLX5_CQE_L2_OK = 1 << 0,
+	MLX5_CQE_L3_OK = 1 << 1,
+	MLX5_CQE_L4_OK = 1 << 2,
+};
+
+enum {
+	MLX5_CQE_L3_HDR_TYPE_NONE = 0x0,
+	MLX5_CQE_L3_HDR_TYPE_IPV6 = 0x1,
+	MLX5_CQE_L3_HDR_TYPE_IPV4 = 0x2,
+};
+
+enum {
+	MLX5_CQE_OWNER_MASK	= 1,
+	MLX5_CQE_REQ		= 0,
+	MLX5_CQE_RESP_WR_IMM	= 1,
+	MLX5_CQE_RESP_SEND	= 2,
+	MLX5_CQE_RESP_SEND_IMM	= 3,
+	MLX5_CQE_RESP_SEND_INV	= 4,
+	MLX5_CQE_RESIZE_CQ	= 5,
+	MLX5_CQE_REQ_ERR	= 13,
+	MLX5_CQE_RESP_ERR	= 14,
+	MLX5_CQE_INVALID	= 15,
+};
+
+enum {
+	MLX5_CQ_DOORBELL			= 0x20
+};
+
+enum {
+	MLX5_CQ_DB_REQ_NOT_SOL	= 1 << 24,
+	MLX5_CQ_DB_REQ_NOT	= 0 << 24,
+};
+
+struct mlx5_err_cqe {
+	uint8_t		rsvd0[32];
+	uint32_t	srqn;
+	uint8_t		rsvd1[18];
+	uint8_t		vendor_err_synd;
+	uint8_t		syndrome;
+	uint32_t	s_wqe_opcode_qpn;
+	uint16_t	wqe_counter;
+	uint8_t		signature;
+	uint8_t		op_own;
+};
+
+struct mlx5_cqe64 {
+	uint8_t		rsvd0[17];
+	uint8_t		ml_path;
+	uint8_t		rsvd20[4];
+	uint16_t	slid;
+	uint32_t	flags_rqpn;
+	uint8_t		hds_ip_ext;
+	uint8_t		l4_hdr_type_etc;
+	uint16_t	vlan_info;
+	uint32_t	srqn_uidx;
+	uint32_t	imm_inval_pkey;
+	uint8_t		rsvd40[4];
+	uint32_t	byte_cnt;
+	__be64		timestamp;
+	uint32_t	sop_drop_qpn;
+	uint16_t	wqe_counter;
+	uint8_t		signature;
+	uint8_t		op_own;
+};
+
+/*
+ * WQE related part
+ */
+enum {
+	MLX5_INVALID_LKEY	= 0x100,
+};
+
+enum {
+	MLX5_EXTENDED_UD_AV	= 0x80000000,
+};
+
+enum {
+	MLX5_WQE_CTRL_CQ_UPDATE	= 2 << 2,
+	MLX5_WQE_CTRL_SOLICITED	= 1 << 1,
+	MLX5_WQE_CTRL_FENCE	= 4 << 5,
+	MLX5_WQE_CTRL_INITIATOR_SMALL_FENCE = 1 << 5,
+};
+
+enum {
+	MLX5_SEND_WQE_BB	= 64,
+	MLX5_SEND_WQE_SHIFT	= 6,
+};
+
+enum {
+	MLX5_INLINE_SEG	= 0x80000000,
+};
+
+enum {
+	MLX5_ETH_WQE_L3_CSUM = (1 << 6),
+	MLX5_ETH_WQE_L4_CSUM = (1 << 7),
+};
+
+struct mlx5_wqe_srq_next_seg {
+	uint8_t			rsvd0[2];
+	uint16_t		next_wqe_index;
+	uint8_t			signature;
+	uint8_t			rsvd1[11];
+};
+
+struct mlx5_wqe_data_seg {
+	uint32_t		byte_count;
+	uint32_t		lkey;
+	uint64_t		addr;
+};
+
+struct mlx5_wqe_ctrl_seg {
+	uint32_t	opmod_idx_opcode;
+	uint32_t	qpn_ds;
+	uint8_t		signature;
+	uint8_t		rsvd[2];
+	uint8_t		fm_ce_se;
+	uint32_t	imm;
+};
+
+struct mlx5_wqe_av {
+	union {
+		struct {
+			uint32_t	qkey;
+			uint32_t	reserved;
+		} qkey;
+		uint64_t	dc_key;
+	} key;
+	uint32_t	dqp_dct;
+	uint8_t		stat_rate_sl;
+	uint8_t		fl_mlid;
+	uint16_t	rlid;
+	uint8_t		reserved0[4];
+	uint8_t		rmac[6];
+	uint8_t		tclass;
+	uint8_t		hop_limit;
+	uint32_t	grh_gid_fl;
+	uint8_t		rgid[16];
+};
+
+struct mlx5_wqe_datagram_seg {
+	struct mlx5_wqe_av	av;
+};
+
+struct mlx5_wqe_raddr_seg {
+	uint64_t	raddr;
+	uint32_t	rkey;
+	uint32_t	reserved;
+};
+
+struct mlx5_wqe_atomic_seg {
+	uint64_t	swap_add;
+	uint64_t	compare;
+};
+
+struct mlx5_wqe_inl_data_seg {
+	uint32_t	byte_count;
+};
+
+struct mlx5_wqe_eth_seg {
+	uint32_t	rsvd0;
+	uint8_t		cs_flags;
+	uint8_t		rsvd1;
+	uint16_t	mss;
+	uint32_t	rsvd2;
+	uint16_t	inline_hdr_sz;
+	uint8_t		inline_hdr_start[2];
+	uint8_t		inline_hdr[16];
+};
+
+#endif /* _MLX5DV_H_ */
diff --git a/providers/mlx5/qp.c b/providers/mlx5/qp.c
index e82b1a0..177b9ae 100644
--- a/providers/mlx5/qp.c
+++ b/providers/mlx5/qp.c
@@ -212,7 +212,7 @@ static void set_datagram_seg(struct mlx5_wqe_datagram_seg *dseg,
 			     struct ibv_send_wr *wr)
 {
 	memcpy(&dseg->av, &to_mah(wr->wr.ud.ah)->av, sizeof dseg->av);
-	dseg->av.dqp_dct = htonl(wr->wr.ud.remote_qpn | MLX5_EXTENED_UD_AV);
+	dseg->av.dqp_dct = htonl(wr->wr.ud.remote_qpn | MLX5_EXTENDED_UD_AV);
 	dseg->av.key.qkey.qkey = htonl(wr->wr.ud.remote_qkey);
 }
 
diff --git a/providers/mlx5/wqe.h b/providers/mlx5/wqe.h
index f097b77..13597bc 100644
--- a/providers/mlx5/wqe.h
+++ b/providers/mlx5/wqe.h
@@ -33,34 +33,6 @@
 #ifndef WQE_H
 #define WQE_H
 
-enum {
-	MLX5_WQE_CTRL_CQ_UPDATE	= 2 << 2,
-	MLX5_WQE_CTRL_SOLICITED	= 1 << 1,
-	MLX5_WQE_CTRL_FENCE	= 4 << 5,
-	MLX5_WQE_CTRL_INITIATOR_SMALL_FENCE = 1 << 5,
-};
-
-enum {
-	MLX5_INVALID_LKEY	= 0x100,
-};
-
-enum {
-	MLX5_EXTENED_UD_AV	= 0x80000000,
-};
-
-struct mlx5_wqe_srq_next_seg {
-	uint8_t			rsvd0[2];
-	uint16_t		next_wqe_index;
-	uint8_t			signature;
-	uint8_t			rsvd1[11];
-};
-
-struct mlx5_wqe_data_seg {
-	uint32_t		byte_count;
-	uint32_t		lkey;
-	uint64_t		addr;
-};
-
 struct mlx5_sg_copy_ptr {
 	int	index;
 	int	offset;
@@ -76,36 +48,6 @@ struct mlx5_eqe_qp_srq {
 	uint32_t	qp_srq_n;
 };
 
-enum {
-	MLX5_ETH_L2_INLINE_HEADER_SIZE	= 18,
-	MLX5_ETH_L2_MIN_HEADER_SIZE	= 14,
-};
-
-enum {
-	MLX5_ETH_WQE_L3_CSUM = (1 << 6),
-	MLX5_ETH_WQE_L4_CSUM = (1 << 7),
-};
-
-struct mlx5_wqe_eth_seg {
-	uint32_t	rsvd0;
-	uint8_t		cs_flags;
-	uint8_t		rsvd1;
-	uint16_t	mss;
-	uint32_t	rsvd2;
-	uint16_t	inline_hdr_sz;
-	uint8_t		inline_hdr_start[2];
-	uint8_t		inline_hdr[16];
-};
-
-struct mlx5_wqe_ctrl_seg {
-	uint32_t	opmod_idx_opcode;
-	uint32_t	qpn_ds;
-	uint8_t		signature;
-	uint8_t		rsvd[2];
-	uint8_t		fm_ce_se;
-	uint32_t	imm;
-};
-
 struct mlx5_wqe_xrc_seg {
 	uint32_t	xrc_srqn;
 	uint8_t		rsvd[12];
@@ -118,42 +60,9 @@ struct mlx5_wqe_masked_atomic_seg {
 	uint64_t	compare_mask;
 };
 
-struct mlx5_wqe_av {
-	union {
-		struct {
-			uint32_t	qkey;
-			uint32_t	reserved;
-		} qkey;
-		uint64_t	dc_key;
-	} key;
-	uint32_t	dqp_dct;
-	uint8_t		stat_rate_sl;
-	uint8_t		fl_mlid;
-	uint16_t	rlid;
-	uint8_t		reserved0[10];
-	uint8_t		tclass;
-	uint8_t		hop_limit;
-	uint32_t	grh_gid_fl;
-	uint8_t		rgid[16];
-};
-
-struct mlx5_wqe_datagram_seg {
-	struct mlx5_wqe_av	av;
-};
-
-struct mlx5_wqe_raddr_seg {
-	uint64_t	raddr;
-	uint32_t	rkey;
-	uint32_t	reserved;
-};
-
-struct mlx5_wqe_atomic_seg {
-	uint64_t	swap_add;
-	uint64_t	compare;
-};
-
-struct mlx5_wqe_inl_data_seg {
-	uint32_t	byte_count;
+enum {
+	MLX5_ETH_L2_INLINE_HEADER_SIZE	= 18,
+	MLX5_ETH_L2_MIN_HEADER_SIZE	= 14,
 };
 
 enum {
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V1 rdma-core 2/7] mlx5: Add CQE fields

[Yishai Hadas]

From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

Introduce inline calls to get/set for most
commonly used CQE data path fields.

Signed-off-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 providers/mlx5/mlx5dv.h | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/providers/mlx5/mlx5dv.h b/providers/mlx5/mlx5dv.h
index 34502a8..5709039 100644
--- a/providers/mlx5/mlx5dv.h
+++ b/providers/mlx5/mlx5dv.h
@@ -37,6 +37,13 @@
 #include <linux/types.h>
 #include <arpa/inet.h>
 
+/* Always inline the functions */
+#ifdef __GNUC__
+#define MLX5DV_ALWAYS_INLINE inline __attribute__((always_inline))
+#else
+#define MLX5DV_ALWAYS_INLINE inline
+#endif
+
 enum {
 	MLX5_RCV_DBR	= 0,
 	MLX5_SND_DBR	= 1,
@@ -264,6 +271,37 @@ struct mlx5_cqe64 {
 	uint8_t		op_own;
 };
 
+static MLX5DV_ALWAYS_INLINE
+uint8_t mlx5dv_get_cqe_owner(struct mlx5_cqe64 *cqe)
+{
+	return cqe->op_own & 0x1;
+}
+
+static MLX5DV_ALWAYS_INLINE
+void mlx5dv_set_cqe_owner(struct mlx5_cqe64 *cqe, uint8_t val)
+{
+	cqe->op_own = (val & 0x1) | (cqe->op_own & ~0x1);
+}
+
+/* Solicited event */
+static MLX5DV_ALWAYS_INLINE
+uint8_t mlx5dv_get_cqe_se(struct mlx5_cqe64 *cqe)
+{
+	return (cqe->op_own >> 1) & 0x1;
+}
+
+static MLX5DV_ALWAYS_INLINE
+uint8_t mlx5dv_get_cqe_format(struct mlx5_cqe64 *cqe)
+{
+	return (cqe->op_own >> 2) & 0x3;
+}
+
+static MLX5DV_ALWAYS_INLINE
+uint8_t mlx5dv_get_cqe_opcode(struct mlx5_cqe64 *cqe)
+{
+	return cqe->op_own >> 4;
+}
+
 /*
  * WQE related part
  */
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V1 rdma-core 3/7] mlx5: Add WQE segments implementation

[Yishai Hadas]

From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

Add generic and x86 SSE3 versions of WQE
segments implementations.

Signed-off-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 providers/mlx5/mlx5dv.h | 179 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 179 insertions(+)

diff --git a/providers/mlx5/mlx5dv.h b/providers/mlx5/mlx5dv.h
index 5709039..4fae71b 100644
--- a/providers/mlx5/mlx5dv.h
+++ b/providers/mlx5/mlx5dv.h
@@ -35,7 +35,14 @@
 
 /* For __be64 type */
 #include <linux/types.h>
+/* For htonl/htons */
 #include <arpa/inet.h>
+/* Some system requires this file instead of arpa/inet.h */
+#include <netinet/in.h>
+#if defined(__SSE3__)
+#include <emmintrin.h>
+#include <tmmintrin.h>
+#endif /* defined(__SSE3__) */
 
 /* Always inline the functions */
 #ifdef __GNUC__
@@ -406,4 +413,176 @@ struct mlx5_wqe_eth_seg {
 	uint8_t		inline_hdr[16];
 };
 
+/*
+ * Control segment - contains some control information for the current WQE.
+ *
+ * Output:
+ *	seg	  - control segment to be filled
+ * Input:
+ *	pi	  - WQEBB number of the first block of this WQE.
+ *		    This number should wrap at 0xffff, regardless of
+ *		    size of the WQ.
+ *	opcode	  - Opcode of this WQE. Encodes the type of operation
+ *		    to be executed on the QP.
+ *	opmod	  - Opcode modifier.
+ *	qp_num	  - QP/SQ number this WQE is posted to.
+ *	fm_ce_se  - FM (fence mode), CE (completion and event mode)
+ *		    and SE (solicited event).
+ *	ds	  - WQE size in octowords (16-byte units). DS accounts for all
+ *		    the segments in the WQE as summarized in WQE construction.
+ *	signature - WQE signature.
+ *	imm	  - Immediate data/Invalidation key/UMR mkey.
+ */
+static MLX5DV_ALWAYS_INLINE
+void mlx5dv_set_ctrl_seg(struct mlx5_wqe_ctrl_seg *seg, uint16_t pi,
+			 uint8_t opcode, uint8_t opmod, uint32_t qp_num,
+			 uint8_t fm_ce_se, uint8_t ds,
+			 uint8_t signature, uint32_t imm)
+{
+	seg->opmod_idx_opcode	= htonl(((uint32_t)opmod << 24) | ((uint32_t)pi << 8) | opcode);
+	seg->qpn_ds		= htonl((qp_num << 8) | ds);
+	seg->fm_ce_se		= fm_ce_se;
+	seg->signature		= signature;
+	/*
+	 * The caller should prepare "imm" in advance based on WR opcode.
+	 * For IBV_WR_SEND_WITH_IMM and IBV_WR_RDMA_WRITE_WITH_IMM,
+	 * the "imm" should be assigned as is.
+	 * For the IBV_WR_SEND_WITH_INV, it should be htonl(imm).
+	 */
+	seg->imm		= imm;
+}
+
+/* x86 optimized version of mlx5dv_set_ctrl_seg()
+ *
+ * This is useful when doing calculations on large data sets
+ * for parallel calculations.
+ *
+ * It doesn't suit for serialized algorithms.
+ */
+#if defined(__SSE3__)
+static MLX5DV_ALWAYS_INLINE
+void mlx5dv_x86_set_ctrl_seg(struct mlx5_wqe_ctrl_seg *seg, uint16_t pi,
+			     uint8_t opcode, uint8_t opmod, uint32_t qp_num,
+			     uint8_t fm_ce_se, uint8_t ds,
+			     uint8_t signature, uint32_t imm)
+{
+	__m128i val  = _mm_set_epi32(imm, qp_num, (ds << 16) | pi,
+				     (signature << 24) | (opcode << 16) | (opmod << 8) | fm_ce_se);
+	__m128i mask = _mm_set_epi8(15, 14, 13, 12,	/* immediate */
+				     0,			/* signal/fence_mode */
+				     0x80, 0x80,	/* reserved */
+				     3,			/* signature */
+				     6,			/* data size */
+				     8, 9, 10,		/* QP num */
+				     2,			/* opcode */
+				     4, 5,		/* sw_pi in BE */
+				     1			/* opmod */
+				     );
+	*(__m128i *) seg = _mm_shuffle_epi8(val, mask);
+}
+#endif /* defined(__SSE3__) */
+
+/*
+ * Datagram Segment - contains address information required in order
+ * to form a datagram message.
+ *
+ * Output:
+ *	seg		- datagram segment to be filled.
+ * Input:
+ *	key		- Q_key/access key.
+ *	dqp_dct		- Destination QP number for UD and DCT for DC.
+ *	ext		- Address vector extension.
+ *	stat_rate_sl	- Maximum static rate control, SL/ethernet priority.
+ *	fl_mlid		- Force loopback and source LID for IB.
+ *	rlid		- Remote LID
+ *	rmac		- Remote MAC
+ *	tclass		- GRH tclass/IPv6 tclass/IPv4 ToS
+ *	hop_limit	- GRH hop limit/IPv6 hop limit/IPv4 TTL
+ *	grh_gid_fi	- GRH, source GID address and IPv6 flow label.
+ *	rgid		- Remote GID/IP address.
+ */
+static MLX5DV_ALWAYS_INLINE
+void mlx5dv_set_dgram_seg(struct mlx5_wqe_datagram_seg *seg,
+			  uint64_t key, uint32_t dqp_dct,
+			  uint8_t ext, uint8_t stat_rate_sl,
+			  uint8_t fl_mlid, uint16_t rlid,
+			  uint8_t *rmac, uint8_t tclass,
+			  uint8_t hop_limit, uint32_t grh_gid_fi,
+			  uint8_t *rgid)
+{
+
+	/* Always put 64 bits, in q_key, the reserved part will be 0 */
+	seg->av.key.dc_key	= htobe64(key);
+	seg->av.dqp_dct		= htonl(((uint32_t)ext << 31) | dqp_dct);
+	seg->av.stat_rate_sl	= stat_rate_sl;
+	seg->av.fl_mlid		= fl_mlid;
+	seg->av.rlid		= htons(rlid);
+	memcpy(seg->av.rmac, rmac, 6);
+	seg->av.tclass		= tclass;
+	seg->av.hop_limit	= hop_limit;
+	seg->av.grh_gid_fl	= htonl(grh_gid_fi);
+	memcpy(seg->av.rgid, rgid, 16);
+}
+
+/*
+ * Data Segments - contain pointers and a byte count for the scatter/gather list.
+ * They can optionally contain data, which will save a memory read access for
+ * gather Work Requests.
+ */
+static MLX5DV_ALWAYS_INLINE
+void mlx5dv_set_data_seg(struct mlx5_wqe_data_seg *seg,
+			 uint32_t length, uint32_t lkey,
+			 uintptr_t address)
+{
+	seg->byte_count = htonl(length);
+	seg->lkey       = htonl(lkey);
+	seg->addr       = htobe64(address);
+}
+/*
+ * x86 optimized version of mlx5dv_set_data_seg()
+ *
+ * This is useful when doing calculations on large data sets
+ * for parallel calculations.
+ *
+ * It doesn't suit for serialized algorithms.
+ */
+#if defined(__SSE3__)
+static MLX5DV_ALWAYS_INLINE
+void mlx5dv_x86_set_data_seg(struct mlx5_wqe_data_seg *seg,
+			     uint32_t length, uint32_t lkey,
+			     uintptr_t address)
+{
+	__m128i val  = _mm_set_epi32((uint32_t)address, (uint32_t)(address >> 32), lkey, length);
+	__m128i mask = _mm_set_epi8(12, 13, 14, 15,	/* local address low */
+				     8, 9, 10, 11,	/* local address high */
+				     4, 5, 6, 7,	/* l_key */
+				     0, 1, 2, 3		/* byte count */
+				     );
+	*(__m128i *) seg = _mm_shuffle_epi8(val, mask);
+}
+#endif /* defined(__SSE3__) */
+
+/*
+ * Eth Segment - contains packet headers and information for stateless L2, L3, L4 offloading.
+ *
+ * Output:
+ *	 seg		 - Eth segment to be filled.
+ * Input:
+ *	cs_flags	 - l3cs/l3cs_inner/l4cs/l4cs_inner.
+ *	mss		 - Maximum segment size. For TSO WQEs, the number of bytes
+ *			   in the TCP payload to be transmitted in each packet. Must
+ *			   be 0 on non TSO WQEs.
+ *	inline_hdr_sz	 - Length of the inlined packet headers.
+ *	inline_hdr_start - Inlined packet header.
+ */
+static MLX5DV_ALWAYS_INLINE
+void mlx5dv_set_eth_seg(struct mlx5_wqe_eth_seg *seg, uint8_t cs_flags,
+			uint16_t mss, uint16_t inline_hdr_sz,
+			uint8_t *inline_hdr_start)
+{
+	seg->cs_flags		= cs_flags;
+	seg->mss		= htons(mss);
+	seg->inline_hdr_sz	= htons(inline_hdr_sz);
+	memcpy(seg->inline_hdr_start, inline_hdr_start, inline_hdr_sz);
+}
 #endif /* _MLX5DV_H_ */
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V1 rdma-core 4/7] mlx5: Use macro for CQE version 1

[Yishai Hadas]

From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

There is macro defined to mark CQE version 1, so let's use it.

Signed-off-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 providers/mlx5/mlx5.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/providers/mlx5/mlx5.c b/providers/mlx5/mlx5.c
index 1cf5a5f..980e4df 100644
--- a/providers/mlx5/mlx5.c
+++ b/providers/mlx5/mlx5.c
@@ -771,7 +771,7 @@ static int mlx5_init_context(struct verbs_device *vdev,
 
 	context->cqe_version = resp.cqe_version;
 	if (context->cqe_version) {
-		if (context->cqe_version == 1)
+		if (context->cqe_version == MLX5_CQE_VERSION_V1)
 			mlx5_ctx_ops.poll_cq = mlx5_poll_cq_v1;
 		else
 			 goto err_free_bf;
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V1 rdma-core 5/7] mlx5: Convert to use predefined get CQE opcodes

[Yishai Hadas]

From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

Signed-off-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 providers/mlx5/cq.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/providers/mlx5/cq.c b/providers/mlx5/cq.c
index 88bca75..d14d956 100644
--- a/providers/mlx5/cq.c
+++ b/providers/mlx5/cq.c
@@ -87,7 +87,7 @@ static void *get_sw_cqe(struct mlx5_cq *cq, int n)
 
 	cqe64 = (cq->cqe_sz == 64) ? cqe : cqe + 64;
 
-	if (likely((cqe64->op_own) >> 4 != MLX5_CQE_INVALID) &&
+	if (likely(mlx5dv_get_cqe_opcode(cqe64) != MLX5_CQE_INVALID) &&
 	    !((cqe64->op_own & MLX5_CQE_OWNER_MASK) ^ !!(n & (cq->ibv_cq.cqe + 1)))) {
 		return cqe;
 	} else {
@@ -547,7 +547,7 @@ static inline int mlx5_parse_cqe(struct mlx5_cq *cq,
 		wc->qp_num = qpn;
 	}
 
-	opcode = cqe64->op_own >> 4;
+	opcode = mlx5dv_get_cqe_opcode(cqe64);
 	switch (opcode) {
 	case MLX5_CQE_REQ:
 	{
@@ -1056,7 +1056,7 @@ static inline enum ibv_wc_opcode mlx5_cq_read_wc_opcode(struct ibv_cq_ex *ibcq)
 {
 	struct mlx5_cq *cq = to_mcq(ibv_cq_ex_to_cq(ibcq));
 
-	switch (cq->cqe64->op_own >> 4) {
+	switch (mlx5dv_get_cqe_opcode(cq->cqe64)) {
 	case MLX5_CQE_RESP_WR_IMM:
 		return IBV_WC_RECV_RDMA_WITH_IMM;
 	case MLX5_CQE_RESP_SEND:
@@ -1114,7 +1114,7 @@ static inline int mlx5_cq_read_wc_flags(struct ibv_cq_ex *ibcq)
 				  MLX5_CQE_L3_HDR_TYPE_IPV4)) <<
 				IBV_WC_IP_CSUM_OK_SHIFT;
 
-	switch (cq->cqe64->op_own >> 4) {
+	switch (mlx5dv_get_cqe_opcode(cq->cqe64)) {
 	case MLX5_CQE_RESP_WR_IMM:
 	case MLX5_CQE_RESP_SEND_IMM:
 		wc_flags	|= IBV_WC_WITH_IMM;
@@ -1147,7 +1147,7 @@ static inline uint32_t mlx5_cq_read_wc_imm_data(struct ibv_cq_ex *ibcq)
 {
 	struct mlx5_cq *cq = to_mcq(ibv_cq_ex_to_cq(ibcq));
 
-	switch (cq->cqe64->op_own >> 4) {
+	switch (mlx5dv_get_cqe_opcode(cq->cqe64)) {
 	case MLX5_CQE_RESP_SEND_INV:
 		return ntohl(cq->cqe64->imm_inval_pkey);
 	default:
@@ -1329,7 +1329,7 @@ static inline int free_res_cqe(struct mlx5_cqe64 *cqe64, uint32_t rsn,
 {
 	if (cqe_version) {
 		if (is_equal_uidx(cqe64, rsn)) {
-			if (srq && is_responder(cqe64->op_own >> 4))
+			if (srq && is_responder(mlx5dv_get_cqe_opcode(cqe64)))
 				mlx5_free_srq_wqe(srq,
 						  ntohs(cqe64->wqe_counter));
 			return 1;
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V1 rdma-core 6/7] mlx5: Add direct verbs man pages

[Yishai Hadas]

From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

Document mlx5 direct verbs feature and exposed related functions.

Signed-off-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 CMakeLists.txt                           |   1 +
 providers/mlx5/man/CMakeLists.txt        |   5 ++
 providers/mlx5/man/mlx5dv.7              |  44 +++++++++++
 providers/mlx5/man/mlx5dv_init_obj.3     | 129 +++++++++++++++++++++++++++++++
 providers/mlx5/man/mlx5dv_query_device.3 |  48 ++++++++++++
 5 files changed, 227 insertions(+)
 create mode 100644 providers/mlx5/man/CMakeLists.txt
 create mode 100644 providers/mlx5/man/mlx5dv.7
 create mode 100644 providers/mlx5/man/mlx5dv_init_obj.3
 create mode 100644 providers/mlx5/man/mlx5dv_query_device.3

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 45cbc4e..529c0cd 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -377,6 +377,7 @@ add_subdirectory(providers/hns)
 add_subdirectory(providers/i40iw)
 add_subdirectory(providers/mlx4)
 add_subdirectory(providers/mlx5)
+add_subdirectory(providers/mlx5/man)
 add_subdirectory(providers/mthca)
 add_subdirectory(providers/nes)
 add_subdirectory(providers/ocrdma)
diff --git a/providers/mlx5/man/CMakeLists.txt b/providers/mlx5/man/CMakeLists.txt
new file mode 100644
index 0000000..bb1610b
--- /dev/null
+++ b/providers/mlx5/man/CMakeLists.txt
@@ -0,0 +1,5 @@
+rdma_man_pages(
+  mlx5dv_init_obj.3
+  mlx5dv_query_device.3
+  mlx5dv.7
+)
diff --git a/providers/mlx5/man/mlx5dv.7 b/providers/mlx5/man/mlx5dv.7
new file mode 100644
index 0000000..48b21b9
--- /dev/null
+++ b/providers/mlx5/man/mlx5dv.7
@@ -0,0 +1,44 @@
+.\" -*- nroff -*-
+.\" Licensed under the OpenIB.org BSD license (FreeBSD Variant) - See COPYING.md
+.\"
+.TH MLX5DV 7 2017-02-02 1.0.0
+.SH "NAME"
+mlx5dv \- Direct verbs for mlx5 devices
+.br
+This is low level access to mlx5 devices to perform data path operations,
+without general branching performed by \fBibv_post_send\fR(3).
+
+.SH "DESCRIPTION"
+The libibverbs API is an abstract one. It is agnostic to any underlying
+provider specific implementation. While this abstraction has the advantage
+of user applications portability it has a performance penalty. For some
+applications optimizing performance is more important than portability.
+
+The mlx5 direct verbs API is intended for such applications.
+It exposes mlx5 specific low level data path (send/receive/completion)
+operations, allowing the application to bypass the libibverbs data path API.
+
+This interface consists from one hardware specific header file
+with relevant inline functions and conversion logic from ibverbs structures
+to mlx5 specific structures.
+
+The direct include of mlx5dv.h together with linkage to mlx5 library will
+allow usage of this new interface.
+
+Once an application uses the direct flow the locking scheme is fully managed
+by itself. There is an expectation that no mixed flows in the data path for both
+direct/non-direct access will be by same application.
+
+.SH "NOTES"
+All Mellanox NIC devices starting from Connect-IB (Connect-IB,
+ConnectX-4, ConnectX-4Lx, ConnectX-5, ...) implement the mlx5 API,
+thus using the mlx5 direct verbs does not limit the applications
+to a single NIC HW device thus keeping some level of portability.
+
+.SH "SEE ALSO"
+.BR ibv_post_send (3),
+.BR verbs (7)
+
+.SH "AUTHORS"
+.TP
+Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
diff --git a/providers/mlx5/man/mlx5dv_init_obj.3 b/providers/mlx5/man/mlx5dv_init_obj.3
new file mode 100644
index 0000000..41b0790
--- /dev/null
+++ b/providers/mlx5/man/mlx5dv_init_obj.3
@@ -0,0 +1,129 @@
+.\" -*- nroff -*-
+.\" Licensed under the OpenIB.org BSD license (FreeBSD Variant) - See COPYING.md
+.\"
+.TH MLX5DV_INIT_OBJ 3 2017-02-02 1.0.0
+.SH "NAME"
+mlx5dv_init_obj \- Initialize mlx5 direct verbs object from ibv_xxx structures
+.SH "SYNOPSIS"
+.nf
+.B #include <infiniband/mlx5dv.h>
+.sp
+.BI "int mlx5dv_init_obj(struct mlx5dv_obj *obj, uint64_t obj_type);
+.fi
+.SH "DESCRIPTION"
+.B mlx5dv_init_obj()
+This function will initialize mlx5dv_xxx structs based on supplied type. The information
+for initialization is taken from ibv_xx structs supplied as part of input.
+
+Request information of CQ marks its owned by direct verbs for all consumer index
+related actions. The initialization type can be combination of several types together.
+.PP
+.nf
+struct mlx5dv_qp {
+.in +8
+uint32_t                *dbrec;
+struct {
+.in +8
+void            *buf;
+uint32_t        wqe_cnt;
+uint32_t        stride;
+.in -8
+} sq;
+struct {
+.in +8
+void            *buf;
+uint32_t        wqe_cnt;
+uint32_t        stride;
+.in -8
+} rq;
+struct {
+.in +8
+void            *reg;
+uint32_t        size;
+.in -8
+} bf;
+uint64_t                comp_mask;
+.in -8
+};
+
+struct mlx5dv_cq {
+.in +8
+void                    *buf;
+uint32_t                *dbrec;
+uint32_t                cqe_cnt;
+uint32_t                cqe_size;
+void                    *uar;
+uint32_t                cqn;
+uint64_t                comp_mask;
+.in -8
+};
+
+struct mlx5dv_srq {
+.in +8
+void                    *buf;
+uint32_t                *dbrec;
+uint32_t                stride;
+uint32_t                head;
+uint32_t                tail;
+uint64_t                comp_mask;
+.in -8
+};
+
+struct mlx5dv_rwq {
+.in +8
+void            *buf;
+uint32_t        *dbrec;
+uint32_t        wqe_cnt;
+uint32_t        stride;
+uint64_t        comp_mask;
+.in -8
+};
+
+struct mlx5dv_obj {
+.in +8
+struct {
+.in +8
+struct ibv_qp           *in;
+struct mlx5dv_qp        *out;
+.in -8
+} qp;
+struct {
+.in +8
+struct ibv_cq           *in;
+struct mlx5dv_cq        *out;
+.in -8
+} cq;
+struct {
+.in +8
+struct ibv_srq          *in;
+struct mlx5dv_srq       *out;
+.in -8
+} srq;
+struct {
+.in +8
+struct ibv_wq           *in;
+struct mlx5dv_rwq       *out;
+.in -8
+} rwq;
+.in -8
+};
+
+enum mlx5dv_obj_type {
+.in +8
+MLX5DV_OBJ_QP   = 1 << 0,
+MLX5DV_OBJ_CQ   = 1 << 1,
+MLX5DV_OBJ_SRQ  = 1 << 2,
+MLX5DV_OBJ_RWQ  = 1 << 3,
+.in -8
+};
+.fi
+.SH "RETURN VALUE"
+0 on success.
+.SH "NOTES"
+The information if doorbell is blueflame is based on mlx5dv_qp->bf->size,
+in case of 0 it's not a BF.
+.SH "SEE ALSO"
+.BR mlx5dv (7)
+.SH "AUTHORS"
+.TP
+Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
diff --git a/providers/mlx5/man/mlx5dv_query_device.3 b/providers/mlx5/man/mlx5dv_query_device.3
new file mode 100644
index 0000000..1aeba3d
--- /dev/null
+++ b/providers/mlx5/man/mlx5dv_query_device.3
@@ -0,0 +1,48 @@
+.\" -*- nroff -*-
+.\" Licensed under the OpenIB.org BSD license (FreeBSD Variant) - See COPYING.md
+.\"
+.TH MLX5DV_QUERY_DEVICE 3 2017-02-02 1.0.0
+.SH "NAME"
+mlx5dv_query_device \- Query device capabilities specific to mlx5
+.SH "SYNOPSIS"
+.nf
+.B #include <infiniband/mlx5dv.h>
+.sp
+.BI "int mlx5dv_query_device(struct ibv_context *ctx_in,
+.BI "                        struct mlx5dv_context *attrs_out);
+.fi
+.SH "DESCRIPTION"
+.B mlx5dv_query_device()
+Query HW device-specific information which is important for data-path, but isn't provided by
+\fBibv_query_device\fR(3).
+.PP
+This function returns version, flags and compatibility mask. The version represents the format
+of the internal hardware structures that mlx5dv.h represents. Additions of new fields to the existed
+structures are handled by comp_mask field.
+.PP
+.nf
+struct mlx5dv_context {
+.in +8
+uint8_t         version;
+uint64_t        flags;
+uint64_t        comp_mask;
+.in -8
+};
+
+enum mlx5dv_context_flags {
+.in +8
+/*
+ * This flag indicates if CQE version 0 or 1 is needed.
+ */
+ MLX5DV_CONTEXT_FLAGS_CQE_V1 = (1 << 0),
+.in -8
+};
+.fi
+.SH "RETURN VALUE"
+0 on success.
+.SH "SEE ALSO"
+.BR mlx5dv (7),
+.BR ibv_query_device (3)
+.SH "AUTHORS"
+.TP
+Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V1 rdma-core 7/7] mlx5: Export mlx5 direct verbs interface

[Yishai Hadas]

From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

Install direct verbs header file into /usr/include/infiniband/
folder and allow for possible users to explicitly include it
into their applications.

Signed-off-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
Signed-off-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 buildlib/rdma_functions.cmake    | 51 ++++++++++++++++++++++++++++++++++++++++
 buildlib/relpath                 |  7 ++++++
 debian/ibverbs-providers.install |  4 ++++
 debian/ibverbs-providers.symbols |  4 ++++
 providers/mlx5/CMakeLists.txt    |  7 +++++-
 providers/mlx5/libmlx5.map       |  8 +++++++
 redhat/rdma-core.spec            |  3 +++
 7 files changed, 83 insertions(+), 1 deletion(-)
 create mode 100644 buildlib/relpath
 create mode 100644 debian/ibverbs-providers.symbols
 create mode 100644 providers/mlx5/libmlx5.map

diff --git a/buildlib/rdma_functions.cmake b/buildlib/rdma_functions.cmake
index 5256ad9..5012e22 100644
--- a/buildlib/rdma_functions.cmake
+++ b/buildlib/rdma_functions.cmake
@@ -72,6 +72,57 @@ function(rdma_library DEST VERSION_SCRIPT SOVERSION VERSION)
   install(TARGETS ${DEST} DESTINATION "${CMAKE_INSTALL_LIBDIR}")
 endfunction()
 
+# Create a special provider with exported symbols in it The shared provider
+# exists as a normal system library with the normal shared library SONAME and
+# other convections. The system library is symlinked into the
+# VERBS_PROVIDER_DIR so it can be dlopened as a provider as well.
+function(rdma_shared_provider DEST VERSION_SCRIPT SOVERSION VERSION)
+  # Installed driver file
+  file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/${DEST}.driver" "driver ${DEST}\n")
+  install(FILES "${CMAKE_CURRENT_BINARY_DIR}/${DEST}.driver" DESTINATION "${CONFIG_DIR}")
+
+  # Uninstalled driver file
+  file(MAKE_DIRECTORY "${BUILD_ETC}/libibverbs.d/")
+  file(WRITE "${BUILD_ETC}/libibverbs.d/${DEST}.driver" "driver ${BUILD_LIB}/lib${DEST}\n")
+
+  # Create a static provider library
+  if (ENABLE_STATIC)
+    add_library(${DEST} STATIC ${ARGN})
+    set_target_properties(${DEST} PROPERTIES LIBRARY_OUTPUT_DIRECTORY "${BUILD_LIB}")
+    install(TARGETS ${DEST} DESTINATION "${CMAKE_INSTALL_LIBDIR}")
+
+    list(APPEND RDMA_STATIC_LIBS ${DEST}-rdmav2 ${DEST})
+    set(RDMA_STATIC_LIBS "${RDMA_STATIC_LIBS}" CACHE INTERNAL "")
+  endif()
+
+  # Create the plugin shared library
+  add_library(${DEST} SHARED ${ARGN})
+  rdma_set_library_map(${DEST} ${VERSION_SCRIPT})
+
+  target_link_libraries(${DEST} LINK_PRIVATE ${COMMON_LIBS_PIC})
+  target_link_libraries(${DEST} LINK_PRIVATE ibverbs)
+  target_link_libraries(${DEST} LINK_PRIVATE ${CMAKE_THREAD_LIBS_INIT})
+  set_target_properties(${DEST} PROPERTIES
+    SOVERSION ${SOVERSION}
+    VERSION ${VERSION}
+    LIBRARY_OUTPUT_DIRECTORY "${BUILD_LIB}")
+  install(TARGETS ${DEST} DESTINATION "${CMAKE_INSTALL_LIBDIR}")
+
+  # Compute a relative symlink from VERBS_PROVIDER_DIR to LIBDIR
+  execute_process(COMMAND python ${CMAKE_SOURCE_DIR}/buildlib/relpath
+    "${CMAKE_INSTALL_FULL_LIBDIR}/lib${DEST}.so.${VERSION}"
+    "${VERBS_PROVIDER_DIR}"
+    OUTPUT_VARIABLE DEST_LINK_PATH OUTPUT_STRIP_TRAILING_WHITESPACE)
+  rdma_install_symlink("${DEST_LINK_PATH}" "${VERBS_PROVIDER_DIR}/lib${DEST}-rdmav2.so")
+
+  # cmake doesn't create target DESTINATION directories until everything is
+  # finished, do it manually here so we can create the in-tree symlink.
+  execute_process(COMMAND "${CMAKE_COMMAND}" -E make_directory "${BUILD_LIB}")
+  execute_process(COMMAND "${CMAKE_COMMAND}" -E create_symlink
+    "lib${DEST}.so.${VERSION}"
+    "${BUILD_LIB}/lib${DEST}-rdmav2.so")
+endfunction()
+
 # Create a provider shared library for libibverbs
 function(rdma_provider DEST)
   # Installed driver file
diff --git a/buildlib/relpath b/buildlib/relpath
new file mode 100644
index 0000000..b80cf63
--- /dev/null
+++ b/buildlib/relpath
@@ -0,0 +1,7 @@
+#!/usr/bin/env python
+# Copyright 2017 Mellanox Technologies, Inc. See COPYING.
+
+import os
+import sys
+
+print(os.path.relpath(sys.argv[1], sys.argv[2]))
diff --git a/debian/ibverbs-providers.install b/debian/ibverbs-providers.install
index 70244ee..7bacc2e 100644
--- a/debian/ibverbs-providers.install
+++ b/debian/ibverbs-providers.install
@@ -1,8 +1,12 @@
+usr/include/infiniband/mlx5dv.h
 etc/libibverbs.d/
 etc/modprobe.d/truescale.conf
 usr/bin/rxe_cfg
 usr/lib/*/libibverbs/lib*-rdmav2.so
+usr/lib/*/libmlx5.*
 usr/lib/truescale-serdes.cmds
 usr/share/doc/rdma-core/rxe.md usr/share/doc/ibverbs-providers/
+usr/share/man/man3/mlx5dv_*.3
+usr/share/man/man7/mlx5dv.7
 usr/share/man/man7/rxe.7
 usr/share/man/man8/rxe_cfg.8
diff --git a/debian/ibverbs-providers.symbols b/debian/ibverbs-providers.symbols
new file mode 100644
index 0000000..b114deb
--- /dev/null
+++ b/debian/ibverbs-providers.symbols
@@ -0,0 +1,4 @@
+libmlx5.so.1 ibverbs-providers #MINVER#
+ MLX5_1.0@MLX5_1.0 13-1
+ mlx5dv_init_obj@MLX5_1.0 13-1
+ mlx5dv_query_device@MLX5_1.0 13-1
diff --git a/providers/mlx5/CMakeLists.txt b/providers/mlx5/CMakeLists.txt
index 44df83b..25e10af 100644
--- a/providers/mlx5/CMakeLists.txt
+++ b/providers/mlx5/CMakeLists.txt
@@ -10,7 +10,8 @@ if (MLX5_MW_DEBUG)
   add_definitions("-DMW_DEBUG")
 endif()
 
-rdma_provider(mlx5
+rdma_shared_provider(mlx5 libmlx5.map
+  1 1.0.${PACKAGE_VERSION}
   buf.c
   cq.c
   dbrec.c
@@ -19,3 +20,7 @@ rdma_provider(mlx5
   srq.c
   verbs.c
 )
+
+publish_headers(infiniband
+  mlx5dv.h
+)
diff --git a/providers/mlx5/libmlx5.map b/providers/mlx5/libmlx5.map
new file mode 100644
index 0000000..7ce3eb1
--- /dev/null
+++ b/providers/mlx5/libmlx5.map
@@ -0,0 +1,8 @@
+/* Export symbols should be added below according to
+   Documentation/versioning.md document. */
+MLX5_1.0 {
+	global:
+		mlx5dv_query_device;
+		mlx5dv_init_obj;
+	local: *;
+};
diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec
index 9bdfdff..ade4b4a 100644
--- a/redhat/rdma-core.spec
+++ b/redhat/rdma-core.spec
@@ -359,12 +359,15 @@ rm -rf %{buildroot}/%{_initrddir}/
 %dir %{_libdir}/libibverbs
 %{_libdir}/libibverbs*.so.*
 %{_libdir}/libibverbs/*.so
+%{_libdir}/libmlx5.so*
 %config(noreplace) %{_sysconfdir}/libibverbs.d/*.driver
 %doc %{_docdir}/%{name}-%{version}/libibverbs.md
 %doc %{_docdir}/%{name}-%{version}/rxe.md
 %{_bindir}/rxe_cfg
 %{_mandir}/man7/rxe*
 %{_mandir}/man8/rxe*
+%{_mandir}/man3/mlx5dv*
+%{_mandir}/man7/mlx5dv*
 
 %files -n libibverbs-utils
 %{_bindir}/ibv_*
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 rdma-core 7/7] mlx5: Export mlx5 direct verbs interface

[Jason Gunthorpe]

On Sun, Feb 12, 2017 at 04:16:52PM +0200, Yishai Hadas wrote:
> diff --git a/debian/ibverbs-providers.install b/debian/ibverbs-providers.install
> index 70244ee..7bacc2e 100644
> +++ b/debian/ibverbs-providers.install
> @@ -1,8 +1,12 @@
> +usr/include/infiniband/mlx5dv.h
>  etc/libibverbs.d/
>  etc/modprobe.d/truescale.conf
>  usr/bin/rxe_cfg
>  usr/lib/*/libibverbs/lib*-rdmav2.so
> +usr/lib/*/libmlx5.*
>  usr/lib/truescale-serdes.cmds
>  usr/share/doc/rdma-core/rxe.md usr/share/doc/ibverbs-providers/
> +usr/share/man/man3/mlx5dv_*.3
> +usr/share/man/man7/mlx5dv.7

These need to go in libibverbs-dev, as does the libmlx5*.so.

Someone needs to figure out if Debian policy requires a dedicated
package for the now linkable libmlx5

> diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec
> index 9bdfdff..ade4b4a 100644
> +++ b/redhat/rdma-core.spec
> @@ -359,12 +359,15 @@ rm -rf %{buildroot}/%{_initrddir}/
>  %dir %{_libdir}/libibverbs
>  %{_libdir}/libibverbs*.so.*
>  %{_libdir}/libibverbs/*.so
> +%{_libdir}/libmlx5.so*
>  %config(noreplace) %{_sysconfdir}/libibverbs.d/*.driver
>  %doc %{_docdir}/%{name}-%{version}/libibverbs.md
>  %doc %{_docdir}/%{name}-%{version}/rxe.md
>  %{_bindir}/rxe_cfg
>  %{_mandir}/man7/rxe*
>  %{_mandir}/man8/rxe*
> +%{_mandir}/man3/mlx5dv*
> +%{_mandir}/man7/mlx5dv*

Same comment here, the devel items should be in rdma-core-devel for rpm.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 rdma-core 6/7] mlx5: Add direct verbs man pages

[Jason Gunthorpe]

On Sun, Feb 12, 2017 at 04:16:51PM +0200, Yishai Hadas wrote:

> +.\" -*- nroff -*-
> +.\" Licensed under the OpenIB.org BSD license (FreeBSD Variant) - See COPYING.md

Please use the MIT variant for all new files

> +.fi
> +.SH "RETURN VALUE"
> +0 on success.

'or the value of errno on failure (which indicates the failure reason)'

and in other places.

> +.BI "int mlx5dv_query_device(struct ibv_context *ctx_in,
> +.BI "                        struct mlx5dv_context *attrs_out);

This isn't going to work with comp_mask, at a minimum you need to add
a size_t attrs_len argument.

I recommend against adding new complex queries like this - they don't
work well from an ABI perspective and not being performance critical
do not require this comp_max madness

Eg just use:

int mlx5dv_query_cq_format(struct ibv_context *ctx_in)

We still need man pages for all the inline functions :|

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 rdma-core 6/7] mlx5: Add direct verbs man pages

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1579 bytes --]

On Mon, Feb 13, 2017 at 09:54:45AM -0700, Jason Gunthorpe wrote:
> On Sun, Feb 12, 2017 at 04:16:51PM +0200, Yishai Hadas wrote:
>
> > +.\" -*- nroff -*-
> > +.\" Licensed under the OpenIB.org BSD license (FreeBSD Variant) - See COPYING.md
>
> Please use the MIT variant for all new files
>
> > +.fi
> > +.SH "RETURN VALUE"
> > +0 on success.
>
> 'or the value of errno on failure (which indicates the failure reason)'
>
> and in other places.
>
> > +.BI "int mlx5dv_query_device(struct ibv_context *ctx_in,
> > +.BI "                        struct mlx5dv_context *attrs_out);
>
> This isn't going to work with comp_mask, at a minimum you need to add
> a size_t attrs_len argument.
>
> I recommend against adding new complex queries like this - they don't
> work well from an ABI perspective and not being performance critical
> do not require this comp_max madness
>
> Eg just use:
>
> int mlx5dv_query_cq_format(struct ibv_context *ctx_in)

This will end with bazillion small exported functions and I don't think
that it is clean way to provide API. Especially for function which
should query_device.

>
> We still need man pages for all the inline functions :|

There is documentation in the include file itself. It is hard to imagine
users are working with these functions without good knowledge of mlx5
HW. They don't need man pages :).

>
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH V1 rdma-core 7/7] mlx5: Export mlx5 direct verbs interface

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1874 bytes --]

On Mon, Feb 13, 2017 at 09:49:44AM -0700, Jason Gunthorpe wrote:
> On Sun, Feb 12, 2017 at 04:16:52PM +0200, Yishai Hadas wrote:
> > diff --git a/debian/ibverbs-providers.install b/debian/ibverbs-providers.install
> > index 70244ee..7bacc2e 100644
> > +++ b/debian/ibverbs-providers.install
> > @@ -1,8 +1,12 @@
> > +usr/include/infiniband/mlx5dv.h
> >  etc/libibverbs.d/
> >  etc/modprobe.d/truescale.conf
> >  usr/bin/rxe_cfg
> >  usr/lib/*/libibverbs/lib*-rdmav2.so
> > +usr/lib/*/libmlx5.*
> >  usr/lib/truescale-serdes.cmds
> >  usr/share/doc/rdma-core/rxe.md usr/share/doc/ibverbs-providers/
> > +usr/share/man/man3/mlx5dv_*.3
> > +usr/share/man/man7/mlx5dv.7
>
> These need to go in libibverbs-dev, as does the libmlx5*.so.

I'm not sure, at the end libmlx5 is a provider.

>
> Someone needs to figure out if Debian policy requires a dedicated
> package for the now linkable libmlx5

Talat will work on it, meanwhile we will keep it as is.

>
> > diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec
> > index 9bdfdff..ade4b4a 100644
> > +++ b/redhat/rdma-core.spec
> > @@ -359,12 +359,15 @@ rm -rf %{buildroot}/%{_initrddir}/
> >  %dir %{_libdir}/libibverbs
> >  %{_libdir}/libibverbs*.so.*
> >  %{_libdir}/libibverbs/*.so
> > +%{_libdir}/libmlx5.so*
> >  %config(noreplace) %{_sysconfdir}/libibverbs.d/*.driver
> >  %doc %{_docdir}/%{name}-%{version}/libibverbs.md
> >  %doc %{_docdir}/%{name}-%{version}/rxe.md
> >  %{_bindir}/rxe_cfg
> >  %{_mandir}/man7/rxe*
> >  %{_mandir}/man8/rxe*
> > +%{_mandir}/man3/mlx5dv*
> > +%{_mandir}/man7/mlx5dv*
>
> Same comment here, the devel items should be in rdma-core-devel for rpm.
>
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH V1 rdma-core 7/7] mlx5: Export mlx5 direct verbs interface

[Jason Gunthorpe]

On Mon, Feb 13, 2017 at 08:21:21PM +0200, Leon Romanovsky wrote:
> On Mon, Feb 13, 2017 at 09:49:44AM -0700, Jason Gunthorpe wrote:
> > On Sun, Feb 12, 2017 at 04:16:52PM +0200, Yishai Hadas wrote:
> > > diff --git a/debian/ibverbs-providers.install b/debian/ibverbs-providers.install
> > > index 70244ee..7bacc2e 100644
> > > +++ b/debian/ibverbs-providers.install
> > > @@ -1,8 +1,12 @@
> > > +usr/include/infiniband/mlx5dv.h
> > >  etc/libibverbs.d/
> > >  etc/modprobe.d/truescale.conf
> > >  usr/bin/rxe_cfg
> > >  usr/lib/*/libibverbs/lib*-rdmav2.so
> > > +usr/lib/*/libmlx5.*
> > >  usr/lib/truescale-serdes.cmds
> > >  usr/share/doc/rdma-core/rxe.md usr/share/doc/ibverbs-providers/
> > > +usr/share/man/man3/mlx5dv_*.3
> > > +usr/share/man/man7/mlx5dv.7
> >
> > These need to go in libibverbs-dev, as does the libmlx5*.so.
> 
> I'm not sure, at the end libmlx5 is a provider.

Devel items need to go in dev pacakges. That means headers, devel man
pages and the main .so symlink.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 rdma-core 6/7] mlx5: Add direct verbs man pages

[Jason Gunthorpe]

On Mon, Feb 13, 2017 at 08:13:38PM +0200, Leon Romanovsky wrote:
> On Mon, Feb 13, 2017 at 09:54:45AM -0700, Jason Gunthorpe wrote:
> > On Sun, Feb 12, 2017 at 04:16:51PM +0200, Yishai Hadas wrote:
> >
> > > +.\" -*- nroff -*-
> > > +.\" Licensed under the OpenIB.org BSD license (FreeBSD Variant) - See COPYING.md
> >
> > Please use the MIT variant for all new files
> >
> > > +.fi
> > > +.SH "RETURN VALUE"
> > > +0 on success.
> >
> > 'or the value of errno on failure (which indicates the failure reason)'
> >
> > and in other places.
> >
> > > +.BI "int mlx5dv_query_device(struct ibv_context *ctx_in,
> > > +.BI "                        struct mlx5dv_context *attrs_out);
> >
> > This isn't going to work with comp_mask, at a minimum you need to add
> > a size_t attrs_len argument.
> >
> > I recommend against adding new complex queries like this - they don't
> > work well from an ABI perspective and not being performance critical
> > do not require this comp_max madness
> >
> > Eg just use:
> >
> > int mlx5dv_query_cq_format(struct ibv_context *ctx_in)
> 
> This will end with bazillion small exported functions and I don't think
> that it is clean way to provide API. Especially for function which
> should query_device.

There is nothing really wrong with that, and it is better than
fighting with that horrid comp_mask stuff, especially when the first
attempt is done wrong :|

You could also use a getsockopt style of multiplexer.

Overall 'query_device' was a design mistake from an ABI perspective,
do not copy it.

In any case, it cannot stay the way it is in this series.

> > We still need man pages for all the inline functions :|
> 
> There is documentation in the include file itself. It is hard to imagine
> users are working with these functions without good knowledge of mlx5
> HW. They don't need man pages :).

Hrm, not really excited to see an API that is only useful if you have
NDA'd mellanox documentation..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 rdma-core 6/7] mlx5: Add direct verbs man pages

[Yishai Hadas]

On 2/13/2017 8:45 PM, Jason Gunthorpe wrote:
> On Mon, Feb 13, 2017 at 08:13:38PM +0200, Leon Romanovsky wrote:
>> On Mon, Feb 13, 2017 at 09:54:45AM -0700, Jason Gunthorpe wrote:
>>> On Sun, Feb 12, 2017 at 04:16:51PM +0200, Yishai Hadas wrote:
>>>
>>>> +.\" -*- nroff -*-
>>>> +.\" Licensed under the OpenIB.org BSD license (FreeBSD Variant) - See COPYING.md
>>>
>>> Please use the MIT variant for all new files
>>>
>>>> +.fi
>>>> +.SH "RETURN VALUE"
>>>> +0 on success.
>>>
>>> 'or the value of errno on failure (which indicates the failure reason)'
>>>
>>> and in other places.
>>>
>>>> +.BI "int mlx5dv_query_device(struct ibv_context *ctx_in,
>>>> +.BI "                        struct mlx5dv_context *attrs_out);
>>>
>>> This isn't going to work with comp_mask, at a minimum you need to add
>>> a size_t attrs_len argument.
>>>
>>> I recommend against adding new complex queries like this - they don't
>>> work well from an ABI perspective and not being performance critical
>>> do not require this comp_max madness
>>>
>>> Eg just use:
>>>
>>> int mlx5dv_query_cq_format(struct ibv_context *ctx_in)
>>
>> This will end with bazillion small exported functions and I don't think
>> that it is clean way to provide API. Especially for function which
>> should query_device.
>
> There is nothing really wrong with that, and it is better than
> fighting with that horrid comp_mask stuff, especially when the first
> attempt is done wrong :|
>
> You could also use a getsockopt style of multiplexer.
>
> Overall 'query_device' was a design mistake from an ABI perspective,
> do not copy it.
>
> In any case, it cannot stay the way it is in this series.

There is nothing wrong with current usage of comp_mask, it just follows 
the usage of ibv_query_rt_values_ex where it's used as both in/out.(see 
below from its man page)
We can update the man pages to say it explicitly so that it will be 
fully clear.

That way looks quite more simple than managing an input size which may 
still require a comp_mask as an output to mark which values were set in 
case few fields can have a valid value of 0 but might be ignored as of 
legacy driver.


 From man page:
----------------------
int ibv_query_rt_values_ex(struct ibv_context *context,
                                   struct ibv_values_ex *values);

DESCRIPTION
ibv_query_rt_values_ex() returns certain real time values of a device 
context.  The argument attr is a pointer to an ibv_device_attr_ex 
struct, as defined in <infiniband/verbs.h>.

struct ibv_values_ex {
uint32_t      comp_mask;    /* Compatibility mask that defines the 
query/queried fields [in/out] */
  struct timespec      raw_clock;    /* HW raw clock */
};

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 rdma-core 6/7] mlx5: Add direct verbs man pages

[Jason Gunthorpe]

On Mon, Feb 13, 2017 at 11:52:33PM +0200, Yishai Hadas wrote:

> There is nothing wrong with current usage of comp_mask, it just
> follows the usage of ibv_query_rt_values_ex where it's used as both
> in/out.(see below from its man page) We can update the man pages to
> say it explicitly so that it will be fully clear.

And I complained about that when you did it too - it is still *horrible*

> That way looks quite more simple than managing an input size which may still
> require a comp_mask as an output to mark which values were set in case few
> fields can have a valid value of 0 but might be ignored as of legacy driver.

The best is to just have unique functions or a sane multiplexor, stop
copying this bad idea.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 rdma-core 7/7] mlx5: Export mlx5 direct verbs interface

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1725 bytes --]

On Mon, Feb 13, 2017 at 11:42:50AM -0700, Jason Gunthorpe wrote:
> On Mon, Feb 13, 2017 at 08:21:21PM +0200, Leon Romanovsky wrote:
> > On Mon, Feb 13, 2017 at 09:49:44AM -0700, Jason Gunthorpe wrote:
> > > On Sun, Feb 12, 2017 at 04:16:52PM +0200, Yishai Hadas wrote:
> > > > diff --git a/debian/ibverbs-providers.install b/debian/ibverbs-providers.install
> > > > index 70244ee..7bacc2e 100644
> > > > +++ b/debian/ibverbs-providers.install
> > > > @@ -1,8 +1,12 @@
> > > > +usr/include/infiniband/mlx5dv.h
> > > >  etc/libibverbs.d/
> > > >  etc/modprobe.d/truescale.conf
> > > >  usr/bin/rxe_cfg
> > > >  usr/lib/*/libibverbs/lib*-rdmav2.so
> > > > +usr/lib/*/libmlx5.*
> > > >  usr/lib/truescale-serdes.cmds
> > > >  usr/share/doc/rdma-core/rxe.md usr/share/doc/ibverbs-providers/
> > > > +usr/share/man/man3/mlx5dv_*.3
> > > > +usr/share/man/man7/mlx5dv.7
> > >
> > > These need to go in libibverbs-dev, as does the libmlx5*.so.
> >
> > I'm not sure, at the end libmlx5 is a provider.
>
> Devel items need to go in dev pacakges. That means headers, devel man
> pages and the main .so symlink.

Sorry, but I'm still missing how mlx5 installation will be handled.

The ibverbs-provider installs different *-rdmav2.so files and it should
install libmlx5-rdmav2.so among them.

If we move libmlx5.so to libibverbs-dev, how will symlink be created?
What will user see after installing libibverbs + ibverbs-provider?
And what will be in case of libibverbs-dev + ibverbs-provider?

Thanks

>
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH V1 rdma-core 7/7] mlx5: Export mlx5 direct verbs interface

[Jason Gunthorpe]

On Tue, Feb 14, 2017 at 07:52:19AM +0200, Leon Romanovsky wrote:

> Sorry, but I'm still missing how mlx5 installation will be handled.
> 
> The ibverbs-provider installs different *-rdmav2.so files and it should
> install libmlx5-rdmav2.so among them.

No, the symlink from -rdmav2.so needs to go to a real file not another
symlink.

> If we move libmlx5.so to libibverbs-dev, how will symlink be
> created?

libmlx5.so is the 'devel' symlink, it is only used by the compiler
during linking which is why it must be placed in the -dev package.

> What will user see after installing libibverbs + ibverbs-provider?

libmlx5-rdmav2.so should be ../libmlx5.so.1.0.13

> And what will be in case of libibverbs-dev + ibverbs-provider?

Then you get libmlx5.so -> libmlx5.so.1.0.13 and the man pages and
header files. libibverbs-dev will need to depend on the
ibverbs-provider package also.

This is all also true for RPM systems.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html