[PATCH V1 rdma-core 0/3] Verbs support for source QPN

[PATCH V1 rdma-core 0/3] Verbs support for source QPN

[Yishai Hadas]

This patch set is the supplementary part of the kernel series that was accepted
upstream into 4.14.

It enables user space applications to create a QP with a given source QP
number. The created QP will use the source QPN as its wire QP number.

This functionality is implemented by mlx5 driver as part of this series to
allow user space applications to accelerate traffic which is typically handled
by IPoIB ULP.

PR was sent:
https://github.com/linux-rdma/rdma-core/pull/212

Changes from V0:
Patch #3:
- Improve commit log to better describe the general case.
- Rename MLX5_IPOIB_INLINE_MAX_HEADER_SIZE to more general name as it really is.

Yishai Hadas (3):
  verbs: Enable creating QP with a given source QP number
  mlx5: Add support for managing QP with a given source QP number
  mlx5: Add support for sending UD packets with source QPN

 libibverbs/cmd.c                  |  6 +++-
 libibverbs/kern-abi.h             |  2 +-
 libibverbs/man/ibv_create_qp_ex.3 |  1 +
 libibverbs/verbs.h                |  2 ++
 providers/mlx5/mlx5.h             |  8 ++++-
 providers/mlx5/qp.c               | 68 ++++++++++++++++++++++++++++++++++++++-
 providers/mlx5/verbs.c            | 46 +++++++++++++++++++++-----
 providers/mlx5/wqe.h              |  6 ++++
 8 files changed, 127 insertions(+), 12 deletions(-)

-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V1 rdma-core 1/3] verbs: Enable creating QP with a given source QP number

[Yishai Hadas]

Enable QP creation with a given source QP number.
The created QP will use the source QPN as its wire QP number.

This comes as a pre-patch for downstream patches in this series to
allow user space applications to accelerate traffic which is
typically handled by IPoIB ULP.

In the IPoIB use case, the source QPN can be achieved by using some
networking tool as of ifconfig, ip link show on the interface and use
the value of 3 bytes from the hardware address starting after the first
byte.

Signed-off-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 libibverbs/cmd.c                  | 6 +++++-
 libibverbs/kern-abi.h             | 2 +-
 libibverbs/man/ibv_create_qp_ex.3 | 1 +
 libibverbs/verbs.h                | 2 ++
 4 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/libibverbs/cmd.c b/libibverbs/cmd.c
index 7a266ee..9d2140f 100644
--- a/libibverbs/cmd.c
+++ b/libibverbs/cmd.c
@@ -927,7 +927,8 @@ static void create_qp_handle_resp_common(struct ibv_context *context,
 enum {
 	CREATE_QP_EX2_SUP_CREATE_FLAGS = IBV_QP_CREATE_BLOCK_SELF_MCAST_LB |
 					 IBV_QP_CREATE_SCATTER_FCS |
-					 IBV_QP_CREATE_CVLAN_STRIPPING,
+					 IBV_QP_CREATE_CVLAN_STRIPPING |
+					 IBV_QP_CREATE_SOURCE_QPN,
 };
 
 int ibv_cmd_create_qp_ex2(struct ibv_context *context,
@@ -967,6 +968,9 @@ int ibv_cmd_create_qp_ex2(struct ibv_context *context,
 				    sizeof(qp_attr->create_flags))
 			return EINVAL;
 		cmd->create_flags = qp_attr->create_flags;
+
+		if (qp_attr->create_flags & IBV_QP_CREATE_SOURCE_QPN)
+			cmd->source_qpn = qp_attr->source_qpn;
 	}
 
 	if (qp_attr->comp_mask & IBV_QP_INIT_ATTR_IND_TABLE) {
diff --git a/libibverbs/kern-abi.h b/libibverbs/kern-abi.h
index 8d42faf..3a0e02a 100644
--- a/libibverbs/kern-abi.h
+++ b/libibverbs/kern-abi.h
@@ -679,7 +679,7 @@ struct ibv_create_qp_ex {
 	__u32 comp_mask;
 	__u32 create_flags;
 	__u32 ind_tbl_handle;
-	__u32 reserved1;
+	__u32 source_qpn;
 };
 
 struct ibv_create_qp_resp_ex {
diff --git a/libibverbs/man/ibv_create_qp_ex.3 b/libibverbs/man/ibv_create_qp_ex.3
index 99ae497..3272c09 100644
--- a/libibverbs/man/ibv_create_qp_ex.3
+++ b/libibverbs/man/ibv_create_qp_ex.3
@@ -38,6 +38,7 @@ enum ibv_qp_create_flags create_flags;	/* Creation flags for this QP */
 uint16_t                max_tso_header; /* Maximum TSO header size */
 struct ibv_rwq_ind_table *rwq_ind_tbl;  /* Indirection table to be associated with the QP */
 struct ibv_rx_hash_conf  rx_hash_conf;  /* RX hash configuration to be used */
+uint32_t                source_qpn;     /* Source QP number, creation flag IBV_QP_CREATE_SOURCE_QPN should be set */
 .in -8
 };
 .sp
diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h
index 922a011..3c8b2c9 100644
--- a/libibverbs/verbs.h
+++ b/libibverbs/verbs.h
@@ -790,6 +790,7 @@ enum ibv_qp_create_flags {
 	IBV_QP_CREATE_BLOCK_SELF_MCAST_LB	= 1 << 1,
 	IBV_QP_CREATE_SCATTER_FCS		= 1 << 8,
 	IBV_QP_CREATE_CVLAN_STRIPPING		= 1 << 9,
+	IBV_QP_CREATE_SOURCE_QPN		= 1 << 10,
 };
 
 struct ibv_rx_hash_conf {
@@ -817,6 +818,7 @@ struct ibv_qp_init_attr_ex {
 	uint16_t		max_tso_header;
 	struct ibv_rwq_ind_table       *rwq_ind_tbl;
 	struct ibv_rx_hash_conf	rx_hash_conf;
+	uint32_t		source_qpn;
 };
 
 enum ibv_qp_open_attr_mask {
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V1 rdma-core 2/3] mlx5: Add support for managing QP with a given source QP number

[Yishai Hadas]

Add support to create a QP with source QPN (e.g. IPoIB QPN) over UD QP.

It includes:
- Input checking that basic QP type is UD.
- Create and manage driver resources as done for RAW QP.

Signed-off-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 providers/mlx5/mlx5.h  |  8 +++++++-
 providers/mlx5/qp.c    |  3 ++-
 providers/mlx5/verbs.c | 36 +++++++++++++++++++++++++++++++-----
 3 files changed, 40 insertions(+), 7 deletions(-)

diff --git a/providers/mlx5/mlx5.h b/providers/mlx5/mlx5.h
index ad36cbf..ae39d17 100644
--- a/providers/mlx5/mlx5.h
+++ b/providers/mlx5/mlx5.h
@@ -150,7 +150,8 @@ enum {
 };
 
 enum {
-	MLX5_CSUM_SUPPORT_RAW_OVER_ETH  = (1 <<  0),
+	MLX5_CSUM_SUPPORT_RAW_OVER_ETH  = (1 << 0),
+	MLX5_CSUM_SUPPORT_UNDERLAY_UD   = (1 << 1),
 	/*
 	 * Only report rx checksum when the validation
 	 * is valid.
@@ -396,6 +397,10 @@ struct mlx5_mr {
 	uint32_t			alloc_flags;
 };
 
+enum mlx5_qp_flags {
+	MLX5_QP_FLAGS_USE_UNDERLAY = 0x01,
+};
+
 struct mlx5_qp {
 	struct mlx5_resource            rsc; /* This struct must be first */
 	struct verbs_qp			verbs_qp;
@@ -421,6 +426,7 @@ struct mlx5_qp {
 	uint32_t			max_tso;
 	uint16_t			max_tso_header;
 	int                             rss_qp;
+	uint32_t			flags; /* Use enum mlx5_qp_flags */
 };
 
 struct mlx5_ah {
diff --git a/providers/mlx5/qp.c b/providers/mlx5/qp.c
index 52da8c1..2960ba0 100644
--- a/providers/mlx5/qp.c
+++ b/providers/mlx5/qp.c
@@ -1199,7 +1199,8 @@ out:
 		 * This is only for Raw Packet QPs since they are represented
 		 * differently in the hardware.
 		 */
-		if (likely(!(ibqp->qp_type == IBV_QPT_RAW_PACKET &&
+		if (likely(!((ibqp->qp_type == IBV_QPT_RAW_PACKET ||
+			      qp->flags & MLX5_QP_FLAGS_USE_UNDERLAY) &&
 			     ibqp->state < IBV_QPS_RTR)))
 			qp->db[MLX5_RCV_DBR] = htobe32(qp->rq.head & 0xffff);
 	}
diff --git a/providers/mlx5/verbs.c b/providers/mlx5/verbs.c
index fc63ae9..a469c93 100644
--- a/providers/mlx5/verbs.c
+++ b/providers/mlx5/verbs.c
@@ -1096,7 +1096,8 @@ static int mlx5_alloc_qp_buf(struct ibv_context *context,
 
 	memset(qp->buf.buf, 0, qp->buf_size);
 
-	if (attr->qp_type == IBV_QPT_RAW_PACKET) {
+	if (attr->qp_type == IBV_QPT_RAW_PACKET ||
+	    qp->flags & MLX5_QP_FLAGS_USE_UNDERLAY) {
 		size_t aligned_sq_buf_size = align(qp->sq_buf_size,
 						   to_mdev(context->device)->page_size);
 		/* For Raw Packet QP, allocate a separate buffer for the SQ */
@@ -1257,6 +1258,17 @@ static struct ibv_qp *create_qp(struct ibv_context *context,
 	ibqp = (struct ibv_qp *)&qp->verbs_qp;
 	qp->ibv_qp = ibqp;
 
+	if ((attr->comp_mask & IBV_QP_INIT_ATTR_CREATE_FLAGS) &&
+		(attr->create_flags & IBV_QP_CREATE_SOURCE_QPN)) {
+
+		if (attr->qp_type != IBV_QPT_UD) {
+			errno = EINVAL;
+			goto err;
+		}
+
+		qp->flags |= MLX5_QP_FLAGS_USE_UNDERLAY;
+	}
+
 	memset(&cmd, 0, sizeof(cmd));
 	memset(&resp, 0, sizeof(resp));
 	memset(&resp_ex, 0, sizeof(resp_ex));
@@ -1282,7 +1294,8 @@ static struct ibv_qp *create_qp(struct ibv_context *context,
 		goto err;
 	}
 
-	if (attr->qp_type == IBV_QPT_RAW_PACKET) {
+	if (attr->qp_type == IBV_QPT_RAW_PACKET ||
+	    qp->flags & MLX5_QP_FLAGS_USE_UNDERLAY) {
 		qp->buf_size = qp->sq.offset;
 		qp->sq_buf_size = ret - qp->buf_size;
 		qp->sq.offset = 0;
@@ -1296,7 +1309,8 @@ static struct ibv_qp *create_qp(struct ibv_context *context,
 		goto err;
 	}
 
-	if (attr->qp_type == IBV_QPT_RAW_PACKET) {
+	if (attr->qp_type == IBV_QPT_RAW_PACKET ||
+	    qp->flags & MLX5_QP_FLAGS_USE_UNDERLAY) {
 		qp->sq_start = qp->sq_buf.buf;
 		qp->sq.qend = qp->sq_buf.buf +
 				(qp->sq.wqe_cnt << qp->sq.wqe_shift);
@@ -1322,7 +1336,8 @@ static struct ibv_qp *create_qp(struct ibv_context *context,
 	qp->db[MLX5_SND_DBR] = 0;
 
 	cmd.buf_addr = (uintptr_t) qp->buf.buf;
-	cmd.sq_buf_addr = (attr->qp_type == IBV_QPT_RAW_PACKET) ?
+	cmd.sq_buf_addr = (attr->qp_type == IBV_QPT_RAW_PACKET ||
+			   qp->flags & MLX5_QP_FLAGS_USE_UNDERLAY) ?
 			  (uintptr_t) qp->sq_buf.buf : 0;
 	cmd.db_addr  = (uintptr_t) qp->db;
 	cmd.sq_wqe_count = qp->sq.wqe_cnt;
@@ -1560,6 +1575,16 @@ int mlx5_modify_qp(struct ibv_qp *qp, struct ibv_qp_attr *attr,
 	if (mqp->rss_qp)
 		return ENOSYS;
 
+	if (mqp->flags & MLX5_QP_FLAGS_USE_UNDERLAY) {
+		if (attr_mask & ~(IBV_QP_STATE | IBV_QP_CUR_STATE))
+			return EINVAL;
+
+		/* Underlay QP is UD over infiniband */
+		if (context->cached_device_cap_flags & IBV_DEVICE_UD_IP_CSUM)
+			mqp->qp_cap_cache |= MLX5_CSUM_SUPPORT_UNDERLAY_UD |
+					     MLX5_RX_CSUM_VALID;
+	}
+
 	if (attr_mask & IBV_QP_PORT) {
 		switch (qp->qp_type) {
 		case IBV_QPT_RAW_PACKET:
@@ -1621,7 +1646,8 @@ int mlx5_modify_qp(struct ibv_qp *qp, struct ibv_qp_attr *attr,
 	if (!ret &&
 	    (attr_mask & IBV_QP_STATE) &&
 	    attr->qp_state == IBV_QPS_RTR &&
-	    qp->qp_type == IBV_QPT_RAW_PACKET) {
+	    (qp->qp_type == IBV_QPT_RAW_PACKET ||
+	     mqp->flags & MLX5_QP_FLAGS_USE_UNDERLAY)) {
 		mlx5_spin_lock(&mqp->rq.lock);
 		mqp->db[MLX5_RCV_DBR] = htobe32(mqp->rq.head & 0xffff);
 		mlx5_spin_unlock(&mqp->rq.lock);
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V1 rdma-core 3/3] mlx5: Add support for sending UD packets with source QPN

[Yishai Hadas]

It includes:
- Update WQE size to match the addition required segments.
- Build a proper packet as part of the post send flow.

The specific IPoIB handling as of copying from the application gather
data into the header is some device limitation to support sending IPoIB
packets. This is not required in the general case but will work as well.
Future devices may work properly without that need, once be ready we may
have some device capability exposed from kernel and clean this specific
handling.

Signed-off-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 providers/mlx5/qp.c    | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++
 providers/mlx5/verbs.c | 10 +++++---
 providers/mlx5/wqe.h   |  6 +++++
 3 files changed, 78 insertions(+), 3 deletions(-)

diff --git a/providers/mlx5/qp.c b/providers/mlx5/qp.c
index 2960ba0..d5e677f 100644
--- a/providers/mlx5/qp.c
+++ b/providers/mlx5/qp.c
@@ -604,6 +604,63 @@ static inline int set_tso_eth_seg(void **seg, struct ibv_send_wr *wr,
 	return 0;
 }
 
+static inline int mlx5_post_send_underlay(struct mlx5_qp *qp, struct ibv_send_wr *wr,
+					  void **pseg, int *total_size,
+					  struct mlx5_sg_copy_ptr *sg_copy_ptr)
+{
+	struct mlx5_wqe_eth_seg *eseg;
+	int inl_hdr_copy_size;
+	void *seg = *pseg;
+	int size = 0;
+
+	if (unlikely(wr->opcode == IBV_WR_SEND_WITH_IMM))
+		return EINVAL;
+
+	memset(seg, 0, sizeof(struct mlx5_wqe_eth_pad));
+	size += sizeof(struct mlx5_wqe_eth_pad);
+	seg += sizeof(struct mlx5_wqe_eth_pad);
+	eseg = seg;
+	*((uint64_t *)eseg) = 0;
+	eseg->rsvd2 = 0;
+
+	if (wr->send_flags & IBV_SEND_IP_CSUM) {
+		if (!(qp->qp_cap_cache & MLX5_CSUM_SUPPORT_UNDERLAY_UD))
+			return EINVAL;
+
+		eseg->cs_flags |= MLX5_ETH_WQE_L3_CSUM | MLX5_ETH_WQE_L4_CSUM;
+	}
+
+	if (likely(wr->sg_list[0].length >= MLX5_SOURCE_QPN_INLINE_MAX_HEADER_SIZE))
+		/* Copying the minimum required data unless inline mode is set */
+		inl_hdr_copy_size = (wr->send_flags & IBV_SEND_INLINE) ?
+				MLX5_SOURCE_QPN_INLINE_MAX_HEADER_SIZE :
+				MLX5_IPOIB_INLINE_MIN_HEADER_SIZE;
+	else {
+		inl_hdr_copy_size = MLX5_IPOIB_INLINE_MIN_HEADER_SIZE;
+		/* We expect at least 4 bytes as part of first entry to hold the IPoIB header */
+		if (unlikely(wr->sg_list[0].length < inl_hdr_copy_size))
+			return EINVAL;
+	}
+
+	memcpy(eseg->inline_hdr_start, (void *)(uintptr_t)wr->sg_list[0].addr,
+	       inl_hdr_copy_size);
+	eseg->inline_hdr_sz = htobe16(inl_hdr_copy_size);
+	size += sizeof(struct mlx5_wqe_eth_seg);
+	seg += sizeof(struct mlx5_wqe_eth_seg);
+
+	/* If we copied all the sge into the inline-headers, then we need to
+	 * start copying from the next sge into the data-segment.
+	 */
+	if (unlikely(wr->sg_list[0].length == inl_hdr_copy_size))
+		sg_copy_ptr->index++;
+	else
+		sg_copy_ptr->offset = inl_hdr_copy_size;
+
+	*pseg = seg;
+	*total_size += (size / 16);
+	return 0;
+}
+
 static inline int _mlx5_post_send(struct ibv_qp *ibqp, struct ibv_send_wr *wr,
 				  struct ibv_send_wr **bad_wr)
 {
@@ -806,6 +863,14 @@ static inline int _mlx5_post_send(struct ibv_qp *ibqp, struct ibv_send_wr *wr,
 			size += sizeof(struct mlx5_wqe_datagram_seg) / 16;
 			if (unlikely((seg == qend)))
 				seg = mlx5_get_send_wqe(qp, 0);
+
+			if (unlikely(qp->flags & MLX5_QP_FLAGS_USE_UNDERLAY)) {
+				err = mlx5_post_send_underlay(qp, wr, &seg, &size, &sg_copy_ptr);
+				if (unlikely(err)) {
+					*bad_wr = wr;
+					goto out;
+				}
+			}
 			break;
 
 		case IBV_QPT_RAW_PACKET:
diff --git a/providers/mlx5/verbs.c b/providers/mlx5/verbs.c
index a469c93..5a360a9 100644
--- a/providers/mlx5/verbs.c
+++ b/providers/mlx5/verbs.c
@@ -756,7 +756,7 @@ int mlx5_destroy_srq(struct ibv_srq *srq)
 	return 0;
 }
 
-static int sq_overhead(enum ibv_qp_type	qp_type)
+static int sq_overhead(struct mlx5_qp *qp, enum ibv_qp_type qp_type)
 {
 	size_t size = 0;
 	size_t mw_bind_size =
@@ -781,6 +781,10 @@ static int sq_overhead(enum ibv_qp_type	qp_type)
 	case IBV_QPT_UD:
 		size = sizeof(struct mlx5_wqe_ctrl_seg) +
 			sizeof(struct mlx5_wqe_datagram_seg);
+
+		if (qp->flags & MLX5_QP_FLAGS_USE_UNDERLAY)
+			size += (sizeof(struct mlx5_wqe_eth_seg) + sizeof(struct mlx5_wqe_eth_pad));
+
 		break;
 
 	case IBV_QPT_XRC_SEND:
@@ -814,7 +818,7 @@ static int mlx5_calc_send_wqe(struct mlx5_context *ctx,
 	int max_gather;
 	int tot_size;
 
-	size = sq_overhead(attr->qp_type);
+	size = sq_overhead(qp, attr->qp_type);
 	if (size < 0)
 		return size;
 
@@ -887,7 +891,7 @@ static int mlx5_calc_sq_size(struct mlx5_context *ctx,
 		return -EINVAL;
 	}
 
-	qp->max_inline_data = wqe_size - sq_overhead(attr->qp_type) -
+	qp->max_inline_data = wqe_size - sq_overhead(qp, attr->qp_type) -
 		sizeof(struct mlx5_wqe_inl_data_seg);
 	attr->cap.max_inline_data = qp->max_inline_data;
 
diff --git a/providers/mlx5/wqe.h b/providers/mlx5/wqe.h
index 063dc9a..5bf536c 100644
--- a/providers/mlx5/wqe.h
+++ b/providers/mlx5/wqe.h
@@ -50,6 +50,10 @@ struct mlx5_eqe_qp_srq {
 	uint32_t	qp_srq_n;
 };
 
+struct mlx5_wqe_eth_pad {
+	uint8_t rsvd0[16];
+};
+
 struct mlx5_wqe_xrc_seg {
 	__be32		xrc_srqn;
 	uint8_t		rsvd[12];
@@ -63,6 +67,8 @@ struct mlx5_wqe_masked_atomic_seg {
 };
 
 enum {
+	MLX5_IPOIB_INLINE_MIN_HEADER_SIZE	= 4,
+	MLX5_SOURCE_QPN_INLINE_MAX_HEADER_SIZE	= 18,
 	MLX5_ETH_L2_INLINE_HEADER_SIZE	= 18,
 	MLX5_ETH_L2_MIN_HEADER_SIZE	= 14,
 };
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 rdma-core 1/3] verbs: Enable creating QP with a given source QP number

[Jason Gunthorpe]

On Thu, Sep 07, 2017 at 03:12:05PM +0300, Yishai Hadas wrote:
> diff --git a/libibverbs/man/ibv_create_qp_ex.3 b/libibverbs/man/ibv_create_qp_ex.3
> index 99ae497..3272c09 100644
> +++ b/libibverbs/man/ibv_create_qp_ex.3
> @@ -38,6 +38,7 @@ enum ibv_qp_create_flags create_flags;	/* Creation flags for this QP */
>  uint16_t                max_tso_header; /* Maximum TSO header size */
>  struct ibv_rwq_ind_table *rwq_ind_tbl;  /* Indirection table to be associated with the QP */
>  struct ibv_rx_hash_conf  rx_hash_conf;  /* RX hash configuration to be used */
> +uint32_t                source_qpn;     /* Source QP number, creation flag IBV_QP_CREATE_SOURCE_QPN should be set */

I think more discussion in the man page is required for this
feature. Eg that it is a SEND ONLY QP and only works for UD.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 rdma-core 1/3] verbs: Enable creating QP with a given source QP number

[Alex Rosenbaum]

On Thu, Sep 7, 2017 at 6:16 PM, Jason Gunthorpe
<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> On Thu, Sep 07, 2017 at 03:12:05PM +0300, Yishai Hadas wrote:
>> +uint32_t                source_qpn;     /* Source QP number, creation flag IBV_QP_CREATE_SOURCE_QPN should be set */
>
> I think more discussion in the man page is required for this
> feature. Eg that it is a SEND ONLY QP and only works for UD.

Verbs UD QP can handle receive of ingress packets sent with
BTH.DestQPN = ibv_qp->qp_num. This is applicable also in this case.

Also, future implementation might include RC source qpn use case. This
can be accomplished by setting the HCA's QPN to be the same as the
'source qpn' value, and returning that to the user.

We prefer not to limit the API definition in the man page.

Alex
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 rdma-core 1/3] verbs: Enable creating QP with a given source QP number

[Jason Gunthorpe]

On Thu, Sep 07, 2017 at 09:46:39PM +0300, Alex Rosenbaum wrote:
> On Thu, Sep 7, 2017 at 6:16 PM, Jason Gunthorpe
> <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> > On Thu, Sep 07, 2017 at 03:12:05PM +0300, Yishai Hadas wrote:
> >> +uint32_t                source_qpn;     /* Source QP number, creation flag IBV_QP_CREATE_SOURCE_QPN should be set */
> >
> > I think more discussion in the man page is required for this
> > feature. Eg that it is a SEND ONLY QP and only works for UD.
> 
> Verbs UD QP can handle receive of ingress packets sent with
> BTH.DestQPN = ibv_qp->qp_num. This is applicable also in this case.
> 
> Also, future implementation might include RC source qpn use case. This
> can be accomplished by setting the HCA's QPN to be the same as the
> 'source qpn' value, and returning that to the user.
> 
> We prefer not to limit the API definition in the man page.

Well, you have to define what the API does, specifically.

If you expect it to RX then you need to specify that. Does it steal
the traffic? Does it get a copy? Do you need to use flow steering to
do that?

Same for RC.

There is nothing wrong with specifying the API limit that exist today,
it helps people understand what the feature is and how to use it.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 rdma-core 1/3] verbs: Enable creating QP with a given source QP number

[Alex Rosenbaum]

On Thu, Sep 7, 2017 at 10:29 PM, Jason Gunthorpe
<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> On Thu, Sep 07, 2017 at 09:46:39PM +0300, Alex Rosenbaum wrote:
>> On Thu, Sep 7, 2017 at 6:16 PM, Jason Gunthorpe
>> <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
>> > On Thu, Sep 07, 2017 at 03:12:05PM +0300, Yishai Hadas wrote:
>> >> +uint32_t                source_qpn;     /* Source QP number, creation flag IBV_QP_CREATE_SOURCE_QPN should be set */
>> >
>> > I think more discussion in the man page is required for this
>> > feature. Eg that it is a SEND ONLY QP and only works for UD.
>>
>> Verbs UD QP can handle receive of ingress packets sent with
>> BTH.DestQPN = ibv_qp->qp_num. This is applicable also in this case.
>>
>> Also, future implementation might include RC source qpn use case. This
>> can be accomplished by setting the HCA's QPN to be the same as the
>> 'source qpn' value, and returning that to the user.
>>
>> We prefer not to limit the API definition in the man page.
>
> Well, you have to define what the API does, specifically.
>
> If you expect it to RX then you need to specify that.

Like any other QP, this too can have a max_recv_wr defined by user. I
don't think we need to specifically mention it, because it is not a
limition.

> Does it steal
> the traffic? Does it get a copy? Do you need to use flow steering to
> do that?

Yes, this QP can support flow steering based on
IBV_DEVICE_MANAGED_FLOW_STEERING.

> Same for RC.
>
> There is nothing wrong with specifying the API limit that exist today,
> it helps people understand what the feature is and how to use it.

RC is currently not supported. Do you prefer we update the man page
with the UD limitation? and once RC or other limitation are removed we
update it again?

Alex
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 rdma-core 1/3] verbs: Enable creating QP with a given source QP number

[Jason Gunthorpe]

On Sun, Sep 10, 2017 at 09:54:45AM +0300, Alex Rosenbaum wrote:
> On Thu, Sep 7, 2017 at 10:29 PM, Jason Gunthorpe
> <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> > On Thu, Sep 07, 2017 at 09:46:39PM +0300, Alex Rosenbaum wrote:
> >> On Thu, Sep 7, 2017 at 6:16 PM, Jason Gunthorpe
> >> <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> >> > On Thu, Sep 07, 2017 at 03:12:05PM +0300, Yishai Hadas wrote:
> >> >> +uint32_t                source_qpn;     /* Source QP number, creation flag IBV_QP_CREATE_SOURCE_QPN should be set */
> >> >
> >> > I think more discussion in the man page is required for this
> >> > feature. Eg that it is a SEND ONLY QP and only works for UD.
> >>
> >> Verbs UD QP can handle receive of ingress packets sent with
> >> BTH.DestQPN = ibv_qp->qp_num. This is applicable also in this case.
> >>
> >> Also, future implementation might include RC source qpn use case. This
> >> can be accomplished by setting the HCA's QPN to be the same as the
> >> 'source qpn' value, and returning that to the user.
> >>
> >> We prefer not to limit the API definition in the man page.
> >
> > Well, you have to define what the API does, specifically.
> >
> > If you expect it to RX then you need to specify that.
> 
> Like any other QP, this too can have a max_recv_wr defined by user. I
> don't think we need to specifically mention it, because it is not a
> limition.

You need to mention what impact source_qpn has on the RX, assuming the
user does nothing else.

>From what I can gather, without flow steering, RX should not be
possible.

This needs to be described.

> > There is nothing wrong with specifying the API limit that exist today,
> > it helps people understand what the feature is and how to use it.
> 
> RC is currently not supported. Do you prefer we update the man page
> with the UD limitation? and once RC or other limitation are removed we
> update it again?

Yes. Describe what it does. The man page is not a spec, it documents
what the software actually does.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 rdma-core 1/3] verbs: Enable creating QP with a given source QP number

[Yishai Hadas]

On 9/11/2017 7:28 PM, Jason Gunthorpe wrote:
> You need to mention what impact source_qpn has on the RX, assuming the
> user does nothing else.
>
> From what I can gather, without flow steering, RX should not be
> possible.
>

Correct.

> Yes. Describe what it does. The man page is not a spec, it documents
> what the software actually does.
>

Man page was updated in the PR to include above notes.

PR:
https://github.com/linux-rdma/rdma-core/pull/212
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 rdma-core 0/3] Verbs support for source QPN

[Yishai Hadas]

On 9/7/2017 3:12 PM, Yishai Hadas wrote:
> This patch set is the supplementary part of the kernel series that was accepted
> upstream into 4.14.
> 
> It enables user space applications to create a QP with a given source QP
> number. The created QP will use the source QPN as its wire QP number.
> 
> This functionality is implemented by mlx5 driver as part of this series to
> allow user space applications to accelerate traffic which is typically handled
> by IPoIB ULP.
> 
> PR was sent:
> https://github.com/linux-rdma/rdma-core/pull/212
> 
> Changes from V0:
> Patch #3:
> - Improve commit log to better describe the general case.
> - Rename MLX5_IPOIB_INLINE_MAX_HEADER_SIZE to more general name as it really is.
> 
> Yishai Hadas (3):
>    verbs: Enable creating QP with a given source QP number
>    mlx5: Add support for managing QP with a given source QP number
>    mlx5: Add support for sending UD packets with source QPN
> 
>   libibverbs/cmd.c                  |  6 +++-
>   libibverbs/kern-abi.h             |  2 +-
>   libibverbs/man/ibv_create_qp_ex.3 |  1 +
>   libibverbs/verbs.h                |  2 ++
>   providers/mlx5/mlx5.h             |  8 ++++-
>   providers/mlx5/qp.c               | 68 ++++++++++++++++++++++++++++++++++++++-
>   providers/mlx5/verbs.c            | 46 +++++++++++++++++++++-----
>   providers/mlx5/wqe.h              |  6 ++++
>   8 files changed, 127 insertions(+), 12 deletions(-)
> 

PR is up-to-date with latest notes regarding the man page.
Series was merged.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html