[PATCH libibverbs 0/3] Add cross-channel support

[PATCH libibverbs 0/3] Add cross-channel support

[Leon Romanovsky]

From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

This patchset adds supplementary part of cross-channel support [1]
to libibverbs.

These patches were added on top of "Completion timestamping" [2]
and "Expose QP block self multicast loopback creation flag" [3]
series.

[1] http://thread.gmane.org/gmane.linux.drivers.rdma/31715
[2] https://www.mail-archive.com/linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg28895.html
[3] http://comments.gmane.org/gmane.linux.drivers.rdma/30158

Leon Romanovsky (3):
  libibverbs: Add CQ ignore overrun creation flag
  libibverbs: Add cross-channel QP initialization flags
  libibverbs: Export cross-channel capability flag

 include/infiniband/verbs.h | 11 ++++++++---
 man/ibv_create_cq_ex.3     | 10 ++++++++--
 src/cmd.c                  |  5 ++++-
 3 files changed, 20 insertions(+), 6 deletions(-)

-- 
1.7.12.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH libibverbs 1/3] libibverbs: Add CQ ignore overrun creation flag

[Leon Romanovsky]

From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

A CQ overrun is checked while posting a completion, and if
encountered, the QP is transferred to the appropriate error
state.

CQ update (and error discovery) are not synchronized with
WQE execution. Thus, a positive acknowledgement can be sent
to the remote request, and subsequently a CQ overrun may be
detected. Hardware may not write the CQE as a result of the
error, and the CQ will be transitioned to an error state as
well as the QP that caused the CQ error.

To avoid these errors, software usually ensures that the
number of WQEs posted to a QP does not exceed the number
of entries in the CQ associated with this QP. When multiple
QPs post completions into the same CQ, this check is cumulative.

CQ ignore overrun flag instructs the HW doesn't ring doorbells
on updates of CQ consumer counter (poll for completion) or
request completion notifications (Arm CQ).

Signed-off-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 include/infiniband/verbs.h |  7 ++++---
 man/ibv_create_cq_ex.3     | 10 ++++++++--
 2 files changed, 12 insertions(+), 5 deletions(-)

diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h
index c3e863850d4e..d8069d52ae13 100644
--- a/include/infiniband/verbs.h
+++ b/include/infiniband/verbs.h
@@ -1201,12 +1201,13 @@ struct ibv_context {
 };
 
 enum ibv_create_cq_attr {
-	IBV_CREATE_CQ_ATTR_FLAGS	= 1 << 0,
-	IBV_CREATE_CQ_ATTR_RESERVED	= 1 << 1
+	IBV_CREATE_CQ_ATTR_FLAGS        = 1 << 0,
+	IBV_CREATE_CQ_ATTR_RESERVED     = 1 << 1
 };
 
 enum ibv_create_cq_attr_flags {
-	IBV_CREATE_CQ_ATTR_COMPLETION_TIMESTAMP	= 1 << 0,
+	IBV_CREATE_CQ_ATTR_COMPLETION_TIMESTAMP = 1 << 0,
+	IBV_CREATE_CQ_ATTR_IGNORE_OVERRUN       = 1 << 1
 };
 
 struct ibv_create_cq_attr_ex {
diff --git a/man/ibv_create_cq_ex.3 b/man/ibv_create_cq_ex.3
index 9f9e049b0d43..8670bd0a828b 100644
--- a/man/ibv_create_cq_ex.3
+++ b/man/ibv_create_cq_ex.3
@@ -42,13 +42,19 @@ enum ibv_wc_flags_ex {
         IBV_WC_EX_WITH_SLID                  = 1 << 7,  /* Require slid in WC */
         IBV_WC_EX_WITH_SL                    = 1 << 8,  /* Require sl in WC */
         IBV_WC_EX_WITH_DLID_PATH_BITS        = 1 << 9,  /* Require dlid path bits in WC */
-        IBV_WC_EX_WITH_COMPLETION_TIMESTAMP  = 1 << 10, /* Require completion timestamp in WC /*
+        IBV_WC_EX_WITH_COMPLETION_TIMESTAMP  = 1 << 10, /* Require completion timestamp in WC */
 };
 
+enum ibv_create_cq_attr_flags {
+	IBV_CREATE_CQ_ATTR_COMPLETION_TIMESTAMP	= 1 << 0,
+	IBV_CREATE_CQ_ATTR_IGNORE_OVERRUN	= 1 << 1
+};
 
 enum ibv_create_cq_attr {
-        IBV_CREATE_CQ_ATTR_FLAGS             = 1 << 0,
+	IBV_CREATE_CQ_ATTR_FLAGS	= 1 << 0,
+	IBV_CREATE_CQ_ATTR_RESERVED	= 1 << 1
 };
+
 .SH "RETURN VALUE"
 .B ibv_create_cq_ex()
 returns a pointer to the CQ, or NULL if the request fails.
-- 
1.7.12.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH libibverbs 2/3] libibverbs: Add cross-channel QP initialization flags

[Leon Romanovsky]

From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

The cross-channel feature allows to execute WQEs that involve
synchronization of I/O operations' on different QPs.

These QPs should be configured to work as a "sync master queue" or
“sync slave queues”.

The QP property flags to indicate if queues are slave or master:
   * IB_QP_CREATE_MANAGED_SEND indicates that posted send work requests
     will not be executed immediately and requires enabling.
   * IB_QP_CREATE_MANAGED_RECV indicates that posted receive work
     requests will not be executed immediately and requires enabling.
   * IB_QP_CREATE_CROSS_CHANNEL declares the QP to work in cross-channel
     mode. If IB_QP_CREATE_MANAGED_SEND and IB_QP_CREATE_MANAGED_RECV are
     not provided, this QP will be sync master queue, else it will be sync
     slave.

Signed-off-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 include/infiniband/verbs.h | 3 +++
 src/cmd.c                  | 5 ++++-
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h
index d8069d52ae13..2b9b50c87f45 100644
--- a/include/infiniband/verbs.h
+++ b/include/infiniband/verbs.h
@@ -769,6 +769,9 @@ enum ibv_qp_init_attr_mask {
 
 enum ibv_qp_create_flags {
 	IBV_QP_CREATE_BLOCK_SELF_MCAST_LB       = 1 << 1,
+	IBV_QP_CREATE_CROSS_CHANNEL		= 1 << 2,
+	IBV_QP_CREATE_MANAGED_SEND		= 1 << 3,
+	IBV_QP_CREATE_MANAGED_RECV		= 1 << 4
 };
 
 struct ibv_qp_init_attr_ex {
diff --git a/src/cmd.c b/src/cmd.c
index 675777a8ee5a..ae33491befb4 100644
--- a/src/cmd.c
+++ b/src/cmd.c
@@ -815,7 +815,10 @@ static void create_qp_handle_resp_common(struct ibv_context *context,
 }
 
 enum {
-	CREATE_QP_EX2_SUP_CREATE_FLAGS = IBV_QP_CREATE_BLOCK_SELF_MCAST_LB,
+	CREATE_QP_EX2_SUP_CREATE_FLAGS = IBV_QP_CREATE_BLOCK_SELF_MCAST_LB |
+		IBV_QP_CREATE_CROSS_CHANNEL |
+		IBV_QP_CREATE_MANAGED_SEND |
+		IBV_QP_CREATE_MANAGED_RECV
 };
 
 int ibv_cmd_create_qp_ex2(struct ibv_context *context,
-- 
1.7.12.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH libibverbs 3/3] libibverbs: Export cross-channel capability flag

[Leon Romanovsky]

From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

The device implements capability to execute complex
I/O operations across multiple I/O channels - synchronize
execution of I/O operations on one I/O channel with
execution of I/O operations on another channel(s).

Cross-channel operations support is indicated by the HCA_CAP.cd.

Signed-off-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 include/infiniband/verbs.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h
index 2b9b50c87f45..879f4cca8173 100644
--- a/include/infiniband/verbs.h
+++ b/include/infiniband/verbs.h
@@ -121,6 +121,7 @@ enum ibv_device_cap_flags {
 	IBV_DEVICE_XRC			= 1 << 20,
 	IBV_DEVICE_RC_IP_CSUM		= 1 << 25,
 	IBV_DEVICE_RAW_IP_CSUM		= 1 << 26,
+	IBV_DEVICE_CROSS_CHANNEL	= 1 << 27,
 	IBV_DEVICE_MANAGED_FLOW_STEERING = 1 << 29
 };
 
-- 
1.7.12.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH libibverbs 2/3] libibverbs: Add cross-channel QP initialization flags

[Or Gerlitz]

On 12/20/2015 1:22 PM, Leon Romanovsky wrote:
> ---
>   include/infiniband/verbs.h | 3 +++
>   src/cmd.c                  | 5 ++++-
>   2 files changed, 7 insertions(+), 1 deletion(-)

You need to document the new creation flags in a man page. If this 
doesn't exist, this should be added to Eran's submission and you will 
enhance that.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH libibverbs 0/3] Add cross-channel support

[Jason Gunthorpe]

On Sun, Dec 20, 2015 at 01:22:41PM +0200, Leon Romanovsky wrote:
> From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> 
> This patchset adds supplementary part of cross-channel support [1]
> to libibverbs.

All of this needs some kind of man page update

And considering this is a non-standard feature it had better be a
really good man page with examples and so forth.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html