[PATCH V1 libibverbs 0/3] Add support for TCP segmentation offload (TSO)

[PATCH V1 libibverbs 0/3] Add support for TCP segmentation offload (TSO)

[Yishai Hadas]

Hi Doug,

This V1 series addresses a note coming from the list to replace a macro
by a static inline function, details below.

The relevant kernel patches regarding capabilities were already taken into your
tree for coming kernel of 4.8.

The series was tested successfully with mlx5 driver (lib, kernel) and can be
accessed also from my openfabrics GIT at:
git://openfabrics.org/~yishaih/libibverbs.git branch: tso_v1

Yishai

In general,
TSO enables an application to pass a packet with large chunk of data than the
MTU to the NIC and let it breaks into small packets as part of its send flow.
The TSO engine will split the packet into separate packets and insert the
relevant user specified L2/L3/L4 headers automatically.
This is a technique for increasing outbound throughput of high-bandwidth
network connections by reducing CPU overhead.

For some extra information about TSO, below link can be helpful:
https://en.wikipedia.org/wiki/Large_segment_offload

Verbs usage,
TSO traffic is performed by ibv_post_send with opcode IBV_WR_TSO.
The following information should be supplied within the associated send WR:
- A pointer to the packet header.
- Header size.
- The maximum segment size (mss) that hardware should generate in
  its TSO engine.

The above information required to send a TSO packet is similar to the IB kernel
API.  http://lxr.free-electrons.com/source/include/rdma/ib_verbs.h#L1237

The TSO capabilities as of the supported QP types, max TSO payload can be
achieved upon query device.
 
When creating a TSO eligible QP, user should supply the maximum TSO header size
in order to let providers preparing their SQ buffer accordingly. This need is
similar to other capabilities that are given as part of QP creation for that
purpose.

Changes from V0:
patch #2: Change 'ibv_is_qpt_supported' to be a static inline function instead
          of a macro.

Bodong Wang (3):
  Fix ibv_cmd_query_device_ex to return valid output
  Add support for TCP segmentation offload (TSO)
  Update man pages for TSO support

 examples/devinfo.c         | 20 ++++++++++++++++++++
 include/infiniband/verbs.h | 37 +++++++++++++++++++++++++++++--------
 man/ibv_create_qp_ex.3     |  1 +
 man/ibv_post_send.3        | 36 ++++++++++++++++++++++++------------
 man/ibv_query_device_ex.3  | 22 ++++++++++++++--------
 src/cmd.c                  |  9 +--------
 6 files changed, 89 insertions(+), 36 deletions(-)

-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V1 libibverbs 1/3] Fix ibv_cmd_query_device_ex to return valid output

[Yishai Hadas]

From: Bodong Wang <bodong-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

Make sure to return valid output by memset the extended fields to zero.
No need to deal with specific fields any more.

Currently, extended fields will be assigned to some value or 0 depending
on response from the command. When adding a new extended field, relevant
variables must be cleared if no response got from the kernel.

Signed-off-by: Bodong Wang <bodong-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 src/cmd.c | 9 +--------
 1 file changed, 1 insertion(+), 8 deletions(-)

diff --git a/src/cmd.c b/src/cmd.c
index cb9e34c..4b3304f 100644
--- a/src/cmd.c
+++ b/src/cmd.c
@@ -163,6 +163,7 @@ int ibv_cmd_query_device_ex(struct ibv_context *context,
 	cmd->comp_mask = 0;
 	cmd->reserved = 0;
 	memset(attr->orig_attr.fw_ver, 0, sizeof(attr->orig_attr.fw_ver));
+	memset(&attr->comp_mask, 0, attr_size - sizeof(attr->orig_attr));
 	err = write(context->cmd_fd, cmd, cmd_size);
 	if (err != cmd_size)
 		return errno;
@@ -184,8 +185,6 @@ int ibv_cmd_query_device_ex(struct ibv_context *context,
 				resp->odp_caps.per_transport_caps.uc_odp_caps;
 			attr->odp_caps.per_transport_caps.ud_odp_caps =
 				resp->odp_caps.per_transport_caps.ud_odp_caps;
-		} else {
-			memset(&attr->odp_caps, 0, sizeof(attr->odp_caps));
 		}
 	}
 
@@ -196,8 +195,6 @@ int ibv_cmd_query_device_ex(struct ibv_context *context,
 		    offsetof(struct ibv_query_device_resp_ex, timestamp_mask) +
 		    sizeof(resp->timestamp_mask))
 			attr->completion_timestamp_mask = resp->timestamp_mask;
-		else
-			attr->completion_timestamp_mask = 0;
 	}
 
 	if (attr_size >= offsetof(struct ibv_device_attr_ex, hca_core_clock) +
@@ -206,8 +203,6 @@ int ibv_cmd_query_device_ex(struct ibv_context *context,
 		    offsetof(struct ibv_query_device_resp_ex, hca_core_clock) +
 		    sizeof(resp->hca_core_clock))
 			attr->hca_core_clock = resp->hca_core_clock;
-		else
-			attr->hca_core_clock = 0;
 	}
 
 	if (attr_size >= offsetof(struct ibv_device_attr_ex, device_cap_flags_ex) +
@@ -216,8 +211,6 @@ int ibv_cmd_query_device_ex(struct ibv_context *context,
 		    offsetof(struct ibv_query_device_resp_ex, device_cap_flags_ex) +
 		    sizeof(resp->device_cap_flags_ex))
 			attr->device_cap_flags_ex = resp->device_cap_flags_ex;
-		else
-			attr->device_cap_flags_ex = 0;
 	}
 
 	return 0;
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V1 libibverbs 2/3] Add support for TCP segmentation offload (TSO)

[Yishai Hadas]

From: Bodong Wang <bodong-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

1) Add a structure to define a TSO packet as part of struct ibv_send_wr.
2) Add IBV_WR_TSO opcode to be used as part of post_send.
3) Add IBV_WC_TSO to be used as part of poll_cq to report a TSO completion.
4) Add IBV_QP_INIT_ATTR_MAX_TSO_HEADER to define the maximum TSO header size
   when creating a QP. This is needed to let providers prepare their SQ buffer
   to fit application's usage.
5) Report TSO capabilities when querying a device.

In order to preserve the size of ibv_send_wr structure and prevents some
performance penalty, the TSO definition was added under a union with the
memory window stuff, those options are mutual exclusive.

The TSO definition should include:
- A pointer to the packet header.
- Header size.
- The maximum segment size (mss) that the hardware should generate in
  its TSO engine.

Signed-off-by: Bodong Wang <bodong-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 examples/devinfo.c         | 20 ++++++++++++++++++++
 include/infiniband/verbs.h | 37 +++++++++++++++++++++++++++++--------
 2 files changed, 49 insertions(+), 8 deletions(-)

diff --git a/examples/devinfo.c b/examples/devinfo.c
index 915ebb3..c497650 100644
--- a/examples/devinfo.c
+++ b/examples/devinfo.c
@@ -345,6 +345,25 @@ static void print_device_cap_flags_ex(uint64_t device_cap_flags_ex)
 		       ex_flags & unknown_flags);
 }
 
+static void print_tso_caps(const struct ibv_tso_caps *caps)
+{
+	uint32_t unknown_general_caps = ~(1 << IBV_QPT_RAW_PACKET |
+					  1 << IBV_QPT_UD);
+	printf("\ttso_caps:\n");
+	printf("\tmax_tso:\t\t\t%d\n", caps->max_tso);
+
+	if (caps->max_tso) {
+		printf("\tsupported_qp:\n");
+		if (ibv_is_qpt_supported(caps->supported_qpts, IBV_QPT_RAW_PACKET))
+			printf("\t\t\t\t\tSUPPORT_RAW_PACKET\n");
+		if (ibv_is_qpt_supported(caps->supported_qpts, IBV_QPT_UD))
+			printf("\t\t\t\t\tSUPPORT_UD\n");
+		if (caps->supported_qpts & unknown_general_caps)
+			printf("\t\t\t\t\tUnknown flags: 0x%" PRIX32 "\n",
+			       caps->supported_qpts & unknown_general_caps);
+	}
+}
+
 static int print_hca_cap(struct ibv_device *ib_dev, uint8_t ib_port)
 {
 	struct ibv_context *ctx;
@@ -445,6 +464,7 @@ static int print_hca_cap(struct ibv_device *ib_dev, uint8_t ib_port)
 
 		printf("\tdevice_cap_flags_ex:\t\t0x%" PRIX64 "\n", device_attr.device_cap_flags_ex);
 		print_device_cap_flags_ex(device_attr.device_cap_flags_ex);
+		print_tso_caps(&device_attr.tso_caps);
 	}
 
 	for (port = 1; port <= device_attr.orig_attr.phys_port_cnt; ++port) {
diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h
index ccf1d51..ec541e3 100644
--- a/include/infiniband/verbs.h
+++ b/include/infiniband/verbs.h
@@ -209,6 +209,11 @@ enum ibv_odp_general_caps {
 	IBV_ODP_SUPPORT = 1 << 0,
 };
 
+struct ibv_tso_caps {
+	uint32_t max_tso;
+	uint32_t supported_qpts;
+};
+
 struct ibv_device_attr_ex {
 	struct ibv_device_attr	orig_attr;
 	uint32_t		comp_mask;
@@ -216,6 +221,7 @@ struct ibv_device_attr_ex {
 	uint64_t		completion_timestamp_mask;
 	uint64_t		hca_core_clock;
 	uint64_t		device_cap_flags_ex;
+	struct ibv_tso_caps	tso_caps;
 };
 
 enum ibv_mtu {
@@ -357,6 +363,7 @@ enum ibv_wc_opcode {
 	IBV_WC_FETCH_ADD,
 	IBV_WC_BIND_MW,
 	IBV_WC_LOCAL_INV,
+	IBV_WC_TSO,
 /*
  * Set value of IBV_WC_RECV so consumers can test if a completion is a
  * receive by testing (opcode & IBV_WC_RECV).
@@ -636,7 +643,8 @@ enum ibv_qp_init_attr_mask {
 	IBV_QP_INIT_ATTR_PD		= 1 << 0,
 	IBV_QP_INIT_ATTR_XRCD		= 1 << 1,
 	IBV_QP_INIT_ATTR_CREATE_FLAGS	= 1 << 2,
-	IBV_QP_INIT_ATTR_RESERVED	= 1 << 3
+	IBV_QP_INIT_ATTR_MAX_TSO_HEADER = 1 << 3,
+	IBV_QP_INIT_ATTR_RESERVED	= 1 << 4
 };
 
 enum ibv_qp_create_flags {
@@ -657,7 +665,7 @@ struct ibv_qp_init_attr_ex {
 	struct ibv_pd	       *pd;
 	struct ibv_xrcd	       *xrcd;
 	uint32_t                create_flags;
-
+	uint16_t		max_tso_header;
 };
 
 enum ibv_qp_open_attr_mask {
@@ -756,6 +764,7 @@ enum ibv_wr_opcode {
 	IBV_WR_LOCAL_INV,
 	IBV_WR_BIND_MW,
 	IBV_WR_SEND_WITH_INV,
+	IBV_WR_TSO,
 };
 
 enum ibv_send_flags {
@@ -802,12 +811,18 @@ struct ibv_send_wr {
 			uint32_t    remote_srqn;
 		} xrc;
 	} qp_type;
-	struct {
-		struct ibv_mw	*mw;
-		uint32_t		rkey;
-		struct ibv_mw_bind_info	bind_info;
-	} bind_mw;
-
+	union {
+		struct {
+			struct ibv_mw	*mw;
+			uint32_t		rkey;
+			struct ibv_mw_bind_info	bind_info;
+		} bind_mw;
+		struct {
+			void		       *hdr;
+			uint16_t		hdr_sz;
+			uint16_t		mss;
+		} tso;
+	};
 };
 
 struct ibv_recv_wr {
@@ -1967,6 +1982,12 @@ int ibv_resolve_eth_l2_from_gid(struct ibv_context *context,
 				struct ibv_ah_attr *attr,
 				uint8_t eth_mac[ETHERNET_LL_SIZE],
 				uint16_t *vid);
+
+static inline int ibv_is_qpt_supported(uint32_t caps, enum ibv_qp_type qpt)
+{
+	return !!(caps & (1 << qpt));
+}
+
 END_C_DECLS
 
 #  undef __attribute_const
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V1 libibverbs 3/3] Update man pages for TSO support

[Yishai Hadas]

From: Bodong Wang <bodong-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

Update relevant man pages for TSO:
- Add max_tso_header as part of ibv_create_qp_ex man page.
- Add IBV_WR_TSO opcode and update send operation support table for
  RAW_PACKET QP as part of ibv_post_send man page.
- Add TSO capabilities as part of ibv_query_device_ex man page.

In addition, fixed a typo as part of updating ibv_post_send related to
ibv_odp_caps.

Signed-off-by: Bodong Wang <bodong-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 man/ibv_create_qp_ex.3    |  1 +
 man/ibv_post_send.3       | 36 ++++++++++++++++++++++++------------
 man/ibv_query_device_ex.3 | 22 ++++++++++++++--------
 3 files changed, 39 insertions(+), 20 deletions(-)

diff --git a/man/ibv_create_qp_ex.3 b/man/ibv_create_qp_ex.3
index f772a57..a2a67a4 100644
--- a/man/ibv_create_qp_ex.3
+++ b/man/ibv_create_qp_ex.3
@@ -34,6 +34,7 @@ uint32_t                comp_mask;	/* Identifies valid fields */
 struct ibv_pd          *pd;		/* PD to be associated with the QP */
 struct ibv_xrcd        *xrcd;		/* XRC domain to be associated with the target QP */
 enum ibv_qp_create_flags create_flags;	/* Creation flags for this QP */
+uint16_t                max_tso_header; /* Maximum TSO header size */
 .in -8
 };
 .sp
diff --git a/man/ibv_post_send.3 b/man/ibv_post_send.3
index f918afb..0d3fa66 100644
--- a/man/ibv_post_send.3
+++ b/man/ibv_post_send.3
@@ -69,6 +69,8 @@ uint32_t remote_srqn;            /* Number of the remote SRQ */
 } xrc;
 .in -8
 } qp_type;
+union {
+.in +8
 struct {
 .in +8
 struct ibv_mw            *mw;             /* Memory window (MW) of type 2 to bind */
@@ -76,6 +78,15 @@ uint32_t                 rkey;            /* The desired new rkey of the MW */
 struct ibv_mw_bind_info  bind_info;       /* MW additional bind information */
 .in -8
 } bind_mw;
+struct {
+.in +8
+void			*hdr;	/* Pointer address of inline header */
+uint16_t		hdr_sz;	/* Inline header size */
+uint16_t		mss;	/* Maximum segment size for each TSO fragment */
+.in -8
+} tso;
+.in -8
+};
 .in -8
 };
 .fi
@@ -104,18 +115,19 @@ uint32_t                lkey;                   /* Key of the local Memory Regio
 Each QP Transport Service Type supports a specific set of opcodes, as shown in the following table:
 .PP
 .nf
-OPCODE                      | IBV_QPT_UD | IBV_QPT_UC | IBV_QPT_RC | IBV_QPT_XRC_SEND
-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+\-\-\-\-\-\-\-\-\-\-\-\-+\-\-\-\-\-\-\-\-\-\-\-\-+\-\-\-\-\-\-\-\-\-\-\-
-IBV_WR_SEND                 |     X      |     X      |     X     |     X
-IBV_WR_SEND_WITH_IMM        |     X      |     X      |     X     |     X
-IBV_WR_RDMA_WRITE           |            |     X      |     X     |     X
-IBV_WR_RDMA_WRITE_WITH_IMM  |            |     X      |     X     |     X
-IBV_WR_RDMA_READ            |            |            |     X     |     X
-IBV_WR_ATOMIC_CMP_AND_SWP   |            |            |     X     |     X
-IBV_WR_ATOMIC_FETCH_AND_ADD |            |            |     X     |     X
-IBV_WR_LOCAL_INV            |            |     X      |     X     |     X
-IBV_WR_BIND_MW              |            |     X      |     X     |     X
-IBV_WR_SEND_WITH_INV        |            |     X      |     X     |     X
+OPCODE                      | IBV_QPT_UD | IBV_QPT_UC | IBV_QPT_RC | IBV_QPT_XRC_SEND | IBV_QPT_RAW_PACKET
+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+\-\-\-\-\-\-\-\-\-\-\-\-+\-\-\-\-\-\-\-\-\-\-\-\-+\-\-\-\-\-\-\-\-\-\-\-\-+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
+IBV_WR_SEND                 |     X      |     X      |     X      |         X        |         X
+IBV_WR_SEND_WITH_IMM        |     X      |     X      |     X      |         X        |
+IBV_WR_RDMA_WRITE           |            |     X      |     X      |         X        |
+IBV_WR_RDMA_WRITE_WITH_IMM  |            |     X      |     X      |         X        |
+IBV_WR_RDMA_READ            |            |            |     X      |         X        |
+IBV_WR_ATOMIC_CMP_AND_SWP   |            |            |     X      |         X        |
+IBV_WR_ATOMIC_FETCH_AND_ADD |            |            |     X      |         X        |
+IBV_WR_LOCAL_INV            |            |     X      |     X      |         X        |
+IBV_WR_BIND_MW              |            |     X      |     X      |         X        |
+IBV_WR_SEND_WITH_INV        |            |     X      |     X      |         X        |
+IBV_WR_TSO                  |            |            |            |                  |         X
 .fi
 .PP
 The attribute send_flags describes the properties of the \s-1WR\s0. It is either 0 or the bitwise \s-1OR\s0 of one or more of the following flags:
diff --git a/man/ibv_query_device_ex.3 b/man/ibv_query_device_ex.3
index 5097be0..d06b40f 100644
--- a/man/ibv_query_device_ex.3
+++ b/man/ibv_query_device_ex.3
@@ -26,17 +26,18 @@ uint32_t               comp_mask;                  /* Compatibility mask that de
 struct ibv_odp_caps    odp_caps;                   /* On-Demand Paging capabilities */
 uint64_t               completion_timestamp_mask;  /* Completion timestamp mask (0 = unsupported) */
 uint64_t               hca_core_clock;             /* The frequency (in kHZ) of the HCA (0 = unsupported) */
-uint64_t               device_cap_flags_ex;    /* Extended device capability flags */
+uint64_t               device_cap_flags_ex;        /* Extended device capability flags */
+struct ibv_tso_caps    tso_caps;                   /* TCP segmentation offload capabilities */
 .in -8
 };
 
-struct ibv_exp_odp_caps {
-	uint64_t	general_odp_caps;  /* Mask with enum ibv_odp_general_cap_bits */
-	struct {
-		uint32_t	rc_odp_caps;      /* Mask with enum ibv_odp_tranport_cap_bits to know which operations are supported. */
-		uint32_t	uc_odp_caps;      /* Mask with enum ibv_odp_tranport_cap_bits to know which operations are supported. */
-		uint32_t	ud_odp_caps;      /* Mask with enum ibv_odp_tranport_cap_bits to know which operations are supported. */
-	} per_transport_caps;
+struct ibv_odp_caps {
+        uint64_t general_odp_caps;    /* Mask with enum ibv_odp_general_cap_bits */
+        struct {
+                uint32_t rc_odp_caps; /* Mask with enum ibv_odp_tranport_cap_bits to know which operations are supported. */
+                uint32_t uc_odp_caps; /* Mask with enum ibv_odp_tranport_cap_bits to know which operations are supported. */
+                uint32_t ud_odp_caps; /* Mask with enum ibv_odp_tranport_cap_bits to know which operations are supported. */
+        } per_transport_caps;
 };
 
 enum ibv_odp_general_cap_bits {
@@ -51,6 +52,11 @@ enum ibv_odp_transport_cap_bits {
         IBV_ODP_SUPPORT_ATOMIC   = 1 << 4, /* RDMA-Atomic operations support on-demand paging */
 };
 
+struct ibv_tso_caps {
+        uint32_t max_tso;        /* Maximum payload size in bytes supported for segmentation by TSO engine.*/
+        uint32_t supported_qpts; /* Bitmap showing which QP types are supported by TSO operation. */
+};
+
 .fi
 .SH "RETURN VALUE"
 .B ibv_query_device_ex()
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 libibverbs 0/3] Add support for TCP segmentation offload (TSO)

[Doug Ledford]

[-- Attachment #1.1: Type: text/plain, Size: 2970 bytes --]

On 8/21/2016 12:47 PM, Yishai Hadas wrote:
> Hi Doug,
> 
> This V1 series addresses a note coming from the list to replace a macro
> by a static inline function, details below.
> 
> The relevant kernel patches regarding capabilities were already taken into your
> tree for coming kernel of 4.8.
> 
> The series was tested successfully with mlx5 driver (lib, kernel) and can be
> accessed also from my openfabrics GIT at:
> git://openfabrics.org/~yishaih/libibverbs.git branch: tso_v1
> 
> Yishai
> 
> In general,
> TSO enables an application to pass a packet with large chunk of data than the
> MTU to the NIC and let it breaks into small packets as part of its send flow.
> The TSO engine will split the packet into separate packets and insert the
> relevant user specified L2/L3/L4 headers automatically.
> This is a technique for increasing outbound throughput of high-bandwidth
> network connections by reducing CPU overhead.
> 
> For some extra information about TSO, below link can be helpful:
> https://en.wikipedia.org/wiki/Large_segment_offload
> 
> Verbs usage,
> TSO traffic is performed by ibv_post_send with opcode IBV_WR_TSO.
> The following information should be supplied within the associated send WR:
> - A pointer to the packet header.
> - Header size.
> - The maximum segment size (mss) that hardware should generate in
>   its TSO engine.
> 
> The above information required to send a TSO packet is similar to the IB kernel
> API.  http://lxr.free-electrons.com/source/include/rdma/ib_verbs.h#L1237
> 
> The TSO capabilities as of the supported QP types, max TSO payload can be
> achieved upon query device.
>  
> When creating a TSO eligible QP, user should supply the maximum TSO header size
> in order to let providers preparing their SQ buffer accordingly. This need is
> similar to other capabilities that are given as part of QP creation for that
> purpose.
> 
> Changes from V0:
> patch #2: Change 'ibv_is_qpt_supported' to be a static inline function instead
>           of a macro.
> 
> Bodong Wang (3):
>   Fix ibv_cmd_query_device_ex to return valid output
>   Add support for TCP segmentation offload (TSO)
>   Update man pages for TSO support
> 
>  examples/devinfo.c         | 20 ++++++++++++++++++++
>  include/infiniband/verbs.h | 37 +++++++++++++++++++++++++++++--------
>  man/ibv_create_qp_ex.3     |  1 +
>  man/ibv_post_send.3        | 36 ++++++++++++++++++++++++------------
>  man/ibv_query_device_ex.3  | 22 ++++++++++++++--------
>  src/cmd.c                  |  9 +--------
>  6 files changed, 89 insertions(+), 36 deletions(-)
> 

The third patch, to the man pages, only added raw eth to the supported
type matrix for TSO.  However, the actual support clearly was intended
to work with UD as well.  I modified the man page patch to correct that.
 Applied.

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
    GPG Key ID: 0E572FDD


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]