[PATCH V1 for-next 0/2] Add support for reporting LSO capabilities

[PATCH V1 for-next 0/2] Add support for reporting LSO capabilities

[Matan Barak]

Hi Doug,

This patch series enables hardware to report Large Segment Offload(LSO)
capabilities from kernel to user space. LSO enabled NIC will accept big chuck of
data with size greater than MTU. The LSO engine will split them into
separate packets and insert the headers automatically. With LSO, CPU is
offloaded from dealing with much less throughput of data.

For detailed info about LSO, please refer to the following link:
https://en.wikipedia.org/wiki/Large_segment_offload

The capabilities are exposed to user space through query_device, ib_device_attr,
and their relevant code is updated to build the pipe. The following capabilities
will be reported:

1. The maximum payload size supported for segmentation by LSO engine.
2. QP types which support LSO operations.

The capabilities of mlx5-related hardware are reported accordingly.

This series conflicts with scatter FCS series that was posted earlier
in this kernel cycle. The resolution for this conflict is available
in Mellanox's git.

Regards,
Bodong and Matan.

Changes from V0:
 * Reported a wrong QP type.
 * Fixed grammer mistakes in commit message.

Bodong Wang (2):
  IB/core: Report LSO capabilities when querying device
  IB/mlx5: Report mlx5 LSO capabilities when querying device

 drivers/infiniband/core/uverbs_cmd.c |  7 +++++++
 drivers/infiniband/hw/mlx5/main.c    | 13 +++++++++++--
 include/rdma/ib_verbs.h              |  6 ++++++
 include/uapi/rdma/ib_user_verbs.h    |  6 ++++++
 4 files changed, 30 insertions(+), 2 deletions(-)

-- 
2.5.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Matan Barak]

From: Bodong Wang <bodong-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

LSO capabilities include the max LSO packet size in bytes and which QP
types does the hardware support.

Relevant verbs to report LSO capabilities from kernel to user space are
updated accordingly.

Signed-off-by: Bodong Wang <bodong-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/infiniband/core/uverbs_cmd.c | 7 +++++++
 include/rdma/ib_verbs.h              | 6 ++++++
 include/uapi/rdma/ib_user_verbs.h    | 6 ++++++
 3 files changed, 19 insertions(+)

diff --git a/drivers/infiniband/core/uverbs_cmd.c b/drivers/infiniband/core/uverbs_cmd.c
index 6fdc7ec..b0fb94b 100644
--- a/drivers/infiniband/core/uverbs_cmd.c
+++ b/drivers/infiniband/core/uverbs_cmd.c
@@ -3655,6 +3655,13 @@ int ib_uverbs_ex_query_device(struct ib_uverbs_file *file,
 	resp.hca_core_clock = attr.hca_core_clock;
 	resp.response_length += sizeof(resp.hca_core_clock);
 
+	if (ucore->outlen < resp.response_length + sizeof(resp.lso_caps))
+		goto end;
+
+	resp.lso_caps.max_lso = attr.lso_caps.max_lso;
+	resp.lso_caps.supported_qpts = attr.lso_caps.supported_qpts;
+	resp.response_length += sizeof(resp.lso_caps);
+
 end:
 	err = ib_copy_to_udata(ucore, &resp, resp.response_length);
 	return err;
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index fb2cef4..d5a466f 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -271,6 +271,11 @@ struct ib_cq_init_attr {
 	u32		flags;
 };
 
+struct ib_lso_caps {
+	u32 max_lso;
+	u32 supported_qpts; /* Use enum ib_qp_type */
+};
+
 struct ib_device_attr {
 	u64			fw_ver;
 	__be64			sys_image_guid;
@@ -317,6 +322,7 @@ struct ib_device_attr {
 	struct ib_odp_caps	odp_caps;
 	uint64_t		timestamp_mask;
 	uint64_t		hca_core_clock; /* in KHZ */
+	struct ib_lso_caps	lso_caps;
 };
 
 enum ib_mtu {
diff --git a/include/uapi/rdma/ib_user_verbs.h b/include/uapi/rdma/ib_user_verbs.h
index 8126c14..fa039f5 100644
--- a/include/uapi/rdma/ib_user_verbs.h
+++ b/include/uapi/rdma/ib_user_verbs.h
@@ -219,6 +219,11 @@ struct ib_uverbs_odp_caps {
 	__u32 reserved;
 };
 
+struct ib_uverbs_lso_caps {
+	__u32 max_lso;
+	__u32 supported_qpts; /* Use enum ib_qp_type */
+};
+
 struct ib_uverbs_ex_query_device_resp {
 	struct ib_uverbs_query_device_resp base;
 	__u32 comp_mask;
@@ -226,6 +231,7 @@ struct ib_uverbs_ex_query_device_resp {
 	struct ib_uverbs_odp_caps odp_caps;
 	__u64 timestamp_mask;
 	__u64 hca_core_clock; /* in KHZ */
+	struct ib_uverbs_lso_caps lso_caps;
 };
 
 struct ib_uverbs_query_port {
-- 
2.5.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V1 for-next 2/2] IB/mlx5: Report mlx5 LSO capabilities when querying device

[Matan Barak]

From: Bodong Wang <bodong-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

mlx5 based hardware supports LSO for Raw Ethernet QPs.
LSO (large segment offload) applied here is mainly used
for TCP (also known as TSO).

Signed-off-by: Bodong Wang <bodong-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/infiniband/hw/mlx5/main.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index 5acf346..3716001e 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -454,6 +454,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
 	int max_rq_sg;
 	int max_sq_sg;
 	u64 min_page_size = 1ull << MLX5_CAP_GEN(mdev, log_pg_sz);
+	u64 max_lso;
 
 	if (uhw->inlen || uhw->outlen)
 		return -EINVAL;
@@ -508,10 +509,18 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
 	if (MLX5_CAP_GEN(mdev, block_lb_mc))
 		props->device_cap_flags |= IB_DEVICE_BLOCK_MULTICAST_LOOPBACK;
 
-	if (MLX5_CAP_GEN(dev->mdev, eth_net_offloads) &&
-	    (MLX5_CAP_ETH(dev->mdev, csum_cap)))
+	if (MLX5_CAP_GEN(dev->mdev, eth_net_offloads)) {
+		if (MLX5_CAP_ETH(mdev, csum_cap))
 			props->device_cap_flags |= IB_DEVICE_RAW_IP_CSUM;
 
+		max_lso = MLX5_CAP_ETH(mdev, max_lso_cap);
+		if (max_lso) {
+			props->lso_caps.max_lso = 1 << max_lso;
+			props->lso_caps.supported_qpts |=
+				1 << IB_QPT_RAW_PACKET;
+		}
+	}
+
 	if (MLX5_CAP_GEN(mdev, ipoib_basic_offloads)) {
 		props->device_cap_flags |= IB_DEVICE_UD_IP_CSUM;
 		props->device_cap_flags |= IB_DEVICE_UD_TSO;
-- 
2.5.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Christoph Lameter]

On Wed, 27 Apr 2016, Matan Barak wrote:

> --- a/include/rdma/ib_verbs.h
> +++ b/include/rdma/ib_verbs.h
> @@ -271,6 +271,11 @@ struct ib_cq_init_attr {
>  	u32		flags;
>  };
>
> +struct ib_lso_caps {
> +	u32 max_lso;
> +	u32 supported_qpts; /* Use enum ib_qp_type */

So this is a bitmap of the qps supported? And thus we have a limit now of
32 QP types. That needs to be documented and  you need a BUILD_BUG_ON
somewhere in case IB_QPT_MAX becomes larger than 32. Also needs to be
better commented so we know this is a bitmap.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 0/2] Add support for reporting LSO capabilities

[Jason Gunthorpe]

On Wed, Apr 27, 2016 at 05:04:50PM +0300, Matan Barak wrote:

> This patch series enables hardware to report Large Segment Offload(LSO)
> capabilities from kernel to user space. LSO enabled NIC will accept big chuck of
> data with size greater than MTU. The LSO engine will split them into
> separate packets and insert the headers automatically. With LSO, CPU is
> offloaded from dealing with much less throughput of data.

I also feel like we should stop changing the current uAPI until we get
the new one in place..

Eg, the gross thing Christoph points out about the QP type bitmap can
probably go away in the new uAPI.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 0/2] Add support for reporting LSO capabilities

[Christoph Lameter]

On Wed, 27 Apr 2016, Jason Gunthorpe wrote:

> Eg, the gross thing Christoph points out about the QP type bitmap can
> probably go away in the new uAPI.

But this functionality would be great to have for us.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 0/2] Add support for reporting LSO capabilities

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1256 bytes --]

On Wed, Apr 27, 2016 at 10:42:30AM -0600, Jason Gunthorpe wrote:
> On Wed, Apr 27, 2016 at 05:04:50PM +0300, Matan Barak wrote:
> 
> > This patch series enables hardware to report Large Segment Offload(LSO)
> > capabilities from kernel to user space. LSO enabled NIC will accept big chuck of
> > data with size greater than MTU. The LSO engine will split them into
> > separate packets and insert the headers automatically. With LSO, CPU is
> > offloaded from dealing with much less throughput of data.
> 
> I also feel like we should stop changing the current uAPI until we get
> the new one in place.

This functionality doesn't relate to ABI change and don't change
current/future interfaces.

There is no need to stop changes like this.

> 
> Eg, the gross thing Christoph points out about the QP type bitmap can
> probably go away in the new uAPI.

We are taking part in this new ABI effort and have full responsibility to
adapt our drivers (mlx4/mlx5) to new ABI whenever it comes.

> 
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH V1 for-next 0/2] Add support for reporting LSO capabilities

[Jason Gunthorpe]

On Wed, Apr 27, 2016 at 08:09:03PM +0300, Leon Romanovsky wrote:
> On Wed, Apr 27, 2016 at 10:42:30AM -0600, Jason Gunthorpe wrote:
> > On Wed, Apr 27, 2016 at 05:04:50PM +0300, Matan Barak wrote:
> > 
> > > This patch series enables hardware to report Large Segment Offload(LSO)
> > > capabilities from kernel to user space. LSO enabled NIC will accept big chuck of
> > > data with size greater than MTU. The LSO engine will split them into
> > > separate packets and insert the headers automatically. With LSO, CPU is
> > > offloaded from dealing with much less throughput of data.
> > 
> > I also feel like we should stop changing the current uAPI until we get
> > the new one in place.
> 
> This functionality doesn't relate to ABI change and don't change
> current/future interfaces.

What do you mean it doesn't change current interfaces? That is all
this patch does.

> There is no need to stop changes like this.

Why do we need to keep churning the uAPI now that we are committed to
deprecating it?

It seems like the discussion is closing on something implementable.

And again, we discussed at the Collab Summit that ABI patches like
this would be OK'd by a multi-vendor group - that hasn't happened yet
for this patch either.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 0/2] Add support for reporting LSO capabilities

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1272 bytes --]

On Wed, Apr 27, 2016 at 11:14:51AM -0600, Jason Gunthorpe wrote:
> On Wed, Apr 27, 2016 at 08:09:03PM +0300, Leon Romanovsky wrote:
> > On Wed, Apr 27, 2016 at 10:42:30AM -0600, Jason Gunthorpe wrote:
> > > On Wed, Apr 27, 2016 at 05:04:50PM +0300, Matan Barak wrote:
> > > 
> > > > This patch series enables hardware to report Large Segment Offload(LSO)
> > > > capabilities from kernel to user space. LSO enabled NIC will accept big chuck of
> > > > data with size greater than MTU. The LSO engine will split them into
> > > > separate packets and insert the headers automatically. With LSO, CPU is
> > > > offloaded from dealing with much less throughput of data.
> > > 
> > > I also feel like we should stop changing the current uAPI until we get
> > > the new one in place.
> > 
> > This functionality doesn't relate to ABI change and don't change
> > current/future interfaces.
> 
> What do you mean it doesn't change current interfaces? That is all
> this patch does.

It uses the same old write/read mechanism. The patch doesn't change it.

> 
> > There is no need to stop changes like this.
> 
> Why do we need to keep churning the uAPI now that we are committed to
> deprecating it?

We are committed to change write/read interface.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

RE: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Hefty, Sean]

> > +struct ib_lso_caps {
> > +	u32 max_lso;
> > +	u32 supported_qpts; /* Use enum ib_qp_type */
> 
> So this is a bitmap of the qps supported? And thus we have a limit now of
> 32 QP types. That needs to be documented and  you need a BUILD_BUG_ON
> somewhere in case IB_QPT_MAX becomes larger than 32. Also needs to be
> better commented so we know this is a bitmap.

I think it would make more sense to indicate what protocol, or specific aspect of a protocol, are being offloaded.  Some qp types support LSO as a matter of practice.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 0/2] Add support for reporting LSO capabilities

[Matan Barak]

On 27/04/2016 20:14, Jason Gunthorpe wrote:
> On Wed, Apr 27, 2016 at 08:09:03PM +0300, Leon Romanovsky wrote:
>> On Wed, Apr 27, 2016 at 10:42:30AM -0600, Jason Gunthorpe wrote:
>>> On Wed, Apr 27, 2016 at 05:04:50PM +0300, Matan Barak wrote:
>>>
>>>> This patch series enables hardware to report Large Segment Offload(LSO)
>>>> capabilities from kernel to user space. LSO enabled NIC will accept big chuck of
>>>> data with size greater than MTU. The LSO engine will split them into
>>>> separate packets and insert the headers automatically. With LSO, CPU is
>>>> offloaded from dealing with much less throughput of data.
>>>
>>> I also feel like we should stop changing the current uAPI until we get
>>> the new one in place.
>>
>> This functionality doesn't relate to ABI change and don't change
>> current/future interfaces.
>
> What do you mean it doesn't change current interfaces? That is all
> this patch does.
>

This patch indeed changes the ABI, but as far as I understand, we want 
to keep libibverbs API. If so, all current libibverbs API's features 
should somehow go through this new uAPI/uABI and LSO should probably be 
one of these features as well. Do we want to halt all progress in the 
RDMA world before we stabilize a new uAPI?


>> There is no need to stop changes like this.
>
> Why do we need to keep churning the uAPI now that we are committed to
> deprecating it?
>

Will there be another libibverbs release before this uAPI change is 
completed? We all want to see this new uAPI change landing ASAP and 
we're all committed to this, but being realistic, it probably won't go 
into 4.7 kernel cycle, while small changes like this could.

> It seems like the discussion is closing on something implementable.
>
> And again, we discussed at the Collab Summit that ABI patches like
> this would be OK'd by a multi-vendor group - that hasn't happened yet
> for this patch either.
>

We're always open for discussions regarding features like this.

> Jason
>

Matan

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Matan Barak]

On 28/04/2016 00:54, Hefty, Sean wrote:
>>> +struct ib_lso_caps {
>>> +	u32 max_lso;
>>> +	u32 supported_qpts; /* Use enum ib_qp_type */
>>
>> So this is a bitmap of the qps supported? And thus we have a limit now of
>> 32 QP types. That needs to be documented and  you need a BUILD_BUG_ON
>> somewhere in case IB_QPT_MAX becomes larger than 32. Also needs to be
>> better commented so we know this is a bitmap.
>
> I think it would make more sense to indicate what protocol, or specific aspect of a protocol, are being offloaded.  Some qp types support LSO as a matter of practice.
>

For a lot of applications, we want to hide the lower level protocol. For 
example, we try hard to hide whether IB/RoCE is used by an application. 
Thus, is it wise to expose this thing in API?
A user creates a UD QP, he doesn't know or care whether the lower layer 
is Ethernet (RoCE v1/V2) or IB.

I agree that the max_lso could be different for various QP types and 
maybe probing a QP type would be favorable that getting this information 
in a generic query_device.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Hefty, Sean]

> For a lot of applications, we want to hide the lower level protocol. For
> example, we try hard to hide whether IB/RoCE is used by an application.
> Thus, is it wise to expose this thing in API?

IMO - yes.

An application must know what protocol it is using in order to communicate at all.  Even sockets exposes the protocol to the app.  If an app truly doesn't care, it still needs a generic way to determine if it is using matching protocols.

> A user creates a UD QP, he doesn't know or care whether the lower layer
> is Ethernet (RoCE v1/V2) or IB.

He cares if that there's more than one choice available.  There are no guarantees that a ud qp can communicate with some other ud qp.

> I agree that the max_lso could be different for various QP types and
> maybe probing a QP type would be favorable that getting this information
> in a generic query_device.

The qp type is being used as a stand-in for the transport protocol.  This worked when the APIs were developed and the only transport was IB.  But it is not true today.

LSO ultimately applies to a specific protocol.  IMO that's what needs to be captured.  Applications can still check for this in a generic fashion by matching which protocol (value) it is using against the LSO protocol support.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Matan Barak]

On 28/04/2016 19:35, Hefty, Sean wrote:
>> For a lot of applications, we want to hide the lower level protocol. For
>> example, we try hard to hide whether IB/RoCE is used by an application.
>> Thus, is it wise to expose this thing in API?
>
> IMO - yes.
>
> An application must know what protocol it is using in order to communicate at all.  Even sockets exposes the protocol to the app.  If an app truly doesn't care, it still needs a generic way to determine if it is using matching protocols.
>

I think socket protocols resembles QP types more than how we use 
protocols in the IB world.

>> A user creates a UD QP, he doesn't know or care whether the lower layer
>> is Ethernet (RoCE v1/V2) or IB.
>
> He cares if that there's more than one choice available.  There are no guarantees that a ud qp can communicate with some other ud qp.
>

One of the core questions here is where do we draw the line between the 
application and the administrator. For example, we could say that an 
administrator could configure the protocol we use as default for the 
fabric and that an application with the right privileges could override 
this default protocol.

Saying that, maybe we should query a QP for its attributes only after it 
is created and a protocol is somehow assigned to it. What do you think?

>> I agree that the max_lso could be different for various QP types and
>> maybe probing a QP type would be favorable that getting this information
>> in a generic query_device.
>
> The qp type is being used as a stand-in for the transport protocol.  This worked when the APIs were developed and the only transport was IB.  But it is not true today.
>

Agree, a QP type hides now several protocols. However, most applications 
that were written in this IB-only verbs world functions great under 
other protocols. It sometimes depend on an administrator configurations 
and such, but still.

> LSO ultimately applies to a specific protocol.  IMO that's what needs to be captured.  Applications can still check for this in a generic fashion by matching which protocol (value) it is using against the LSO protocol support.
>

If a protocol is already assigned when an application queries a QP type, 
we get that a QP type uniquely defines a protocol for this application 
and thus querying a QP type is enough.
If that's not the case, an application needs to somehow ask for a 
specific protocol (and it might needs to be privileged to do so).

So, what do you think about stating an optional protocol in create_qp 
and putting LSO in query_qp?

Thanks for taking a look.

> - Sean
>

Matan

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Hefty, Sean]

> So, what do you think about stating an optional protocol in create_qp
> and putting LSO in query_qp?

This makes sense to me, and I think will work well as devices become more complex (e.g. the qlogic NIC that supports both iWarp and RoCE).
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Jason Gunthorpe]

On Mon, May 02, 2016 at 05:20:17PM +0000, Hefty, Sean wrote:
> > So, what do you think about stating an optional protocol in create_qp
> > and putting LSO in query_qp?
> 
> This makes sense to me, and I think will work well as devices become
> more complex (e.g. the qlogic NIC that supports both iWarp and
> RoCE).

What is the actual issue here?

LSO should just be programmable UD fragmentation. It should be
indistinguishable from a user app sending multiple send WRs to the UD
QP. Surely it doesn't have anything to do with the underlying wire
protocol?

That said, I haven't seen anything talking about how to program the
segmentation behavior (presumably it is assuming IP datagrams or
something?). It is absolutely necessary to specify the segmentation
method that the HW is going to use, there is lots of variety there.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Hefty, Sean]

> > > So, what do you think about stating an optional protocol in create_qp
> > > and putting LSO in query_qp?
> >
> > This makes sense to me, and I think will work well as devices become
> > more complex (e.g. the qlogic NIC that supports both iWarp and
> > RoCE).
> 
> What is the actual issue here?

My issue is how this is being exposed.  RC QPs already do LSO.  What does this change mean in that context?  What is actually being offloaded?

When I see LSO, I think of TCP LSO.  Is LSO the most appropriate term here?

> LSO should just be programmable UD fragmentation. It should be
> indistinguishable from a user app sending multiple send WRs to the UD
> QP. Surely it doesn't have anything to do with the underlying wire
> protocol?

IMO - the fragmentation has everything to do with the underlying protocol.  Are we fragmenting into IP datagrams?  IB UD datagrams?  UDP packets?  Is re-assembly occurring on the opposite side (assuming no)?  Is the receiver seeing multiple messages (assuming yes)?  How many messages, and where did the breaks occur?  If the wire protocol doesn't matter, why does the qp type?

> That said, I haven't seen anything talking about how to program the
> segmentation behavior (presumably it is assuming IP datagrams or
> something?). It is absolutely necessary to specify the segmentation
> method that the HW is going to use, there is lots of variety there.

I'm hoping that the protocol conveys this information.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Jason Gunthorpe]

On Mon, May 02, 2016 at 06:13:45PM +0000, Hefty, Sean wrote:
> > > > So, what do you think about stating an optional protocol in create_qp
> > > > and putting LSO in query_qp?
> > >
> > > This makes sense to me, and I think will work well as devices become
> > > more complex (e.g. the qlogic NIC that supports both iWarp and
> > > RoCE).
> > 
> > What is the actual issue here?
> 
> My issue is how this is being exposed.  RC QPs already do LSO.  What
> does this change mean in that context?  What is actually being
> offloaded?

No, you are mixing layers.

LSO is only defined at the WR layer.

LSO says, 'take this WR, in HW perform a split and transform operation
and create multiple SEND WRs'.

The end result is still SEND, LSO does not change anything about the
layers below the WR. A SEND on a UD, RC, or RAW QP is still exactly the
same as before LSO was involved.

Critically, the operation of LSO should be indistinguishable on the
wire from actually generating the final WRs directly by the
application in the SQ.

Thus, RC/UC QPs do not do LSO. Their message segmentation protocol is
something totally different.

> > LSO should just be programmable UD fragmentation. It should be
> > indistinguishable from a user app sending multiple send WRs to the UD
> > QP. Surely it doesn't have anything to do with the underlying wire
> > protocol?
> 
> IMO - the fragmentation has everything to do with the underlying
> protocol.  Are we fragmenting into IP datagrams?  IB UD datagrams?
> UDP packets?

It really doesn't.

It is perfectly legitimate to perform an Ethernet IPv4 TCP LSO
transformation on a RC QP for IB (for instance). You'd get Ethernet
frames split up within IB RC SEND packets. eg for ethernet over IB
type applications.

The LSO transformation requested and underlying QP are totally
orthogonal concepts.

> Is re-assembly occurring on the opposite side (assuming no)?  Is the
> receiver seeing multiple messages (assuming yes)?  How many
> messages, and where did the breaks occur?  If the wire protocol
> doesn't matter, why does the qp type?

Typically an IP focused LSO will work like this:

 'SEND' WR for 32k.
 Split it into 1500 byte chunks, and make a new SEND for each chunk
 Replicate bytes 0->NN of header onto every new SEND
 Edit bytes XX->YY assuming they are an IPv4 header (update lengths,
  checksum, sequence, etc)
 Edit bytes YY->ZZ assuming they are a TCP or UDP header
  (length,checksum,etc)

In all cases the LSO is simply a transformation of a single SEND into
multiple SEND according to certain rules.

The rules *must* be specified in the API. I think this is what you are
thinking about when you say 'protocol', but this is not IB or iWarp,
it is 'LSO packet format protocol #1' or something.

IIRC there is quite a lot of variety here in Ethernet hardware, and
some hardware can specify the LSO format on each WR.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Hefty, Sean]

> LSO says, 'take this WR, in HW perform a split and transform operation
> and create multiple SEND WRs'.
> 
> The end result is still SEND, LSO does not change anything about the
> layers below the WR. A SEND on a UD, RC, or RAW QP is still exactly the
> same as before LSO was involved.

Okay - this isn't what I think about when I read LSO at all (i.e. large segment offload, not large send [WR] offload).  We have chained WRs.  What your describing sounds like some sort of optimized version of that.

> Critically, the operation of LSO should be indistinguishable on the
> wire from actually generating the final WRs directly by the
> application in the SQ.

This concept should apply to the receive side as well.

And generically, I don't think we even need to restrict WR offload to being the same type of operation.  E.g. WRITE_WITH_SEND type of operation.

> Typically an IP focused LSO will work like this:
> 
>  'SEND' WR for 32k.
>  Split it into 1500 byte chunks, and make a new SEND for each chunk
>  Replicate bytes 0->NN of header onto every new SEND
>  Edit bytes XX->YY assuming they are an IPv4 header (update lengths,
>   checksum, sequence, etc)
>  Edit bytes YY->ZZ assuming they are a TCP or UDP header
>   (length,checksum,etc)
> 
> In all cases the LSO is simply a transformation of a single SEND into
> multiple SEND according to certain rules.
> 
> The rules *must* be specified in the API. I think this is what you are
> thinking about when you say 'protocol', but this is not IB or iWarp,
> it is 'LSO packet format protocol #1' or something.

I was under the impression that this was actually working at a protocol level that was above the transport level of the QP -- i.e. TCP/UDP/IP segmentation offload.
 
- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Jason Gunthorpe]

On Wed, May 04, 2016 at 04:49:47PM +0000, Hefty, Sean wrote:

> Okay - this isn't what I think about when I read LSO at all
> (i.e. large segment offload, not large send [WR] offload).  We have
> chained WRs.  What your describing sounds like some sort of
> optimized version of that.

Yes.

But critically the typical LSO engine will have the ability to do header
replication and editing as it splits up the send.

> This concept should apply to the receive side as well.

The receive side is asymmetric in this case, LRO is a different
animal.

> > The rules *must* be specified in the API. I think this is what you are
> > thinking about when you say 'protocol', but this is not IB or iWarp,
> > it is 'LSO packet format protocol #1' or something.
> 
> I was under the impression that this was actually working at a
> protocol level that was above the transport level of the QP --
> i.e. TCP/UDP/IP segmentation offload.

Sort of. The typical LSO rules the NIC vendors have designed are
intended for TCP and UDP packet segmentation.

But in our verbs context, we'd talk about this process as a
transformation of 1xLSO_SEND WRs into NxSEND WRs and maintain strict
layering. The use of LSO can not do anything the user could not do
just by creating SENDs on their own.

Further, a LSO rule set should always operate the same, no matter what
sort of QP it is running on - the LSO rules don't change just because a
QP is iwarp or IB.

Matan: I think the take away here is that the API needs a lot more
work on specifying exactly what 'LSO' means, in terms of what it
actually does, and we almost certainly need some way to negotiate
different rule-sets, as I doubt all NIC vendors agree on how this
works. Even Linux net has a few variations over time (TSO, LSO/GSO)
supported by the core code. And it is easy to predict that things like
FCoX, SCTP and so forth will have their own unique rule sets.

This should probably be done after creating the QP (and AH?), as a NIC
may not be able to do certain LSO rules with certain QP/AH
configurations.

I'd probably also use the driver-specific channel to communicate the
NIC's capability and prototype some kind of API in libibverbs.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Matan Barak]

On 04/05/2016 21:11, Jason Gunthorpe wrote:
> On Wed, May 04, 2016 at 04:49:47PM +0000, Hefty, Sean wrote:
>
>> Okay - this isn't what I think about when I read LSO at all
>> (i.e. large segment offload, not large send [WR] offload).  We have
>> chained WRs.  What your describing sounds like some sort of
>> optimized version of that.
>
> Yes.
>
> But critically the typical LSO engine will have the ability to do header
> replication and editing as it splits up the send.
>
>> This concept should apply to the receive side as well.
>
> The receive side is asymmetric in this case, LRO is a different
> animal.
>
>>> The rules *must* be specified in the API. I think this is what you are
>>> thinking about when you say 'protocol', but this is not IB or iWarp,
>>> it is 'LSO packet format protocol #1' or something.
>>
>> I was under the impression that this was actually working at a
>> protocol level that was above the transport level of the QP --
>> i.e. TCP/UDP/IP segmentation offload.
>
> Sort of. The typical LSO rules the NIC vendors have designed are
> intended for TCP and UDP packet segmentation.
>
> But in our verbs context, we'd talk about this process as a
> transformation of 1xLSO_SEND WRs into NxSEND WRs and maintain strict
> layering. The use of LSO can not do anything the user could not do
> just by creating SENDs on their own.
>
> Further, a LSO rule set should always operate the same, no matter what
> sort of QP it is running on - the LSO rules don't change just because a
> QP is iwarp or IB.
>

Agree

> Matan: I think the take away here is that the API needs a lot more
> work on specifying exactly what 'LSO' means, in terms of what it
> actually does, and we almost certainly need some way to negotiate
> different rule-sets, as I doubt all NIC vendors agree on how this
> works. Even Linux net has a few variations over time (TSO, LSO/GSO)
> supported by the core code. And it is easy to predict that things like
> FCoX, SCTP and so forth will have their own unique rule sets.
>

I agree that we're lacking especially in the query capabilities area here.

> This should probably be done after creating the QP (and AH?), as a NIC
> may not be able to do certain LSO rules with certain QP/AH
> configurations.
>

So I guess we want something that after a user creates a QP, he can 
query which protocols (IP, TCP, etc) could be used in the NIC's 
segmentation offload mechanism.

> I'd probably also use the driver-specific channel to communicate the
> NIC's capability and prototype some kind of API in libibverbs.
>

If we would like to expose this feature in the uAPI, why would we want 
to use the vendor specific channel instead of the standard core channel?

> Jason
>

Matan
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Jason Gunthorpe]

On Thu, May 05, 2016 at 11:21:29AM +0300, Matan Barak wrote:
> >This should probably be done after creating the QP (and AH?), as a NIC
> >may not be able to do certain LSO rules with certain QP/AH
> >configurations.
> >
> 
> So I guess we want something that after a user creates a QP, he can query
> which protocols (IP, TCP, etc) could be used in the NIC's segmentation
> offload mechanism.

This is where I get upset with the process we are following here,
without input from other hardware architects in other companies, it is
hard to design something truly common.

> >I'd probably also use the driver-specific channel to communicate the
> >NIC's capability and prototype some kind of API in libibverbs.
> 
> If we would like to expose this feature in the uAPI, why would we want to
> use the vendor specific channel instead of the standard core channel?

Because we don't really know what we are doing, we have only one
example of hardware that implements this and we can change libibverbs
with alot less pain than changing the kernel's uAPI.

If you use the driver channel, and are smart about it, you can
generically expose everything the card can do and can stop churning
the kernel every time a new QP feature comes up. How is that not
better for everyone?

When multiple vendors actually implement a verbs feature (or agree to
implement one via IBTA/IETF/etc) then it becomes much safer to
enshrine it forever in a common kernel uAPI.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Christoph Lameter]

On Thu, 5 May 2016, Jason Gunthorpe wrote:

> On Thu, May 05, 2016 at 11:21:29AM +0300, Matan Barak wrote:
> > >This should probably be done after creating the QP (and AH?), as a NIC
> > >may not be able to do certain LSO rules with certain QP/AH
> > >configurations.
> > >
> >
> > So I guess we want something that after a user creates a QP, he can query
> > which protocols (IP, TCP, etc) could be used in the NIC's segmentation
> > offload mechanism.
>
> This is where I get upset with the process we are following here,
> without input from other hardware architects in other companies, it is
> hard to design something truly common.

Presumably the other vendors are listening in on this conversation. The
lack of alternate proposals and objections by them has often been
considered silent approval in the past.

> When multiple vendors actually implement a verbs feature (or agree to
> implement one via IBTA/IETF/etc) then it becomes much safer to
> enshrine it forever in a common kernel uAPI.

On the other hand nothing is going to happen if one vendor does not push
ahead. There were multiple implementations in the IP stack as well until
things settled. We need to be able to do the same and not stifle
innovation by making a vendor to wait until the competition comes
up with something similar.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Jason Gunthorpe]

On Thu, May 05, 2016 at 09:20:49PM -0500, Christoph Lameter wrote:

> > This is where I get upset with the process we are following here,
> > without input from other hardware architects in other companies,
> > it is hard to design something truly common.
> 
> Presumably the other vendors are listening in on this conversation. The
> lack of alternate proposals and objections by them has often been
> considered silent approval in the past.

I know the various hardware architects that would be involved with a
IBTA process/etc do not monitor this list.

AFAIK, no other linux community works in a way where the Linux centric
mailing list sets hardware standards. Eg linux-scsi doesn't drive the
T10 agenda, linux-pci doesn't drive the PCISIG, etc. At best they
inspire work in those other communities.

> > When multiple vendors actually implement a verbs feature (or agree to
> > implement one via IBTA/IETF/etc) then it becomes much safer to
> > enshrine it forever in a common kernel uAPI.
> 
> On the other hand nothing is going to happen if one vendor does not push
> ahead. There were multiple implementations in the IP stack as well until
> things settled. We need to be able to do the same and not stifle
> innovation by making a vendor to wait until the competition comes
> up with something similar.

Agreed.

I'm just saying, do it in user space and leave the common kernel uAPI
alone until a more obvious consensus is reached. That should speed
everything up..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Christoph Lameter]

On Fri, 6 May 2016, Jason Gunthorpe wrote:

> > > This is where I get upset with the process we are following here,
> > > without input from other hardware architects in other companies,
> > > it is hard to design something truly common.
> >
> > Presumably the other vendors are listening in on this conversation. The
> > lack of alternate proposals and objections by them has often been
> > considered silent approval in the past.
>
> I know the various hardware architects that would be involved with a
> IBTA process/etc do not monitor this list.

An IBTA process? How would that be relevant here?

> > On the other hand nothing is going to happen if one vendor does not push
> > ahead. There were multiple implementations in the IP stack as well until
> > things settled. We need to be able to do the same and not stifle
> > innovation by making a vendor to wait until the competition comes
> > up with something similar.
>
> Agreed.
>
> I'm just saying, do it in user space and leave the common kernel uAPI
> alone until a more obvious consensus is reached. That should speed
> everything up..

How would you do this in user space?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Jason Gunthorpe]

On Mon, May 09, 2016 at 09:27:32AM -0500, Christoph Lameter wrote:
> On Fri, 6 May 2016, Jason Gunthorpe wrote:
> 
> > > > This is where I get upset with the process we are following here,
> > > > without input from other hardware architects in other companies,
> > > > it is hard to design something truly common.
> > >
> > > Presumably the other vendors are listening in on this conversation. The
> > > lack of alternate proposals and objections by them has often been
> > > considered silent approval in the past.
> >
> > I know the various hardware architects that would be involved with a
> > IBTA process/etc do not monitor this list.
> 
> An IBTA process? How would that be relevant here?

What do you mean? IBTA is the only multi-vendor body left working on
standardizing the hardware specification for verbs.

These various recent patches are adding new hardware features to
verbs.

If you want hardware knowledgeable people to help, then you need to go
to the forums they are active in.

This isn't just a software exercise.

> > > On the other hand nothing is going to happen if one vendor does not push
> > > ahead. There were multiple implementations in the IP stack as well until
> > > things settled. We need to be able to do the same and not stifle
> > > innovation by making a vendor to wait until the competition comes
> > > up with something similar.
> >
> > Agreed.
> >
> > I'm just saying, do it in user space and leave the common kernel uAPI
> > alone until a more obvious consensus is reached. That should speed
> > everything up..
> 
> How would you do this in user space?

Use udata to get the driver to do the little bits needed.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Hefty, Sean]

> > An IBTA process? How would that be relevant here?
> 
> What do you mean? IBTA is the only multi-vendor body left working on
> standardizing the hardware specification for verbs.
> 
> These various recent patches are adding new hardware features to
> verbs.
> 
> If you want hardware knowledgeable people to help, then you need to go
> to the forums they are active in.
> 
> This isn't just a software exercise.

Previous changes to the uABI (for example, XRC) were not accepted until they were standardized by the IBTA.  An exception that I'm aware of are the flow specs.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1327 bytes --]

On Mon, May 09, 2016 at 10:35:37AM -0600, Jason Gunthorpe wrote:
> On Mon, May 09, 2016 at 09:27:32AM -0500, Christoph Lameter wrote:
> > On Fri, 6 May 2016, Jason Gunthorpe wrote:
> > 
> > > > > This is where I get upset with the process we are following here,
> > > > > without input from other hardware architects in other companies,
> > > > > it is hard to design something truly common.
> > > >
> > > > Presumably the other vendors are listening in on this conversation. The
> > > > lack of alternate proposals and objections by them has often been
> > > > considered silent approval in the past.
> > >
> > > I know the various hardware architects that would be involved with a
> > > IBTA process/etc do not monitor this list.
> > 
> > An IBTA process? How would that be relevant here?
> 
> What do you mean? IBTA is the only multi-vendor body left working on
> standardizing the hardware specification for verbs.
> 
> These various recent patches are adding new hardware features to
> verbs.

These patches are exporting hardware capabilities and not "adding new
hardware features".

> 
> If you want hardware knowledgeable people to help, then you need to go
> to the forums they are active in.

It doesn't make sense to look after these people who have no interest in
linux RDMA at all.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Jason Gunthorpe]

On Mon, May 09, 2016 at 10:16:13PM +0300, Leon Romanovsky wrote:

> > If you want hardware knowledgeable people to help, then you need to go
> > to the forums they are active in.
> 
> It doesn't make sense to look after these people who have no interest in
> linux RDMA at all.

Linux RDMA is only one half of the coin - most of our uAPIs are
backed by a hardware implementation, and thus defining a bad API
directly limits future hardware designs. It is very foolish not to ask
the people who are designing future hardware what they think.

This isn't like other places in the kernel where everything is just
software.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Jason Gunthorpe]

On Mon, May 09, 2016 at 07:11:23PM +0000, Hefty, Sean wrote:
> > > An IBTA process? How would that be relevant here?
> > 
> > What do you mean? IBTA is the only multi-vendor body left working on
> > standardizing the hardware specification for verbs.
> > 
> > These various recent patches are adding new hardware features to
> > verbs.
> > 
> > If you want hardware knowledgeable people to help, then you need to go
> > to the forums they are active in.
> > 
> > This isn't just a software exercise.
> 
> Previous changes to the uABI (for example, XRC) were not accepted
> until they were standardized by the IBTA.  An exception that I'm
> aware of are the flow specs.

We need to decide as a community what we want to do here, because this
uncertain process is getting tiring for everyone.

The Collab Summit discussions ended with a general consensus that some
things would go to the driver-specific channel, while hardware
behavior related to the common uAPI would go through at least a
multi-vendor sign off process - like IBTA.

We didn't talk about how to expose them to apps in userspace once they
were exposed through the kernel.

It is pretty obvious the current 'process' isn't working. This new
idea that the common uAPI should just be a union of every vendor's
unique hardware ideas has not resulted in patches being merged any
faster; and I think the quality of the API is not as good now..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Hefty, Sean]

> We need to decide as a community what we want to do here, because this
> uncertain process is getting tiring for everyone.

Agree greatly

> The Collab Summit discussions ended with a general consensus that some
> things would go to the driver-specific channel, while hardware
> behavior related to the common uAPI would go through at least a
> multi-vendor sign off process - like IBTA.

I agree with this approach.

> We didn't talk about how to expose them to apps in userspace once they
> were exposed through the kernel.

I think this will end up being vendor specific, and I'm not sure each vendor knows what they want to do.

> It is pretty obvious the current 'process' isn't working. This new
> idea that the common uAPI should just be a union of every vendor's
> unique hardware ideas has not resulted in patches being merged any
> faster; and I think the quality of the API is not as good now..

IMO, crafting a common API means a higher level of abstraction.  The lower the API, the more it needs to be driver specific.  I prefer a method that: allows vendors to stay out of each other's way, and quickly allows vendors to export their features to user space.  Either of these options can work for that.  I don't see that a low-level, common API is suitable.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1026 bytes --]

On Mon, May 09, 2016 at 01:57:34PM -0600, Jason Gunthorpe wrote:
> On Mon, May 09, 2016 at 10:16:13PM +0300, Leon Romanovsky wrote:
> 
> > > If you want hardware knowledgeable people to help, then you need to go
> > > to the forums they are active in.
> > 
> > It doesn't make sense to look after these people who have no interest in
> > linux RDMA at all.
> 
> Linux RDMA is only one half of the coin - most of our uAPIs are
> backed by a hardware implementation, and thus defining a bad API
> directly limits future hardware designs. It is very foolish not to ask
> the people who are designing future hardware what they think.

Exactly, this is why it is posted on the open mailing list where
everyone has stage to express their opinion, including these mysterious
hardware architecture who cares about future design, but doesn't care about
its future support.

The intension for submitted new uAPI to be generic from the
beginning and authors are always open to adjust it to answer
reviews feedback.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1726 bytes --]

On Mon, May 09, 2016 at 02:05:07PM -0600, Jason Gunthorpe wrote:
> On Mon, May 09, 2016 at 07:11:23PM +0000, Hefty, Sean wrote:
> > > > An IBTA process? How would that be relevant here?
> > > 
> > > What do you mean? IBTA is the only multi-vendor body left working on
> > > standardizing the hardware specification for verbs.
> > > 
> > > These various recent patches are adding new hardware features to
> > > verbs.
> > > 
> > > If you want hardware knowledgeable people to help, then you need to go
> > > to the forums they are active in.
> > > 
> > > This isn't just a software exercise.
> > 
> > Previous changes to the uABI (for example, XRC) were not accepted
> > until they were standardized by the IBTA.  An exception that I'm
> > aware of are the flow specs.
> 
> We need to decide as a community what we want to do here, because this
> uncertain process is getting tiring for everyone.
> 
> The Collab Summit discussions ended with a general consensus that some
> things would go to the driver-specific channel, while hardware
> behavior related to the common uAPI would go through at least a
> multi-vendor sign off process - like IBTA.

Jason,
We did ALL and much more from the discussed for RSS patches and at the end,
we saw half a word from the maintainer about the future plans for its acceptance.

> 
> We didn't talk about how to expose them to apps in userspace once they
> were exposed through the kernel.
> 
> It is pretty obvious the current 'process' isn't working.

There is no meaning of "process" without people and their execution, and right
now, I don't see any execution of it in the linux RDMA community (fixes
and ULP changes are not merged for a while).

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

RE: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Steve Wise]

> 
> > We need to decide as a community what we want to do here, because this
> > uncertain process is getting tiring for everyone.
> 
> Agree greatly
> 
> > The Collab Summit discussions ended with a general consensus that some
> > things would go to the driver-specific channel, while hardware
> > behavior related to the common uAPI would go through at least a
> > multi-vendor sign off process - like IBTA.
> 
> I agree with this approach.
> 
> > We didn't talk about how to expose them to apps in userspace once they
> > were exposed through the kernel.
> 
> I think this will end up being vendor specific, and I'm not sure each vendor
knows
> what they want to do.
> 
> > It is pretty obvious the current 'process' isn't working. This new
> > idea that the common uAPI should just be a union of every vendor's
> > unique hardware ideas has not resulted in patches being merged any
> > faster; and I think the quality of the API is not as good now..
> 
> IMO, crafting a common API means a higher level of abstraction.  The lower the
> API, the more it needs to be driver specific.  I prefer a method that: allows
vendors
> to stay out of each other's way, and quickly allows vendors to export their
features
> to user space.  Either of these options can work for that.  I don't see that a
low-
> level, common API is suitable.

Hey all,

I've been trying to follow this thread, and there has been comments that other
device owners should speak up.  But I'm not sure what LSO would do for an iWARP
RC QP connection.  Can someone give me a quick summary?  

Because as it stands now, a ~4GB send can be posted in 1 work request, and FW/HW
(in the iWARP DDP layer) deals with segmenting it based on the MTU for the
interface, and then assembling it to deliver as a single RECV completion at the
peer.  What would LSO do beyond this?
 
Sorry for the dumb question.

Steve

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Jason Gunthorpe]

On Tue, May 10, 2016 at 10:03:48AM -0500, Steve Wise wrote:

> I've been trying to follow this thread, and there has been comments that other
> device owners should speak up.  But I'm not sure what LSO would do for an iWARP
> RC QP connection.  Can someone give me a quick summary?  

It is unlikely someone would use LSO and an iWarp RC QP together.

However, an iWarp NIC may want to expose the raw ethernet QP style of
feature, which would probably also want to include the LSO, RSS,
timestamping etc features.

Most likely iWarp cards already include LSO features in their
netdevice drivers? In that instance I'd say, imagine that same LSO
feature works on a verbs QP and ask if the patch covers the way it
works.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Steve Wise]

> 
> > I've been trying to follow this thread, and there has been comments that
other
> > device owners should speak up.  But I'm not sure what LSO would do for an
iWARP
> > RC QP connection.  Can someone give me a quick summary?
> 
> It is unlikely someone would use LSO and an iWarp RC QP together.
> 
> However, an iWarp NIC may want to expose the raw ethernet QP style of
> feature, which would probably also want to include the LSO, RSS,
> timestamping etc features.
> 

Yes, that makes sense.  Chelsio has something like this that is currently
out-of-kernel.  But there is no LSO/RSS support.  And it needs major rewrite to
become acceptable for inclusion upstream.  But I can review the changes with
this in mind.  

> Most likely iWarp cards already include LSO features in their
> netdevice drivers? In that instance I'd say, imagine that same LSO
> feature works on a verbs QP and ask if the patch covers the way it
> works.
>

All of Chelsio's cards support LSO features (and all the other important NIC
features).  I'll think about the proposed API and if it would be sufficient for
a RAW_QP type of service.

Thanks for the summary Jason!  

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Jason Gunthorpe]

> Exactly, this is why it is posted on the open mailing list where
> everyone has stage to express their opinion, including these mysterious
> hardware architecture who cares about future design, but doesn't care about
> its future support.

Look, if you want linux-rdma to be setting hardware standards then get
a consensus from the vendors on this idea.

This was tried at the Collab summit and I heard several representatives
strongly say no.

It is not an inherently unworkable idea.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Hefty, Sean]

> I've been trying to follow this thread, and there has been comments that
> other
> device owners should speak up.  But I'm not sure what LSO would do for an
> iWARP
> RC QP connection.  Can someone give me a quick summary?

Well, it's not clear to me yet what this capability actually does.  Jason described it as some sort of work request 'chaining'.  I tried briefly searching for details on the web, but didn't find any significant documentation.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Jason Gunthorpe]

On Tue, May 10, 2016 at 07:58:01AM +0300, Leon Romanovsky wrote:
> > The Collab Summit discussions ended with a general consensus that some
> > things would go to the driver-specific channel, while hardware
> > behavior related to the common uAPI would go through at least a
> > multi-vendor sign off process - like IBTA.
> 
> We did ALL and much more from the discussed for RSS patches and at the end,
> we saw half a word from the maintainer about the future plans for its acceptance.

Where is the 'multi-vendor sign off' part of the process?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 977 bytes --]

On Tue, May 10, 2016 at 11:07:39AM -0600, Jason Gunthorpe wrote:
> > Exactly, this is why it is posted on the open mailing list where
> > everyone has stage to express their opinion, including these mysterious
> > hardware architecture who cares about future design, but doesn't care about
> > its future support.
> 
> Look, if you want linux-rdma to be setting hardware standards then get
> a consensus from the vendors on this idea.

I want fruitful discussion on this mailing list which can be beneficial
to everyone, right now all discussions can be summarized in the following
sentence: "i don't need it, no one need it".

And to which vendors are you referring exactly? They are all on this mailing list.

> 
> This was tried at the Collab summit and I heard several representatives
> strongly say no.

I may admit that I missed part of the discussions and my knowledge is
limited. Can you post their names and their practical suggestions?

Thanks

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 771 bytes --]

On Tue, May 10, 2016 at 11:18:19AM -0600, Jason Gunthorpe wrote:
> On Tue, May 10, 2016 at 07:58:01AM +0300, Leon Romanovsky wrote:
> > > The Collab Summit discussions ended with a general consensus that some
> > > things would go to the driver-specific channel, while hardware
> > > behavior related to the common uAPI would go through at least a
> > > multi-vendor sign off process - like IBTA.
> > 
> > We did ALL and much more from the discussed for RSS patches and at the end,
> > we saw half a word from the maintainer about the future plans for its acceptance.
> 
> Where is the 'multi-vendor sign off' part of the process?

The steps are described at the following mail [1].

[1] http://marc.info/?l=linux-rdma&m=146218340000784&w=2

> 
> Jason

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Jason Gunthorpe]

On Tue, May 10, 2016 at 08:52:00PM +0300, Leon Romanovsky wrote:
> On Tue, May 10, 2016 at 11:18:19AM -0600, Jason Gunthorpe wrote:
> > On Tue, May 10, 2016 at 07:58:01AM +0300, Leon Romanovsky wrote:
> > > > The Collab Summit discussions ended with a general consensus
> > > > that some things would go to the driver-specific channel,
> > > > while hardware behavior related to the common uAPI would go
> > > > through at least a multi-vendor sign off process - like IBTA.
> > > 
> > > We did ALL and much more from the discussed for RSS patches and
> > > at the end, we saw half a word from the maintainer about the
> > > future plans for its acceptance.
> > 
> > Where is the 'multi-vendor sign off' part of the process?
> 
> The steps are described at the following mail [1].
> 
> [1] http://marc.info/?l=linux-rdma&m=146218340000784&w=2

None of that is what I mean when I say multi-vendor sign off.

I mean a vote showing multi-vendor participation, or active support
from other vendors.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1254 bytes --]

On Tue, May 10, 2016 at 12:18:47PM -0600, Jason Gunthorpe wrote:
> On Tue, May 10, 2016 at 08:52:00PM +0300, Leon Romanovsky wrote:
> > On Tue, May 10, 2016 at 11:18:19AM -0600, Jason Gunthorpe wrote:
> > > On Tue, May 10, 2016 at 07:58:01AM +0300, Leon Romanovsky wrote:
> > > > > The Collab Summit discussions ended with a general consensus
> > > > > that some things would go to the driver-specific channel,
> > > > > while hardware behavior related to the common uAPI would go
> > > > > through at least a multi-vendor sign off process - like IBTA.
> > > > 
> > > > We did ALL and much more from the discussed for RSS patches and
> > > > at the end, we saw half a word from the maintainer about the
> > > > future plans for its acceptance.
> > > 
> > > Where is the 'multi-vendor sign off' part of the process?
> > 
> > The steps are described at the following mail [1].
> > 
> > [1] http://marc.info/?l=linux-rdma&m=146218340000784&w=2
> 
> None of that is what I mean when I say multi-vendor sign off.
> 
> I mean a vote showing multi-vendor participation, or active support
> from other vendors.

They participated in OFA/OFAWG/OFVWG and this ML and didn't said no.
It is more than enough to be convinced.

> 
> Jason

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1430 bytes --]

On Tue, May 10, 2016 at 10:26:55PM +0300, Leon Romanovsky wrote:
> On Tue, May 10, 2016 at 12:18:47PM -0600, Jason Gunthorpe wrote:
> > On Tue, May 10, 2016 at 08:52:00PM +0300, Leon Romanovsky wrote:
> > > On Tue, May 10, 2016 at 11:18:19AM -0600, Jason Gunthorpe wrote:
> > > > On Tue, May 10, 2016 at 07:58:01AM +0300, Leon Romanovsky wrote:
> > > > > > The Collab Summit discussions ended with a general consensus
> > > > > > that some things would go to the driver-specific channel,
> > > > > > while hardware behavior related to the common uAPI would go
> > > > > > through at least a multi-vendor sign off process - like IBTA.
> > > > > 
> > > > > We did ALL and much more from the discussed for RSS patches and
> > > > > at the end, we saw half a word from the maintainer about the
> > > > > future plans for its acceptance.
> > > > 
> > > > Where is the 'multi-vendor sign off' part of the process?
> > > 
> > > The steps are described at the following mail [1].
> > > 
> > > [1] http://marc.info/?l=linux-rdma&m=146218340000784&w=2
> > 
> > None of that is what I mean when I say multi-vendor sign off.
> > 
> > I mean a vote showing multi-vendor participation, or active support
> > from other vendors.
> 
> They participated in OFA/OFAWG/OFVWG and this ML and didn't said no.
> It is more than enough to be convinced.

Sorry for typos,
s/didn't said/didn't say

> 
> > 
> > Jason



[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Christoph Lameter]

On Tue, 10 May 2016, Jason Gunthorpe wrote:

> > Exactly, this is why it is posted on the open mailing list where
> > everyone has stage to express their opinion, including these mysterious
> > hardware architecture who cares about future design, but doesn't care about
> > its future support.
>
> Look, if you want linux-rdma to be setting hardware standards then get
> a consensus from the vendors on this idea.

This is about exporting existing hardware features that are already
available through the standard socket APIs.

> This was tried at the Collab summit and I heard several representatives
> strongly say no.

Hardware standards? I thought we were talking about software and APIs.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Jason Gunthorpe]

On Tue, May 10, 2016 at 07:29:42PM -0500, Christoph Lameter wrote:
> > > Exactly, this is why it is posted on the open mailing list where
> > > everyone has stage to express their opinion, including these mysterious
> > > hardware architecture who cares about future design, but doesn't care about
> > > its future support.
> >
> > Look, if you want linux-rdma to be setting hardware standards then get
> > a consensus from the vendors on this idea.
> 
> This is about exporting existing hardware features that are already
> available through the standard socket APIs.

Eh? How is LSO or RSS available through standard socket APIs? Yes, it
is indirectly through the TCP stack, but the actual hardware feature
is not directly accessible???

> > This was tried at the Collab summit and I heard several representatives
> > strongly say no.
> 
> Hardware standards? I thought we were talking about software and APIs.

The LSO, RSS and Timestamp patches all demand very specific hardware
functionality for a device to implement the API. It is fundamentally
not a 100% software API.

Look at the patches. The LSO patch is so bad Sean and Steve couldn't
even figure out what the heck the HW feature does. I know what it
does, but I can't tell you the fine details of the actual required
hardware behavior.

How on earth is someone else supposed to ever implement hardware that
does this?

I still don't understand why this is being forced into
libibverbs. dpdk would seem to be the prefered way to access these
sorts of features and there is no technical reason why the mlx
dpdk provider needs to use libibverbs. Talk directly to /dev/uverbs0
and use udata to enable the universe of nic specific features.

For instance, the dpdk community has already done the hard work to
figure out how to expose things like LSO to their users - and they
actually have multiple vendors providing these features...

http://dpdk.org/ml/archives/dev/2014-May/002537.html

I guess you are going to tell me you want these features on IB UD
transport and dpdk has no support for L2's other than ethernet?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Bodong Wang]

On 5/10/2016 9:04 PM, Jason Gunthorpe wrote:
> On Tue, May 10, 2016 at 07:29:42PM -0500, Christoph Lameter wrote:
>>> This was tried at the Collab summit and I heard several representatives
>>> strongly say no.
>> Hardware standards? I thought we were talking about software and APIs.
> The LSO, RSS and Timestamp patches all demand very specific hardware
> functionality for a device to implement the API. It is fundamentally
> not a 100% software API.
>
> Look at the patches. The LSO patch is so bad Sean and Steve couldn't
> even figure out what the heck the HW feature does. I know what it
> does, but I can't tell you the fine details of the actual required
> hardware behavior.
>
> How on earth is someone else supposed to ever implement hardware that
> does this?
>
> I still don't understand why this is being forced into
> libibverbs. dpdk would seem to be the prefered way to access these
> sorts of features and there is no technical reason why the mlx
> dpdk provider needs to use libibverbs. Talk directly to /dev/uverbs0
> and use udata to enable the universe of nic specific features.
There are other user space libraries which are build on top of libibverbs rather than DPDK. E.g, VMA.

>
> For instance, the dpdk community has already done the hard work to
> figure out how to expose things like LSO to their users - and they
> actually have multiple vendors providing these features...
>
> http://dpdk.org/ml/archives/dev/2014-May/002537.html
>
> I guess you are going to tell me you want these features on IB UD
> transport and dpdk has no support for L2's other than ethernet?
>
> Jason
>
The patches from kernel are purely for hardware capabilities report. The policy and logic about how to implement the LSO are living in libibverbs and hardware related libraies(e.g, libmlx5). Review these patches together might make things more clear. They're not pushed to community for review yet because we still can't reach the agreement from this very first step.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 846 bytes --]

On Tue, May 10, 2016 at 08:04:29PM -0600, Jason Gunthorpe wrote:
> Look at the patches. The LSO patch is so bad Sean and Steve couldn't
> even figure out what the heck the HW feature does. I know what it
> does, but I can't tell you the fine details of the actual required
> hardware behavior.

These people are very busy and you can't expect from them to know
everything, If someone doesn't know/understand, he will ask, exactly
as they, you and me are doing all the time.

Just to remind you, that YOUR's patch [1] didn't meet YOUR own quality
criteria and can be called bad one, just because very respectful people
asked their questions which very expected to be answered in the patch
itself, but they don't [2].

[1] e6bd18f57aad ("IB/security: Restrict use of the write() interface")
[2] http://marc.info/?l=linux-rdma&m=146264707224127&w=2

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Christoph Lameter]

On Tue, 10 May 2016, Jason Gunthorpe wrote:

> I still don't understand why this is being forced into
> libibverbs. dpdk would seem to be the prefered way to access these
> sorts of features and there is no technical reason why the mlx
> dpdk provider needs to use libibverbs. Talk directly to /dev/uverbs0
> and use udata to enable the universe of nic specific features.

Well then the trouble is how to integrate that into QPs and the rest of
the RDMA infrastructure.

>
> For instance, the dpdk community has already done the hard work to
> figure out how to expose things like LSO to their users - and they
> actually have multiple vendors providing these features...
>
> http://dpdk.org/ml/archives/dev/2014-May/002537.html
>
> I guess you are going to tell me you want these features on IB UD
> transport and dpdk has no support for L2's other than ethernet?

Our software is based on raw ethernet frame processing through QPs. Thus
the desire to have the extras that the hardware can provide. And yes we
want to have an API that can be used to interface with hardware from
multiple vendors.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Jason Gunthorpe]

On Wed, May 11, 2016 at 09:23:30AM -0500, Christoph Lameter wrote:
> On Tue, 10 May 2016, Jason Gunthorpe wrote:
> 
> > I still don't understand why this is being forced into
> > libibverbs. dpdk would seem to be the prefered way to access these
> > sorts of features and there is no technical reason why the mlx
> > dpdk provider needs to use libibverbs. Talk directly to /dev/uverbs0
> > and use udata to enable the universe of nic specific features.
> 
> Well then the trouble is how to integrate that into QPs and the rest of
> the RDMA infrastructure.

Granted, yes, dpdk obviously hasn't had much work in that area,
but it also doesn't seem all that complex.

You basically want to attach a dpdk pmd to something other than
ethernet hardware. Eg a IB UD QP using ipoib. At the end of the day
that is just a few more parameters when starting the pmd, and a
different hwaddress length. To me that sounds much simpler than
squeezing all of this endless stuff into libibverbs.

Assuming that could be done, the dpdk community seems to have a much
better handle on how to design and expose these IP acceleration
features. All we need to do in RDMA land is just let the mlx4 pmd and
mlx4 kernel driver communicate privately to do the special
setup. We've already largely done that with udata. I don't think it
would be contentious at all if mlx4 developed some 'dpdk'
driver-specific calls. Much like how hfi1 has psm driver specific
calls.

Overall, dpdk looks alot better for building IP centric apps - it has
more support utilities to work with IP packets, more optimization, and
more device support for this kind of function. libibverbs always seemed
like a poor fit to me..

.. and having something micro-optimized to only do IP would certainly
speed up wr and wc processing in the pmd, code paths and structure
members to support other opcodes can just be purged entirely. That is
the cache line efficiency you were talking about in the other thread.

> Our software is based on raw ethernet frame processing through QPs. Thus
> the desire to have the extras that the hardware can provide. And yes we
> want to have an API that can be used to interface with hardware from
> multiple vendors.

Well, then you need multiple vendors to agree they could potentially
build hardware that matches these proposed uAPIs. Which is more or less
what I've been more or less asking for :)

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Jason Gunthorpe]

On Wed, May 11, 2016 at 07:31:34AM +0300, Leon Romanovsky wrote:
> On Tue, May 10, 2016 at 08:04:29PM -0600, Jason Gunthorpe wrote:
> > Look at the patches. The LSO patch is so bad Sean and Steve couldn't
> > even figure out what the heck the HW feature does. I know what it
> > does, but I can't tell you the fine details of the actual required
> > hardware behavior.
> 
> These people are very busy and you can't expect from them to know
> everything, If someone doesn't know/understand, he will ask, exactly
> as they, you and me are doing all the time.
> 
> Just to remind you, that YOUR's patch [1] didn't meet YOUR own quality
> criteria and can be called bad one, just because very respectful people
> asked their questions which very expected to be answered in the patch
> itself, but they don't [2].

Oh come on, can you please get serious about this? Stubbornly taking a
baseless and totally obstinate position is not working with the
community.

A security patch, which is deliberately a little obtuse to avoid
telling the world how to write an exploit, is not at all the same as
describing a complex new uAPI feature that is expected to be
implementable by other parties.

I can't believe you'd even try to use that as an excuse.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1798 bytes --]

On Wed, May 11, 2016 at 11:36:08AM -0600, Jason Gunthorpe wrote:
> On Wed, May 11, 2016 at 07:31:34AM +0300, Leon Romanovsky wrote:
> > On Tue, May 10, 2016 at 08:04:29PM -0600, Jason Gunthorpe wrote:
> > > Look at the patches. The LSO patch is so bad Sean and Steve couldn't
> > > even figure out what the heck the HW feature does. I know what it
> > > does, but I can't tell you the fine details of the actual required
> > > hardware behavior.
> > 
> > These people are very busy and you can't expect from them to know
> > everything, If someone doesn't know/understand, he will ask, exactly
> > as they, you and me are doing all the time.
> > 
> > Just to remind you, that YOUR's patch [1] didn't meet YOUR own quality
> > criteria and can be called bad one, just because very respectful people
> > asked their questions which very expected to be answered in the patch
> > itself, but they don't [2].
> 
> Oh come on, can you please get serious about this? Stubbornly taking a
> baseless and totally obstinate position is not working with the
> community.
> 
> A security patch, which is deliberately a little obtuse to avoid
> telling the world how to write an exploit, is not at all the same as
> describing a complex new uAPI feature that is expected to be
> implementable by other parties.
> 
> I can't believe you'd even try to use that as an excuse.

It is not an excuse and never supposed to be.

My position is very concrete on this topic: post code on ML, start
discussion, adjust the code and merge it.

It works for all other code posted on various linux kernel ML and
I don't see why it can't work for RDMA too.

And again, there is openness and readiness of all participants in this
ML to answer feedback and post adjusted code.

> 
> Jason

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Jason Gunthorpe]

On Wed, May 11, 2016 at 03:47:14AM +0000, Bodong Wang wrote:

> > I still don't understand why this is being forced into
> > libibverbs. dpdk would seem to be the prefered way to access these
> > sorts of features and there is no technical reason why the mlx
> > dpdk provider needs to use libibverbs. Talk directly to /dev/uverbs0
> > and use udata to enable the universe of nic specific features.

> There are other user space libraries which are build on top of
> libibverbs rather than DPDK. E.g, VMA.

I looked at the VMA manual a bit, and honestly, it looks a lot like
DPDK. It could hook into the kernel the same way (via udata and
/dev/uverbs0)

> The patches from kernel are purely for hardware capabilities
> report. The policy and logic about how to implement the LSO are
> living in libibverbs and hardware related libraies(e.g,
> libmlx5). Review these patches together might make things more
> clear. They're not pushed to community for review yet because we
> still can't reach the agreement from this very first step.

That seems like a great argument that this doesn't belong in the
common api. If it is just driver capabilities, and they are not well
defined, then put them in mlx4's udata.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Christoph Lameter]

On Wed, 11 May 2016, Jason Gunthorpe wrote:

> You basically want to attach a dpdk pmd to something other than
> ethernet hardware. Eg a IB UD QP using ipoib. At the end of the day
> that is just a few more parameters when starting the pmd, and a
> different hwaddress length. To me that sounds much simpler than
> squeezing all of this endless stuff into libibverbs.

DPDK handles all network traffic for a nic. What we have with the raw
ethernet QPs is the ability to offload individual Ethernet data streams to
user space. That is not possible with DPDK and very much a QP based
feature.

> Overall, dpdk looks alot better for building IP centric apps - it has
> more support utilities to work with IP packets, more optimization, and
> more device support for this kind of function. libibverbs always seemed
> like a poor fit to me..

It does not allow the kernel to handle the ethernet traffic in general but
requires a full implementation of an IP stack with all the extras.

> Well, then you need multiple vendors to agree they could potentially
> build hardware that matches these proposed uAPIs. Which is more or less
> what I've been more or less asking for :)

I did not see disagreement from the other vendors.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/2] IB/core: Report LSO capabilities when querying device

[Jason Gunthorpe]

On Thu, May 12, 2016 at 08:34:41AM -0500, Christoph Lameter wrote:
> On Wed, 11 May 2016, Jason Gunthorpe wrote:
> 
> > You basically want to attach a dpdk pmd to something other than
> > ethernet hardware. Eg a IB UD QP using ipoib. At the end of the day
> > that is just a few more parameters when starting the pmd, and a
> > different hwaddress length. To me that sounds much simpler than
> > squeezing all of this endless stuff into libibverbs.
> 
> DPDK handles all network traffic for a nic.

Eh? The DPDK documentation for the mlx4 PMD disagrees:

 This capability allows the PMD to coexist with kernel network
 interfaces which remain functional, although they stop receiving
 unicast packets as long as they share the same MAC address.

The mlx4 PMD *is* QP based, clearly there is no fundamental obstacle
to providing the dpdk API over a flow-steering QP hardware model.

>From that point it becomes a problem of configuring the QP's flow
steering to get the exact split you need, which is exactly the same as
what is being done with verbs today.

Don't be confused that the typical example of dpdk uses a sr-iov
hardware driver, that is just reflecting a limitation of the NIC
hardware available, better hardware like mlx4 can and does use a QP
model to steer the packets instead.

> > build hardware that matches these proposed uAPIs. Which is more or less
> > what I've been more or less asking for :)
> 
> I did not see disagreement from the other vendors.

You don't find Steve's remarks troubling then?

I view Chelsio as one the vendors that could actually implement the
hardware side of these proposed APIs with a firmware update and he
hasn't even looked at them, and has yet to understand what they even
do..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html