[PATCH RFC] Documentation/infiniband: Add docs for rdma-helpers

[PATCH RFC] Documentation/infiniband: Add docs for rdma-helpers

[Michael Wang]

This is the following patch for:
  https://lkml.org/lkml/2015/5/5/417
which try to document the settled rdma_cap_XX().

Highlights:
  There could be many missing/mistakes/misunderstanding, please don't
  be hesitate to point out the issues, any suggestions to improve or
  complete the description are very welcomed ;-)

Signed-off-by: Michael Wang <yun.wang@profitbricks.com>
---
 Documentation/infiniband/rdma_helpers.txt | 76 +++++++++++++++++++++++++++++++
 1 file changed, 76 insertions(+)
 create mode 100644 Documentation/infiniband/rdma_helpers.txt

diff --git a/Documentation/infiniband/rdma_helpers.txt b/Documentation/infiniband/rdma_helpers.txt
new file mode 100644
index 0000000..abc75ec
--- /dev/null
+++ b/Documentation/infiniband/rdma_helpers.txt
@@ -0,0 +1,76 @@
+RDMA HELPERS
+
+  For core layer, below helpers are used to check if a paticular capability
+  is supported by the port.
+
+    rdma_cap_ib_mad		- Infiniband Management Datagrams.
+    rdma_cap_ib_smi		- Infiniband Subnet Management Interface.
+    rdma_cap_ib_cm		- Infiniband Communication Manager.
+    rdma_cap_iw_cm		- IWARP Communication Manager.
+    rdma_cap_ib_sa		- Infiniband Subnet Administration.
+    rdma_cap_ib_mcast		- Infiniband Multicast.
+    rdma_cap_read_multi_sge	- RDMA Read Multiple Scatter-Gather Entries.
+    rdma_cap_af_ib		- Native Infiniband Address.
+    rdma_cap_eth_ah		- Ethernet Address Handler.
+
+USAGE
+
+  if (rdma_cap_XX(device, i)) {
+	/* The port i of device support XX */
+	...
+  } else {
+	/* The port i of device don't support XX */
+	...
+  }
+
+  rdma_cap_ib_mad
+  ---------------
+    Management Datagrams (MAD) is the prototype of management packet
+    to be used by all the kinds of infiniband managers, use the helper
+    to verify the port before utilize related features.
+
+  rdma_cap_ib_smi
+  ---------------
+    Subnet Management Interface (SMI) will handle SMP packet from SM
+    in an infiniband fabric, use the helper to verify the port before
+    utilize related features.
+
+  rdma_cap_ib_cm
+  ---------------
+    Communication Manager (CM) will handle the connections between
+    adaptors, currently there are two different implementation,
+    IB or IWARP, use the helper to verify whether the port using
+    IB-CM or not
+
+  rdma_cap_iw_cm
+  ---------------
+    IWARP has it's own implemented CM which is different from infiniband,
+    use the helper to check whether the port using IWARP-CM or not.
+
+  rdma_cap_ib_sa
+  ---------------
+    Subnet Administration (SA) is the database built by SM in an
+    infiniband fabric, use the helper to verify the port before
+    utilize related features.
+
+  rdma_cap_ib_mcast
+  ---------------
+    Multicast is the feature for one QP to send messages to multiple
+    QP in an infiniband fabric, use the helper to verify the port before
+    utilize related features.
+
+  rdma_cap_read_multi_sge
+  ---------------
+    RDMA read operation could support multiple scatter-gather entries,
+    use the helper to verify wthether the port support this feature
+    or not.
+
+  rdma_cap_af_ib
+  ---------------
+    RDMA address format could be ethernet or infiniband, use the helper
+    to verify whether the port support infiniband format or not.
+
+  rdma_cap_eth_ah
+  ---------------
+    Infiniband address handler format is special in ethernet fabric, use
+    the helper to verify whether the port is using ethernet format or not.
-- 
2.1.0

Re: [PATCH RFC] Documentation/infiniband: Add docs for rdma-helpers

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 6039 bytes --]

On Wed, 2015-05-13 at 15:24 +0200, Michael Wang wrote:
> This is the following patch for:
>   https://lkml.org/lkml/2015/5/5/417
> which try to document the settled rdma_cap_XX().
> 
> Highlights:
>   There could be many missing/mistakes/misunderstanding, please don't
>   be hesitate to point out the issues, any suggestions to improve or
>   complete the description are very welcomed ;-)
> 
> Signed-off-by: Michael Wang <yun.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>
> ---
>  Documentation/infiniband/rdma_helpers.txt | 76 +++++++++++++++++++++++++++++++
>  1 file changed, 76 insertions(+)
>  create mode 100644 Documentation/infiniband/rdma_helpers.txt
> 
> diff --git a/Documentation/infiniband/rdma_helpers.txt b/Documentation/infiniband/rdma_helpers.txt
> new file mode 100644
> index 0000000..abc75ec
> --- /dev/null
> +++ b/Documentation/infiniband/rdma_helpers.txt
> @@ -0,0 +1,76 @@
> +RDMA HELPERS
> +
> +  For core layer, below helpers are used to check if a paticular capability
> +  is supported by the port.

The following helpers are used to check the specific capabilities of a
particular port before utilizing those capabilities.

> +
> +    rdma_cap_ib_mad		- Infiniband Management Datagrams.
> +    rdma_cap_ib_smi		- Infiniband Subnet Management Interface.
> +    rdma_cap_ib_cm		- Infiniband Communication Manager.
InfiniBand Connection Management
> +    rdma_cap_iw_cm		- IWARP Communication Manager.
iWARP Connection Management
> +    rdma_cap_ib_sa		- Infiniband Subnet Administration.
> +    rdma_cap_ib_mcast		- Infiniband Multicast.
InfiniBand Multicast join/leave protocol
> +    rdma_cap_read_multi_sge	- RDMA Read Multiple Scatter-Gather Entries.
RDMA Read verb supports more than 1 sge in the work request
> +    rdma_cap_af_ib		- Native Infiniband Address.
> +    rdma_cap_eth_ah		- Ethernet Address Handler.
Queue Pair is InfiniBand transport, but uses Ethernet address instead of
native InfiniBand address (aka, this is a RoCE QP, and that means
ethertype 0x8915 + GRH for RoCEv1 and IP/UDP to well known UDP port for
RoCEv2)
> +
> +USAGE
> +
> +  if (rdma_cap_XX(device, i)) {
> +	/* The port i of device support XX */
> +	...
> +  } else {
> +	/* The port i of device don't support XX */
> +	...
> +  }
> +
> +  rdma_cap_ib_mad
> +  ---------------
> +    Management Datagrams (MAD) is the prototype of management packet
> +    to be used by all the kinds of infiniband managers, use the helper
> +    to verify the port before utilize related features.
Management Datagrams (MAD) are a required part of the InfiniBand
specification and are supported on all InfiniBand devices.  A slightly
extended version are also supported on OPA interfaces.

I would drop all instances of "use the helper to verify..." as that's
redundant.  This whole doc is about using the helpers to verify things.

> +
> +  rdma_cap_ib_smi
> +  ---------------
> +    Subnet Management Interface (SMI) will handle SMP packet from SM
> +    in an infiniband fabric, use the helper to verify the port before
> +    utilize related features.
> +
> +  rdma_cap_ib_cm
> +  ---------------
> +    Communication Manager (CM) will handle the connections between
       ^Connection Manager (CM) service, used to ease the process of
connecting to a remote host.  The IB CM can be used to connect to remote
hosts using either InfiniBand or RoCE connections.  iWARP has its own
connection manager, see below.
> +    adaptors, currently there are two different implementation,
> +    IB or IWARP, use the helper to verify whether the port using
> +    IB-CM or not
> +
> +  rdma_cap_iw_cm
> +  ---------------
> +    IWARP has it's own implemented CM which is different from infiniband,
      iWARP connection manager.  Similar to the IB Connection Manager,
but only used on iWARP devices.
> +    use the helper to check whether the port using IWARP-CM or not.
> +
> +  rdma_cap_ib_sa
> +  ---------------
> +    Subnet Administration (SA) is the database built by SM in an
> +    infiniband fabric, use the helper to verify the port before
> +    utilize related features.
> +
> +  rdma_cap_ib_mcast
> +  ---------------
> +    Multicast is the feature for one QP to send messages to multiple
> +    QP in an infiniband fabric, use the helper to verify the port before
> +    utilize related features.

InfiniBand (and OPA) use a different multicast mechanism than
traditional IP multicast found on Ethernet devices.  If this capability
is true, then traditional IPv4/IPv6 multicast is handled by the IPoIB
layer and direct multicast joins and leaves are handled per the
InfiniBand specifications.

> +
> +  rdma_cap_read_multi_sge
> +  ---------------
> +    RDMA read operation could support multiple scatter-gather entries,
> +    use the helper to verify wthether the port support this feature
> +    or not.

Certain devices (iWARP in particular) have restrictions on the number of
scatter gather elements that can be present in an RDMA READ work
request.  This is true if the device does not have that restriction.

> +  rdma_cap_af_ib
> +  ---------------
> +    RDMA address format could be ethernet or infiniband, use the helper
> +    to verify whether the port support infiniband format or not.

Many code paths for traditional InfiniBand and RoCE links are the same,
but need minor differences to accommodate the different addresses on the
two types of connections.  This helper is true when the address of the
specific connection is of the InfiniBand native variety.

> +
> +  rdma_cap_eth_ah
> +  ---------------
> +    Infiniband address handler format is special in ethernet fabric, use
> +    the helper to verify whether the port is using ethernet format or not.

This helper is true when the address of the specific connection is of
the Ethernet (RoCE) variety.

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH RFC] Documentation/infiniband: Add docs for rdma-helpers

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 5987 bytes --]

On Wed, 2015-05-13 at 15:24 +0200, Michael Wang wrote:
> This is the following patch for:
>   https://lkml.org/lkml/2015/5/5/417
> which try to document the settled rdma_cap_XX().
> 
> Highlights:
>   There could be many missing/mistakes/misunderstanding, please don't
>   be hesitate to point out the issues, any suggestions to improve or
>   complete the description are very welcomed ;-)
> 
> Signed-off-by: Michael Wang <yun.wang@profitbricks.com>
> ---
>  Documentation/infiniband/rdma_helpers.txt | 76 +++++++++++++++++++++++++++++++
>  1 file changed, 76 insertions(+)
>  create mode 100644 Documentation/infiniband/rdma_helpers.txt
> 
> diff --git a/Documentation/infiniband/rdma_helpers.txt b/Documentation/infiniband/rdma_helpers.txt
> new file mode 100644
> index 0000000..abc75ec
> --- /dev/null
> +++ b/Documentation/infiniband/rdma_helpers.txt
> @@ -0,0 +1,76 @@
> +RDMA HELPERS
> +
> +  For core layer, below helpers are used to check if a paticular capability
> +  is supported by the port.

The following helpers are used to check the specific capabilities of a
particular port before utilizing those capabilities.

> +
> +    rdma_cap_ib_mad		- Infiniband Management Datagrams.
> +    rdma_cap_ib_smi		- Infiniband Subnet Management Interface.
> +    rdma_cap_ib_cm		- Infiniband Communication Manager.
InfiniBand Connection Management
> +    rdma_cap_iw_cm		- IWARP Communication Manager.
iWARP Connection Management
> +    rdma_cap_ib_sa		- Infiniband Subnet Administration.
> +    rdma_cap_ib_mcast		- Infiniband Multicast.
InfiniBand Multicast join/leave protocol
> +    rdma_cap_read_multi_sge	- RDMA Read Multiple Scatter-Gather Entries.
RDMA Read verb supports more than 1 sge in the work request
> +    rdma_cap_af_ib		- Native Infiniband Address.
> +    rdma_cap_eth_ah		- Ethernet Address Handler.
Queue Pair is InfiniBand transport, but uses Ethernet address instead of
native InfiniBand address (aka, this is a RoCE QP, and that means
ethertype 0x8915 + GRH for RoCEv1 and IP/UDP to well known UDP port for
RoCEv2)
> +
> +USAGE
> +
> +  if (rdma_cap_XX(device, i)) {
> +	/* The port i of device support XX */
> +	...
> +  } else {
> +	/* The port i of device don't support XX */
> +	...
> +  }
> +
> +  rdma_cap_ib_mad
> +  ---------------
> +    Management Datagrams (MAD) is the prototype of management packet
> +    to be used by all the kinds of infiniband managers, use the helper
> +    to verify the port before utilize related features.
Management Datagrams (MAD) are a required part of the InfiniBand
specification and are supported on all InfiniBand devices.  A slightly
extended version are also supported on OPA interfaces.

I would drop all instances of "use the helper to verify..." as that's
redundant.  This whole doc is about using the helpers to verify things.

> +
> +  rdma_cap_ib_smi
> +  ---------------
> +    Subnet Management Interface (SMI) will handle SMP packet from SM
> +    in an infiniband fabric, use the helper to verify the port before
> +    utilize related features.
> +
> +  rdma_cap_ib_cm
> +  ---------------
> +    Communication Manager (CM) will handle the connections between
       ^Connection Manager (CM) service, used to ease the process of
connecting to a remote host.  The IB CM can be used to connect to remote
hosts using either InfiniBand or RoCE connections.  iWARP has its own
connection manager, see below.
> +    adaptors, currently there are two different implementation,
> +    IB or IWARP, use the helper to verify whether the port using
> +    IB-CM or not
> +
> +  rdma_cap_iw_cm
> +  ---------------
> +    IWARP has it's own implemented CM which is different from infiniband,
      iWARP connection manager.  Similar to the IB Connection Manager,
but only used on iWARP devices.
> +    use the helper to check whether the port using IWARP-CM or not.
> +
> +  rdma_cap_ib_sa
> +  ---------------
> +    Subnet Administration (SA) is the database built by SM in an
> +    infiniband fabric, use the helper to verify the port before
> +    utilize related features.
> +
> +  rdma_cap_ib_mcast
> +  ---------------
> +    Multicast is the feature for one QP to send messages to multiple
> +    QP in an infiniband fabric, use the helper to verify the port before
> +    utilize related features.

InfiniBand (and OPA) use a different multicast mechanism than
traditional IP multicast found on Ethernet devices.  If this capability
is true, then traditional IPv4/IPv6 multicast is handled by the IPoIB
layer and direct multicast joins and leaves are handled per the
InfiniBand specifications.

> +
> +  rdma_cap_read_multi_sge
> +  ---------------
> +    RDMA read operation could support multiple scatter-gather entries,
> +    use the helper to verify wthether the port support this feature
> +    or not.

Certain devices (iWARP in particular) have restrictions on the number of
scatter gather elements that can be present in an RDMA READ work
request.  This is true if the device does not have that restriction.

> +  rdma_cap_af_ib
> +  ---------------
> +    RDMA address format could be ethernet or infiniband, use the helper
> +    to verify whether the port support infiniband format or not.

Many code paths for traditional InfiniBand and RoCE links are the same,
but need minor differences to accommodate the different addresses on the
two types of connections.  This helper is true when the address of the
specific connection is of the InfiniBand native variety.

> +
> +  rdma_cap_eth_ah
> +  ---------------
> +    Infiniband address handler format is special in ethernet fabric, use
> +    the helper to verify whether the port is using ethernet format or not.

This helper is true when the address of the specific connection is of
the Ethernet (RoCE) variety.

-- 
Doug Ledford <dledford@redhat.com>
              GPG KeyID: 0E572FDD


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH RFC] Documentation/infiniband: Add docs for rdma-helpers

[Jason Gunthorpe]

On Wed, May 13, 2015 at 03:24:32PM +0200, Michael Wang wrote:
> This is the following patch for:
>   https://lkml.org/lkml/2015/5/5/417
> which try to document the settled rdma_cap_XX().
> 
> Highlights:
>   There could be many missing/mistakes/misunderstanding, please don't
>   be hesitate to point out the issues, any suggestions to improve or
>   complete the description are very welcomed ;-)

I'd rather see this in the kdoc for each function.

Thanks,
Jason

RE: [PATCH RFC] Documentation/infiniband: Add docs for rdma-helpers

[Hefty, Sean]

> > +  rdma_cap_ib_cm
> > +  ---------------
> > +    Communication Manager (CM) will handle the connections between
>        ^Connection Manager (CM) service, used to ease the process of

In IB terms, this is communication manager.  It also handles transport level address resolution for UD QPs.

> > +  rdma_cap_eth_ah
> > +  ---------------
> > +    Infiniband address handler format is special in ethernet fabric,
> use
> > +    the helper to verify whether the port is using ethernet format or
> not.
> 
> This helper is true when the address of the specific connection is of
> the Ethernet (RoCE) variety.

This is used for connectionless communication.

RE: [PATCH RFC] Documentation/infiniband: Add docs for rdma-helpers

[Hefty, Sean]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 820 bytes --]

> > +  rdma_cap_ib_cm
> > +  ---------------
> > +    Communication Manager (CM) will handle the connections between
>        ^Connection Manager (CM) service, used to ease the process of

In IB terms, this is communication manager.  It also handles transport level address resolution for UD QPs.

> > +  rdma_cap_eth_ah
> > +  ---------------
> > +    Infiniband address handler format is special in ethernet fabric,
> use
> > +    the helper to verify whether the port is using ethernet format or
> not.
> 
> This helper is true when the address of the specific connection is of
> the Ethernet (RoCE) variety.

This is used for connectionless communication.
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

Re: [PATCH RFC] Documentation/infiniband: Add docs for rdma-helpers

[Michael Wang]

On 05/13/2015 05:59 PM, Jason Gunthorpe wrote:
> On Wed, May 13, 2015 at 03:24:32PM +0200, Michael Wang wrote:
>> This is the following patch for:
>>   https://lkml.org/lkml/2015/5/5/417
>> which try to document the settled rdma_cap_XX().
>>
>> Highlights:
>>   There could be many missing/mistakes/misunderstanding, please don't
>>   be hesitate to point out the issues, any suggestions to improve or
>>   complete the description are very welcomed ;-)
> 
> I'd rather see this in the kdoc for each function.

I used to thought you mean the kernel documentation like
this... my misunderstanding but this is the usual way to
document kernel stuff, isn't it?

BTW, could you give more details on the kdoc?

Regards,
Michael Wang

> 
> Thanks,
> Jason
> 

Re: [PATCH RFC] Documentation/infiniband: Add docs for rdma-helpers

[Michael Wang]

On 05/13/2015 05:11 PM, Doug Ledford wrote:
[snip]
>> +
>> +  For core layer, below helpers are used to check if a paticular capability
>> +  is supported by the port.
> 
> The following helpers are used to check the specific capabilities of a
> particular port before utilizing those capabilities.

Will be in next version :-)

> 
>> +
>> +    rdma_cap_ib_mad		- Infiniband Management Datagrams.
>> +    rdma_cap_ib_smi		- Infiniband Subnet Management Interface.
>> +    rdma_cap_ib_cm		- Infiniband Communication Manager.
> InfiniBand Connection Management

Me too used to think it's 'connection', while I found some docs explain
this as 'communication'... but anyway, 'connection' sounds
more close to what it did in kernel :-)

>> +    rdma_cap_iw_cm		- IWARP Communication Manager.
> iWARP Connection Management
>> +    rdma_cap_ib_sa		- Infiniband Subnet Administration.
>> +    rdma_cap_ib_mcast		- Infiniband Multicast.
> InfiniBand Multicast join/leave protocol
>> +    rdma_cap_read_multi_sge	- RDMA Read Multiple Scatter-Gather Entries.
> RDMA Read verb supports more than 1 sge in the work request

Will be in next version :-)

>> +    rdma_cap_af_ib		- Native Infiniband Address.
>> +    rdma_cap_eth_ah		- Ethernet Address Handler.
> Queue Pair is InfiniBand transport, but uses Ethernet address instead of
> native InfiniBand address (aka, this is a RoCE QP, and that means
> ethertype 0x8915 + GRH for RoCEv1 and IP/UDP to well known UDP port for
> RoCEv2)

Shall we put this long description into USAGE? Here maybe list
all the helpers to give some quick overview with a brief
description, what's your opinion?

>> +
>> +USAGE
>> +
>> +  if (rdma_cap_XX(device, i)) {
>> +	/* The port i of device support XX */
>> +	...
>> +  } else {
>> +	/* The port i of device don't support XX */
>> +	...
>> +  }
>> +
>> +  rdma_cap_ib_mad
>> +  ---------------
>> +    Management Datagrams (MAD) is the prototype of management packet
>> +    to be used by all the kinds of infiniband managers, use the helper
>> +    to verify the port before utilize related features.
> Management Datagrams (MAD) are a required part of the InfiniBand
> specification and are supported on all InfiniBand devices.  A slightly
> extended version are also supported on OPA interfaces.
> 
> I would drop all instances of "use the helper to verify..." as that's
> redundant.  This whole doc is about using the helpers to verify things.

Agree, will be dropped in next version.

And all the comments below make sense, will be merged ;-)

Regards,
Michael Wang

> 
>> +
>> +  rdma_cap_ib_smi
>> +  ---------------
>> +    Subnet Management Interface (SMI) will handle SMP packet from SM
>> +    in an infiniband fabric, use the helper to verify the port before
>> +    utilize related features.
>> +
>> +  rdma_cap_ib_cm
>> +  ---------------
>> +    Communication Manager (CM) will handle the connections between
>        ^Connection Manager (CM) service, used to ease the process of
> connecting to a remote host.  The IB CM can be used to connect to remote
> hosts using either InfiniBand or RoCE connections.  iWARP has its own
> connection manager, see below.
>> +    adaptors, currently there are two different implementation,
>> +    IB or IWARP, use the helper to verify whether the port using
>> +    IB-CM or not
>> +
>> +  rdma_cap_iw_cm
>> +  ---------------
>> +    IWARP has it's own implemented CM which is different from infiniband,
>       iWARP connection manager.  Similar to the IB Connection Manager,
> but only used on iWARP devices.
>> +    use the helper to check whether the port using IWARP-CM or not.
>> +
>> +  rdma_cap_ib_sa
>> +  ---------------
>> +    Subnet Administration (SA) is the database built by SM in an
>> +    infiniband fabric, use the helper to verify the port before
>> +    utilize related features.
>> +
>> +  rdma_cap_ib_mcast
>> +  ---------------
>> +    Multicast is the feature for one QP to send messages to multiple
>> +    QP in an infiniband fabric, use the helper to verify the port before
>> +    utilize related features.
> 
> InfiniBand (and OPA) use a different multicast mechanism than
> traditional IP multicast found on Ethernet devices.  If this capability
> is true, then traditional IPv4/IPv6 multicast is handled by the IPoIB
> layer and direct multicast joins and leaves are handled per the
> InfiniBand specifications.
> 
>> +
>> +  rdma_cap_read_multi_sge
>> +  ---------------
>> +    RDMA read operation could support multiple scatter-gather entries,
>> +    use the helper to verify wthether the port support this feature
>> +    or not.
> 
> Certain devices (iWARP in particular) have restrictions on the number of
> scatter gather elements that can be present in an RDMA READ work
> request.  This is true if the device does not have that restriction.
> 
>> +  rdma_cap_af_ib
>> +  ---------------
>> +    RDMA address format could be ethernet or infiniband, use the helper
>> +    to verify whether the port support infiniband format or not.
> 
> Many code paths for traditional InfiniBand and RoCE links are the same,
> but need minor differences to accommodate the different addresses on the
> two types of connections.  This helper is true when the address of the
> specific connection is of the InfiniBand native variety.
> 
>> +
>> +  rdma_cap_eth_ah
>> +  ---------------
>> +    Infiniband address handler format is special in ethernet fabric, use
>> +    the helper to verify whether the port is using ethernet format or not.
> 
> This helper is true when the address of the specific connection is of
> the Ethernet (RoCE) variety.
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH RFC] Documentation/infiniband: Add docs for rdma-helpers

[Michael Wang]

On 05/13/2015 05:11 PM, Doug Ledford wrote:
[snip]
>> +
>> +  For core layer, below helpers are used to check if a paticular capability
>> +  is supported by the port.
> 
> The following helpers are used to check the specific capabilities of a
> particular port before utilizing those capabilities.

Will be in next version :-)

> 
>> +
>> +    rdma_cap_ib_mad		- Infiniband Management Datagrams.
>> +    rdma_cap_ib_smi		- Infiniband Subnet Management Interface.
>> +    rdma_cap_ib_cm		- Infiniband Communication Manager.
> InfiniBand Connection Management

Me too used to think it's 'connection', while I found some docs explain
this as 'communication'... but anyway, 'connection' sounds
more close to what it did in kernel :-)

>> +    rdma_cap_iw_cm		- IWARP Communication Manager.
> iWARP Connection Management
>> +    rdma_cap_ib_sa		- Infiniband Subnet Administration.
>> +    rdma_cap_ib_mcast		- Infiniband Multicast.
> InfiniBand Multicast join/leave protocol
>> +    rdma_cap_read_multi_sge	- RDMA Read Multiple Scatter-Gather Entries.
> RDMA Read verb supports more than 1 sge in the work request

Will be in next version :-)

>> +    rdma_cap_af_ib		- Native Infiniband Address.
>> +    rdma_cap_eth_ah		- Ethernet Address Handler.
> Queue Pair is InfiniBand transport, but uses Ethernet address instead of
> native InfiniBand address (aka, this is a RoCE QP, and that means
> ethertype 0x8915 + GRH for RoCEv1 and IP/UDP to well known UDP port for
> RoCEv2)

Shall we put this long description into USAGE? Here maybe list
all the helpers to give some quick overview with a brief
description, what's your opinion?

>> +
>> +USAGE
>> +
>> +  if (rdma_cap_XX(device, i)) {
>> +	/* The port i of device support XX */
>> +	...
>> +  } else {
>> +	/* The port i of device don't support XX */
>> +	...
>> +  }
>> +
>> +  rdma_cap_ib_mad
>> +  ---------------
>> +    Management Datagrams (MAD) is the prototype of management packet
>> +    to be used by all the kinds of infiniband managers, use the helper
>> +    to verify the port before utilize related features.
> Management Datagrams (MAD) are a required part of the InfiniBand
> specification and are supported on all InfiniBand devices.  A slightly
> extended version are also supported on OPA interfaces.
> 
> I would drop all instances of "use the helper to verify..." as that's
> redundant.  This whole doc is about using the helpers to verify things.

Agree, will be dropped in next version.

And all the comments below make sense, will be merged ;-)

Regards,
Michael Wang

> 
>> +
>> +  rdma_cap_ib_smi
>> +  ---------------
>> +    Subnet Management Interface (SMI) will handle SMP packet from SM
>> +    in an infiniband fabric, use the helper to verify the port before
>> +    utilize related features.
>> +
>> +  rdma_cap_ib_cm
>> +  ---------------
>> +    Communication Manager (CM) will handle the connections between
>        ^Connection Manager (CM) service, used to ease the process of
> connecting to a remote host.  The IB CM can be used to connect to remote
> hosts using either InfiniBand or RoCE connections.  iWARP has its own
> connection manager, see below.
>> +    adaptors, currently there are two different implementation,
>> +    IB or IWARP, use the helper to verify whether the port using
>> +    IB-CM or not
>> +
>> +  rdma_cap_iw_cm
>> +  ---------------
>> +    IWARP has it's own implemented CM which is different from infiniband,
>       iWARP connection manager.  Similar to the IB Connection Manager,
> but only used on iWARP devices.
>> +    use the helper to check whether the port using IWARP-CM or not.
>> +
>> +  rdma_cap_ib_sa
>> +  ---------------
>> +    Subnet Administration (SA) is the database built by SM in an
>> +    infiniband fabric, use the helper to verify the port before
>> +    utilize related features.
>> +
>> +  rdma_cap_ib_mcast
>> +  ---------------
>> +    Multicast is the feature for one QP to send messages to multiple
>> +    QP in an infiniband fabric, use the helper to verify the port before
>> +    utilize related features.
> 
> InfiniBand (and OPA) use a different multicast mechanism than
> traditional IP multicast found on Ethernet devices.  If this capability
> is true, then traditional IPv4/IPv6 multicast is handled by the IPoIB
> layer and direct multicast joins and leaves are handled per the
> InfiniBand specifications.
> 
>> +
>> +  rdma_cap_read_multi_sge
>> +  ---------------
>> +    RDMA read operation could support multiple scatter-gather entries,
>> +    use the helper to verify wthether the port support this feature
>> +    or not.
> 
> Certain devices (iWARP in particular) have restrictions on the number of
> scatter gather elements that can be present in an RDMA READ work
> request.  This is true if the device does not have that restriction.
> 
>> +  rdma_cap_af_ib
>> +  ---------------
>> +    RDMA address format could be ethernet or infiniband, use the helper
>> +    to verify whether the port support infiniband format or not.
> 
> Many code paths for traditional InfiniBand and RoCE links are the same,
> but need minor differences to accommodate the different addresses on the
> two types of connections.  This helper is true when the address of the
> specific connection is of the InfiniBand native variety.
> 
>> +
>> +  rdma_cap_eth_ah
>> +  ---------------
>> +    Infiniband address handler format is special in ethernet fabric, use
>> +    the helper to verify whether the port is using ethernet format or not.
> 
> This helper is true when the address of the specific connection is of
> the Ethernet (RoCE) variety.
> 

Re: [PATCH RFC] Documentation/infiniband: Add docs for rdma-helpers

[Michael Wang]

On 05/13/2015 06:42 PM, Hefty, Sean wrote:
>>> +  rdma_cap_ib_cm
>>> +  ---------------
>>> +    Communication Manager (CM) will handle the connections between
>>        ^Connection Manager (CM) service, used to ease the process of
> 
> In IB terms, this is communication manager.  It also handles transport level address resolution for UD QPs.

I could find both 'connection' and 'communication' in different docs,
while 'connection' is more related to verbs, 'communication' is more
close to specification.

IMHO 'connection' make more sense, after all, all the transport between
adaptors could named as communication, while connection management is exactly
what CM did in kernel.

Doug, what's your opinion?

> 
>>> +  rdma_cap_eth_ah
>>> +  ---------------
>>> +    Infiniband address handler format is special in ethernet fabric,
>> use
>>> +    the helper to verify whether the port is using ethernet format or
>> not.
>>
>> This helper is true when the address of the specific connection is of
>> the Ethernet (RoCE) variety.
> 
> This is used for connectionless communication.

Could you please give more details on this?

Regards,
Michael Wang

> 

Re: [PATCH RFC] Documentation/infiniband: Add docs for rdma-helpers

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 6766 bytes --]

On Fri, 2015-05-15 at 10:12 +0200, Michael Wang wrote:
> 
> On 05/13/2015 05:11 PM, Doug Ledford wrote:
> [snip]
> >> +
> >> +  For core layer, below helpers are used to check if a paticular capability
> >> +  is supported by the port.
> > 
> > The following helpers are used to check the specific capabilities of a
> > particular port before utilizing those capabilities.
> 
> Will be in next version :-)
> 
> > 
> >> +
> >> +    rdma_cap_ib_mad		- Infiniband Management Datagrams.
> >> +    rdma_cap_ib_smi		- Infiniband Subnet Management Interface.
> >> +    rdma_cap_ib_cm		- Infiniband Communication Manager.
> > InfiniBand Connection Management
> 
> Me too used to think it's 'connection', while I found some docs explain
> this as 'communication'... but anyway, 'connection' sounds
> more close to what it did in kernel :-)

That's kind of what I thought.  Anyway, it's communication management
(which to me is a gross abuse of the english language for which the IBTA
should be appropriately chastised), but that doesn't mean that lower
down in the more descriptive area of text that we can't call out that
this is really for establishing a connection and that once your
connection is established and you *truly* want to communicate, this does
nothing.

> >> +    rdma_cap_iw_cm		- IWARP Communication Manager.
> > iWARP Connection Management
> >> +    rdma_cap_ib_sa		- Infiniband Subnet Administration.
> >> +    rdma_cap_ib_mcast		- Infiniband Multicast.
> > InfiniBand Multicast join/leave protocol
> >> +    rdma_cap_read_multi_sge	- RDMA Read Multiple Scatter-Gather Entries.
> > RDMA Read verb supports more than 1 sge in the work request
> 
> Will be in next version :-)
> 
> >> +    rdma_cap_af_ib		- Native Infiniband Address.
> >> +    rdma_cap_eth_ah		- Ethernet Address Handler.
> > Queue Pair is InfiniBand transport, but uses Ethernet address instead of
> > native InfiniBand address (aka, this is a RoCE QP, and that means
> > ethertype 0x8915 + GRH for RoCEv1 and IP/UDP to well known UDP port for
> > RoCEv2)
> 
> Shall we put this long description into USAGE? Here maybe list
> all the helpers to give some quick overview with a brief
> description, what's your opinion?

Given how we have a more complete description of this below, it need not
have such a lengthy description here.
> 
> >> +
> >> +USAGE
> >> +
> >> +  if (rdma_cap_XX(device, i)) {
> >> +	/* The port i of device support XX */
> >> +	...
> >> +  } else {
> >> +	/* The port i of device don't support XX */
> >> +	...
> >> +  }
> >> +
> >> +  rdma_cap_ib_mad
> >> +  ---------------
> >> +    Management Datagrams (MAD) is the prototype of management packet
> >> +    to be used by all the kinds of infiniband managers, use the helper
> >> +    to verify the port before utilize related features.
> > Management Datagrams (MAD) are a required part of the InfiniBand
> > specification and are supported on all InfiniBand devices.  A slightly
> > extended version are also supported on OPA interfaces.
> > 
> > I would drop all instances of "use the helper to verify..." as that's
> > redundant.  This whole doc is about using the helpers to verify things.
> 
> Agree, will be dropped in next version.
> 
> And all the comments below make sense, will be merged ;-)
> 
> Regards,
> Michael Wang
> 
> > 
> >> +
> >> +  rdma_cap_ib_smi
> >> +  ---------------
> >> +    Subnet Management Interface (SMI) will handle SMP packet from SM
> >> +    in an infiniband fabric, use the helper to verify the port before
> >> +    utilize related features.
> >> +
> >> +  rdma_cap_ib_cm
> >> +  ---------------
> >> +    Communication Manager (CM) will handle the connections between
> >        ^Connection Manager (CM) service, used to ease the process of
> > connecting to a remote host.  The IB CM can be used to connect to remote
> > hosts using either InfiniBand or RoCE connections.  iWARP has its own
> > connection manager, see below.
> >> +    adaptors, currently there are two different implementation,
> >> +    IB or IWARP, use the helper to verify whether the port using
> >> +    IB-CM or not
> >> +
> >> +  rdma_cap_iw_cm
> >> +  ---------------
> >> +    IWARP has it's own implemented CM which is different from infiniband,
> >       iWARP connection manager.  Similar to the IB Connection Manager,
> > but only used on iWARP devices.
> >> +    use the helper to check whether the port using IWARP-CM or not.
> >> +
> >> +  rdma_cap_ib_sa
> >> +  ---------------
> >> +    Subnet Administration (SA) is the database built by SM in an
> >> +    infiniband fabric, use the helper to verify the port before
> >> +    utilize related features.
> >> +
> >> +  rdma_cap_ib_mcast
> >> +  ---------------
> >> +    Multicast is the feature for one QP to send messages to multiple
> >> +    QP in an infiniband fabric, use the helper to verify the port before
> >> +    utilize related features.
> > 
> > InfiniBand (and OPA) use a different multicast mechanism than
> > traditional IP multicast found on Ethernet devices.  If this capability
> > is true, then traditional IPv4/IPv6 multicast is handled by the IPoIB
> > layer and direct multicast joins and leaves are handled per the
> > InfiniBand specifications.
> > 
> >> +
> >> +  rdma_cap_read_multi_sge
> >> +  ---------------
> >> +    RDMA read operation could support multiple scatter-gather entries,
> >> +    use the helper to verify wthether the port support this feature
> >> +    or not.
> > 
> > Certain devices (iWARP in particular) have restrictions on the number of
> > scatter gather elements that can be present in an RDMA READ work
> > request.  This is true if the device does not have that restriction.
> > 
> >> +  rdma_cap_af_ib
> >> +  ---------------
> >> +    RDMA address format could be ethernet or infiniband, use the helper
> >> +    to verify whether the port support infiniband format or not.
> > 
> > Many code paths for traditional InfiniBand and RoCE links are the same,
> > but need minor differences to accommodate the different addresses on the
> > two types of connections.  This helper is true when the address of the
> > specific connection is of the InfiniBand native variety.
> > 
> >> +
> >> +  rdma_cap_eth_ah
> >> +  ---------------
> >> +    Infiniband address handler format is special in ethernet fabric, use
> >> +    the helper to verify whether the port is using ethernet format or not.
> > 
> > This helper is true when the address of the specific connection is of
> > the Ethernet (RoCE) variety.
> > 


-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH RFC] Documentation/infiniband: Add docs for rdma-helpers

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 6737 bytes --]

On Fri, 2015-05-15 at 10:12 +0200, Michael Wang wrote:
> 
> On 05/13/2015 05:11 PM, Doug Ledford wrote:
> [snip]
> >> +
> >> +  For core layer, below helpers are used to check if a paticular capability
> >> +  is supported by the port.
> > 
> > The following helpers are used to check the specific capabilities of a
> > particular port before utilizing those capabilities.
> 
> Will be in next version :-)
> 
> > 
> >> +
> >> +    rdma_cap_ib_mad		- Infiniband Management Datagrams.
> >> +    rdma_cap_ib_smi		- Infiniband Subnet Management Interface.
> >> +    rdma_cap_ib_cm		- Infiniband Communication Manager.
> > InfiniBand Connection Management
> 
> Me too used to think it's 'connection', while I found some docs explain
> this as 'communication'... but anyway, 'connection' sounds
> more close to what it did in kernel :-)

That's kind of what I thought.  Anyway, it's communication management
(which to me is a gross abuse of the english language for which the IBTA
should be appropriately chastised), but that doesn't mean that lower
down in the more descriptive area of text that we can't call out that
this is really for establishing a connection and that once your
connection is established and you *truly* want to communicate, this does
nothing.

> >> +    rdma_cap_iw_cm		- IWARP Communication Manager.
> > iWARP Connection Management
> >> +    rdma_cap_ib_sa		- Infiniband Subnet Administration.
> >> +    rdma_cap_ib_mcast		- Infiniband Multicast.
> > InfiniBand Multicast join/leave protocol
> >> +    rdma_cap_read_multi_sge	- RDMA Read Multiple Scatter-Gather Entries.
> > RDMA Read verb supports more than 1 sge in the work request
> 
> Will be in next version :-)
> 
> >> +    rdma_cap_af_ib		- Native Infiniband Address.
> >> +    rdma_cap_eth_ah		- Ethernet Address Handler.
> > Queue Pair is InfiniBand transport, but uses Ethernet address instead of
> > native InfiniBand address (aka, this is a RoCE QP, and that means
> > ethertype 0x8915 + GRH for RoCEv1 and IP/UDP to well known UDP port for
> > RoCEv2)
> 
> Shall we put this long description into USAGE? Here maybe list
> all the helpers to give some quick overview with a brief
> description, what's your opinion?

Given how we have a more complete description of this below, it need not
have such a lengthy description here.
> 
> >> +
> >> +USAGE
> >> +
> >> +  if (rdma_cap_XX(device, i)) {
> >> +	/* The port i of device support XX */
> >> +	...
> >> +  } else {
> >> +	/* The port i of device don't support XX */
> >> +	...
> >> +  }
> >> +
> >> +  rdma_cap_ib_mad
> >> +  ---------------
> >> +    Management Datagrams (MAD) is the prototype of management packet
> >> +    to be used by all the kinds of infiniband managers, use the helper
> >> +    to verify the port before utilize related features.
> > Management Datagrams (MAD) are a required part of the InfiniBand
> > specification and are supported on all InfiniBand devices.  A slightly
> > extended version are also supported on OPA interfaces.
> > 
> > I would drop all instances of "use the helper to verify..." as that's
> > redundant.  This whole doc is about using the helpers to verify things.
> 
> Agree, will be dropped in next version.
> 
> And all the comments below make sense, will be merged ;-)
> 
> Regards,
> Michael Wang
> 
> > 
> >> +
> >> +  rdma_cap_ib_smi
> >> +  ---------------
> >> +    Subnet Management Interface (SMI) will handle SMP packet from SM
> >> +    in an infiniband fabric, use the helper to verify the port before
> >> +    utilize related features.
> >> +
> >> +  rdma_cap_ib_cm
> >> +  ---------------
> >> +    Communication Manager (CM) will handle the connections between
> >        ^Connection Manager (CM) service, used to ease the process of
> > connecting to a remote host.  The IB CM can be used to connect to remote
> > hosts using either InfiniBand or RoCE connections.  iWARP has its own
> > connection manager, see below.
> >> +    adaptors, currently there are two different implementation,
> >> +    IB or IWARP, use the helper to verify whether the port using
> >> +    IB-CM or not
> >> +
> >> +  rdma_cap_iw_cm
> >> +  ---------------
> >> +    IWARP has it's own implemented CM which is different from infiniband,
> >       iWARP connection manager.  Similar to the IB Connection Manager,
> > but only used on iWARP devices.
> >> +    use the helper to check whether the port using IWARP-CM or not.
> >> +
> >> +  rdma_cap_ib_sa
> >> +  ---------------
> >> +    Subnet Administration (SA) is the database built by SM in an
> >> +    infiniband fabric, use the helper to verify the port before
> >> +    utilize related features.
> >> +
> >> +  rdma_cap_ib_mcast
> >> +  ---------------
> >> +    Multicast is the feature for one QP to send messages to multiple
> >> +    QP in an infiniband fabric, use the helper to verify the port before
> >> +    utilize related features.
> > 
> > InfiniBand (and OPA) use a different multicast mechanism than
> > traditional IP multicast found on Ethernet devices.  If this capability
> > is true, then traditional IPv4/IPv6 multicast is handled by the IPoIB
> > layer and direct multicast joins and leaves are handled per the
> > InfiniBand specifications.
> > 
> >> +
> >> +  rdma_cap_read_multi_sge
> >> +  ---------------
> >> +    RDMA read operation could support multiple scatter-gather entries,
> >> +    use the helper to verify wthether the port support this feature
> >> +    or not.
> > 
> > Certain devices (iWARP in particular) have restrictions on the number of
> > scatter gather elements that can be present in an RDMA READ work
> > request.  This is true if the device does not have that restriction.
> > 
> >> +  rdma_cap_af_ib
> >> +  ---------------
> >> +    RDMA address format could be ethernet or infiniband, use the helper
> >> +    to verify whether the port support infiniband format or not.
> > 
> > Many code paths for traditional InfiniBand and RoCE links are the same,
> > but need minor differences to accommodate the different addresses on the
> > two types of connections.  This helper is true when the address of the
> > specific connection is of the InfiniBand native variety.
> > 
> >> +
> >> +  rdma_cap_eth_ah
> >> +  ---------------
> >> +    Infiniband address handler format is special in ethernet fabric, use
> >> +    the helper to verify whether the port is using ethernet format or not.
> > 
> > This helper is true when the address of the specific connection is of
> > the Ethernet (RoCE) variety.
> > 


-- 
Doug Ledford <dledford@redhat.com>
              GPG KeyID: 0E572FDD


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH RFC] Documentation/infiniband: Add docs for rdma-helpers

[Michael Wang]

On 05/15/2015 04:27 PM, Doug Ledford wrote:
[snip]
>>
>> Me too used to think it's 'connection', while I found some docs explain
>> this as 'communication'... but anyway, 'connection' sounds
>> more close to what it did in kernel :-)
> 
> That's kind of what I thought.  Anyway, it's communication management
> (which to me is a gross abuse of the english language for which the IBTA
> should be appropriately chastised), but that doesn't mean that lower
> down in the more descriptive area of text that we can't call out that
> this is really for establishing a connection and that once your
> connection is established and you *truly* want to communicate, this does
> nothing.

I see :-) we can reserve the communication management as the definition of CM,
to obey the standard, meanwhile give some description related to connection
below in the long description.

> 
[snip]
>> Shall we put this long description into USAGE? Here maybe list
>> all the helpers to give some quick overview with a brief
>> description, what's your opinion?
> 
> Given how we have a more complete description of this below, it need not
> have such a lengthy description here.

Got it :-)

Regards,
Michael Wang

>>
>>>> +
>>>> +USAGE
>>>> +
>>>> +  if (rdma_cap_XX(device, i)) {
>>>> +	/* The port i of device support XX */
>>>> +	...
>>>> +  } else {
>>>> +	/* The port i of device don't support XX */
>>>> +	...
>>>> +  }
>>>> +
>>>> +  rdma_cap_ib_mad
>>>> +  ---------------
>>>> +    Management Datagrams (MAD) is the prototype of management packet
>>>> +    to be used by all the kinds of infiniband managers, use the helper
>>>> +    to verify the port before utilize related features.
>>> Management Datagrams (MAD) are a required part of the InfiniBand
>>> specification and are supported on all InfiniBand devices.  A slightly
>>> extended version are also supported on OPA interfaces.
>>>
>>> I would drop all instances of "use the helper to verify..." as that's
>>> redundant.  This whole doc is about using the helpers to verify things.
>>
>> Agree, will be dropped in next version.
>>
>> And all the comments below make sense, will be merged ;-)
>>
>> Regards,
>> Michael Wang
>>
>>>
>>>> +
>>>> +  rdma_cap_ib_smi
>>>> +  ---------------
>>>> +    Subnet Management Interface (SMI) will handle SMP packet from SM
>>>> +    in an infiniband fabric, use the helper to verify the port before
>>>> +    utilize related features.
>>>> +
>>>> +  rdma_cap_ib_cm
>>>> +  ---------------
>>>> +    Communication Manager (CM) will handle the connections between
>>>        ^Connection Manager (CM) service, used to ease the process of
>>> connecting to a remote host.  The IB CM can be used to connect to remote
>>> hosts using either InfiniBand or RoCE connections.  iWARP has its own
>>> connection manager, see below.
>>>> +    adaptors, currently there are two different implementation,
>>>> +    IB or IWARP, use the helper to verify whether the port using
>>>> +    IB-CM or not
>>>> +
>>>> +  rdma_cap_iw_cm
>>>> +  ---------------
>>>> +    IWARP has it's own implemented CM which is different from infiniband,
>>>       iWARP connection manager.  Similar to the IB Connection Manager,
>>> but only used on iWARP devices.
>>>> +    use the helper to check whether the port using IWARP-CM or not.
>>>> +
>>>> +  rdma_cap_ib_sa
>>>> +  ---------------
>>>> +    Subnet Administration (SA) is the database built by SM in an
>>>> +    infiniband fabric, use the helper to verify the port before
>>>> +    utilize related features.
>>>> +
>>>> +  rdma_cap_ib_mcast
>>>> +  ---------------
>>>> +    Multicast is the feature for one QP to send messages to multiple
>>>> +    QP in an infiniband fabric, use the helper to verify the port before
>>>> +    utilize related features.
>>>
>>> InfiniBand (and OPA) use a different multicast mechanism than
>>> traditional IP multicast found on Ethernet devices.  If this capability
>>> is true, then traditional IPv4/IPv6 multicast is handled by the IPoIB
>>> layer and direct multicast joins and leaves are handled per the
>>> InfiniBand specifications.
>>>
>>>> +
>>>> +  rdma_cap_read_multi_sge
>>>> +  ---------------
>>>> +    RDMA read operation could support multiple scatter-gather entries,
>>>> +    use the helper to verify wthether the port support this feature
>>>> +    or not.
>>>
>>> Certain devices (iWARP in particular) have restrictions on the number of
>>> scatter gather elements that can be present in an RDMA READ work
>>> request.  This is true if the device does not have that restriction.
>>>
>>>> +  rdma_cap_af_ib
>>>> +  ---------------
>>>> +    RDMA address format could be ethernet or infiniband, use the helper
>>>> +    to verify whether the port support infiniband format or not.
>>>
>>> Many code paths for traditional InfiniBand and RoCE links are the same,
>>> but need minor differences to accommodate the different addresses on the
>>> two types of connections.  This helper is true when the address of the
>>> specific connection is of the InfiniBand native variety.
>>>
>>>> +
>>>> +  rdma_cap_eth_ah
>>>> +  ---------------
>>>> +    Infiniband address handler format is special in ethernet fabric, use
>>>> +    the helper to verify whether the port is using ethernet format or not.
>>>
>>> This helper is true when the address of the specific connection is of
>>> the Ethernet (RoCE) variety.
>>>
> 
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH RFC] Documentation/infiniband: Add docs for rdma-helpers

[Michael Wang]

On 05/15/2015 04:27 PM, Doug Ledford wrote:
[snip]
>>
>> Me too used to think it's 'connection', while I found some docs explain
>> this as 'communication'... but anyway, 'connection' sounds
>> more close to what it did in kernel :-)
> 
> That's kind of what I thought.  Anyway, it's communication management
> (which to me is a gross abuse of the english language for which the IBTA
> should be appropriately chastised), but that doesn't mean that lower
> down in the more descriptive area of text that we can't call out that
> this is really for establishing a connection and that once your
> connection is established and you *truly* want to communicate, this does
> nothing.

I see :-) we can reserve the communication management as the definition of CM,
to obey the standard, meanwhile give some description related to connection
below in the long description.

> 
[snip]
>> Shall we put this long description into USAGE? Here maybe list
>> all the helpers to give some quick overview with a brief
>> description, what's your opinion?
> 
> Given how we have a more complete description of this below, it need not
> have such a lengthy description here.

Got it :-)

Regards,
Michael Wang

>>
>>>> +
>>>> +USAGE
>>>> +
>>>> +  if (rdma_cap_XX(device, i)) {
>>>> +	/* The port i of device support XX */
>>>> +	...
>>>> +  } else {
>>>> +	/* The port i of device don't support XX */
>>>> +	...
>>>> +  }
>>>> +
>>>> +  rdma_cap_ib_mad
>>>> +  ---------------
>>>> +    Management Datagrams (MAD) is the prototype of management packet
>>>> +    to be used by all the kinds of infiniband managers, use the helper
>>>> +    to verify the port before utilize related features.
>>> Management Datagrams (MAD) are a required part of the InfiniBand
>>> specification and are supported on all InfiniBand devices.  A slightly
>>> extended version are also supported on OPA interfaces.
>>>
>>> I would drop all instances of "use the helper to verify..." as that's
>>> redundant.  This whole doc is about using the helpers to verify things.
>>
>> Agree, will be dropped in next version.
>>
>> And all the comments below make sense, will be merged ;-)
>>
>> Regards,
>> Michael Wang
>>
>>>
>>>> +
>>>> +  rdma_cap_ib_smi
>>>> +  ---------------
>>>> +    Subnet Management Interface (SMI) will handle SMP packet from SM
>>>> +    in an infiniband fabric, use the helper to verify the port before
>>>> +    utilize related features.
>>>> +
>>>> +  rdma_cap_ib_cm
>>>> +  ---------------
>>>> +    Communication Manager (CM) will handle the connections between
>>>        ^Connection Manager (CM) service, used to ease the process of
>>> connecting to a remote host.  The IB CM can be used to connect to remote
>>> hosts using either InfiniBand or RoCE connections.  iWARP has its own
>>> connection manager, see below.
>>>> +    adaptors, currently there are two different implementation,
>>>> +    IB or IWARP, use the helper to verify whether the port using
>>>> +    IB-CM or not
>>>> +
>>>> +  rdma_cap_iw_cm
>>>> +  ---------------
>>>> +    IWARP has it's own implemented CM which is different from infiniband,
>>>       iWARP connection manager.  Similar to the IB Connection Manager,
>>> but only used on iWARP devices.
>>>> +    use the helper to check whether the port using IWARP-CM or not.
>>>> +
>>>> +  rdma_cap_ib_sa
>>>> +  ---------------
>>>> +    Subnet Administration (SA) is the database built by SM in an
>>>> +    infiniband fabric, use the helper to verify the port before
>>>> +    utilize related features.
>>>> +
>>>> +  rdma_cap_ib_mcast
>>>> +  ---------------
>>>> +    Multicast is the feature for one QP to send messages to multiple
>>>> +    QP in an infiniband fabric, use the helper to verify the port before
>>>> +    utilize related features.
>>>
>>> InfiniBand (and OPA) use a different multicast mechanism than
>>> traditional IP multicast found on Ethernet devices.  If this capability
>>> is true, then traditional IPv4/IPv6 multicast is handled by the IPoIB
>>> layer and direct multicast joins and leaves are handled per the
>>> InfiniBand specifications.
>>>
>>>> +
>>>> +  rdma_cap_read_multi_sge
>>>> +  ---------------
>>>> +    RDMA read operation could support multiple scatter-gather entries,
>>>> +    use the helper to verify wthether the port support this feature
>>>> +    or not.
>>>
>>> Certain devices (iWARP in particular) have restrictions on the number of
>>> scatter gather elements that can be present in an RDMA READ work
>>> request.  This is true if the device does not have that restriction.
>>>
>>>> +  rdma_cap_af_ib
>>>> +  ---------------
>>>> +    RDMA address format could be ethernet or infiniband, use the helper
>>>> +    to verify whether the port support infiniband format or not.
>>>
>>> Many code paths for traditional InfiniBand and RoCE links are the same,
>>> but need minor differences to accommodate the different addresses on the
>>> two types of connections.  This helper is true when the address of the
>>> specific connection is of the InfiniBand native variety.
>>>
>>>> +
>>>> +  rdma_cap_eth_ah
>>>> +  ---------------
>>>> +    Infiniband address handler format is special in ethernet fabric, use
>>>> +    the helper to verify whether the port is using ethernet format or not.
>>>
>>> This helper is true when the address of the specific connection is of
>>> the Ethernet (RoCE) variety.
>>>
> 
> 

Re: [PATCH RFC] Documentation/infiniband: Add docs for rdma-helpers

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 2547 bytes --]

On Fri, 2015-05-15 at 10:31 +0200, Michael Wang wrote:
> 
> On 05/13/2015 06:42 PM, Hefty, Sean wrote:
> >>> +  rdma_cap_ib_cm
> >>> +  ---------------
> >>> +    Communication Manager (CM) will handle the connections between
> >>        ^Connection Manager (CM) service, used to ease the process of
> > 
> > In IB terms, this is communication manager.  It also handles transport level address resolution for UD QPs.
> 
> I could find both 'connection' and 'communication' in different docs,
> while 'connection' is more related to verbs, 'communication' is more
> close to specification.
> 
> IMHO 'connection' make more sense, after all, all the transport between
> adaptors could named as communication, while connection management is exactly
> what CM did in kernel.
> 
> Doug, what's your opinion?

See my previous email.  I can understand using the official name, but I
can also definitely see calling out in the more verbose section of text
that what this does is more close related to connection management (and
I would argue that transport level address resolution for a disconnected
queue pair is a form of connection management too, it's just connection
management on a packet by packet basis).

> > 
> >>> +  rdma_cap_eth_ah
> >>> +  ---------------
> >>> +    Infiniband address handler format is special in ethernet fabric,
> >> use
> >>> +    the helper to verify whether the port is using ethernet format or
> >> not.
> >>
> >> This helper is true when the address of the specific connection is of
> >> the Ethernet (RoCE) variety.
> > 
> > This is used for connectionless communication.
> 
> Could you please give more details on this?

The test itself doesn't mean that.  It means we need a RoCE address
(it's true when transport is IB and link layer is Ethernet).  That we
*use* it during connectionless communication because we have to generate
our own address vector for the packet while during connected queue pair
use the address vector is created by the card using the queue pair
information is just the circumstance of its use.  And even though a
disconnected queue pair isn't solidly connected to a remote endpoint, it
is solidly bound to an adapter that requires either an IB or Ethernet
address family.  Maybe this to resolve your issue with the wording:

This helper is true when the address family of this queue pair is of the
Ethernet (RoCE) variety.


-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH RFC] Documentation/infiniband: Add docs for rdma-helpers

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 2518 bytes --]

On Fri, 2015-05-15 at 10:31 +0200, Michael Wang wrote:
> 
> On 05/13/2015 06:42 PM, Hefty, Sean wrote:
> >>> +  rdma_cap_ib_cm
> >>> +  ---------------
> >>> +    Communication Manager (CM) will handle the connections between
> >>        ^Connection Manager (CM) service, used to ease the process of
> > 
> > In IB terms, this is communication manager.  It also handles transport level address resolution for UD QPs.
> 
> I could find both 'connection' and 'communication' in different docs,
> while 'connection' is more related to verbs, 'communication' is more
> close to specification.
> 
> IMHO 'connection' make more sense, after all, all the transport between
> adaptors could named as communication, while connection management is exactly
> what CM did in kernel.
> 
> Doug, what's your opinion?

See my previous email.  I can understand using the official name, but I
can also definitely see calling out in the more verbose section of text
that what this does is more close related to connection management (and
I would argue that transport level address resolution for a disconnected
queue pair is a form of connection management too, it's just connection
management on a packet by packet basis).

> > 
> >>> +  rdma_cap_eth_ah
> >>> +  ---------------
> >>> +    Infiniband address handler format is special in ethernet fabric,
> >> use
> >>> +    the helper to verify whether the port is using ethernet format or
> >> not.
> >>
> >> This helper is true when the address of the specific connection is of
> >> the Ethernet (RoCE) variety.
> > 
> > This is used for connectionless communication.
> 
> Could you please give more details on this?

The test itself doesn't mean that.  It means we need a RoCE address
(it's true when transport is IB and link layer is Ethernet).  That we
*use* it during connectionless communication because we have to generate
our own address vector for the packet while during connected queue pair
use the address vector is created by the card using the queue pair
information is just the circumstance of its use.  And even though a
disconnected queue pair isn't solidly connected to a remote endpoint, it
is solidly bound to an adapter that requires either an IB or Ethernet
address family.  Maybe this to resolve your issue with the wording:

This helper is true when the address family of this queue pair is of the
Ethernet (RoCE) variety.


-- 
Doug Ledford <dledford@redhat.com>
              GPG KeyID: 0E572FDD


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH RFC] Documentation/infiniband: Add docs for rdma-helpers

[Michael Wang]

On 05/15/2015 04:40 PM, Doug Ledford wrote:
[snip]
> 
> The test itself doesn't mean that.  It means we need a RoCE address
> (it's true when transport is IB and link layer is Ethernet).  That we
> *use* it during connectionless communication because we have to generate
> our own address vector for the packet while during connected queue pair
> use the address vector is created by the card using the queue pair
> information is just the circumstance of its use.  And even though a
> disconnected queue pair isn't solidly connected to a remote endpoint, it
> is solidly bound to an adapter that requires either an IB or Ethernet
> address family.  Maybe this to resolve your issue with the wording:

Thanks for the explain :-) The term 'connectionless' still sounds a little
strange to me when it's just means no HW support on creating address vector,
but I can understand the concept.

> 
> This helper is true when the address family of this queue pair is of the
> Ethernet (RoCE) variety.

Sounds good, will be merged in next version :-)

Regards,
Michael Wang

> 
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH RFC] Documentation/infiniband: Add docs for rdma-helpers

[Michael Wang]

On 05/15/2015 04:40 PM, Doug Ledford wrote:
[snip]
> 
> The test itself doesn't mean that.  It means we need a RoCE address
> (it's true when transport is IB and link layer is Ethernet).  That we
> *use* it during connectionless communication because we have to generate
> our own address vector for the packet while during connected queue pair
> use the address vector is created by the card using the queue pair
> information is just the circumstance of its use.  And even though a
> disconnected queue pair isn't solidly connected to a remote endpoint, it
> is solidly bound to an adapter that requires either an IB or Ethernet
> address family.  Maybe this to resolve your issue with the wording:

Thanks for the explain :-) The term 'connectionless' still sounds a little
strange to me when it's just means no HW support on creating address vector,
but I can understand the concept.

> 
> This helper is true when the address family of this queue pair is of the
> Ethernet (RoCE) variety.

Sounds good, will be merged in next version :-)

Regards,
Michael Wang

> 
>