[Question] Is address format "gid" supported by RDMACM with RoCE?

[Question] Is address format "gid" supported by RDMACM with RoCE?

[Sylvain Didelot]

Hi,

I have configured one of my Mellanox network adapters for RoCE:
---
CA 'roceP1p1s0f1'
    CA type: MT4123
    Number of ports: 1
    Firmware version: 20.32.1010
    Hardware version: 0
    Node GUID: 0xb8cef603002d1707
    System image GUID: 0xb8cef603002d1706
    Port 1:
        State: Active
        Physical state: LinkUp
        Rate: 100
        Base lid: 0
        LMC: 0
        SM lid: 0
        Capability mask: 0x00010000
        Port GUID: 0xbacef6fffe2d1707
        Link layer: Ethernet
---

The Infiniband stack was installed from the official Ubuntu repository
(20.04.4 LTS):
---
$ apt search ibverbs
Sorting... Done
Full Text Search... Done
ibverbs-providers/focal,now 28.0-1ubuntu1 arm64 [installed]
  User space provider drivers for libibverbs

ibverbs-utils/focal,now 28.0-1ubuntu1 arm64 [installed]
  Examples for the libibverbs library

libibverbs-dev/focal,now 28.0-1ubuntu1 arm64 [installed]
  Development files for the libibverbs library

libibverbs1/focal,now 28.0-1ubuntu1 arm64 [installed]
  Library for direct userspace use of RDMA (InfiniBand/iWARP)

librdmacm-dev/focal,now 28.0-1ubuntu1 arm64 [installed]
  Development files for the librdmacm library

librdmacm1/focal,now 28.0-1ubuntu1 arm64 [installed]
  Library for managing RDMA connections

rdmacm-utils/focal,now 28.0-1ubuntu1 arm64 [installed]
  Examples for the librdmacm library
---

When I start the ucmatose server with the address format "gid", the
tool fails binding with the error "No such device"

Here is an example of the output:
---
$ cat /sys/class/infiniband/roceP1p1s0f1/ports/1/gids/0
fe80:0000:0000:0000:bace:f6ff:fe2d:1707

$ ucmatose -b fe80:0000:0000:0000:bace:f6ff:fe2d:1707 -P ib -f gid
cmatose: starting server
cmatose: bind address failed: No such device
test complete
return status -1
---

Does rdmacm support connection establishment using GID with RoCE? Or
Is it a known limitation for RoCE?
FYI, the same experiment without RoCE (Link layer: Infiniband) works perfectly.

Thanks for your help and your feedback.

Sylvain Didelot

Re: [Question] Is address format "gid" supported by RDMACM with RoCE?

[Mark Zhang]

On 3/7/2022 9:57 PM, Sylvain Didelot wrote:
> External email: Use caution opening links or attachments
> 
> 
> Hi,
> 
> I have configured one of my Mellanox network adapters for RoCE:
> ---
> CA 'roceP1p1s0f1'
>      CA type: MT4123
>      Number of ports: 1
>      Firmware version: 20.32.1010
>      Hardware version: 0
>      Node GUID: 0xb8cef603002d1707
>      System image GUID: 0xb8cef603002d1706
>      Port 1:
>          State: Active
>          Physical state: LinkUp
>          Rate: 100
>          Base lid: 0
>          LMC: 0
>          SM lid: 0
>          Capability mask: 0x00010000
>          Port GUID: 0xbacef6fffe2d1707
>          Link layer: Ethernet
> ---
> 
> The Infiniband stack was installed from the official Ubuntu repository
> (20.04.4 LTS):
> ---
> $ apt search ibverbs
> Sorting... Done
> Full Text Search... Done
> ibverbs-providers/focal,now 28.0-1ubuntu1 arm64 [installed]
>    User space provider drivers for libibverbs
> 
> ibverbs-utils/focal,now 28.0-1ubuntu1 arm64 [installed]
>    Examples for the libibverbs library
> 
> libibverbs-dev/focal,now 28.0-1ubuntu1 arm64 [installed]
>    Development files for the libibverbs library
> 
> libibverbs1/focal,now 28.0-1ubuntu1 arm64 [installed]
>    Library for direct userspace use of RDMA (InfiniBand/iWARP)
> 
> librdmacm-dev/focal,now 28.0-1ubuntu1 arm64 [installed]
>    Development files for the librdmacm library
> 
> librdmacm1/focal,now 28.0-1ubuntu1 arm64 [installed]
>    Library for managing RDMA connections
> 
> rdmacm-utils/focal,now 28.0-1ubuntu1 arm64 [installed]
>    Examples for the librdmacm library
> ---
> 
> When I start the ucmatose server with the address format "gid", the
> tool fails binding with the error "No such device"
> 
> Here is an example of the output:
> ---
> $ cat /sys/class/infiniband/roceP1p1s0f1/ports/1/gids/0
> fe80:0000:0000:0000:bace:f6ff:fe2d:1707
> 
> $ ucmatose -b fe80:0000:0000:0000:bace:f6ff:fe2d:1707 -P ib -f gid
> cmatose: starting server
> cmatose: bind address failed: No such device
> test complete
> return status -1
> ---
> 
> Does rdmacm support connection establishment using GID with RoCE? Or
> Is it a known limitation for RoCE?
> FYI, the same experiment without RoCE (Link layer: Infiniband) works perfectly.
> 
> Thanks for your help and your feedback.
> 
> Sylvain Didelot

I think ucmatose doesn't support RoCE when using "-f gid", as in this 
case ai_family is set to AF_IB.

Re: [Question] Is address format "gid" supported by RDMACM with RoCE?

[Sylvain Didelot]

Does it mean that RoCE always requires the network interface to have
an IP address as it cannot use the GID to establish connections?

On Tue, Mar 8, 2022 at 11:49 AM Mark Zhang <markzhang@nvidia.com> wrote:
>
> On 3/7/2022 9:57 PM, Sylvain Didelot wrote:
> > External email: Use caution opening links or attachments
> >
> >
> > Hi,
> >
> > I have configured one of my Mellanox network adapters for RoCE:
> > ---
> > CA 'roceP1p1s0f1'
> >      CA type: MT4123
> >      Number of ports: 1
> >      Firmware version: 20.32.1010
> >      Hardware version: 0
> >      Node GUID: 0xb8cef603002d1707
> >      System image GUID: 0xb8cef603002d1706
> >      Port 1:
> >          State: Active
> >          Physical state: LinkUp
> >          Rate: 100
> >          Base lid: 0
> >          LMC: 0
> >          SM lid: 0
> >          Capability mask: 0x00010000
> >          Port GUID: 0xbacef6fffe2d1707
> >          Link layer: Ethernet
> > ---
> >
> > The Infiniband stack was installed from the official Ubuntu repository
> > (20.04.4 LTS):
> > ---
> > $ apt search ibverbs
> > Sorting... Done
> > Full Text Search... Done
> > ibverbs-providers/focal,now 28.0-1ubuntu1 arm64 [installed]
> >    User space provider drivers for libibverbs
> >
> > ibverbs-utils/focal,now 28.0-1ubuntu1 arm64 [installed]
> >    Examples for the libibverbs library
> >
> > libibverbs-dev/focal,now 28.0-1ubuntu1 arm64 [installed]
> >    Development files for the libibverbs library
> >
> > libibverbs1/focal,now 28.0-1ubuntu1 arm64 [installed]
> >    Library for direct userspace use of RDMA (InfiniBand/iWARP)
> >
> > librdmacm-dev/focal,now 28.0-1ubuntu1 arm64 [installed]
> >    Development files for the librdmacm library
> >
> > librdmacm1/focal,now 28.0-1ubuntu1 arm64 [installed]
> >    Library for managing RDMA connections
> >
> > rdmacm-utils/focal,now 28.0-1ubuntu1 arm64 [installed]
> >    Examples for the librdmacm library
> > ---
> >
> > When I start the ucmatose server with the address format "gid", the
> > tool fails binding with the error "No such device"
> >
> > Here is an example of the output:
> > ---
> > $ cat /sys/class/infiniband/roceP1p1s0f1/ports/1/gids/0
> > fe80:0000:0000:0000:bace:f6ff:fe2d:1707
> >
> > $ ucmatose -b fe80:0000:0000:0000:bace:f6ff:fe2d:1707 -P ib -f gid
> > cmatose: starting server
> > cmatose: bind address failed: No such device
> > test complete
> > return status -1
> > ---
> >
> > Does rdmacm support connection establishment using GID with RoCE? Or
> > Is it a known limitation for RoCE?
> > FYI, the same experiment without RoCE (Link layer: Infiniband) works perfectly.
> >
> > Thanks for your help and your feedback.
> >
> > Sylvain Didelot
>
> I think ucmatose doesn't support RoCE when using "-f gid", as in this
> case ai_family is set to AF_IB.

Re: [Question] Is address format "gid" supported by RDMACM with RoCE?

[Mark Zhang]

On 3/8/2022 6:54 PM, Sylvain Didelot wrote:
> External email: Use caution opening links or attachments
> 
> 
> Does it mean that RoCE always requires the network interface to have
> an IP address as it cannot use the GID to establish connections?
> 

I think it's just a ucmatose limitation..

> On Tue, Mar 8, 2022 at 11:49 AM Mark Zhang <markzhang@nvidia.com> wrote:
>>
>> On 3/7/2022 9:57 PM, Sylvain Didelot wrote:
>>> External email: Use caution opening links or attachments
>>>
>>>
>>> Hi,
>>>
>>> I have configured one of my Mellanox network adapters for RoCE:
>>> ---
>>> CA 'roceP1p1s0f1'
>>>       CA type: MT4123
>>>       Number of ports: 1
>>>       Firmware version: 20.32.1010
>>>       Hardware version: 0
>>>       Node GUID: 0xb8cef603002d1707
>>>       System image GUID: 0xb8cef603002d1706
>>>       Port 1:
>>>           State: Active
>>>           Physical state: LinkUp
>>>           Rate: 100
>>>           Base lid: 0
>>>           LMC: 0
>>>           SM lid: 0
>>>           Capability mask: 0x00010000
>>>           Port GUID: 0xbacef6fffe2d1707
>>>           Link layer: Ethernet
>>> ---
>>>
>>> The Infiniband stack was installed from the official Ubuntu repository
>>> (20.04.4 LTS):
>>> ---
>>> $ apt search ibverbs
>>> Sorting... Done
>>> Full Text Search... Done
>>> ibverbs-providers/focal,now 28.0-1ubuntu1 arm64 [installed]
>>>     User space provider drivers for libibverbs
>>>
>>> ibverbs-utils/focal,now 28.0-1ubuntu1 arm64 [installed]
>>>     Examples for the libibverbs library
>>>
>>> libibverbs-dev/focal,now 28.0-1ubuntu1 arm64 [installed]
>>>     Development files for the libibverbs library
>>>
>>> libibverbs1/focal,now 28.0-1ubuntu1 arm64 [installed]
>>>     Library for direct userspace use of RDMA (InfiniBand/iWARP)
>>>
>>> librdmacm-dev/focal,now 28.0-1ubuntu1 arm64 [installed]
>>>     Development files for the librdmacm library
>>>
>>> librdmacm1/focal,now 28.0-1ubuntu1 arm64 [installed]
>>>     Library for managing RDMA connections
>>>
>>> rdmacm-utils/focal,now 28.0-1ubuntu1 arm64 [installed]
>>>     Examples for the librdmacm library
>>> ---
>>>
>>> When I start the ucmatose server with the address format "gid", the
>>> tool fails binding with the error "No such device"
>>>
>>> Here is an example of the output:
>>> ---
>>> $ cat /sys/class/infiniband/roceP1p1s0f1/ports/1/gids/0
>>> fe80:0000:0000:0000:bace:f6ff:fe2d:1707
>>>
>>> $ ucmatose -b fe80:0000:0000:0000:bace:f6ff:fe2d:1707 -P ib -f gid
>>> cmatose: starting server
>>> cmatose: bind address failed: No such device
>>> test complete
>>> return status -1
>>> ---
>>>
>>> Does rdmacm support connection establishment using GID with RoCE? Or
>>> Is it a known limitation for RoCE?
>>> FYI, the same experiment without RoCE (Link layer: Infiniband) works perfectly.
>>>
>>> Thanks for your help and your feedback.
>>>
>>> Sylvain Didelot
>>
>> I think ucmatose doesn't support RoCE when using "-f gid", as in this
>> case ai_family is set to AF_IB.

Re: [Question] Is address format "gid" supported by RDMACM with RoCE?

[Sylvain Didelot]

Thanks for your answers, Mark.

The Nvidia documentation[1] states the following: "RoCE has two
addressing modes: MAC based GIDs, and IP address based GIDs."
Do you know which address family (AF_*) should be used to establish
connection with MAC based GIDs?
Is there an example or some documentation somewhere?

I can fix ucmatose myself, but I have no idea what's missing exactly.

[1]: https://docs.nvidia.com/networking/pages/viewpage.action?pageId=15046549

On Tue, Mar 8, 2022 at 1:43 PM Mark Zhang <markzhang@nvidia.com> wrote:
>
> On 3/8/2022 6:54 PM, Sylvain Didelot wrote:
> > External email: Use caution opening links or attachments
> >
> >
> > Does it mean that RoCE always requires the network interface to have
> > an IP address as it cannot use the GID to establish connections?
> >
>
> I think it's just a ucmatose limitation..
>
> > On Tue, Mar 8, 2022 at 11:49 AM Mark Zhang <markzhang@nvidia.com> wrote:
> >>
> >> On 3/7/2022 9:57 PM, Sylvain Didelot wrote:
> >>> External email: Use caution opening links or attachments
> >>>
> >>>
> >>> Hi,
> >>>
> >>> I have configured one of my Mellanox network adapters for RoCE:
> >>> ---
> >>> CA 'roceP1p1s0f1'
> >>>       CA type: MT4123
> >>>       Number of ports: 1
> >>>       Firmware version: 20.32.1010
> >>>       Hardware version: 0
> >>>       Node GUID: 0xb8cef603002d1707
> >>>       System image GUID: 0xb8cef603002d1706
> >>>       Port 1:
> >>>           State: Active
> >>>           Physical state: LinkUp
> >>>           Rate: 100
> >>>           Base lid: 0
> >>>           LMC: 0
> >>>           SM lid: 0
> >>>           Capability mask: 0x00010000
> >>>           Port GUID: 0xbacef6fffe2d1707
> >>>           Link layer: Ethernet
> >>> ---
> >>>
> >>> The Infiniband stack was installed from the official Ubuntu repository
> >>> (20.04.4 LTS):
> >>> ---
> >>> $ apt search ibverbs
> >>> Sorting... Done
> >>> Full Text Search... Done
> >>> ibverbs-providers/focal,now 28.0-1ubuntu1 arm64 [installed]
> >>>     User space provider drivers for libibverbs
> >>>
> >>> ibverbs-utils/focal,now 28.0-1ubuntu1 arm64 [installed]
> >>>     Examples for the libibverbs library
> >>>
> >>> libibverbs-dev/focal,now 28.0-1ubuntu1 arm64 [installed]
> >>>     Development files for the libibverbs library
> >>>
> >>> libibverbs1/focal,now 28.0-1ubuntu1 arm64 [installed]
> >>>     Library for direct userspace use of RDMA (InfiniBand/iWARP)
> >>>
> >>> librdmacm-dev/focal,now 28.0-1ubuntu1 arm64 [installed]
> >>>     Development files for the librdmacm library
> >>>
> >>> librdmacm1/focal,now 28.0-1ubuntu1 arm64 [installed]
> >>>     Library for managing RDMA connections
> >>>
> >>> rdmacm-utils/focal,now 28.0-1ubuntu1 arm64 [installed]
> >>>     Examples for the librdmacm library
> >>> ---
> >>>
> >>> When I start the ucmatose server with the address format "gid", the
> >>> tool fails binding with the error "No such device"
> >>>
> >>> Here is an example of the output:
> >>> ---
> >>> $ cat /sys/class/infiniband/roceP1p1s0f1/ports/1/gids/0
> >>> fe80:0000:0000:0000:bace:f6ff:fe2d:1707
> >>>
> >>> $ ucmatose -b fe80:0000:0000:0000:bace:f6ff:fe2d:1707 -P ib -f gid
> >>> cmatose: starting server
> >>> cmatose: bind address failed: No such device
> >>> test complete
> >>> return status -1
> >>> ---
> >>>
> >>> Does rdmacm support connection establishment using GID with RoCE? Or
> >>> Is it a known limitation for RoCE?
> >>> FYI, the same experiment without RoCE (Link layer: Infiniband) works perfectly.
> >>>
> >>> Thanks for your help and your feedback.
> >>>
> >>> Sylvain Didelot
> >>
> >> I think ucmatose doesn't support RoCE when using "-f gid", as in this
> >> case ai_family is set to AF_IB.
>