* [Question] Is address format "gid" supported by RDMACM with RoCE?
@ 2022-03-07 13:57 Sylvain Didelot
2022-03-08 10:49 ` Mark Zhang
0 siblings, 1 reply; 5+ messages in thread
From: Sylvain Didelot @ 2022-03-07 13:57 UTC (permalink / raw)
To: linux-rdma
Hi,
I have configured one of my Mellanox network adapters for RoCE:
---
CA 'roceP1p1s0f1'
CA type: MT4123
Number of ports: 1
Firmware version: 20.32.1010
Hardware version: 0
Node GUID: 0xb8cef603002d1707
System image GUID: 0xb8cef603002d1706
Port 1:
State: Active
Physical state: LinkUp
Rate: 100
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: 0xbacef6fffe2d1707
Link layer: Ethernet
---
The Infiniband stack was installed from the official Ubuntu repository
(20.04.4 LTS):
---
$ apt search ibverbs
Sorting... Done
Full Text Search... Done
ibverbs-providers/focal,now 28.0-1ubuntu1 arm64 [installed]
User space provider drivers for libibverbs
ibverbs-utils/focal,now 28.0-1ubuntu1 arm64 [installed]
Examples for the libibverbs library
libibverbs-dev/focal,now 28.0-1ubuntu1 arm64 [installed]
Development files for the libibverbs library
libibverbs1/focal,now 28.0-1ubuntu1 arm64 [installed]
Library for direct userspace use of RDMA (InfiniBand/iWARP)
librdmacm-dev/focal,now 28.0-1ubuntu1 arm64 [installed]
Development files for the librdmacm library
librdmacm1/focal,now 28.0-1ubuntu1 arm64 [installed]
Library for managing RDMA connections
rdmacm-utils/focal,now 28.0-1ubuntu1 arm64 [installed]
Examples for the librdmacm library
---
When I start the ucmatose server with the address format "gid", the
tool fails binding with the error "No such device"
Here is an example of the output:
---
$ cat /sys/class/infiniband/roceP1p1s0f1/ports/1/gids/0
fe80:0000:0000:0000:bace:f6ff:fe2d:1707
$ ucmatose -b fe80:0000:0000:0000:bace:f6ff:fe2d:1707 -P ib -f gid
cmatose: starting server
cmatose: bind address failed: No such device
test complete
return status -1
---
Does rdmacm support connection establishment using GID with RoCE? Or
Is it a known limitation for RoCE?
FYI, the same experiment without RoCE (Link layer: Infiniband) works perfectly.
Thanks for your help and your feedback.
Sylvain Didelot
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Question] Is address format "gid" supported by RDMACM with RoCE?
2022-03-07 13:57 [Question] Is address format "gid" supported by RDMACM with RoCE? Sylvain Didelot
@ 2022-03-08 10:49 ` Mark Zhang
2022-03-08 10:54 ` Sylvain Didelot
0 siblings, 1 reply; 5+ messages in thread
From: Mark Zhang @ 2022-03-08 10:49 UTC (permalink / raw)
To: Sylvain Didelot, linux-rdma
On 3/7/2022 9:57 PM, Sylvain Didelot wrote:
> External email: Use caution opening links or attachments
>
>
> Hi,
>
> I have configured one of my Mellanox network adapters for RoCE:
> ---
> CA 'roceP1p1s0f1'
> CA type: MT4123
> Number of ports: 1
> Firmware version: 20.32.1010
> Hardware version: 0
> Node GUID: 0xb8cef603002d1707
> System image GUID: 0xb8cef603002d1706
> Port 1:
> State: Active
> Physical state: LinkUp
> Rate: 100
> Base lid: 0
> LMC: 0
> SM lid: 0
> Capability mask: 0x00010000
> Port GUID: 0xbacef6fffe2d1707
> Link layer: Ethernet
> ---
>
> The Infiniband stack was installed from the official Ubuntu repository
> (20.04.4 LTS):
> ---
> $ apt search ibverbs
> Sorting... Done
> Full Text Search... Done
> ibverbs-providers/focal,now 28.0-1ubuntu1 arm64 [installed]
> User space provider drivers for libibverbs
>
> ibverbs-utils/focal,now 28.0-1ubuntu1 arm64 [installed]
> Examples for the libibverbs library
>
> libibverbs-dev/focal,now 28.0-1ubuntu1 arm64 [installed]
> Development files for the libibverbs library
>
> libibverbs1/focal,now 28.0-1ubuntu1 arm64 [installed]
> Library for direct userspace use of RDMA (InfiniBand/iWARP)
>
> librdmacm-dev/focal,now 28.0-1ubuntu1 arm64 [installed]
> Development files for the librdmacm library
>
> librdmacm1/focal,now 28.0-1ubuntu1 arm64 [installed]
> Library for managing RDMA connections
>
> rdmacm-utils/focal,now 28.0-1ubuntu1 arm64 [installed]
> Examples for the librdmacm library
> ---
>
> When I start the ucmatose server with the address format "gid", the
> tool fails binding with the error "No such device"
>
> Here is an example of the output:
> ---
> $ cat /sys/class/infiniband/roceP1p1s0f1/ports/1/gids/0
> fe80:0000:0000:0000:bace:f6ff:fe2d:1707
>
> $ ucmatose -b fe80:0000:0000:0000:bace:f6ff:fe2d:1707 -P ib -f gid
> cmatose: starting server
> cmatose: bind address failed: No such device
> test complete
> return status -1
> ---
>
> Does rdmacm support connection establishment using GID with RoCE? Or
> Is it a known limitation for RoCE?
> FYI, the same experiment without RoCE (Link layer: Infiniband) works perfectly.
>
> Thanks for your help and your feedback.
>
> Sylvain Didelot
I think ucmatose doesn't support RoCE when using "-f gid", as in this
case ai_family is set to AF_IB.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Question] Is address format "gid" supported by RDMACM with RoCE?
2022-03-08 10:49 ` Mark Zhang
@ 2022-03-08 10:54 ` Sylvain Didelot
2022-03-08 12:43 ` Mark Zhang
0 siblings, 1 reply; 5+ messages in thread
From: Sylvain Didelot @ 2022-03-08 10:54 UTC (permalink / raw)
To: Mark Zhang; +Cc: linux-rdma
Does it mean that RoCE always requires the network interface to have
an IP address as it cannot use the GID to establish connections?
On Tue, Mar 8, 2022 at 11:49 AM Mark Zhang <markzhang@nvidia.com> wrote:
>
> On 3/7/2022 9:57 PM, Sylvain Didelot wrote:
> > External email: Use caution opening links or attachments
> >
> >
> > Hi,
> >
> > I have configured one of my Mellanox network adapters for RoCE:
> > ---
> > CA 'roceP1p1s0f1'
> > CA type: MT4123
> > Number of ports: 1
> > Firmware version: 20.32.1010
> > Hardware version: 0
> > Node GUID: 0xb8cef603002d1707
> > System image GUID: 0xb8cef603002d1706
> > Port 1:
> > State: Active
> > Physical state: LinkUp
> > Rate: 100
> > Base lid: 0
> > LMC: 0
> > SM lid: 0
> > Capability mask: 0x00010000
> > Port GUID: 0xbacef6fffe2d1707
> > Link layer: Ethernet
> > ---
> >
> > The Infiniband stack was installed from the official Ubuntu repository
> > (20.04.4 LTS):
> > ---
> > $ apt search ibverbs
> > Sorting... Done
> > Full Text Search... Done
> > ibverbs-providers/focal,now 28.0-1ubuntu1 arm64 [installed]
> > User space provider drivers for libibverbs
> >
> > ibverbs-utils/focal,now 28.0-1ubuntu1 arm64 [installed]
> > Examples for the libibverbs library
> >
> > libibverbs-dev/focal,now 28.0-1ubuntu1 arm64 [installed]
> > Development files for the libibverbs library
> >
> > libibverbs1/focal,now 28.0-1ubuntu1 arm64 [installed]
> > Library for direct userspace use of RDMA (InfiniBand/iWARP)
> >
> > librdmacm-dev/focal,now 28.0-1ubuntu1 arm64 [installed]
> > Development files for the librdmacm library
> >
> > librdmacm1/focal,now 28.0-1ubuntu1 arm64 [installed]
> > Library for managing RDMA connections
> >
> > rdmacm-utils/focal,now 28.0-1ubuntu1 arm64 [installed]
> > Examples for the librdmacm library
> > ---
> >
> > When I start the ucmatose server with the address format "gid", the
> > tool fails binding with the error "No such device"
> >
> > Here is an example of the output:
> > ---
> > $ cat /sys/class/infiniband/roceP1p1s0f1/ports/1/gids/0
> > fe80:0000:0000:0000:bace:f6ff:fe2d:1707
> >
> > $ ucmatose -b fe80:0000:0000:0000:bace:f6ff:fe2d:1707 -P ib -f gid
> > cmatose: starting server
> > cmatose: bind address failed: No such device
> > test complete
> > return status -1
> > ---
> >
> > Does rdmacm support connection establishment using GID with RoCE? Or
> > Is it a known limitation for RoCE?
> > FYI, the same experiment without RoCE (Link layer: Infiniband) works perfectly.
> >
> > Thanks for your help and your feedback.
> >
> > Sylvain Didelot
>
> I think ucmatose doesn't support RoCE when using "-f gid", as in this
> case ai_family is set to AF_IB.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Question] Is address format "gid" supported by RDMACM with RoCE?
2022-03-08 10:54 ` Sylvain Didelot
@ 2022-03-08 12:43 ` Mark Zhang
2022-03-08 13:04 ` Sylvain Didelot
0 siblings, 1 reply; 5+ messages in thread
From: Mark Zhang @ 2022-03-08 12:43 UTC (permalink / raw)
To: Sylvain Didelot; +Cc: linux-rdma
On 3/8/2022 6:54 PM, Sylvain Didelot wrote:
> External email: Use caution opening links or attachments
>
>
> Does it mean that RoCE always requires the network interface to have
> an IP address as it cannot use the GID to establish connections?
>
I think it's just a ucmatose limitation..
> On Tue, Mar 8, 2022 at 11:49 AM Mark Zhang <markzhang@nvidia.com> wrote:
>>
>> On 3/7/2022 9:57 PM, Sylvain Didelot wrote:
>>> External email: Use caution opening links or attachments
>>>
>>>
>>> Hi,
>>>
>>> I have configured one of my Mellanox network adapters for RoCE:
>>> ---
>>> CA 'roceP1p1s0f1'
>>> CA type: MT4123
>>> Number of ports: 1
>>> Firmware version: 20.32.1010
>>> Hardware version: 0
>>> Node GUID: 0xb8cef603002d1707
>>> System image GUID: 0xb8cef603002d1706
>>> Port 1:
>>> State: Active
>>> Physical state: LinkUp
>>> Rate: 100
>>> Base lid: 0
>>> LMC: 0
>>> SM lid: 0
>>> Capability mask: 0x00010000
>>> Port GUID: 0xbacef6fffe2d1707
>>> Link layer: Ethernet
>>> ---
>>>
>>> The Infiniband stack was installed from the official Ubuntu repository
>>> (20.04.4 LTS):
>>> ---
>>> $ apt search ibverbs
>>> Sorting... Done
>>> Full Text Search... Done
>>> ibverbs-providers/focal,now 28.0-1ubuntu1 arm64 [installed]
>>> User space provider drivers for libibverbs
>>>
>>> ibverbs-utils/focal,now 28.0-1ubuntu1 arm64 [installed]
>>> Examples for the libibverbs library
>>>
>>> libibverbs-dev/focal,now 28.0-1ubuntu1 arm64 [installed]
>>> Development files for the libibverbs library
>>>
>>> libibverbs1/focal,now 28.0-1ubuntu1 arm64 [installed]
>>> Library for direct userspace use of RDMA (InfiniBand/iWARP)
>>>
>>> librdmacm-dev/focal,now 28.0-1ubuntu1 arm64 [installed]
>>> Development files for the librdmacm library
>>>
>>> librdmacm1/focal,now 28.0-1ubuntu1 arm64 [installed]
>>> Library for managing RDMA connections
>>>
>>> rdmacm-utils/focal,now 28.0-1ubuntu1 arm64 [installed]
>>> Examples for the librdmacm library
>>> ---
>>>
>>> When I start the ucmatose server with the address format "gid", the
>>> tool fails binding with the error "No such device"
>>>
>>> Here is an example of the output:
>>> ---
>>> $ cat /sys/class/infiniband/roceP1p1s0f1/ports/1/gids/0
>>> fe80:0000:0000:0000:bace:f6ff:fe2d:1707
>>>
>>> $ ucmatose -b fe80:0000:0000:0000:bace:f6ff:fe2d:1707 -P ib -f gid
>>> cmatose: starting server
>>> cmatose: bind address failed: No such device
>>> test complete
>>> return status -1
>>> ---
>>>
>>> Does rdmacm support connection establishment using GID with RoCE? Or
>>> Is it a known limitation for RoCE?
>>> FYI, the same experiment without RoCE (Link layer: Infiniband) works perfectly.
>>>
>>> Thanks for your help and your feedback.
>>>
>>> Sylvain Didelot
>>
>> I think ucmatose doesn't support RoCE when using "-f gid", as in this
>> case ai_family is set to AF_IB.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Question] Is address format "gid" supported by RDMACM with RoCE?
2022-03-08 12:43 ` Mark Zhang
@ 2022-03-08 13:04 ` Sylvain Didelot
0 siblings, 0 replies; 5+ messages in thread
From: Sylvain Didelot @ 2022-03-08 13:04 UTC (permalink / raw)
To: Mark Zhang; +Cc: linux-rdma
Thanks for your answers, Mark.
The Nvidia documentation[1] states the following: "RoCE has two
addressing modes: MAC based GIDs, and IP address based GIDs."
Do you know which address family (AF_*) should be used to establish
connection with MAC based GIDs?
Is there an example or some documentation somewhere?
I can fix ucmatose myself, but I have no idea what's missing exactly.
[1]: https://docs.nvidia.com/networking/pages/viewpage.action?pageId=15046549
On Tue, Mar 8, 2022 at 1:43 PM Mark Zhang <markzhang@nvidia.com> wrote:
>
> On 3/8/2022 6:54 PM, Sylvain Didelot wrote:
> > External email: Use caution opening links or attachments
> >
> >
> > Does it mean that RoCE always requires the network interface to have
> > an IP address as it cannot use the GID to establish connections?
> >
>
> I think it's just a ucmatose limitation..
>
> > On Tue, Mar 8, 2022 at 11:49 AM Mark Zhang <markzhang@nvidia.com> wrote:
> >>
> >> On 3/7/2022 9:57 PM, Sylvain Didelot wrote:
> >>> External email: Use caution opening links or attachments
> >>>
> >>>
> >>> Hi,
> >>>
> >>> I have configured one of my Mellanox network adapters for RoCE:
> >>> ---
> >>> CA 'roceP1p1s0f1'
> >>> CA type: MT4123
> >>> Number of ports: 1
> >>> Firmware version: 20.32.1010
> >>> Hardware version: 0
> >>> Node GUID: 0xb8cef603002d1707
> >>> System image GUID: 0xb8cef603002d1706
> >>> Port 1:
> >>> State: Active
> >>> Physical state: LinkUp
> >>> Rate: 100
> >>> Base lid: 0
> >>> LMC: 0
> >>> SM lid: 0
> >>> Capability mask: 0x00010000
> >>> Port GUID: 0xbacef6fffe2d1707
> >>> Link layer: Ethernet
> >>> ---
> >>>
> >>> The Infiniband stack was installed from the official Ubuntu repository
> >>> (20.04.4 LTS):
> >>> ---
> >>> $ apt search ibverbs
> >>> Sorting... Done
> >>> Full Text Search... Done
> >>> ibverbs-providers/focal,now 28.0-1ubuntu1 arm64 [installed]
> >>> User space provider drivers for libibverbs
> >>>
> >>> ibverbs-utils/focal,now 28.0-1ubuntu1 arm64 [installed]
> >>> Examples for the libibverbs library
> >>>
> >>> libibverbs-dev/focal,now 28.0-1ubuntu1 arm64 [installed]
> >>> Development files for the libibverbs library
> >>>
> >>> libibverbs1/focal,now 28.0-1ubuntu1 arm64 [installed]
> >>> Library for direct userspace use of RDMA (InfiniBand/iWARP)
> >>>
> >>> librdmacm-dev/focal,now 28.0-1ubuntu1 arm64 [installed]
> >>> Development files for the librdmacm library
> >>>
> >>> librdmacm1/focal,now 28.0-1ubuntu1 arm64 [installed]
> >>> Library for managing RDMA connections
> >>>
> >>> rdmacm-utils/focal,now 28.0-1ubuntu1 arm64 [installed]
> >>> Examples for the librdmacm library
> >>> ---
> >>>
> >>> When I start the ucmatose server with the address format "gid", the
> >>> tool fails binding with the error "No such device"
> >>>
> >>> Here is an example of the output:
> >>> ---
> >>> $ cat /sys/class/infiniband/roceP1p1s0f1/ports/1/gids/0
> >>> fe80:0000:0000:0000:bace:f6ff:fe2d:1707
> >>>
> >>> $ ucmatose -b fe80:0000:0000:0000:bace:f6ff:fe2d:1707 -P ib -f gid
> >>> cmatose: starting server
> >>> cmatose: bind address failed: No such device
> >>> test complete
> >>> return status -1
> >>> ---
> >>>
> >>> Does rdmacm support connection establishment using GID with RoCE? Or
> >>> Is it a known limitation for RoCE?
> >>> FYI, the same experiment without RoCE (Link layer: Infiniband) works perfectly.
> >>>
> >>> Thanks for your help and your feedback.
> >>>
> >>> Sylvain Didelot
> >>
> >> I think ucmatose doesn't support RoCE when using "-f gid", as in this
> >> case ai_family is set to AF_IB.
>
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2022-03-08 13:04 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-03-07 13:57 [Question] Is address format "gid" supported by RDMACM with RoCE? Sylvain Didelot
2022-03-08 10:49 ` Mark Zhang
2022-03-08 10:54 ` Sylvain Didelot
2022-03-08 12:43 ` Mark Zhang
2022-03-08 13:04 ` Sylvain Didelot
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.