ibv_rc_pingpong failes on softroce device using GID[0]

ibv_rc_pingpong failes on softroce device using GID[0]

[hui wang]

Hi All
I am using ubuntu 21.10. I used the following command to create a
softroce device rxe0:
sudo rdma link add rxe0 type rxe netdev enp0s3

rxe0's GID[0] is not an real IP address assigned to my enp0s3. It
seems to be something softroce generated from the node_guid (seems to
be generated from mac address of enp0s3).
Is this expected behavior? How to use this GID given that it is not
reachable because it is not a real IP address?

I used ibv_rc_pingpong -d rxe0 -g 0 to setup a server with this GID,
the server failed with "Failed to modify QP to RTR" when client tries
to talk with it with client's GID[0] using command "./ibv_rc_pingpong
-d rxe0 -g 0 10.0.0.233". Is this because the GID[0] is not a real IP?

ibv_rc_pingpong works if i use any other GID, (eg. GID[1] - GID[5])

Here is the output of ibv_devinfo and ip addr. you can see GID[0]
fe80::a00:27ff:fe2e:f7a5 is not a real ip address assigned to enp0s3
(where rxe0 was created).

huiwang@huiwang-VirtualBox:~/rdma-core$ ibv_devinfo -v
hca_id: rxe0
        transport:                      InfiniBand (0)
        fw_ver:                         0.0.0
        node_guid:                      0a00:27ff:fe2e:f7a5
        sys_image_guid:                 0a00:27ff:fe2e:f7a5
....
                        GID[  0]:
fe80::a00:27ff:fe2e:f7a5, RoCE v2
                        GID[  1]:
fe80::9bd3:885:53df:9fc4, RoCE v2
                        GID[  2]:               ::ffff:10.0.0.151, RoCE v2
                        GID[  3]:
2601:647:5380:2210:19df:d855:54a9:cc9f, RoCE v2
                        GID[  4]:
2601:647:5380:2210:e3f3:7e57:504b:7e5c, RoCE v2
                        GID[  5]:
2601:647:5380:2210::3be1, RoCE v2

huiwang@huiwang-VirtualBox:~/rdma-core/build/bin$ ip addr
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel
state UP group default qlen 1000
    link/ether 08:00:27:2e:f7:a5 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.151/64 brd 10.0.0.255 scope global dynamic noprefixroute enp0s3
       valid_lft 143273sec preferred_lft 143273sec
    inet6 2601:647:5380:2210::3be1/128 scope global dynamic noprefixroute
       valid_lft 575276sec preferred_lft 575276sec
    inet6 2601:647:5380:2210:e3f3:7e57:504b:7e5c/64 scope global
temporary dynamic
       valid_lft 255135sec preferred_lft 56812sec
    inet6 2601:647:5380:2210:19df:d855:54a9:cc9f/64 scope global
dynamic mngtmpaddr noprefixroute
       valid_lft 255135sec preferred_lft 255135sec
    inet6 fe80::9bd3:885:53df:9fc4/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

Thanks,
Hui

Re: ibv_rc_pingpong failes on softroce device using GID[0]

[Yanjun Zhu]

在 2022/2/3 13:28, hui wang 写道:
> Hi All
> I am using ubuntu 21.10. I used the following command to create a
> softroce device rxe0:
> sudo rdma link add rxe0 type rxe netdev enp0s3
> 
> rxe0's GID[0] is not an real IP address assigned to my enp0s3. It
> seems to be something softroce generated from the node_guid (seems to
> be generated from mac address of enp0s3).
> Is this expected behavior? How to use this GID given that it is not
> reachable because it is not a real IP address?
> 
> I used ibv_rc_pingpong -d rxe0 -g 0 to setup a server with this GID,
> the server failed with "Failed to modify QP to RTR" when client tries

please check this link:

https://www.spinics.net/lists/linux-rdma/msg108317.html

Zhu Yanjun

> to talk with it with client's GID[0] using command "./ibv_rc_pingpong
> -d rxe0 -g 0 10.0.0.233". Is this because the GID[0] is not a real IP?
> 
> ibv_rc_pingpong works if i use any other GID, (eg. GID[1] - GID[5])
> 
> Here is the output of ibv_devinfo and ip addr. you can see GID[0]
> fe80::a00:27ff:fe2e:f7a5 is not a real ip address assigned to enp0s3
> (where rxe0 was created).
> 
> huiwang@huiwang-VirtualBox:~/rdma-core$ ibv_devinfo -v
> hca_id: rxe0
>          transport:                      InfiniBand (0)
>          fw_ver:                         0.0.0
>          node_guid:                      0a00:27ff:fe2e:f7a5
>          sys_image_guid:                 0a00:27ff:fe2e:f7a5
> ....
>                          GID[  0]:
> fe80::a00:27ff:fe2e:f7a5, RoCE v2
>                          GID[  1]:
> fe80::9bd3:885:53df:9fc4, RoCE v2
>                          GID[  2]:               ::ffff:10.0.0.151, RoCE v2
>                          GID[  3]:
> 2601:647:5380:2210:19df:d855:54a9:cc9f, RoCE v2
>                          GID[  4]:
> 2601:647:5380:2210:e3f3:7e57:504b:7e5c, RoCE v2
>                          GID[  5]:
> 2601:647:5380:2210::3be1, RoCE v2
> 
> huiwang@huiwang-VirtualBox:~/rdma-core/build/bin$ ip addr
> 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel
> state UP group default qlen 1000
>      link/ether 08:00:27:2e:f7:a5 brd ff:ff:ff:ff:ff:ff
>      inet 10.0.0.151/64 brd 10.0.0.255 scope global dynamic noprefixroute enp0s3
>         valid_lft 143273sec preferred_lft 143273sec
>      inet6 2601:647:5380:2210::3be1/128 scope global dynamic noprefixroute
>         valid_lft 575276sec preferred_lft 575276sec
>      inet6 2601:647:5380:2210:e3f3:7e57:504b:7e5c/64 scope global
> temporary dynamic
>         valid_lft 255135sec preferred_lft 56812sec
>      inet6 2601:647:5380:2210:19df:d855:54a9:cc9f/64 scope global
> dynamic mngtmpaddr noprefixroute
>         valid_lft 255135sec preferred_lft 255135sec
>      inet6 fe80::9bd3:885:53df:9fc4/64 scope link noprefixroute
>         valid_lft forever preferred_lft forever
> 
> Thanks,
> Hui