Re: rdma_rxe usage problem

[Alexander Kalentyev <comrad.karlovich@gmail.com>]

Dear Robert,

Thank you for your help!
What I did was :
ipv6calc --in prefix+mac fe80:: XX:XX:XX:XX:XX:XX
(with XX:XX:XX:XX:XX:XX one have to insert the MAC address of the NIC)
And then
udo ip addr add dev <dev> fe80::ZZZZ:ZZZZ:ZZZZ:ZZZZ/64
(with ZZZ:ZZZZ:ZZZZ the address genereted by ipv6calc)
sudo rdma link add rxe0 type rxe netdev <dev>
After that everything is working as expected:
>ibv_rc_pingpong -d rxe0 -g 0 192.168.0.176
  local address:  LID 0x0000, QPN 0x000012, PSN 0xf85048, GID
ZZZZ::ZZZZ:ZZZZ:ZZZZ:ZZZZ
  remote address: LID 0x0000, QPN 0x000011, PSN 0xf7c7b7, GID
ZZZZ::ZZZZ:ZZZZ:ZZZZ:ZZZZ
8192000 bytes in 0.01 seconds = 4743.49 Mbit/sec
1000 iters in 0.01 seconds = 13.82 usec/iter

So, I think it make sense to include this information in a README on
github. Otherwise somebody like me can spend a week trying to figure
out what is going on!

Thanks once again!
Regards,
Alexander Kalentev

ср, 19 янв. 2022 г. в 20:54, Pearson, Robert B <robert.pearson2@hpe.com>:
>
>
>
> -----Original Message-----
> From: Alexander Kalentyev <comrad.karlovich@gmail.com>
> Sent: Wednesday, January 19, 2022 11:53 AM
>
>
> Anyway the ibv_rc_pingpong shows an error:
>
> >ibv_rc_pingpong -d rxe0 -g 0
>   local address:  LID 0x0000, QPN 0x000015, PSN 0x015dd8, GID
> fe80::4a51:c5ff:fef6:e159
> Failed to modify QP to RTR
> Couldn't connect to remote QP
>
> >
> Alexander,
>
> I use a script to restart rxe after changing anything it looks like
>
>         #!/bin/bash
>
>         export LD_LIBRARY_PATH=<path to rdma-core>/rdma-core/build/lib:/usr/lib
>
>         sudo rmmod rdma_rxe
>         sudo modprobe rdma_rxe
>
>         sudo ip link set dev enp0s3 mtu 4500
>         sudo ip addr add dev enp0s3 fe80::0a00:27ff:fe35:5ea7/64
>         sudo rdma link add rxe0 type rxe netdev enp0s3
>
> The important line is adding the ipv6 address which corresponds with the MAC address of
> The ethernet nic which is
>
>         08:00:27:35:5e:a7
>
> Some OSes (like mine) do not create this address automatically but mangle the address.
> But the rdma core driver seems to expect all roce providers to have it.
>
> Hope this helps.
>
> Bob Pearson
> rpearson@hpe.com