Re: Setting up siw devices

[Marcin Mielniczuk <marcin@golem.network>]

While the device is detected by ibv_devices and rping works, I can't get
ibv_rc_pingpong working
(and as far as I understand, RC should be supported by the iWARP driver)

rping works:

    server$  rping -s -a 10.30.10.211 -v
    server ping data: rdma-ping-0:
ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
    server ping data: rdma-ping-1:
BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
    server DISCONNECT EVENT...
    wait for RDMA_READ_ADV state 10

    client$ rping -c -a 10.30.10.211 -C 2 -v
    (output omitted)


But ibv_rc_pingpong doesn't

    server$ ibv_rc_pingpong -d iwp____
      local address:  LID 0x0000, QPN 0x000001, PSN 0xb8aafc, GID ::
    Failed to modify QP to RTS
    Couldn't connect to remote QP
    client$ ibv_rc_pingpong  -d iwp____ 10.30.10.211
      local address:  LID 0x0000, QPN 0x000001, PSN 0x71abc5, GID ::
    client read/write: Protocol not supported
    Couldn't read/write remote address

dmesg says: [   93.957733] iwpm_register_pid: Unable to send a nlmsg
(client = 2)

And OpenMPI/UCX doesn't want to use the transport either:

    [1566564168.338103] [host:23059:0]         select.c:410  UCX  ERROR
no active messages transport to <no debug data>: mm/posix - Destination
is unreachable, mm/sysv - Destination is unreachable, self/self -
Destination is unreachable, rdmacm/sockaddr - no am bcopy, cma/cma - no
am bcopy

Is this a bug or a limitation of siw?
Thanks,
Marcin

On 22.08.2019 18:24, Marcin Mielniczuk wrote:
> Thanks a lot, this did the trick. I think this is worth documenting
> somewhere.
> Would README.md in the rdma-core repo be a good place?
>
> Why
>
> On 22.08.2019 17:52, Leon Romanovsky wrote:
>> On Thu, Aug 22, 2019 at 09:13:25PM +0530, Krishnamraju Eraparaju wrote:
>>> On Thursday, August 08/22/19, 2019 at 17:08:49 +0200, Marcin Mielniczuk wrote:
>>>> Hi,
>>>>
>>>> I'm trying to test the recently merged siw module.
>>>> I'm running kernel 5.3-rc5 (taken from the Ubuntu mainline-kernel
>>>> repository [1]) on Ubuntu 18.04 (bionic).
>>>> I also manually installed rdma-core 25.0 from tarball, using the
>>>> included Debian packaging. I installed all the packages but ibacm.
>>>>
>>>> After booting the new kernel I manually loaded the kernel module by
>>>>
>>>>      modprobe siw
>>>>      modprobe rdma_ucm
>>>>
>>>> Then ibv_devinfo shows: "No IB devices found".
>>>> dmesg only shows:
>>>>      [   29.856751] SoftiWARP attached
>>>>
>>>> According to this tutorial, [2] it should be enough to just load the siw
>>>> module. (unlike RXE, where one needs to use rxe_cfg to set up the
>>>> interface)
>>>> Is this a bug in siw or just a configuration issue on my side?
>>> Have you done "rdma link"?
>>>
>>> rdma link add <NAME> type siw netdev <NETDEV>
>>>
>>> http://man7.org/linux/man-pages/man8/rdma-link.8.html
>> BTW, the same goes for RXE and rxe_cfg is discouraged.
>>
>> Thanks
>>
>>>> Thanks,
>>>> Marcin
>>>>
>>>> [1] https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc5/
>>>> [2] https://budevg.github.io/posts/tutorials/2017/04/29/rdma-101-1.html
>>>>
>