Setting up siw devices

Setting up siw devices

[Marcin Mielniczuk]

[-- Attachment #1.1: Type: text/plain, Size: 919 bytes --]

Hi,

I'm trying to test the recently merged siw module.
I'm running kernel 5.3-rc5 (taken from the Ubuntu mainline-kernel
repository [1]) on Ubuntu 18.04 (bionic).
I also manually installed rdma-core 25.0 from tarball, using the
included Debian packaging. I installed all the packages but ibacm.

After booting the new kernel I manually loaded the kernel module by

     modprobe siw
     modprobe rdma_ucm

Then ibv_devinfo shows: "No IB devices found".
dmesg only shows:
     [   29.856751] SoftiWARP attached

According to this tutorial, [2] it should be enough to just load the siw
module. (unlike RXE, where one needs to use rxe_cfg to set up the
interface)
Is this a bug in siw or just a configuration issue on my side?

Thanks,
Marcin

[1] https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc5/
[2] https://budevg.github.io/posts/tutorials/2017/04/29/rdma-101-1.html


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Setting up siw devices

[Krishnamraju Eraparaju]

On Thursday, August 08/22/19, 2019 at 17:08:49 +0200, Marcin Mielniczuk wrote:
> Hi,
> 
> I'm trying to test the recently merged siw module.
> I'm running kernel 5.3-rc5 (taken from the Ubuntu mainline-kernel
> repository [1]) on Ubuntu 18.04 (bionic).
> I also manually installed rdma-core 25.0 from tarball, using the
> included Debian packaging. I installed all the packages but ibacm.
> 
> After booting the new kernel I manually loaded the kernel module by
> 
>      modprobe siw
>      modprobe rdma_ucm
> 
> Then ibv_devinfo shows: "No IB devices found".
> dmesg only shows:
>      [   29.856751] SoftiWARP attached
> 
> According to this tutorial, [2] it should be enough to just load the siw
> module. (unlike RXE, where one needs to use rxe_cfg to set up the
> interface)
> Is this a bug in siw or just a configuration issue on my side?

Have you done "rdma link"?

rdma link add <NAME> type siw netdev <NETDEV>

http://man7.org/linux/man-pages/man8/rdma-link.8.html


> 
> Thanks,
> Marcin
> 
> [1] https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc5/
> [2] https://budevg.github.io/posts/tutorials/2017/04/29/rdma-101-1.html
> 

Re: Setting up siw devices

[Leon Romanovsky]

On Thu, Aug 22, 2019 at 09:13:25PM +0530, Krishnamraju Eraparaju wrote:
> On Thursday, August 08/22/19, 2019 at 17:08:49 +0200, Marcin Mielniczuk wrote:
> > Hi,
> >
> > I'm trying to test the recently merged siw module.
> > I'm running kernel 5.3-rc5 (taken from the Ubuntu mainline-kernel
> > repository [1]) on Ubuntu 18.04 (bionic).
> > I also manually installed rdma-core 25.0 from tarball, using the
> > included Debian packaging. I installed all the packages but ibacm.
> >
> > After booting the new kernel I manually loaded the kernel module by
> >
> >      modprobe siw
> >      modprobe rdma_ucm
> >
> > Then ibv_devinfo shows: "No IB devices found".
> > dmesg only shows:
> >      [   29.856751] SoftiWARP attached
> >
> > According to this tutorial, [2] it should be enough to just load the siw
> > module. (unlike RXE, where one needs to use rxe_cfg to set up the
> > interface)
> > Is this a bug in siw or just a configuration issue on my side?
>
> Have you done "rdma link"?
>
> rdma link add <NAME> type siw netdev <NETDEV>
>
> http://man7.org/linux/man-pages/man8/rdma-link.8.html

BTW, the same goes for RXE and rxe_cfg is discouraged.

Thanks

>
> >
> > Thanks,
> > Marcin
> >
> > [1] https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc5/
> > [2] https://budevg.github.io/posts/tutorials/2017/04/29/rdma-101-1.html
> >
>
>
>

Re: Setting up siw devices

[Marcin Mielniczuk]

Thanks a lot, this did the trick. I think this is worth documenting
somewhere that this step is needed.
I'll make a PR, would README.md in the rdma-core repo be a good place?

Does <NAME> have any significance? I did:

     sudo rdma link add siw0 type siw netdev enpXsYYfZ

but the resulting device is called iwpXsYYfZ. I couldn't find a trace of
`siw0` anywhere.

On 22.08.2019 17:52, Leon Romanovsky wrote:
> On Thu, Aug 22, 2019 at 09:13:25PM +0530, Krishnamraju Eraparaju wrote:
>> On Thursday, August 08/22/19, 2019 at 17:08:49 +0200, Marcin Mielniczuk wrote:
>>> Hi,
>>>
>>> I'm trying to test the recently merged siw module.
>>> I'm running kernel 5.3-rc5 (taken from the Ubuntu mainline-kernel
>>> repository [1]) on Ubuntu 18.04 (bionic).
>>> I also manually installed rdma-core 25.0 from tarball, using the
>>> included Debian packaging. I installed all the packages but ibacm.
>>>
>>> After booting the new kernel I manually loaded the kernel module by
>>>
>>>      modprobe siw
>>>      modprobe rdma_ucm
>>>
>>> Then ibv_devinfo shows: "No IB devices found".
>>> dmesg only shows:
>>>      [   29.856751] SoftiWARP attached
>>>
>>> According to this tutorial, [2] it should be enough to just load the siw
>>> module. (unlike RXE, where one needs to use rxe_cfg to set up the
>>> interface)
>>> Is this a bug in siw or just a configuration issue on my side?
>> Have you done "rdma link"?
>>
>> rdma link add <NAME> type siw netdev <NETDEV>
>>
>> http://man7.org/linux/man-pages/man8/rdma-link.8.html
> BTW, the same goes for RXE and rxe_cfg is discouraged.
>
> Thanks
>
>>> Thanks,
>>> Marcin
>>>
>>> [1] https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc5/
>>> [2] https://budevg.github.io/posts/tutorials/2017/04/29/rdma-101-1.html
>>>
>>

Re: Setting up siw devices

[Leon Romanovsky]

On Thu, Aug 22, 2019 at 07:05:12PM +0200, Marcin Mielniczuk wrote:
> Thanks a lot, this did the trick. I think this is worth documenting
> somewhere that this step is needed.
> I'll make a PR, would README.md in the rdma-core repo be a good place?

I'm not so sure, but it is better to have in some place instead of not having at all.

>
> Does <NAME> have any significance? I did:
>
>      sudo rdma link add siw0 type siw netdev enpXsYYfZ
>
> but the resulting device is called iwpXsYYfZ. I couldn't find a trace of
> `siw0` anywhere.

I would say that it is a bug in kernel part of SIW, because kernel rename
(the thing which change your siw0 to be iw* name) is looking for absence
of mentioning PCI inside of /sys/class/infiniband/siw0/*
https://github.com/linux-rdma/rdma-core/blob/master/kernel-boot/rdma_rename.c#L378

That rdma-core line works for RXE and SIW should be similar.

Thanks

>
> On 22.08.2019 17:52, Leon Romanovsky wrote:
> > On Thu, Aug 22, 2019 at 09:13:25PM +0530, Krishnamraju Eraparaju wrote:
> >> On Thursday, August 08/22/19, 2019 at 17:08:49 +0200, Marcin Mielniczuk wrote:
> >>> Hi,
> >>>
> >>> I'm trying to test the recently merged siw module.
> >>> I'm running kernel 5.3-rc5 (taken from the Ubuntu mainline-kernel
> >>> repository [1]) on Ubuntu 18.04 (bionic).
> >>> I also manually installed rdma-core 25.0 from tarball, using the
> >>> included Debian packaging. I installed all the packages but ibacm.
> >>>
> >>> After booting the new kernel I manually loaded the kernel module by
> >>>
> >>>      modprobe siw
> >>>      modprobe rdma_ucm
> >>>
> >>> Then ibv_devinfo shows: "No IB devices found".
> >>> dmesg only shows:
> >>>      [   29.856751] SoftiWARP attached
> >>>
> >>> According to this tutorial, [2] it should be enough to just load the siw
> >>> module. (unlike RXE, where one needs to use rxe_cfg to set up the
> >>> interface)
> >>> Is this a bug in siw or just a configuration issue on my side?
> >> Have you done "rdma link"?
> >>
> >> rdma link add <NAME> type siw netdev <NETDEV>
> >>
> >> http://man7.org/linux/man-pages/man8/rdma-link.8.html
> > BTW, the same goes for RXE and rxe_cfg is discouraged.
> >
> > Thanks
> >
> >>> Thanks,
> >>> Marcin
> >>>
> >>> [1] https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc5/
> >>> [2] https://budevg.github.io/posts/tutorials/2017/04/29/rdma-101-1.html
> >>>
> >>
>
>

Re: Setting up siw devices

[Marcin Mielniczuk]

On 22.08.2019 19:21, Leon Romanovsky wrote:
> On Thu, Aug 22, 2019 at 07:05:12PM +0200, Marcin Mielniczuk wrote:
>> Thanks a lot, this did the trick. I think this is worth documenting
>> somewhere that this step is needed.
>> I'll make a PR, would README.md in the rdma-core repo be a good place?
> I'm not so sure, but it is better to have in some place instead of not having at all.
I think it's the first place one would look for some information. I'll
make a PR today or tomorrow.
>> Does <NAME> have any significance? I did:
>>
>>      sudo rdma link add siw0 type siw netdev enpXsYYfZ
>>
>> but the resulting device is called iwpXsYYfZ. I couldn't find a trace of
>> `siw0` anywhere.
> I would say that it is a bug in kernel part of SIW, because kernel rename
> (the thing which change your siw0 to be iw* name) is looking for absence
> of mentioning PCI inside of /sys/class/infiniband/siw0/*
> https://github.com/linux-rdma/rdma-core/blob/master/kernel-boot/rdma_rename.c#L378
I don't have /sys/class/infiniband/siw0 on my system, only
/sys/class/infiniband/iwpXsYYfZ.
iwp probably comes from iWARP.

Regards,
Marcin
> That rdma-core line works for RXE and SIW should be similar.
>
> Thanks
>
>> On 22.08.2019 17:52, Leon Romanovsky wrote:
>>> On Thu, Aug 22, 2019 at 09:13:25PM +0530, Krishnamraju Eraparaju wrote:
>>>> On Thursday, August 08/22/19, 2019 at 17:08:49 +0200, Marcin Mielniczuk wrote:
>>>>> Hi,
>>>>>
>>>>> I'm trying to test the recently merged siw module.
>>>>> I'm running kernel 5.3-rc5 (taken from the Ubuntu mainline-kernel
>>>>> repository [1]) on Ubuntu 18.04 (bionic).
>>>>> I also manually installed rdma-core 25.0 from tarball, using the
>>>>> included Debian packaging. I installed all the packages but ibacm.
>>>>>
>>>>> After booting the new kernel I manually loaded the kernel module by
>>>>>
>>>>>      modprobe siw
>>>>>      modprobe rdma_ucm
>>>>>
>>>>> Then ibv_devinfo shows: "No IB devices found".
>>>>> dmesg only shows:
>>>>>      [   29.856751] SoftiWARP attached
>>>>>
>>>>> According to this tutorial, [2] it should be enough to just load the siw
>>>>> module. (unlike RXE, where one needs to use rxe_cfg to set up the
>>>>> interface)
>>>>> Is this a bug in siw or just a configuration issue on my side?
>>>> Have you done "rdma link"?
>>>>
>>>> rdma link add <NAME> type siw netdev <NETDEV>
>>>>
>>>> http://man7.org/linux/man-pages/man8/rdma-link.8.html
>>> BTW, the same goes for RXE and rxe_cfg is discouraged.
>>>
>>> Thanks
>>>
>>>>> Thanks,
>>>>> Marcin
>>>>>
>>>>> [1] https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc5/
>>>>> [2] https://budevg.github.io/posts/tutorials/2017/04/29/rdma-101-1.html
>>>>>
>>

Re: Setting up siw devices

[Leon Romanovsky]

On Thu, Aug 22, 2019 at 07:58:56PM +0200, Marcin Mielniczuk wrote:
> On 22.08.2019 19:21, Leon Romanovsky wrote:
> > On Thu, Aug 22, 2019 at 07:05:12PM +0200, Marcin Mielniczuk wrote:
> >> Thanks a lot, this did the trick. I think this is worth documenting
> >> somewhere that this step is needed.
> >> I'll make a PR, would README.md in the rdma-core repo be a good place?
> > I'm not so sure, but it is better to have in some place instead of not having at all.
> I think it's the first place one would look for some information. I'll
> make a PR today or tomorrow.
> >> Does <NAME> have any significance? I did:
> >>
> >>      sudo rdma link add siw0 type siw netdev enpXsYYfZ
> >>
> >> but the resulting device is called iwpXsYYfZ. I couldn't find a trace of
> >> `siw0` anywhere.
> > I would say that it is a bug in kernel part of SIW, because kernel rename
> > (the thing which change your siw0 to be iw* name) is looking for absence
> > of mentioning PCI inside of /sys/class/infiniband/siw0/*
> > https://github.com/linux-rdma/rdma-core/blob/master/kernel-boot/rdma_rename.c#L378
> I don't have /sys/class/infiniband/siw0 on my system, only
> /sys/class/infiniband/iwpXsYYfZ.
> iwp probably comes from iWARP.

Your iwpXsYYfZ was siw0 before rdma_rename was executed.

I can't test the patch now, but hope that this change below will fix your problem.

diff --git a/drivers/infiniband/sw/siw/siw_main.c b/drivers/infiniband/sw/siw/siw_main.c
index 05a92f997f60..38c25a26dfb2 100644
--- a/drivers/infiniband/sw/siw/siw_main.c
+++ b/drivers/infiniband/sw/siw/siw_main.c
@@ -326,21 +326,6 @@ static struct siw_device *siw_device_create(struct net_device *netdev)
 	struct device *parent = netdev->dev.parent;
 	int rv;

-	if (!parent) {
-		/*
-		 * The loopback device has no parent device,
-		 * so it appears as a top-level device. To support
-		 * loopback device connectivity, take this device
-		 * as the parent device. Skip all other devices
-		 * w/o parent device.
-		 */
-		if (netdev->type != ARPHRD_LOOPBACK) {
-			pr_warn("siw: device %s error: no parent device\n",
-				netdev->name);
-			return NULL;
-		}
-		parent = &netdev->dev;
-	}
 	sdev = ib_alloc_device(siw_device, base_dev);
 	if (!sdev)
 		return NULL;

Re: Setting up siw devices

[Jason Gunthorpe]

On Thu, Aug 22, 2019 at 09:38:07PM +0300, Leon Romanovsky wrote:
> On Thu, Aug 22, 2019 at 07:58:56PM +0200, Marcin Mielniczuk wrote:
> > On 22.08.2019 19:21, Leon Romanovsky wrote:
> > > On Thu, Aug 22, 2019 at 07:05:12PM +0200, Marcin Mielniczuk wrote:
> > >> Thanks a lot, this did the trick. I think this is worth documenting
> > >> somewhere that this step is needed.
> > >> I'll make a PR, would README.md in the rdma-core repo be a good place?
> > > I'm not so sure, but it is better to have in some place instead of not having at all.
> > I think it's the first place one would look for some information. I'll
> > make a PR today or tomorrow.
> > >> Does <NAME> have any significance? I did:
> > >>
> > >>      sudo rdma link add siw0 type siw netdev enpXsYYfZ
> > >>
> > >> but the resulting device is called iwpXsYYfZ. I couldn't find a trace of
> > >> `siw0` anywhere.
> > > I would say that it is a bug in kernel part of SIW, because kernel rename
> > > (the thing which change your siw0 to be iw* name) is looking for absence
> > > of mentioning PCI inside of /sys/class/infiniband/siw0/*
> > > https://github.com/linux-rdma/rdma-core/blob/master/kernel-boot/rdma_rename.c#L378
> > I don't have /sys/class/infiniband/siw0 on my system, only
> > /sys/class/infiniband/iwpXsYYfZ.
> > iwp probably comes from iWARP.
> 
> Your iwpXsYYfZ was siw0 before rdma_rename was executed.
> 
> I can't test the patch now, but hope that this change below will fix your problem.

I think we should directly blacklist rxe and siw from
renaming. They can only be created with a user-given name so they
should never ever be renamed.

netlink now returns the driver_id and we can use that to trigger it.

Jason

Re: Setting up siw devices

[Leon Romanovsky]

On Thu, Aug 22, 2019 at 04:13:54PM -0300, Jason Gunthorpe wrote:
> On Thu, Aug 22, 2019 at 09:38:07PM +0300, Leon Romanovsky wrote:
> > On Thu, Aug 22, 2019 at 07:58:56PM +0200, Marcin Mielniczuk wrote:
> > > On 22.08.2019 19:21, Leon Romanovsky wrote:
> > > > On Thu, Aug 22, 2019 at 07:05:12PM +0200, Marcin Mielniczuk wrote:
> > > >> Thanks a lot, this did the trick. I think this is worth documenting
> > > >> somewhere that this step is needed.
> > > >> I'll make a PR, would README.md in the rdma-core repo be a good place?
> > > > I'm not so sure, but it is better to have in some place instead of not having at all.
> > > I think it's the first place one would look for some information. I'll
> > > make a PR today or tomorrow.
> > > >> Does <NAME> have any significance? I did:
> > > >>
> > > >>      sudo rdma link add siw0 type siw netdev enpXsYYfZ
> > > >>
> > > >> but the resulting device is called iwpXsYYfZ. I couldn't find a trace of
> > > >> `siw0` anywhere.
> > > > I would say that it is a bug in kernel part of SIW, because kernel rename
> > > > (the thing which change your siw0 to be iw* name) is looking for absence
> > > > of mentioning PCI inside of /sys/class/infiniband/siw0/*
> > > > https://github.com/linux-rdma/rdma-core/blob/master/kernel-boot/rdma_rename.c#L378
> > > I don't have /sys/class/infiniband/siw0 on my system, only
> > > /sys/class/infiniband/iwpXsYYfZ.
> > > iwp probably comes from iWARP.
> >
> > Your iwpXsYYfZ was siw0 before rdma_rename was executed.
> >
> > I can't test the patch now, but hope that this change below will fix your problem.
>
> I think we should directly blacklist rxe and siw from
> renaming. They can only be created with a user-given name so they
> should never ever be renamed.
>
> netlink now returns the driver_id and we can use that to trigger it.

Blacklisting does not play well with new kernel with new devices against old library.

Thanks

>
> Jason

Re: Setting up siw devices

[Marcin Mielniczuk]

With the kernel built with yiour patch the device is still being
renamed. (tbh, I don't even see why this patch would help)
But it's not a big issue for me right now, I can just type in the
renamed interface name.
Thanks

On 22.08.2019 20:38, Leon Romanovsky wrote:
> On Thu, Aug 22, 2019 at 07:58:56PM +0200, Marcin Mielniczuk wrote:
>> On 22.08.2019 19:21, Leon Romanovsky wrote:
>>> On Thu, Aug 22, 2019 at 07:05:12PM +0200, Marcin Mielniczuk wrote:
>>>> Thanks a lot, this did the trick. I think this is worth documenting
>>>> somewhere that this step is needed.
>>>> I'll make a PR, would README.md in the rdma-core repo be a good place?
>>> I'm not so sure, but it is better to have in some place instead of not having at all.
>> I think it's the first place one would look for some information. I'll
>> make a PR today or tomorrow.
>>>> Does <NAME> have any significance? I did:
>>>>
>>>>      sudo rdma link add siw0 type siw netdev enpXsYYfZ
>>>>
>>>> but the resulting device is called iwpXsYYfZ. I couldn't find a trace of
>>>> `siw0` anywhere.
>>> I would say that it is a bug in kernel part of SIW, because kernel rename
>>> (the thing which change your siw0 to be iw* name) is looking for absence
>>> of mentioning PCI inside of /sys/class/infiniband/siw0/*
>>> https://github.com/linux-rdma/rdma-core/blob/master/kernel-boot/rdma_rename.c#L378
>> I don't have /sys/class/infiniband/siw0 on my system, only
>> /sys/class/infiniband/iwpXsYYfZ.
>> iwp probably comes from iWARP.
> Your iwpXsYYfZ was siw0 before rdma_rename was executed.
>
> I can't test the patch now, but hope that this change below will fix your problem.
>
> diff --git a/drivers/infiniband/sw/siw/siw_main.c b/drivers/infiniband/sw/siw/siw_main.c
> index 05a92f997f60..38c25a26dfb2 100644
> --- a/drivers/infiniband/sw/siw/siw_main.c
> +++ b/drivers/infiniband/sw/siw/siw_main.c
> @@ -326,21 +326,6 @@ static struct siw_device *siw_device_create(struct net_device *netdev)
>  	struct device *parent = netdev->dev.parent;
>  	int rv;
>
> -	if (!parent) {
> -		/*
> -		 * The loopback device has no parent device,
> -		 * so it appears as a top-level device. To support
> -		 * loopback device connectivity, take this device
> -		 * as the parent device. Skip all other devices
> -		 * w/o parent device.
> -		 */
> -		if (netdev->type != ARPHRD_LOOPBACK) {
> -			pr_warn("siw: device %s error: no parent device\n",
> -				netdev->name);
> -			return NULL;
> -		}
> -		parent = &netdev->dev;
> -	}
>  	sdev = ib_alloc_device(siw_device, base_dev);
>  	if (!sdev)
>  		return NULL;
>

Re: Setting up siw devices

[Leon Romanovsky]

On Mon, Aug 26, 2019 at 12:02:47PM +0200, Marcin Mielniczuk wrote:
> With the kernel built with yiour patch the device is still being
> renamed. (tbh, I don't even see why this patch would help)

Because I'm afraid that SIW sets wrong "parent" device.

Thanks

Re: Setting up siw devices

[Marcin Mielniczuk]

Btw. I noticed that on Linux 4.15 rxe_cfg created a device called rxe0
and it was not renamed. On the other hand, on 5.3-rc6, rxe_cfg the
device gets renamed from rxe0 to rocepXsXXfX

On 26.08.2019 13:39, Leon Romanovsky wrote:
> On Mon, Aug 26, 2019 at 12:02:47PM +0200, Marcin Mielniczuk wrote:
>> With the kernel built with yiour patch the device is still being
>> renamed. (tbh, I don't even see why this patch would help)
> Because I'm afraid that SIW sets wrong "parent" device.
>
> Thanks

Re: Setting up siw devices

[Marcin Mielniczuk]

While the device is detected by ibv_devices and rping works, I can't get
ibv_rc_pingpong working
(and as far as I understand, RC should be supported by the iWARP driver)

rping works:

    server$  rping -s -a 10.30.10.211 -v
    server ping data: rdma-ping-0:
ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
    server ping data: rdma-ping-1:
BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
    server DISCONNECT EVENT...
    wait for RDMA_READ_ADV state 10

    client$ rping -c -a 10.30.10.211 -C 2 -v
    (output omitted)


But ibv_rc_pingpong doesn't

    server$ ibv_rc_pingpong -d iwp____
      local address:  LID 0x0000, QPN 0x000001, PSN 0xb8aafc, GID ::
    Failed to modify QP to RTS
    Couldn't connect to remote QP
    client$ ibv_rc_pingpong  -d iwp____ 10.30.10.211
      local address:  LID 0x0000, QPN 0x000001, PSN 0x71abc5, GID ::
    client read/write: Protocol not supported
    Couldn't read/write remote address

dmesg says: [   93.957733] iwpm_register_pid: Unable to send a nlmsg
(client = 2)

And OpenMPI/UCX doesn't want to use the transport either:

    [1566564168.338103] [host:23059:0]         select.c:410  UCX  ERROR
no active messages transport to <no debug data>: mm/posix - Destination
is unreachable, mm/sysv - Destination is unreachable, self/self -
Destination is unreachable, rdmacm/sockaddr - no am bcopy, cma/cma - no
am bcopy

Is this a bug or a limitation of siw?
Thanks,
Marcin

On 22.08.2019 18:24, Marcin Mielniczuk wrote:
> Thanks a lot, this did the trick. I think this is worth documenting
> somewhere.
> Would README.md in the rdma-core repo be a good place?
>
> Why
>
> On 22.08.2019 17:52, Leon Romanovsky wrote:
>> On Thu, Aug 22, 2019 at 09:13:25PM +0530, Krishnamraju Eraparaju wrote:
>>> On Thursday, August 08/22/19, 2019 at 17:08:49 +0200, Marcin Mielniczuk wrote:
>>>> Hi,
>>>>
>>>> I'm trying to test the recently merged siw module.
>>>> I'm running kernel 5.3-rc5 (taken from the Ubuntu mainline-kernel
>>>> repository [1]) on Ubuntu 18.04 (bionic).
>>>> I also manually installed rdma-core 25.0 from tarball, using the
>>>> included Debian packaging. I installed all the packages but ibacm.
>>>>
>>>> After booting the new kernel I manually loaded the kernel module by
>>>>
>>>>      modprobe siw
>>>>      modprobe rdma_ucm
>>>>
>>>> Then ibv_devinfo shows: "No IB devices found".
>>>> dmesg only shows:
>>>>      [   29.856751] SoftiWARP attached
>>>>
>>>> According to this tutorial, [2] it should be enough to just load the siw
>>>> module. (unlike RXE, where one needs to use rxe_cfg to set up the
>>>> interface)
>>>> Is this a bug in siw or just a configuration issue on my side?
>>> Have you done "rdma link"?
>>>
>>> rdma link add <NAME> type siw netdev <NETDEV>
>>>
>>> http://man7.org/linux/man-pages/man8/rdma-link.8.html
>> BTW, the same goes for RXE and rxe_cfg is discouraged.
>>
>> Thanks
>>
>>>> Thanks,
>>>> Marcin
>>>>
>>>> [1] https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc5/
>>>> [2] https://budevg.github.io/posts/tutorials/2017/04/29/rdma-101-1.html
>>>>
>

Re: Setting up siw devices

[Jason Gunthorpe]

On Fri, Aug 23, 2019 at 04:17:50PM +0200, Marcin Mielniczuk wrote:
> While the device is detected by ibv_devices and rping works, I can't get
> ibv_rc_pingpong working
> (and as far as I understand, RC should be supported by the iWARP driver)
> 
> rping works:
> 
>     server$  rping -s -a 10.30.10.211 -v
>     server ping data: rdma-ping-0:
> ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
>     server ping data: rdma-ping-1:
> BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
>     server DISCONNECT EVENT...
>     wait for RDMA_READ_ADV state 10
> 
>     client$ rping -c -a 10.30.10.211 -C 2 -v
>     (output omitted)
> 
> 
> But ibv_rc_pingpong doesn't
> 
>     server$ ibv_rc_pingpong -d iwp____
>       local address:  LID 0x0000, QPN 0x000001, PSN 0xb8aafc, GID ::
>     Failed to modify QP to RTS
>     Couldn't connect to remote QP
>     client$ ibv_rc_pingpong  -d iwp____ 10.30.10.211
>       local address:  LID 0x0000, QPN 0x000001, PSN 0x71abc5, GID ::
>     client read/write: Protocol not supported
>     Couldn't read/write remote address

iwarp does not work with the pingpong examples as iwarp requires RDMA
CM and the examples don't use it.

Jason