Re: Setting up siw devices

From: Leon Romanovsky <leon@kernel.org>
To: Marcin Mielniczuk <marcin@golem.network>
Cc: Krishnamraju Eraparaju <krishna2@chelsio.com>,
	linux-rdma@vger.kernel.org
Subject: Re: Setting up siw devices
Date: Thu, 22 Aug 2019 20:21:55 +0300	[thread overview]
Message-ID: <20190822172155.GL29433@mtr-leonro.mtl.com> (raw)
In-Reply-To: <bf42725d-d441-0237-9df5-bd39cb981dcc@golem.network>

On Thu, Aug 22, 2019 at 07:05:12PM +0200, Marcin Mielniczuk wrote:
> Thanks a lot, this did the trick. I think this is worth documenting
> somewhere that this step is needed.
> I'll make a PR, would README.md in the rdma-core repo be a good place?

I'm not so sure, but it is better to have in some place instead of not having at all.

>
> Does <NAME> have any significance? I did:
>
>      sudo rdma link add siw0 type siw netdev enpXsYYfZ
>
> but the resulting device is called iwpXsYYfZ. I couldn't find a trace of
> `siw0` anywhere.

I would say that it is a bug in kernel part of SIW, because kernel rename
(the thing which change your siw0 to be iw* name) is looking for absence
of mentioning PCI inside of /sys/class/infiniband/siw0/*
https://github.com/linux-rdma/rdma-core/blob/master/kernel-boot/rdma_rename.c#L378

That rdma-core line works for RXE and SIW should be similar.

Thanks

>
> On 22.08.2019 17:52, Leon Romanovsky wrote:
> > On Thu, Aug 22, 2019 at 09:13:25PM +0530, Krishnamraju Eraparaju wrote:
> >> On Thursday, August 08/22/19, 2019 at 17:08:49 +0200, Marcin Mielniczuk wrote:
> >>> Hi,
> >>>
> >>> I'm trying to test the recently merged siw module.
> >>> I'm running kernel 5.3-rc5 (taken from the Ubuntu mainline-kernel
> >>> repository [1]) on Ubuntu 18.04 (bionic).
> >>> I also manually installed rdma-core 25.0 from tarball, using the
> >>> included Debian packaging. I installed all the packages but ibacm.
> >>>
> >>> After booting the new kernel I manually loaded the kernel module by
> >>>
> >>>      modprobe siw
> >>>      modprobe rdma_ucm
> >>>
> >>> Then ibv_devinfo shows: "No IB devices found".
> >>> dmesg only shows:
> >>>      [   29.856751] SoftiWARP attached
> >>>
> >>> According to this tutorial, [2] it should be enough to just load the siw
> >>> module. (unlike RXE, where one needs to use rxe_cfg to set up the
> >>> interface)
> >>> Is this a bug in siw or just a configuration issue on my side?
> >> Have you done "rdma link"?
> >>
> >> rdma link add <NAME> type siw netdev <NETDEV>
> >>
> >> http://man7.org/linux/man-pages/man8/rdma-link.8.html
> > BTW, the same goes for RXE and rxe_cfg is discouraged.
> >
> > Thanks
> >
> >>> Thanks,
> >>> Marcin
> >>>
> >>> [1] https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc5/
> >>> [2] https://budevg.github.io/posts/tutorials/2017/04/29/rdma-101-1.html
> >>>
> >>
>
>