* Setting up siw devices
@ 2019-08-22 15:08 Marcin Mielniczuk
2019-08-22 15:43 ` Krishnamraju Eraparaju
0 siblings, 1 reply; 14+ messages in thread
From: Marcin Mielniczuk @ 2019-08-22 15:08 UTC (permalink / raw)
To: linux-rdma
[-- Attachment #1.1: Type: text/plain, Size: 919 bytes --]
Hi,
I'm trying to test the recently merged siw module.
I'm running kernel 5.3-rc5 (taken from the Ubuntu mainline-kernel
repository [1]) on Ubuntu 18.04 (bionic).
I also manually installed rdma-core 25.0 from tarball, using the
included Debian packaging. I installed all the packages but ibacm.
After booting the new kernel I manually loaded the kernel module by
modprobe siw
modprobe rdma_ucm
Then ibv_devinfo shows: "No IB devices found".
dmesg only shows:
[ 29.856751] SoftiWARP attached
According to this tutorial, [2] it should be enough to just load the siw
module. (unlike RXE, where one needs to use rxe_cfg to set up the
interface)
Is this a bug in siw or just a configuration issue on my side?
Thanks,
Marcin
[1] https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc5/
[2] https://budevg.github.io/posts/tutorials/2017/04/29/rdma-101-1.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Setting up siw devices
2019-08-22 15:08 Setting up siw devices Marcin Mielniczuk
@ 2019-08-22 15:43 ` Krishnamraju Eraparaju
2019-08-22 15:52 ` Leon Romanovsky
0 siblings, 1 reply; 14+ messages in thread
From: Krishnamraju Eraparaju @ 2019-08-22 15:43 UTC (permalink / raw)
To: Marcin Mielniczuk; +Cc: linux-rdma
On Thursday, August 08/22/19, 2019 at 17:08:49 +0200, Marcin Mielniczuk wrote:
> Hi,
>
> I'm trying to test the recently merged siw module.
> I'm running kernel 5.3-rc5 (taken from the Ubuntu mainline-kernel
> repository [1]) on Ubuntu 18.04 (bionic).
> I also manually installed rdma-core 25.0 from tarball, using the
> included Debian packaging. I installed all the packages but ibacm.
>
> After booting the new kernel I manually loaded the kernel module by
>
> modprobe siw
> modprobe rdma_ucm
>
> Then ibv_devinfo shows: "No IB devices found".
> dmesg only shows:
> [ 29.856751] SoftiWARP attached
>
> According to this tutorial, [2] it should be enough to just load the siw
> module. (unlike RXE, where one needs to use rxe_cfg to set up the
> interface)
> Is this a bug in siw or just a configuration issue on my side?
Have you done "rdma link"?
rdma link add <NAME> type siw netdev <NETDEV>
http://man7.org/linux/man-pages/man8/rdma-link.8.html
>
> Thanks,
> Marcin
>
> [1] https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc5/
> [2] https://budevg.github.io/posts/tutorials/2017/04/29/rdma-101-1.html
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Setting up siw devices
2019-08-22 15:43 ` Krishnamraju Eraparaju
@ 2019-08-22 15:52 ` Leon Romanovsky
2019-08-22 17:05 ` Marcin Mielniczuk
[not found] ` <b4bf4bc2-8dc7-a2c2-6bd2-ab41d9fbadc9@golem.network>
0 siblings, 2 replies; 14+ messages in thread
From: Leon Romanovsky @ 2019-08-22 15:52 UTC (permalink / raw)
To: Krishnamraju Eraparaju; +Cc: Marcin Mielniczuk, linux-rdma
On Thu, Aug 22, 2019 at 09:13:25PM +0530, Krishnamraju Eraparaju wrote:
> On Thursday, August 08/22/19, 2019 at 17:08:49 +0200, Marcin Mielniczuk wrote:
> > Hi,
> >
> > I'm trying to test the recently merged siw module.
> > I'm running kernel 5.3-rc5 (taken from the Ubuntu mainline-kernel
> > repository [1]) on Ubuntu 18.04 (bionic).
> > I also manually installed rdma-core 25.0 from tarball, using the
> > included Debian packaging. I installed all the packages but ibacm.
> >
> > After booting the new kernel I manually loaded the kernel module by
> >
> > modprobe siw
> > modprobe rdma_ucm
> >
> > Then ibv_devinfo shows: "No IB devices found".
> > dmesg only shows:
> > [ 29.856751] SoftiWARP attached
> >
> > According to this tutorial, [2] it should be enough to just load the siw
> > module. (unlike RXE, where one needs to use rxe_cfg to set up the
> > interface)
> > Is this a bug in siw or just a configuration issue on my side?
>
> Have you done "rdma link"?
>
> rdma link add <NAME> type siw netdev <NETDEV>
>
> http://man7.org/linux/man-pages/man8/rdma-link.8.html
BTW, the same goes for RXE and rxe_cfg is discouraged.
Thanks
>
> >
> > Thanks,
> > Marcin
> >
> > [1] https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc5/
> > [2] https://budevg.github.io/posts/tutorials/2017/04/29/rdma-101-1.html
> >
>
>
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Setting up siw devices
2019-08-22 15:52 ` Leon Romanovsky
@ 2019-08-22 17:05 ` Marcin Mielniczuk
2019-08-22 17:21 ` Leon Romanovsky
[not found] ` <b4bf4bc2-8dc7-a2c2-6bd2-ab41d9fbadc9@golem.network>
1 sibling, 1 reply; 14+ messages in thread
From: Marcin Mielniczuk @ 2019-08-22 17:05 UTC (permalink / raw)
To: Leon Romanovsky, Krishnamraju Eraparaju; +Cc: linux-rdma
Thanks a lot, this did the trick. I think this is worth documenting
somewhere that this step is needed.
I'll make a PR, would README.md in the rdma-core repo be a good place?
Does <NAME> have any significance? I did:
sudo rdma link add siw0 type siw netdev enpXsYYfZ
but the resulting device is called iwpXsYYfZ. I couldn't find a trace of
`siw0` anywhere.
On 22.08.2019 17:52, Leon Romanovsky wrote:
> On Thu, Aug 22, 2019 at 09:13:25PM +0530, Krishnamraju Eraparaju wrote:
>> On Thursday, August 08/22/19, 2019 at 17:08:49 +0200, Marcin Mielniczuk wrote:
>>> Hi,
>>>
>>> I'm trying to test the recently merged siw module.
>>> I'm running kernel 5.3-rc5 (taken from the Ubuntu mainline-kernel
>>> repository [1]) on Ubuntu 18.04 (bionic).
>>> I also manually installed rdma-core 25.0 from tarball, using the
>>> included Debian packaging. I installed all the packages but ibacm.
>>>
>>> After booting the new kernel I manually loaded the kernel module by
>>>
>>> modprobe siw
>>> modprobe rdma_ucm
>>>
>>> Then ibv_devinfo shows: "No IB devices found".
>>> dmesg only shows:
>>> [ 29.856751] SoftiWARP attached
>>>
>>> According to this tutorial, [2] it should be enough to just load the siw
>>> module. (unlike RXE, where one needs to use rxe_cfg to set up the
>>> interface)
>>> Is this a bug in siw or just a configuration issue on my side?
>> Have you done "rdma link"?
>>
>> rdma link add <NAME> type siw netdev <NETDEV>
>>
>> http://man7.org/linux/man-pages/man8/rdma-link.8.html
> BTW, the same goes for RXE and rxe_cfg is discouraged.
>
> Thanks
>
>>> Thanks,
>>> Marcin
>>>
>>> [1] https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc5/
>>> [2] https://budevg.github.io/posts/tutorials/2017/04/29/rdma-101-1.html
>>>
>>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Setting up siw devices
2019-08-22 17:05 ` Marcin Mielniczuk
@ 2019-08-22 17:21 ` Leon Romanovsky
2019-08-22 17:58 ` Marcin Mielniczuk
0 siblings, 1 reply; 14+ messages in thread
From: Leon Romanovsky @ 2019-08-22 17:21 UTC (permalink / raw)
To: Marcin Mielniczuk; +Cc: Krishnamraju Eraparaju, linux-rdma
On Thu, Aug 22, 2019 at 07:05:12PM +0200, Marcin Mielniczuk wrote:
> Thanks a lot, this did the trick. I think this is worth documenting
> somewhere that this step is needed.
> I'll make a PR, would README.md in the rdma-core repo be a good place?
I'm not so sure, but it is better to have in some place instead of not having at all.
>
> Does <NAME> have any significance? I did:
>
> sudo rdma link add siw0 type siw netdev enpXsYYfZ
>
> but the resulting device is called iwpXsYYfZ. I couldn't find a trace of
> `siw0` anywhere.
I would say that it is a bug in kernel part of SIW, because kernel rename
(the thing which change your siw0 to be iw* name) is looking for absence
of mentioning PCI inside of /sys/class/infiniband/siw0/*
https://github.com/linux-rdma/rdma-core/blob/master/kernel-boot/rdma_rename.c#L378
That rdma-core line works for RXE and SIW should be similar.
Thanks
>
> On 22.08.2019 17:52, Leon Romanovsky wrote:
> > On Thu, Aug 22, 2019 at 09:13:25PM +0530, Krishnamraju Eraparaju wrote:
> >> On Thursday, August 08/22/19, 2019 at 17:08:49 +0200, Marcin Mielniczuk wrote:
> >>> Hi,
> >>>
> >>> I'm trying to test the recently merged siw module.
> >>> I'm running kernel 5.3-rc5 (taken from the Ubuntu mainline-kernel
> >>> repository [1]) on Ubuntu 18.04 (bionic).
> >>> I also manually installed rdma-core 25.0 from tarball, using the
> >>> included Debian packaging. I installed all the packages but ibacm.
> >>>
> >>> After booting the new kernel I manually loaded the kernel module by
> >>>
> >>> modprobe siw
> >>> modprobe rdma_ucm
> >>>
> >>> Then ibv_devinfo shows: "No IB devices found".
> >>> dmesg only shows:
> >>> [ 29.856751] SoftiWARP attached
> >>>
> >>> According to this tutorial, [2] it should be enough to just load the siw
> >>> module. (unlike RXE, where one needs to use rxe_cfg to set up the
> >>> interface)
> >>> Is this a bug in siw or just a configuration issue on my side?
> >> Have you done "rdma link"?
> >>
> >> rdma link add <NAME> type siw netdev <NETDEV>
> >>
> >> http://man7.org/linux/man-pages/man8/rdma-link.8.html
> > BTW, the same goes for RXE and rxe_cfg is discouraged.
> >
> > Thanks
> >
> >>> Thanks,
> >>> Marcin
> >>>
> >>> [1] https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc5/
> >>> [2] https://budevg.github.io/posts/tutorials/2017/04/29/rdma-101-1.html
> >>>
> >>
>
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Setting up siw devices
2019-08-22 17:21 ` Leon Romanovsky
@ 2019-08-22 17:58 ` Marcin Mielniczuk
2019-08-22 18:38 ` Leon Romanovsky
0 siblings, 1 reply; 14+ messages in thread
From: Marcin Mielniczuk @ 2019-08-22 17:58 UTC (permalink / raw)
To: Leon Romanovsky; +Cc: Krishnamraju Eraparaju, linux-rdma
On 22.08.2019 19:21, Leon Romanovsky wrote:
> On Thu, Aug 22, 2019 at 07:05:12PM +0200, Marcin Mielniczuk wrote:
>> Thanks a lot, this did the trick. I think this is worth documenting
>> somewhere that this step is needed.
>> I'll make a PR, would README.md in the rdma-core repo be a good place?
> I'm not so sure, but it is better to have in some place instead of not having at all.
I think it's the first place one would look for some information. I'll
make a PR today or tomorrow.
>> Does <NAME> have any significance? I did:
>>
>> sudo rdma link add siw0 type siw netdev enpXsYYfZ
>>
>> but the resulting device is called iwpXsYYfZ. I couldn't find a trace of
>> `siw0` anywhere.
> I would say that it is a bug in kernel part of SIW, because kernel rename
> (the thing which change your siw0 to be iw* name) is looking for absence
> of mentioning PCI inside of /sys/class/infiniband/siw0/*
> https://github.com/linux-rdma/rdma-core/blob/master/kernel-boot/rdma_rename.c#L378
I don't have /sys/class/infiniband/siw0 on my system, only
/sys/class/infiniband/iwpXsYYfZ.
iwp probably comes from iWARP.
Regards,
Marcin
> That rdma-core line works for RXE and SIW should be similar.
>
> Thanks
>
>> On 22.08.2019 17:52, Leon Romanovsky wrote:
>>> On Thu, Aug 22, 2019 at 09:13:25PM +0530, Krishnamraju Eraparaju wrote:
>>>> On Thursday, August 08/22/19, 2019 at 17:08:49 +0200, Marcin Mielniczuk wrote:
>>>>> Hi,
>>>>>
>>>>> I'm trying to test the recently merged siw module.
>>>>> I'm running kernel 5.3-rc5 (taken from the Ubuntu mainline-kernel
>>>>> repository [1]) on Ubuntu 18.04 (bionic).
>>>>> I also manually installed rdma-core 25.0 from tarball, using the
>>>>> included Debian packaging. I installed all the packages but ibacm.
>>>>>
>>>>> After booting the new kernel I manually loaded the kernel module by
>>>>>
>>>>> modprobe siw
>>>>> modprobe rdma_ucm
>>>>>
>>>>> Then ibv_devinfo shows: "No IB devices found".
>>>>> dmesg only shows:
>>>>> [ 29.856751] SoftiWARP attached
>>>>>
>>>>> According to this tutorial, [2] it should be enough to just load the siw
>>>>> module. (unlike RXE, where one needs to use rxe_cfg to set up the
>>>>> interface)
>>>>> Is this a bug in siw or just a configuration issue on my side?
>>>> Have you done "rdma link"?
>>>>
>>>> rdma link add <NAME> type siw netdev <NETDEV>
>>>>
>>>> http://man7.org/linux/man-pages/man8/rdma-link.8.html
>>> BTW, the same goes for RXE and rxe_cfg is discouraged.
>>>
>>> Thanks
>>>
>>>>> Thanks,
>>>>> Marcin
>>>>>
>>>>> [1] https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc5/
>>>>> [2] https://budevg.github.io/posts/tutorials/2017/04/29/rdma-101-1.html
>>>>>
>>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Setting up siw devices
2019-08-22 17:58 ` Marcin Mielniczuk
@ 2019-08-22 18:38 ` Leon Romanovsky
2019-08-22 19:13 ` Jason Gunthorpe
2019-08-26 10:02 ` Marcin Mielniczuk
0 siblings, 2 replies; 14+ messages in thread
From: Leon Romanovsky @ 2019-08-22 18:38 UTC (permalink / raw)
To: Marcin Mielniczuk; +Cc: Krishnamraju Eraparaju, linux-rdma
On Thu, Aug 22, 2019 at 07:58:56PM +0200, Marcin Mielniczuk wrote:
> On 22.08.2019 19:21, Leon Romanovsky wrote:
> > On Thu, Aug 22, 2019 at 07:05:12PM +0200, Marcin Mielniczuk wrote:
> >> Thanks a lot, this did the trick. I think this is worth documenting
> >> somewhere that this step is needed.
> >> I'll make a PR, would README.md in the rdma-core repo be a good place?
> > I'm not so sure, but it is better to have in some place instead of not having at all.
> I think it's the first place one would look for some information. I'll
> make a PR today or tomorrow.
> >> Does <NAME> have any significance? I did:
> >>
> >> sudo rdma link add siw0 type siw netdev enpXsYYfZ
> >>
> >> but the resulting device is called iwpXsYYfZ. I couldn't find a trace of
> >> `siw0` anywhere.
> > I would say that it is a bug in kernel part of SIW, because kernel rename
> > (the thing which change your siw0 to be iw* name) is looking for absence
> > of mentioning PCI inside of /sys/class/infiniband/siw0/*
> > https://github.com/linux-rdma/rdma-core/blob/master/kernel-boot/rdma_rename.c#L378
> I don't have /sys/class/infiniband/siw0 on my system, only
> /sys/class/infiniband/iwpXsYYfZ.
> iwp probably comes from iWARP.
Your iwpXsYYfZ was siw0 before rdma_rename was executed.
I can't test the patch now, but hope that this change below will fix your problem.
diff --git a/drivers/infiniband/sw/siw/siw_main.c b/drivers/infiniband/sw/siw/siw_main.c
index 05a92f997f60..38c25a26dfb2 100644
--- a/drivers/infiniband/sw/siw/siw_main.c
+++ b/drivers/infiniband/sw/siw/siw_main.c
@@ -326,21 +326,6 @@ static struct siw_device *siw_device_create(struct net_device *netdev)
struct device *parent = netdev->dev.parent;
int rv;
- if (!parent) {
- /*
- * The loopback device has no parent device,
- * so it appears as a top-level device. To support
- * loopback device connectivity, take this device
- * as the parent device. Skip all other devices
- * w/o parent device.
- */
- if (netdev->type != ARPHRD_LOOPBACK) {
- pr_warn("siw: device %s error: no parent device\n",
- netdev->name);
- return NULL;
- }
- parent = &netdev->dev;
- }
sdev = ib_alloc_device(siw_device, base_dev);
if (!sdev)
return NULL;
^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: Setting up siw devices
2019-08-22 18:38 ` Leon Romanovsky
@ 2019-08-22 19:13 ` Jason Gunthorpe
2019-08-22 19:24 ` Leon Romanovsky
2019-08-26 10:02 ` Marcin Mielniczuk
1 sibling, 1 reply; 14+ messages in thread
From: Jason Gunthorpe @ 2019-08-22 19:13 UTC (permalink / raw)
To: Leon Romanovsky; +Cc: Marcin Mielniczuk, Krishnamraju Eraparaju, linux-rdma
On Thu, Aug 22, 2019 at 09:38:07PM +0300, Leon Romanovsky wrote:
> On Thu, Aug 22, 2019 at 07:58:56PM +0200, Marcin Mielniczuk wrote:
> > On 22.08.2019 19:21, Leon Romanovsky wrote:
> > > On Thu, Aug 22, 2019 at 07:05:12PM +0200, Marcin Mielniczuk wrote:
> > >> Thanks a lot, this did the trick. I think this is worth documenting
> > >> somewhere that this step is needed.
> > >> I'll make a PR, would README.md in the rdma-core repo be a good place?
> > > I'm not so sure, but it is better to have in some place instead of not having at all.
> > I think it's the first place one would look for some information. I'll
> > make a PR today or tomorrow.
> > >> Does <NAME> have any significance? I did:
> > >>
> > >> sudo rdma link add siw0 type siw netdev enpXsYYfZ
> > >>
> > >> but the resulting device is called iwpXsYYfZ. I couldn't find a trace of
> > >> `siw0` anywhere.
> > > I would say that it is a bug in kernel part of SIW, because kernel rename
> > > (the thing which change your siw0 to be iw* name) is looking for absence
> > > of mentioning PCI inside of /sys/class/infiniband/siw0/*
> > > https://github.com/linux-rdma/rdma-core/blob/master/kernel-boot/rdma_rename.c#L378
> > I don't have /sys/class/infiniband/siw0 on my system, only
> > /sys/class/infiniband/iwpXsYYfZ.
> > iwp probably comes from iWARP.
>
> Your iwpXsYYfZ was siw0 before rdma_rename was executed.
>
> I can't test the patch now, but hope that this change below will fix your problem.
I think we should directly blacklist rxe and siw from
renaming. They can only be created with a user-given name so they
should never ever be renamed.
netlink now returns the driver_id and we can use that to trigger it.
Jason
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Setting up siw devices
2019-08-22 19:13 ` Jason Gunthorpe
@ 2019-08-22 19:24 ` Leon Romanovsky
0 siblings, 0 replies; 14+ messages in thread
From: Leon Romanovsky @ 2019-08-22 19:24 UTC (permalink / raw)
To: Jason Gunthorpe; +Cc: Marcin Mielniczuk, Krishnamraju Eraparaju, linux-rdma
On Thu, Aug 22, 2019 at 04:13:54PM -0300, Jason Gunthorpe wrote:
> On Thu, Aug 22, 2019 at 09:38:07PM +0300, Leon Romanovsky wrote:
> > On Thu, Aug 22, 2019 at 07:58:56PM +0200, Marcin Mielniczuk wrote:
> > > On 22.08.2019 19:21, Leon Romanovsky wrote:
> > > > On Thu, Aug 22, 2019 at 07:05:12PM +0200, Marcin Mielniczuk wrote:
> > > >> Thanks a lot, this did the trick. I think this is worth documenting
> > > >> somewhere that this step is needed.
> > > >> I'll make a PR, would README.md in the rdma-core repo be a good place?
> > > > I'm not so sure, but it is better to have in some place instead of not having at all.
> > > I think it's the first place one would look for some information. I'll
> > > make a PR today or tomorrow.
> > > >> Does <NAME> have any significance? I did:
> > > >>
> > > >> sudo rdma link add siw0 type siw netdev enpXsYYfZ
> > > >>
> > > >> but the resulting device is called iwpXsYYfZ. I couldn't find a trace of
> > > >> `siw0` anywhere.
> > > > I would say that it is a bug in kernel part of SIW, because kernel rename
> > > > (the thing which change your siw0 to be iw* name) is looking for absence
> > > > of mentioning PCI inside of /sys/class/infiniband/siw0/*
> > > > https://github.com/linux-rdma/rdma-core/blob/master/kernel-boot/rdma_rename.c#L378
> > > I don't have /sys/class/infiniband/siw0 on my system, only
> > > /sys/class/infiniband/iwpXsYYfZ.
> > > iwp probably comes from iWARP.
> >
> > Your iwpXsYYfZ was siw0 before rdma_rename was executed.
> >
> > I can't test the patch now, but hope that this change below will fix your problem.
>
> I think we should directly blacklist rxe and siw from
> renaming. They can only be created with a user-given name so they
> should never ever be renamed.
>
> netlink now returns the driver_id and we can use that to trigger it.
Blacklisting does not play well with new kernel with new devices against old library.
Thanks
>
> Jason
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Setting up siw devices
[not found] ` <b4bf4bc2-8dc7-a2c2-6bd2-ab41d9fbadc9@golem.network>
@ 2019-08-23 14:17 ` Marcin Mielniczuk
2019-08-23 14:50 ` Jason Gunthorpe
0 siblings, 1 reply; 14+ messages in thread
From: Marcin Mielniczuk @ 2019-08-23 14:17 UTC (permalink / raw)
To: Leon Romanovsky, Krishnamraju Eraparaju; +Cc: linux-rdma
While the device is detected by ibv_devices and rping works, I can't get
ibv_rc_pingpong working
(and as far as I understand, RC should be supported by the iWARP driver)
rping works:
server$ rping -s -a 10.30.10.211 -v
server ping data: rdma-ping-0:
ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
server ping data: rdma-ping-1:
BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
server DISCONNECT EVENT...
wait for RDMA_READ_ADV state 10
client$ rping -c -a 10.30.10.211 -C 2 -v
(output omitted)
But ibv_rc_pingpong doesn't
server$ ibv_rc_pingpong -d iwp____
local address: LID 0x0000, QPN 0x000001, PSN 0xb8aafc, GID ::
Failed to modify QP to RTS
Couldn't connect to remote QP
client$ ibv_rc_pingpong -d iwp____ 10.30.10.211
local address: LID 0x0000, QPN 0x000001, PSN 0x71abc5, GID ::
client read/write: Protocol not supported
Couldn't read/write remote address
dmesg says: [ 93.957733] iwpm_register_pid: Unable to send a nlmsg
(client = 2)
And OpenMPI/UCX doesn't want to use the transport either:
[1566564168.338103] [host:23059:0] select.c:410 UCX ERROR
no active messages transport to <no debug data>: mm/posix - Destination
is unreachable, mm/sysv - Destination is unreachable, self/self -
Destination is unreachable, rdmacm/sockaddr - no am bcopy, cma/cma - no
am bcopy
Is this a bug or a limitation of siw?
Thanks,
Marcin
On 22.08.2019 18:24, Marcin Mielniczuk wrote:
> Thanks a lot, this did the trick. I think this is worth documenting
> somewhere.
> Would README.md in the rdma-core repo be a good place?
>
> Why
>
> On 22.08.2019 17:52, Leon Romanovsky wrote:
>> On Thu, Aug 22, 2019 at 09:13:25PM +0530, Krishnamraju Eraparaju wrote:
>>> On Thursday, August 08/22/19, 2019 at 17:08:49 +0200, Marcin Mielniczuk wrote:
>>>> Hi,
>>>>
>>>> I'm trying to test the recently merged siw module.
>>>> I'm running kernel 5.3-rc5 (taken from the Ubuntu mainline-kernel
>>>> repository [1]) on Ubuntu 18.04 (bionic).
>>>> I also manually installed rdma-core 25.0 from tarball, using the
>>>> included Debian packaging. I installed all the packages but ibacm.
>>>>
>>>> After booting the new kernel I manually loaded the kernel module by
>>>>
>>>> modprobe siw
>>>> modprobe rdma_ucm
>>>>
>>>> Then ibv_devinfo shows: "No IB devices found".
>>>> dmesg only shows:
>>>> [ 29.856751] SoftiWARP attached
>>>>
>>>> According to this tutorial, [2] it should be enough to just load the siw
>>>> module. (unlike RXE, where one needs to use rxe_cfg to set up the
>>>> interface)
>>>> Is this a bug in siw or just a configuration issue on my side?
>>> Have you done "rdma link"?
>>>
>>> rdma link add <NAME> type siw netdev <NETDEV>
>>>
>>> http://man7.org/linux/man-pages/man8/rdma-link.8.html
>> BTW, the same goes for RXE and rxe_cfg is discouraged.
>>
>> Thanks
>>
>>>> Thanks,
>>>> Marcin
>>>>
>>>> [1] https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc5/
>>>> [2] https://budevg.github.io/posts/tutorials/2017/04/29/rdma-101-1.html
>>>>
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Setting up siw devices
2019-08-23 14:17 ` Marcin Mielniczuk
@ 2019-08-23 14:50 ` Jason Gunthorpe
0 siblings, 0 replies; 14+ messages in thread
From: Jason Gunthorpe @ 2019-08-23 14:50 UTC (permalink / raw)
To: Marcin Mielniczuk; +Cc: Leon Romanovsky, Krishnamraju Eraparaju, linux-rdma
On Fri, Aug 23, 2019 at 04:17:50PM +0200, Marcin Mielniczuk wrote:
> While the device is detected by ibv_devices and rping works, I can't get
> ibv_rc_pingpong working
> (and as far as I understand, RC should be supported by the iWARP driver)
>
> rping works:
>
> server$ rping -s -a 10.30.10.211 -v
> server ping data: rdma-ping-0:
> ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
> server ping data: rdma-ping-1:
> BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
> server DISCONNECT EVENT...
> wait for RDMA_READ_ADV state 10
>
> client$ rping -c -a 10.30.10.211 -C 2 -v
> (output omitted)
>
>
> But ibv_rc_pingpong doesn't
>
> server$ ibv_rc_pingpong -d iwp____
> local address: LID 0x0000, QPN 0x000001, PSN 0xb8aafc, GID ::
> Failed to modify QP to RTS
> Couldn't connect to remote QP
> client$ ibv_rc_pingpong -d iwp____ 10.30.10.211
> local address: LID 0x0000, QPN 0x000001, PSN 0x71abc5, GID ::
> client read/write: Protocol not supported
> Couldn't read/write remote address
iwarp does not work with the pingpong examples as iwarp requires RDMA
CM and the examples don't use it.
Jason
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Setting up siw devices
2019-08-22 18:38 ` Leon Romanovsky
2019-08-22 19:13 ` Jason Gunthorpe
@ 2019-08-26 10:02 ` Marcin Mielniczuk
2019-08-26 11:39 ` Leon Romanovsky
1 sibling, 1 reply; 14+ messages in thread
From: Marcin Mielniczuk @ 2019-08-26 10:02 UTC (permalink / raw)
To: Leon Romanovsky; +Cc: Krishnamraju Eraparaju, linux-rdma
With the kernel built with yiour patch the device is still being
renamed. (tbh, I don't even see why this patch would help)
But it's not a big issue for me right now, I can just type in the
renamed interface name.
Thanks
On 22.08.2019 20:38, Leon Romanovsky wrote:
> On Thu, Aug 22, 2019 at 07:58:56PM +0200, Marcin Mielniczuk wrote:
>> On 22.08.2019 19:21, Leon Romanovsky wrote:
>>> On Thu, Aug 22, 2019 at 07:05:12PM +0200, Marcin Mielniczuk wrote:
>>>> Thanks a lot, this did the trick. I think this is worth documenting
>>>> somewhere that this step is needed.
>>>> I'll make a PR, would README.md in the rdma-core repo be a good place?
>>> I'm not so sure, but it is better to have in some place instead of not having at all.
>> I think it's the first place one would look for some information. I'll
>> make a PR today or tomorrow.
>>>> Does <NAME> have any significance? I did:
>>>>
>>>> sudo rdma link add siw0 type siw netdev enpXsYYfZ
>>>>
>>>> but the resulting device is called iwpXsYYfZ. I couldn't find a trace of
>>>> `siw0` anywhere.
>>> I would say that it is a bug in kernel part of SIW, because kernel rename
>>> (the thing which change your siw0 to be iw* name) is looking for absence
>>> of mentioning PCI inside of /sys/class/infiniband/siw0/*
>>> https://github.com/linux-rdma/rdma-core/blob/master/kernel-boot/rdma_rename.c#L378
>> I don't have /sys/class/infiniband/siw0 on my system, only
>> /sys/class/infiniband/iwpXsYYfZ.
>> iwp probably comes from iWARP.
> Your iwpXsYYfZ was siw0 before rdma_rename was executed.
>
> I can't test the patch now, but hope that this change below will fix your problem.
>
> diff --git a/drivers/infiniband/sw/siw/siw_main.c b/drivers/infiniband/sw/siw/siw_main.c
> index 05a92f997f60..38c25a26dfb2 100644
> --- a/drivers/infiniband/sw/siw/siw_main.c
> +++ b/drivers/infiniband/sw/siw/siw_main.c
> @@ -326,21 +326,6 @@ static struct siw_device *siw_device_create(struct net_device *netdev)
> struct device *parent = netdev->dev.parent;
> int rv;
>
> - if (!parent) {
> - /*
> - * The loopback device has no parent device,
> - * so it appears as a top-level device. To support
> - * loopback device connectivity, take this device
> - * as the parent device. Skip all other devices
> - * w/o parent device.
> - */
> - if (netdev->type != ARPHRD_LOOPBACK) {
> - pr_warn("siw: device %s error: no parent device\n",
> - netdev->name);
> - return NULL;
> - }
> - parent = &netdev->dev;
> - }
> sdev = ib_alloc_device(siw_device, base_dev);
> if (!sdev)
> return NULL;
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Setting up siw devices
2019-08-26 10:02 ` Marcin Mielniczuk
@ 2019-08-26 11:39 ` Leon Romanovsky
2019-08-26 18:21 ` Marcin Mielniczuk
0 siblings, 1 reply; 14+ messages in thread
From: Leon Romanovsky @ 2019-08-26 11:39 UTC (permalink / raw)
To: Marcin Mielniczuk; +Cc: Krishnamraju Eraparaju, linux-rdma
On Mon, Aug 26, 2019 at 12:02:47PM +0200, Marcin Mielniczuk wrote:
> With the kernel built with yiour patch the device is still being
> renamed. (tbh, I don't even see why this patch would help)
Because I'm afraid that SIW sets wrong "parent" device.
Thanks
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Setting up siw devices
2019-08-26 11:39 ` Leon Romanovsky
@ 2019-08-26 18:21 ` Marcin Mielniczuk
0 siblings, 0 replies; 14+ messages in thread
From: Marcin Mielniczuk @ 2019-08-26 18:21 UTC (permalink / raw)
To: Leon Romanovsky; +Cc: Krishnamraju Eraparaju, linux-rdma
Btw. I noticed that on Linux 4.15 rxe_cfg created a device called rxe0
and it was not renamed. On the other hand, on 5.3-rc6, rxe_cfg the
device gets renamed from rxe0 to rocepXsXXfX
On 26.08.2019 13:39, Leon Romanovsky wrote:
> On Mon, Aug 26, 2019 at 12:02:47PM +0200, Marcin Mielniczuk wrote:
>> With the kernel built with yiour patch the device is still being
>> renamed. (tbh, I don't even see why this patch would help)
> Because I'm afraid that SIW sets wrong "parent" device.
>
> Thanks
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2019-08-26 18:21 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-08-22 15:08 Setting up siw devices Marcin Mielniczuk
2019-08-22 15:43 ` Krishnamraju Eraparaju
2019-08-22 15:52 ` Leon Romanovsky
2019-08-22 17:05 ` Marcin Mielniczuk
2019-08-22 17:21 ` Leon Romanovsky
2019-08-22 17:58 ` Marcin Mielniczuk
2019-08-22 18:38 ` Leon Romanovsky
2019-08-22 19:13 ` Jason Gunthorpe
2019-08-22 19:24 ` Leon Romanovsky
2019-08-26 10:02 ` Marcin Mielniczuk
2019-08-26 11:39 ` Leon Romanovsky
2019-08-26 18:21 ` Marcin Mielniczuk
[not found] ` <b4bf4bc2-8dc7-a2c2-6bd2-ab41d9fbadc9@golem.network>
2019-08-23 14:17 ` Marcin Mielniczuk
2019-08-23 14:50 ` Jason Gunthorpe
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).