* NULL pointer dereference in nvmet_rdma_queue_disconnect during bond failover
@ 2019-06-05 18:03 Alex Lyakas
2019-06-06 0:05 ` Sagi Grimberg
0 siblings, 1 reply; 19+ messages in thread
From: Alex Lyakas @ 2019-06-05 18:03 UTC (permalink / raw)
Greetings NVMe community,
I am running kernel 5.1.6, which is the latest stable kernel.
I am testing a nvmf kernel target, configured on top of a bond interface,
for high availability. The bond interface is created on top of two
ConnectX-3 interfaces, which represent two ports of one ConnectX-3 VF (with
this hardware a VF is dual-ported, i.e., a single VF yields two network
interfaces). The bond is configured in active-backup mode. Exact bonding
configuration is given in [1]. The nvmet target configuration doesn't have
anything special and is given in [2].
I create a nvmf connection from a different machine to the nvmet target.
Then I initiate bond failover, by disconnecting a cable that corresponds to
the active bond slave. As a result, I get the following kernel panic:
[ 268.036732] mlx4_en: b1s1: Link Down
[ 268.036739] mlx4_en: b0s1: Link Down
[ 268.036771] mlx4_en: b2s1: Link Down
[ 268.138594] bebond: link status definitely down for interface b1s1,
disabling it
[ 268.138597] bebond: making interface b1s0 the new active one 53500 ms
earlier
[ 268.138671] RDMA CM addr change for ndev bebond used by id
0000000019666fc8
[ 268.138673] RDMA CM addr change for ndev bebond used by id
000000007a8dd02e
[ 268.138674] RDMA CM addr change for ndev bebond used by id
00000000f825cc30
[ 268.138675] RDMA CM addr change for ndev bebond used by id
00000000c575ce3d
[ 268.138733] BUG: unable to handle kernel NULL pointer dereference at
0000000000000148
[ 268.138764] #PF error: [normal kernel read fault]
[ 268.138782] PGD 0 P4D 0
[ 268.138795] Oops: 0000 [#1] SMP PTI
[ 268.138811] CPU: 1 PID: 869 Comm: kworker/u4:5 Not tainted
5.1.6-050106-generic #201905311031
[ 268.138839] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.10.2-1ubuntu1 04/01/2014
[ 268.138885] Workqueue: rdma_cm cma_ndev_work_handler [rdma_cm]
[ 268.138912] RIP: 0010:nvmet_rdma_queue_disconnect+0x19/0x80 [nvmet_rdma]
[ 268.138937] Code: e8 bc fe ff ff e9 68 ff ff ff 0f 1f 80 00 00 00 00 66
66 66 66 90 55 48 89 e5 53 48 89 fb 48 c7 c7 80 10 86 c0 e8 57 1d ff d1 <48>
8b 93 48 01 00 00 48 8d 83 48 01 00 00 48 39 d0 74 3a 48 8b 8b
[ 268.139020] RSP: 0018:ffffb28a0111be08 EFLAGS: 00010246
[ 268.139712] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
0000000000000000
[ 268.140348] RDX: ffff9cc2a7c15c00 RSI: 000000000000000e RDI:
ffffffffc0861080
[ 268.140764] RBP: ffffb28a0111be10 R08: ffff9cc2a7c15c00 R09:
000000000000008c
[ 268.141195] R10: 00000000000001ed R11: 0000000000000001 R12:
ffff9cc2a7c54aa8
[ 268.141616] R13: ffff9cc2a9b55800 R14: ffff9cc2a7c54a80 R15:
0ffff9cc2a78ee60
[ 268.142057] FS: 0000000000000000(0000) GS:ffff9cc2b9b00000(0000)
knlGS:0000000000000000
[ 268.142520] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 268.142962] CR2: 0000000000000148 CR3: 00000001afe30004 CR4:
00000000000606e0
[ 268.143430] Call Trace:
[ 268.143880] nvmet_rdma_cm_handler+0x94/0x292 [nvmet_rdma]
[ 268.144343] cma_ndev_work_handler+0x45/0xb0 [rdma_cm]
[ 268.144792] process_one_work+0x20f/0x410
[ 268.145246] worker_thread+0x34/0x400
[ 268.145689] kthread+0x120/0x140
[ 268.146141] ? process_one_work+0x410/0x410
[ 268.146595] ? __kthread_parkme+0x70/0x70
[ 268.147045] ret_from_fork+0x35/0x40
This is 100% reproducible.
Thanks,
Alex.
[1]
echo +bebond >/sys/class/net/bonding_masters
echo "1" > /proc/sys/net/ipv6/conf/bebond/disable_ipv6
echo "1" > /sys/class/net/bebond/bonding/mode
echo "100" > /sys/class/net/bebond/bonding/miimon
echo "1" > /sys/class/net/bebond/bonding/fail_over_mac
echo "60000" > /sys/class/net/bebond/bonding/updelay
ifconfig b1s1 down
echo "+b1s1" > /sys/class/net/bebond/bonding/slaves
ifconfig b1s0 down
echo "+b1s0" > /sys/class/net/bebond/bonding/slaves
echo "b1s1" > /sys/class/net/bebond/bonding/primary
ip addr add 10.3.3.23/24 dev bebond
[2]
mkdir /sys/kernel/config/nvmet/subsystems/volume-55555555
echo 1 >
/sys/kernel/config/nvmet/subsystems/volume-55555555/attr_allow_any_host
echo 000055555555 >
/sys/kernel/config/nvmet/subsystems/volume-55555555/attr_serial
mkdir /sys/kernel/config/nvmet/subsystems/volume-55555555/namespaces/1
echo 0977dff3-6885-43b3-a948-000055555555 >
/sys/kernel/config/nvmet/subsystems/volume-55555555/namespaces/1/device_uuid
echo -n /dev/loop0 >
/sys/kernel/config/nvmet/subsystems/volume-55555555/namespaces/1/device_path
echo 1 >
/sys/kernel/config/nvmet/subsystems/volume-55555555/namespaces/1/enable
mkdir /sys/kernel/config/nvmet/ports/1
echo -n "ipv4" > /sys/kernel/config/nvmet/ports/1/addr_adrfam
echo -n "rdma" > /sys/kernel/config/nvmet/ports/1/addr_trtype
echo -n 10.3.3.23 > /sys/kernel/config/nvmet/ports/1/addr_traddr
echo -n 4420 > /sys/kernel/config/nvmet/ports/1/addr_trsvcid
ln -s /sys/kernel/config/nvmet/subsystems/volume-55555555
/sys/kernel/config/nvmet/ports/1/subsystems/
^ permalink raw reply [flat|nested] 19+ messages in thread
* NULL pointer dereference in nvmet_rdma_queue_disconnect during bond failover
2019-06-05 18:03 NULL pointer dereference in nvmet_rdma_queue_disconnect during bond failover Alex Lyakas
@ 2019-06-06 0:05 ` Sagi Grimberg
2019-06-06 7:31 ` Max Gurtovoy
0 siblings, 1 reply; 19+ messages in thread
From: Sagi Grimberg @ 2019-06-06 0:05 UTC (permalink / raw)
> Greetings NVMe community,
>
> I am running kernel 5.1.6, which is the latest stable kernel.
>
> I am testing a nvmf kernel target, configured on top of a bond
> interface, for high availability. The bond interface is created on top
> of two ConnectX-3 interfaces, which represent two ports of one
> ConnectX-3 VF (with this hardware a VF is dual-ported, i.e., a single VF
> yields two network interfaces). The bond is configured in active-backup
> mode. Exact bonding configuration is given in [1]. The nvmet target
> configuration doesn't have anything special and is given in [2].
>
> I create a nvmf connection from a different machine to the nvmet target.
> Then I initiate bond failover, by disconnecting a cable that corresponds
> to the active bond slave. As a result, I get the following kernel panic:
Max sent a fix exactly for this. You can test that it works for you
when he sends v2.
Max, care to CC Alex when you send it?
^ permalink raw reply [flat|nested] 19+ messages in thread
* NULL pointer dereference in nvmet_rdma_queue_disconnect during bond failover
2019-06-06 0:05 ` Sagi Grimberg
@ 2019-06-06 7:31 ` Max Gurtovoy
2019-07-03 9:28 ` Alex Lyakas
0 siblings, 1 reply; 19+ messages in thread
From: Max Gurtovoy @ 2019-06-06 7:31 UTC (permalink / raw)
On 6/6/2019 3:05 AM, Sagi Grimberg wrote:
>
>> Greetings NVMe community,
>>
>> I am running kernel 5.1.6, which is the latest stable kernel.
>>
>> I am testing a nvmf kernel target, configured on top of a bond
>> interface, for high availability. The bond interface is created on
>> top of two ConnectX-3 interfaces, which represent two ports of one
>> ConnectX-3 VF (with this hardware a VF is dual-ported, i.e., a single
>> VF yields two network interfaces). The bond is configured in
>> active-backup mode. Exact bonding configuration is given in [1]. The
>> nvmet target configuration doesn't have anything special and is given
>> in [2].
>>
>> I create a nvmf connection from a different machine to the nvmet
>> target. Then I initiate bond failover, by disconnecting a cable that
>> corresponds to the active bond slave. As a result, I get the
>> following kernel panic:
>
> Max sent a fix exactly for this. You can test that it works for you
> when he sends v2.
>
> Max, care to CC Alex when you send it?
Sure, No problem.
>
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.infradead.org%2Fmailman%2Flistinfo%2Flinux-nvme&data=02%7C01%7Cmaxg%40mellanox.com%7C9879366f21df4e7f4ce608d6ea12c41f%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636953763648159480&sdata=UtkpvcEM9%2BBBdj68Kx6bobMWkeGSp1Jz3yJXxH8MLgY%3D&reserved=0
>
^ permalink raw reply [flat|nested] 19+ messages in thread
* NULL pointer dereference in nvmet_rdma_queue_disconnect during bond failover
2019-06-06 7:31 ` Max Gurtovoy
@ 2019-07-03 9:28 ` Alex Lyakas
2019-07-03 12:56 ` Max Gurtovoy
0 siblings, 1 reply; 19+ messages in thread
From: Alex Lyakas @ 2019-07-03 9:28 UTC (permalink / raw)
Hi Max,
Has any patch been sent to resolve the kernel panic in nvmet that we are
seeing?
Thanks,
Alex.
-----Original Message-----
From: Max Gurtovoy
Sent: Thursday, June 06, 2019 10:31 AM
To: linux-nvme at lists.infradead.org
Subject: Re: NULL pointer dereference in nvmet_rdma_queue_disconnect during
bond failover
On 6/6/2019 3:05 AM, Sagi Grimberg wrote:
>
>> Greetings NVMe community,
>>
>> I am running kernel 5.1.6, which is the latest stable kernel.
>>
>> I am testing a nvmf kernel target, configured on top of a bond interface,
>> for high availability. The bond interface is created on top of two
>> ConnectX-3 interfaces, which represent two ports of one ConnectX-3 VF
>> (with this hardware a VF is dual-ported, i.e., a single VF yields two
>> network interfaces). The bond is configured in active-backup mode. Exact
>> bonding configuration is given in [1]. The nvmet target configuration
>> doesn't have anything special and is given in [2].
>>
>> I create a nvmf connection from a different machine to the nvmet target.
>> Then I initiate bond failover, by disconnecting a cable that corresponds
>> to the active bond slave. As a result, I get the following kernel panic:
>
> Max sent a fix exactly for this. You can test that it works for you
> when he sends v2.
>
> Max, care to CC Alex when you send it?
Sure, No problem.
>
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.infradead.org%2Fmailman%2Flistinfo%2Flinux-nvme&data=02%7C01%7Cmaxg%40mellanox.com%7C9879366f21df4e7f4ce608d6ea12c41f%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636953763648159480&sdata=UtkpvcEM9%2BBBdj68Kx6bobMWkeGSp1Jz3yJXxH8MLgY%3D&reserved=0
_______________________________________________
Linux-nvme mailing list
Linux-nvme at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
^ permalink raw reply [flat|nested] 19+ messages in thread
* NULL pointer dereference in nvmet_rdma_queue_disconnect during bond failover
2019-07-03 9:28 ` Alex Lyakas
@ 2019-07-03 12:56 ` Max Gurtovoy
2019-07-03 22:42 ` Sagi Grimberg
0 siblings, 1 reply; 19+ messages in thread
From: Max Gurtovoy @ 2019-07-03 12:56 UTC (permalink / raw)
Hi Alex,
Not yet. Our fix is in the Initiator/Host side and it was merged.
This is on our plate.
In case you would like to send a patch to solve this, we'll review it
of-course.
-Max.
On 7/3/2019 12:28 PM, Alex Lyakas wrote:
> Hi Max,
>
> Has any patch been sent to resolve the kernel panic in nvmet that we
> are seeing?
>
> Thanks,
> Alex.
>
>
> -----Original Message----- From: Max Gurtovoy
> Sent: Thursday, June 06, 2019 10:31 AM
> To: linux-nvme at lists.infradead.org
> Subject: Re: NULL pointer dereference in nvmet_rdma_queue_disconnect
> during bond failover
>
>
> On 6/6/2019 3:05 AM, Sagi Grimberg wrote:
>>
>>> Greetings NVMe community,
>>>
>>> I am running kernel 5.1.6, which is the latest stable kernel.
>>>
>>> I am testing a nvmf kernel target, configured on top of a bond
>>> interface, for high availability. The bond interface is created on
>>> top of two ConnectX-3 interfaces, which represent two ports of one
>>> ConnectX-3 VF (with this hardware a VF is dual-ported, i.e., a
>>> single VF yields two network interfaces). The bond is configured in
>>> active-backup mode. Exact bonding configuration is given in [1]. The
>>> nvmet target configuration doesn't have anything special and is
>>> given in [2].
>>>
>>> I create a nvmf connection from a different machine to the nvmet
>>> target. Then I initiate bond failover, by disconnecting a cable that
>>> corresponds to the active bond slave. As a result, I get the
>>> following kernel panic:
>>
>> Max sent a fix exactly for this. You can test that it works for you
>> when he sends v2.
>>
>> Max, care to CC Alex when you send it?
>
> Sure, No problem.
>
>
>>
>> _______________________________________________
>> Linux-nvme mailing list
>> Linux-nvme at lists.infradead.org
>> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.infradead.org%2Fmailman%2Flistinfo%2Flinux-nvme&data=02%7C01%7Cmaxg%40mellanox.com%7Cb0df220121534ce1fe4b08d6ff98e119%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636977429409072070&sdata=nwWwML3PUy%2BZ0HJeJRKpNyLB0Nm%2BV%2BUxNuweaAWyeyA%3D&reserved=0
>>
>
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.infradead.org%2Fmailman%2Flistinfo%2Flinux-nvme&data=02%7C01%7Cmaxg%40mellanox.com%7Cb0df220121534ce1fe4b08d6ff98e119%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636977429409072070&sdata=nwWwML3PUy%2BZ0HJeJRKpNyLB0Nm%2BV%2BUxNuweaAWyeyA%3D&reserved=0
>
^ permalink raw reply [flat|nested] 19+ messages in thread
* NULL pointer dereference in nvmet_rdma_queue_disconnect during bond failover
2019-07-03 12:56 ` Max Gurtovoy
@ 2019-07-03 22:42 ` Sagi Grimberg
2019-07-12 19:38 ` Sagi Grimberg
0 siblings, 1 reply; 19+ messages in thread
From: Sagi Grimberg @ 2019-07-03 22:42 UTC (permalink / raw)
> Hi Alex,
>
> Not yet. Our fix is in the Initiator/Host side and it was merged.
>
> This is on our plate.
>
> In case you would like to send a patch to solve this, we'll review it
> of-course.
Does the attached untested patch help?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-nvmet-rdma-fix-bonding-failover-possible-NULL-deref.patch
Type: text/x-patch
Size: 8365 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-nvme/attachments/20190703/c697655b/attachment-0001.bin>
^ permalink raw reply [flat|nested] 19+ messages in thread
* NULL pointer dereference in nvmet_rdma_queue_disconnect during bond failover
2019-07-03 22:42 ` Sagi Grimberg
@ 2019-07-12 19:38 ` Sagi Grimberg
2019-07-13 19:44 ` Alex Lyakas
0 siblings, 1 reply; 19+ messages in thread
From: Sagi Grimberg @ 2019-07-12 19:38 UTC (permalink / raw)
>> Hi Alex,
>>
>> Not yet. Our fix is in the Initiator/Host side and it was merged.
>>
>> This is on our plate.
>>
>> In case you would like to send a patch to solve this, we'll review it
>> of-course.
>
> Does the attached untested patch help?
Alex? Max?
^ permalink raw reply [flat|nested] 19+ messages in thread
* NULL pointer dereference in nvmet_rdma_queue_disconnect during bond failover
2019-07-12 19:38 ` Sagi Grimberg
@ 2019-07-13 19:44 ` Alex Lyakas
2019-07-14 7:27 ` Sagi Grimberg
0 siblings, 1 reply; 19+ messages in thread
From: Alex Lyakas @ 2019-07-13 19:44 UTC (permalink / raw)
Hi Sagi,
Which kernel this patch applies to?
At this point the environment I used for nvmf evaluation is not available
for me. I will make an effort to test this patch, and get back to you.
Thanks,
Alex,
-----Original Message-----
From: Sagi Grimberg
Sent: Friday, July 12, 2019 10:38 PM
To: Max Gurtovoy ; Alex Lyakas ; linux-nvme at lists.infradead.org ; Shlomi
Nimrodi ; Israel Rukshin ; tomwu at mellanox.com
Subject: Re: NULL pointer dereference in nvmet_rdma_queue_disconnect during
bond failover
>> Hi Alex,
>>
>> Not yet. Our fix is in the Initiator/Host side and it was merged.
>>
>> This is on our plate.
>>
>> In case you would like to send a patch to solve this, we'll review it
>> of-course.
>
> Does the attached untested patch help?
Alex? Max?
^ permalink raw reply [flat|nested] 19+ messages in thread
* NULL pointer dereference in nvmet_rdma_queue_disconnect during bond failover
2019-07-13 19:44 ` Alex Lyakas
@ 2019-07-14 7:27 ` Sagi Grimberg
2019-08-01 1:08 ` Sagi Grimberg
0 siblings, 1 reply; 19+ messages in thread
From: Sagi Grimberg @ 2019-07-14 7:27 UTC (permalink / raw)
> Hi Sagi,
>
> Which kernel this patch applies to?
its based on the nvme tree, but it should apply cleanly on upstream
5.2...
^ permalink raw reply [flat|nested] 19+ messages in thread
* NULL pointer dereference in nvmet_rdma_queue_disconnect during bond failover
2019-07-14 7:27 ` Sagi Grimberg
@ 2019-08-01 1:08 ` Sagi Grimberg
2019-09-13 18:44 ` Sagi Grimberg
0 siblings, 1 reply; 19+ messages in thread
From: Sagi Grimberg @ 2019-08-01 1:08 UTC (permalink / raw)
>> Hi Sagi,
>>
>> Which kernel this patch applies to?
>
> its based on the nvme tree, but it should apply cleanly on upstream
> 5.2...
Alex, Max? did you retest this?
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: NULL pointer dereference in nvmet_rdma_queue_disconnect during bond failover
2019-08-01 1:08 ` Sagi Grimberg
@ 2019-09-13 18:44 ` Sagi Grimberg
2020-03-30 19:02 ` Alex Lyakas
0 siblings, 1 reply; 19+ messages in thread
From: Sagi Grimberg @ 2019-09-13 18:44 UTC (permalink / raw)
To: Alex Lyakas, Max Gurtovoy, linux-nvme, Shlomi Nimrodi,
Israel Rukshin, tomwu
>>> Hi Sagi,
>>>
>>> Which kernel this patch applies to?
>>
>> its based on the nvme tree, but it should apply cleanly on upstream
>> 5.2...
>
> Alex, Max? did you retest this?
Raising this from the ashes...
Alex, did you test this patch?
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: NULL pointer dereference in nvmet_rdma_queue_disconnect during bond failover
2019-09-13 18:44 ` Sagi Grimberg
@ 2020-03-30 19:02 ` Alex Lyakas
2020-03-30 21:06 ` Max Gurtovoy
2020-03-31 0:21 ` Sagi Grimberg
0 siblings, 2 replies; 19+ messages in thread
From: Alex Lyakas @ 2020-03-30 19:02 UTC (permalink / raw)
To: Sagi Grimberg
Cc: tomwu, Max Gurtovoy, Israel Rukshin, linux-nvme, Shlomi Nimrodi
Hi Sagi,
>
>
> >>> Hi Sagi,
> >>>
> >>> Which kernel this patch applies to?
> >>
> >> its based on the nvme tree, but it should apply cleanly on upstream
> >> 5.2...
> >
> > Alex, Max? did you retest this?
>
> Raising this from the ashes...
>
> Alex, did you test this patch?
Raising from the ashes!
In short: this patch fixes the issue!
More details:
This patch doesn't apply on kernel 5.2. Moreover, I believe this patch
is incomplete, because nvmet_rdma_find_get_device() needs to be fixed
to treat cm_id->context as "struct nvmet_rdma_port" and not as "struct
nvmet_port".
However, since we are working with kernel modules from Mellanox OFED,
I tried applying this patch on OFED 4.7. I discovered that it already
has almost everything this patch introduces. Like "struct
nvmet_rdma_port" and the refactoring of nvmet_rdma_add_port into
nvmet_rdma_enable_port, and nvmet_rdma_remove_port to
nvmet_rdma_disable_port. I ended up with this patch [1].
Tested bond failover, and cm_id is destroyed and re-created as expected [2]
Israel, Max and other Mellanox folks: can we have this fix in OFED 4.9?
Thanks,
Alex.
[1]
diff -ru mlnx-nvme-4.7-orig/target/rdma.c mlnx-nvme-4.7/target/rdma.c
--- mlnx-nvme-4.7-orig/target/rdma.c 2020-01-15 09:58:59.000000000 +0200
+++ mlnx-nvme-4.7/target/rdma.c 2020-03-30 20:49:49.932479383 +0300
@@ -191,6 +191,7 @@
struct nvmet_rdma_rsp *r);
static int nvmet_rdma_alloc_rsp(struct nvmet_rdma_device *ndev,
struct nvmet_rdma_rsp *r);
+static void nvmet_rdma_disable_port(struct nvmet_rdma_port *port);
static const struct nvmet_fabrics_ops nvmet_rdma_ops;
@@ -1544,6 +1545,13 @@
nvmet_rdma_queue_established(queue);
break;
case RDMA_CM_EVENT_ADDR_CHANGE:
+ if (!queue) {
+ struct nvmet_rdma_port *port = cm_id->context;
+
+ pr_warn("RDMA_CM_EVENT_ADDR_CHANGE: cm_id=%p schedule
enable_work\n", cm_id);
+ schedule_delayed_work(&port->enable_work, 0);
+ break;
+ }
case RDMA_CM_EVENT_DISCONNECTED:
case RDMA_CM_EVENT_TIMEWAIT_EXIT:
nvmet_rdma_queue_disconnect(queue);
@@ -1598,6 +1606,8 @@
return PTR_ERR(cm_id);
}
+ pr_info("nvmet_rdma_enable_port: created cm_id=%p\n", cm_id);
+
/*
* Allow both IPv4 and IPv6 sockets to bind a single port
* at the same time.
@@ -1620,7 +1630,7 @@
goto out_destroy_id;
}
- port->cm_id = cm_id;
+ xchg(&port->cm_id, cm_id);
if (cm_id->device)
port->node_guid = cm_id->device->node_guid;
@@ -1640,6 +1650,7 @@
struct nvmet_rdma_port, enable_work);
int ret;
+ nvmet_rdma_disable_port(port);
ret = nvmet_rdma_enable_port(port);
if (ret)
schedule_delayed_work(&port->enable_work, 5 * HZ);
@@ -1707,13 +1718,14 @@
static void nvmet_rdma_disable_port(struct nvmet_rdma_port *port)
{
- struct rdma_cm_id *cm_id = port->cm_id;
+ struct rdma_cm_id *cm_id = xchg(&port->cm_id, NULL);
struct nvmet_port *nport = port->nport;
+ pr_info("nvmet_rdma_disable_port: cm_id=%p\n", cm_id);
+
if (nport->offload && cm_id)
nvmet_rdma_destroy_xrqs(nport);
- port->cm_id = NULL;
if (cm_id)
rdma_destroy_id(cm_id);
}
[2]
Mar 30 21:57:48.030761 qa3-sn2 kernel: [95220.661707] bebond: making
interface be10G2 the new active one
Mar 30 21:57:48.030789 qa3-sn2 kernel: [95220.662003] RDMA CM addr
change for ndev bebond used by id ffff966432c63000
Mar 30 21:57:48.030793 qa3-sn2 kernel: [95220.662007] RDMA CM addr
change for ndev bebond used by id ffff966a6ee85800
Mar 30 21:57:48.030817 qa3-sn2 kernel: [95220.662010] RDMA CM addr
change for ndev bebond used by id ffff966a6ee87400
Mar 30 21:57:48.030821 qa3-sn2 kernel: [95220.662012] RDMA CM addr
change for ndev bebond used by id ffff966a6ee85400
Mar 30 21:57:48.030824 qa3-sn2 kernel: [95220.662015] RDMA CM addr
change for ndev bebond used by id ffff966a6ee83c00
Mar 30 21:57:48.030827 qa3-sn2 kernel: [95220.662017] RDMA CM addr
change for ndev bebond used by id ffff966a6ee84c00
Mar 30 21:57:48.030829 qa3-sn2 kernel: [95220.662025] nvmet_rdma:
RDMA_CM_EVENT_ADDR_CHANGE: cm_id=ffff966432c63000 schedule enable_work
Mar 30 21:57:48.030832 qa3-sn2 kernel: [95220.662069] nvmet_rdma:
nvmet_rdma_disable_port: cm_id=ffff966432c63000
Mar 30 21:57:48.030834 qa3-sn2 kernel: [95220.662093] nvmet_rdma:
nvmet_rdma_enable_port: created cm_id=ffff96658fdab800
Mar 30 21:57:48.030837 qa3-sn2 kernel: [95220.662120] nvmet_rdma:
enabling port 1 (10.3.3.3:4420)
Mar 30 21:57:50.266755 qa3-sn2 kernel: [95222.897752] nvmet: creating
controller 1 for subsystem
nqn.2011-04.com.zadarastorage:volume-00000010 for NQN
iqn.2011-04.com.zadarastorage:2:vc-1.
_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: NULL pointer dereference in nvmet_rdma_queue_disconnect during bond failover
2020-03-30 19:02 ` Alex Lyakas
@ 2020-03-30 21:06 ` Max Gurtovoy
2020-03-31 0:21 ` Sagi Grimberg
1 sibling, 0 replies; 19+ messages in thread
From: Max Gurtovoy @ 2020-03-30 21:06 UTC (permalink / raw)
To: Alex Lyakas, Sagi Grimberg
Cc: tomwu, Shlomi Nimrodi, linux-nvme, Israel Rukshin
On 3/30/2020 10:02 PM, Alex Lyakas wrote:
> Hi Sagi,
>
>>
>>>>> Hi Sagi,
>>>>>
>>>>> Which kernel this patch applies to?
>>>> its based on the nvme tree, but it should apply cleanly on upstream
>>>> 5.2...
>>> Alex, Max? did you retest this?
>> Raising this from the ashes...
>>
>> Alex, did you test this patch?
> Raising from the ashes!
>
> In short: this patch fixes the issue!
>
> More details:
>
> This patch doesn't apply on kernel 5.2. Moreover, I believe this patch
> is incomplete, because nvmet_rdma_find_get_device() needs to be fixed
> to treat cm_id->context as "struct nvmet_rdma_port" and not as "struct
> nvmet_port".
>
> However, since we are working with kernel modules from Mellanox OFED,
> I tried applying this patch on OFED 4.7. I discovered that it already
> has almost everything this patch introduces. Like "struct
> nvmet_rdma_port" and the refactoring of nvmet_rdma_add_port into
> nvmet_rdma_enable_port, and nvmet_rdma_remove_port to
> nvmet_rdma_disable_port. I ended up with this patch [1].
>
> Tested bond failover, and cm_id is destroyed and re-created as expected [2]
>
> Israel, Max and other Mellanox folks: can we have this fix in OFED 4.9?
Alex,
We first need to fix this issue in upstream.
hopefully we can get to it soon.
>
> Thanks,
> Alex.
>
>
> [1]
> diff -ru mlnx-nvme-4.7-orig/target/rdma.c mlnx-nvme-4.7/target/rdma.c
> --- mlnx-nvme-4.7-orig/target/rdma.c 2020-01-15 09:58:59.000000000 +0200
> +++ mlnx-nvme-4.7/target/rdma.c 2020-03-30 20:49:49.932479383 +0300
> @@ -191,6 +191,7 @@
> struct nvmet_rdma_rsp *r);
> static int nvmet_rdma_alloc_rsp(struct nvmet_rdma_device *ndev,
> struct nvmet_rdma_rsp *r);
> +static void nvmet_rdma_disable_port(struct nvmet_rdma_port *port);
>
> static const struct nvmet_fabrics_ops nvmet_rdma_ops;
>
> @@ -1544,6 +1545,13 @@
> nvmet_rdma_queue_established(queue);
> break;
> case RDMA_CM_EVENT_ADDR_CHANGE:
> + if (!queue) {
> + struct nvmet_rdma_port *port = cm_id->context;
> +
> + pr_warn("RDMA_CM_EVENT_ADDR_CHANGE: cm_id=%p schedule
> enable_work\n", cm_id);
> + schedule_delayed_work(&port->enable_work, 0);
> + break;
> + }
> case RDMA_CM_EVENT_DISCONNECTED:
> case RDMA_CM_EVENT_TIMEWAIT_EXIT:
> nvmet_rdma_queue_disconnect(queue);
> @@ -1598,6 +1606,8 @@
> return PTR_ERR(cm_id);
> }
>
> + pr_info("nvmet_rdma_enable_port: created cm_id=%p\n", cm_id);
> +
> /*
> * Allow both IPv4 and IPv6 sockets to bind a single port
> * at the same time.
> @@ -1620,7 +1630,7 @@
> goto out_destroy_id;
> }
>
> - port->cm_id = cm_id;
> + xchg(&port->cm_id, cm_id);
> if (cm_id->device)
> port->node_guid = cm_id->device->node_guid;
>
> @@ -1640,6 +1650,7 @@
> struct nvmet_rdma_port, enable_work);
> int ret;
>
> + nvmet_rdma_disable_port(port);
> ret = nvmet_rdma_enable_port(port);
> if (ret)
> schedule_delayed_work(&port->enable_work, 5 * HZ);
> @@ -1707,13 +1718,14 @@
>
> static void nvmet_rdma_disable_port(struct nvmet_rdma_port *port)
> {
> - struct rdma_cm_id *cm_id = port->cm_id;
> + struct rdma_cm_id *cm_id = xchg(&port->cm_id, NULL);
> struct nvmet_port *nport = port->nport;
>
> + pr_info("nvmet_rdma_disable_port: cm_id=%p\n", cm_id);
> +
> if (nport->offload && cm_id)
> nvmet_rdma_destroy_xrqs(nport);
>
> - port->cm_id = NULL;
> if (cm_id)
> rdma_destroy_id(cm_id);
> }
>
>
> [2]
> Mar 30 21:57:48.030761 qa3-sn2 kernel: [95220.661707] bebond: making
> interface be10G2 the new active one
> Mar 30 21:57:48.030789 qa3-sn2 kernel: [95220.662003] RDMA CM addr
> change for ndev bebond used by id ffff966432c63000
> Mar 30 21:57:48.030793 qa3-sn2 kernel: [95220.662007] RDMA CM addr
> change for ndev bebond used by id ffff966a6ee85800
> Mar 30 21:57:48.030817 qa3-sn2 kernel: [95220.662010] RDMA CM addr
> change for ndev bebond used by id ffff966a6ee87400
> Mar 30 21:57:48.030821 qa3-sn2 kernel: [95220.662012] RDMA CM addr
> change for ndev bebond used by id ffff966a6ee85400
> Mar 30 21:57:48.030824 qa3-sn2 kernel: [95220.662015] RDMA CM addr
> change for ndev bebond used by id ffff966a6ee83c00
> Mar 30 21:57:48.030827 qa3-sn2 kernel: [95220.662017] RDMA CM addr
> change for ndev bebond used by id ffff966a6ee84c00
> Mar 30 21:57:48.030829 qa3-sn2 kernel: [95220.662025] nvmet_rdma:
> RDMA_CM_EVENT_ADDR_CHANGE: cm_id=ffff966432c63000 schedule enable_work
> Mar 30 21:57:48.030832 qa3-sn2 kernel: [95220.662069] nvmet_rdma:
> nvmet_rdma_disable_port: cm_id=ffff966432c63000
> Mar 30 21:57:48.030834 qa3-sn2 kernel: [95220.662093] nvmet_rdma:
> nvmet_rdma_enable_port: created cm_id=ffff96658fdab800
> Mar 30 21:57:48.030837 qa3-sn2 kernel: [95220.662120] nvmet_rdma:
> enabling port 1 (10.3.3.3:4420)
> Mar 30 21:57:50.266755 qa3-sn2 kernel: [95222.897752] nvmet: creating
> controller 1 for subsystem
> nqn.2011-04.com.zadarastorage:volume-00000010 for NQN
> iqn.2011-04.com.zadarastorage:2:vc-1.
_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: NULL pointer dereference in nvmet_rdma_queue_disconnect during bond failover
2020-03-30 19:02 ` Alex Lyakas
2020-03-30 21:06 ` Max Gurtovoy
@ 2020-03-31 0:21 ` Sagi Grimberg
2020-03-31 8:16 ` Shlomi Nimrodi
2020-04-02 9:13 ` Alex Lyakas
1 sibling, 2 replies; 19+ messages in thread
From: Sagi Grimberg @ 2020-03-31 0:21 UTC (permalink / raw)
To: Alex Lyakas
Cc: Shlomi Nimrodi, tomwu, Israel Rukshin, linux-nvme, Max Gurtovoy
Hey Alex,
>>> Alex, Max? did you retest this?
>>
>> Raising this from the ashes...
>>
>> Alex, did you test this patch?
>
> Raising from the ashes!
>
> In short: this patch fixes the issue!
Thanks for following up..
>
> More details:
>
> This patch doesn't apply on kernel 5.2. Moreover, I believe this patch
> is incomplete, because nvmet_rdma_find_get_device() needs to be fixed
> to treat cm_id->context as "struct nvmet_rdma_port" and not as "struct
> nvmet_port".
Does patch [1] apply on kernel 5.2?
> However, since we are working with kernel modules from Mellanox OFED,
> I tried applying this patch on OFED 4.7. I discovered that it already
> has almost everything this patch introduces. Like "struct
> nvmet_rdma_port" and the refactoring of nvmet_rdma_add_port into
> nvmet_rdma_enable_port, and nvmet_rdma_remove_port to
> nvmet_rdma_disable_port. I ended up with this patch [1].
>
> Tested bond failover, and cm_id is destroyed and re-created as expected [2]
>
> Israel, Max and other Mellanox folks: can we have this fix in OFED 4.9?
>
For MOFED issues you can follow-up with Max and Israel offline. If you
can test upstream or even 5.2 stable that would be beneficial as I can
add your Tested-by tag.
Thanks.
[1]:
--
Author: Sagi Grimberg <sagi@grimberg.me>
Date: Wed Jul 3 15:33:01 2019 -0700
nvmet-rdma: fix bonding failover possible NULL deref
RDMA_CM_EVENT_ADDR_CHANGE event occur in the case of bonding failover
on normal as well as on listening cm_ids. Hence this event will
immediately trigger a NULL dereference trying to disconnect a queue
for a cm_id that actually belongs to the port.
To fix this we provide a different handler for the listener cm_ids
that will defer a work to disable+(re)enable the port which essentially
destroys and setups another listener cm_id
Reported-by: Alex Lyakas <alex@zadara.com>
Signed-off-by: Sagi Grimberg <sagi@grimberg.me>
diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index 9e1b8c61f54e..8dac89b7aa12 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -105,6 +105,13 @@ struct nvmet_rdma_queue {
struct list_head queue_list;
};
+struct nvmet_rdma_port {
+ struct nvmet_port *nport;
+ struct sockaddr_storage addr;
+ struct rdma_cm_id *cm_id;
+ struct delayed_work repair_work;
+};
+
struct nvmet_rdma_device {
struct ib_device *device;
struct ib_pd *pd;
@@ -1272,6 +1279,7 @@ static int nvmet_rdma_cm_accept(struct rdma_cm_id
*cm_id,
static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id,
struct rdma_cm_event *event)
{
+ struct nvmet_rdma_port *port = cm_id->context;
struct nvmet_rdma_device *ndev;
struct nvmet_rdma_queue *queue;
int ret = -EINVAL;
@@ -1287,7 +1295,7 @@ static int nvmet_rdma_queue_connect(struct
rdma_cm_id *cm_id,
ret = -ENOMEM;
goto put_device;
}
- queue->port = cm_id->context;
+ queue->port = port->nport;
if (queue->host_qid == 0) {
/* Let inflight controller teardown complete */
@@ -1412,7 +1420,7 @@ static void nvmet_rdma_queue_connect_fail(struct
rdma_cm_id *cm_id,
static int nvmet_rdma_device_removal(struct rdma_cm_id *cm_id,
struct nvmet_rdma_queue *queue)
{
- struct nvmet_port *port;
+ struct nvmet_rdma_port *port;
if (queue) {
/*
@@ -1431,7 +1439,7 @@ static int nvmet_rdma_device_removal(struct
rdma_cm_id *cm_id,
* cm_id destroy. use atomic xchg to make sure
* we don't compete with remove_port.
*/
- if (xchg(&port->priv, NULL) != cm_id)
+ if (xchg(&port->cm_id, NULL) != cm_id)
return 0;
/*
@@ -1462,6 +1470,13 @@ static int nvmet_rdma_cm_handler(struct
rdma_cm_id *cm_id,
nvmet_rdma_queue_established(queue);
break;
case RDMA_CM_EVENT_ADDR_CHANGE:
+ if (!queue) {
+ struct nvmet_rdma_port *port = cm_id->context;
+
+ schedule_delayed_work(&port->repair_work, 0);
+ break;
+ }
+ /* FALLTHROUGH */
case RDMA_CM_EVENT_DISCONNECTED:
case RDMA_CM_EVENT_TIMEWAIT_EXIT:
nvmet_rdma_queue_disconnect(queue);
@@ -1504,42 +1519,19 @@ static void nvmet_rdma_delete_ctrl(struct
nvmet_ctrl *ctrl)
mutex_unlock(&nvmet_rdma_queue_mutex);
}
-static int nvmet_rdma_add_port(struct nvmet_port *port)
+static void nvmet_rdma_disable_port(struct nvmet_rdma_port *port)
{
- struct rdma_cm_id *cm_id;
- struct sockaddr_storage addr = { };
- __kernel_sa_family_t af;
- int ret;
+ struct rdma_cm_id *cm_id = xchg(&port->cm_id, NULL);
- switch (port->disc_addr.adrfam) {
- case NVMF_ADDR_FAMILY_IP4:
- af = AF_INET;
- break;
- case NVMF_ADDR_FAMILY_IP6:
- af = AF_INET6;
- break;
- default:
- pr_err("address family %d not supported\n",
- port->disc_addr.adrfam);
- return -EINVAL;
- }
-
- if (port->inline_data_size < 0) {
- port->inline_data_size =
NVMET_RDMA_DEFAULT_INLINE_DATA_SIZE;
- } else if (port->inline_data_size >
NVMET_RDMA_MAX_INLINE_DATA_SIZE) {
- pr_warn("inline_data_size %u is too large, reducing to
%u\n",
- port->inline_data_size,
- NVMET_RDMA_MAX_INLINE_DATA_SIZE);
- port->inline_data_size = NVMET_RDMA_MAX_INLINE_DATA_SIZE;
- }
+ if (cm_id)
+ rdma_destroy_id(cm_id);
+}
- ret = inet_pton_with_scope(&init_net, af, port->disc_addr.traddr,
- port->disc_addr.trsvcid, &addr);
- if (ret) {
- pr_err("malformed ip/port passed: %s:%s\n",
- port->disc_addr.traddr, port->disc_addr.trsvcid);
- return ret;
- }
+static int nvmet_rdma_enable_port(struct nvmet_rdma_port *port)
+{
+ struct sockaddr *addr = (struct sockaddr *)&port->addr;
+ struct rdma_cm_id *cm_id;
+ int ret;
cm_id = rdma_create_id(&init_net, nvmet_rdma_cm_handler, port,
RDMA_PS_TCP, IB_QPT_RC);
@@ -1558,23 +1550,19 @@ static int nvmet_rdma_add_port(struct nvmet_port
*port)
goto out_destroy_id;
}
- ret = rdma_bind_addr(cm_id, (struct sockaddr *)&addr);
+ ret = rdma_bind_addr(cm_id, addr);
if (ret) {
- pr_err("binding CM ID to %pISpcs failed (%d)\n",
- (struct sockaddr *)&addr, ret);
+ pr_err("binding CM ID to %pISpcs failed (%d)\n", addr, ret);
goto out_destroy_id;
}
ret = rdma_listen(cm_id, 128);
if (ret) {
- pr_err("listening to %pISpcs failed (%d)\n",
- (struct sockaddr *)&addr, ret);
+ pr_err("listening to %pISpcs failed (%d)\n", addr, ret);
goto out_destroy_id;
}
- pr_info("enabling port %d (%pISpcs)\n",
- le16_to_cpu(port->disc_addr.portid), (struct sockaddr
*)&addr);
- port->priv = cm_id;
+ port->cm_id = cm_id;
return 0;
out_destroy_id:
@@ -1582,18 +1570,92 @@ static int nvmet_rdma_add_port(struct nvmet_port
*port)
return ret;
}
-static void nvmet_rdma_remove_port(struct nvmet_port *port)
+static void nvmet_rdma_repair_port_work(struct work_struct *w)
{
- struct rdma_cm_id *cm_id = xchg(&port->priv, NULL);
+ struct nvmet_rdma_port *port = container_of(to_delayed_work(w),
+ struct nvmet_rdma_port, repair_work);
+ int ret;
- if (cm_id)
- rdma_destroy_id(cm_id);
+ nvmet_rdma_disable_port(port);
+ ret = nvmet_rdma_enable_port(port);
+ if (ret)
+ schedule_delayed_work(&port->repair_work, 5 * HZ);
+}
+
+static int nvmet_rdma_add_port(struct nvmet_port *nport)
+{
+ struct nvmet_rdma_port *port;
+ __kernel_sa_family_t af;
+ int ret;
+
+ port = kzalloc(sizeof(*port), GFP_KERNEL);
+ if (!port)
+ return -ENOMEM;
+
+ nport->priv = port;
+ port->nport = nport;
+ INIT_DELAYED_WORK(&port->repair_work, nvmet_rdma_repair_port_work);
+
+ switch (nport->disc_addr.adrfam) {
+ case NVMF_ADDR_FAMILY_IP4:
+ af = AF_INET;
+ break;
+ case NVMF_ADDR_FAMILY_IP6:
+ af = AF_INET6;
+ break;
+ default:
+ pr_err("address family %d not supported\n",
+ nport->disc_addr.adrfam);
+ ret = -EINVAL;
+ goto out_free_port;
+ }
+
+ if (nport->inline_data_size < 0) {
+ nport->inline_data_size =
NVMET_RDMA_DEFAULT_INLINE_DATA_SIZE;
+ } else if (nport->inline_data_size >
NVMET_RDMA_MAX_INLINE_DATA_SIZE) {
+ pr_warn("inline_data_size %u is too large, reducing to
%u\n",
+ nport->inline_data_size,
+ NVMET_RDMA_MAX_INLINE_DATA_SIZE);
+ nport->inline_data_size = NVMET_RDMA_MAX_INLINE_DATA_SIZE;
+ }
+
+ ret = inet_pton_with_scope(&init_net, af, nport->disc_addr.traddr,
+ nport->disc_addr.trsvcid, &port->addr);
+ if (ret) {
+ pr_err("malformed ip/port passed: %s:%s\n",
+ nport->disc_addr.traddr, nport->disc_addr.trsvcid);
+ goto out_free_port;
+ }
+
+ ret = nvmet_rdma_enable_port(port);
+ if(ret)
+ goto out_free_port;
+
+ pr_info("enabling port %d (%pISpcs)\n",
+ le16_to_cpu(nport->disc_addr.portid),
+ (struct sockaddr *)&port->addr);
+
+ return 0;
+
+out_free_port:
+ kfree(port);
+ return ret;
+}
+
+static void nvmet_rdma_remove_port(struct nvmet_port *nport)
+{
+ struct nvmet_rdma_port *port = nport->priv;
+
+ cancel_delayed_work_sync(&port->repair_work);
+ nvmet_rdma_disable_port(port);
+ kfree(port);
}
static void nvmet_rdma_disc_port_addr(struct nvmet_req *req,
- struct nvmet_port *port, char *traddr)
+ struct nvmet_port *nport, char *traddr)
{
- struct rdma_cm_id *cm_id = port->priv;
+ struct nvmet_rdma_port *port = nport->priv;
+ struct rdma_cm_id *cm_id = port->cm_id;
if (inet_addr_is_any((struct sockaddr
*)&cm_id->route.addr.src_addr)) {
struct nvmet_rdma_rsp *rsp =
@@ -1603,7 +1665,7 @@ static void nvmet_rdma_disc_port_addr(struct
nvmet_req *req,
sprintf(traddr, "%pISc", addr);
} else {
- memcpy(traddr, port->disc_addr.traddr, NVMF_TRADDR_SIZE);
+ memcpy(traddr, nport->disc_addr.traddr, NVMF_TRADDR_SIZE);
}
}
--
_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
^ permalink raw reply related [flat|nested] 19+ messages in thread
* RE: NULL pointer dereference in nvmet_rdma_queue_disconnect during bond failover
2020-03-31 0:21 ` Sagi Grimberg
@ 2020-03-31 8:16 ` Shlomi Nimrodi
2020-04-02 9:13 ` Alex Lyakas
1 sibling, 0 replies; 19+ messages in thread
From: Shlomi Nimrodi @ 2020-03-31 8:16 UTC (permalink / raw)
To: Sagi Grimberg, Alex Lyakas
Cc: Tom Wu, Nitzan Carmi, Israel Rukshin, linux-nvme, Max Gurtovoy
++
-----Original Message-----
From: Sagi Grimberg <sagi@grimberg.me>
Sent: Tuesday, March 31, 2020 03:22
To: Alex Lyakas <alex@zadara.com>
Cc: Tom Wu <tomwu@mellanox.com>; Max Gurtovoy <maxg@mellanox.com>; Israel Rukshin <israelr@mellanox.com>; linux-nvme <linux-nvme@lists.infradead.org>; Shlomi Nimrodi <shlomin@mellanox.com>
Subject: Re: NULL pointer dereference in nvmet_rdma_queue_disconnect during bond failover
Hey Alex,
>>> Alex, Max? did you retest this?
>>
>> Raising this from the ashes...
>>
>> Alex, did you test this patch?
>
> Raising from the ashes!
>
> In short: this patch fixes the issue!
Thanks for following up..
>
> More details:
>
> This patch doesn't apply on kernel 5.2. Moreover, I believe this patch
> is incomplete, because nvmet_rdma_find_get_device() needs to be fixed
> to treat cm_id->context as "struct nvmet_rdma_port" and not as "struct
> nvmet_port".
Does patch [1] apply on kernel 5.2?
> However, since we are working with kernel modules from Mellanox OFED,
> I tried applying this patch on OFED 4.7. I discovered that it already
> has almost everything this patch introduces. Like "struct
> nvmet_rdma_port" and the refactoring of nvmet_rdma_add_port into
> nvmet_rdma_enable_port, and nvmet_rdma_remove_port to
> nvmet_rdma_disable_port. I ended up with this patch [1].
>
> Tested bond failover, and cm_id is destroyed and re-created as
> expected [2]
>
> Israel, Max and other Mellanox folks: can we have this fix in OFED 4.9?
>
For MOFED issues you can follow-up with Max and Israel offline. If you can test upstream or even 5.2 stable that would be beneficial as I can add your Tested-by tag.
Thanks.
[1]:
--
Author: Sagi Grimberg <sagi@grimberg.me>
Date: Wed Jul 3 15:33:01 2019 -0700
nvmet-rdma: fix bonding failover possible NULL deref
RDMA_CM_EVENT_ADDR_CHANGE event occur in the case of bonding failover
on normal as well as on listening cm_ids. Hence this event will
immediately trigger a NULL dereference trying to disconnect a queue
for a cm_id that actually belongs to the port.
To fix this we provide a different handler for the listener cm_ids
that will defer a work to disable+(re)enable the port which essentially
destroys and setups another listener cm_id
Reported-by: Alex Lyakas <alex@zadara.com>
Signed-off-by: Sagi Grimberg <sagi@grimberg.me>
diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c index 9e1b8c61f54e..8dac89b7aa12 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -105,6 +105,13 @@ struct nvmet_rdma_queue {
struct list_head queue_list;
};
+struct nvmet_rdma_port {
+ struct nvmet_port *nport;
+ struct sockaddr_storage addr;
+ struct rdma_cm_id *cm_id;
+ struct delayed_work repair_work;
+};
+
struct nvmet_rdma_device {
struct ib_device *device;
struct ib_pd *pd;
@@ -1272,6 +1279,7 @@ static int nvmet_rdma_cm_accept(struct rdma_cm_id *cm_id,
static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id,
struct rdma_cm_event *event)
{
+ struct nvmet_rdma_port *port = cm_id->context;
struct nvmet_rdma_device *ndev;
struct nvmet_rdma_queue *queue;
int ret = -EINVAL;
@@ -1287,7 +1295,7 @@ static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id,
ret = -ENOMEM;
goto put_device;
}
- queue->port = cm_id->context;
+ queue->port = port->nport;
if (queue->host_qid == 0) {
/* Let inflight controller teardown complete */
@@ -1412,7 +1420,7 @@ static void nvmet_rdma_queue_connect_fail(struct
rdma_cm_id *cm_id,
static int nvmet_rdma_device_removal(struct rdma_cm_id *cm_id,
struct nvmet_rdma_queue *queue)
{
- struct nvmet_port *port;
+ struct nvmet_rdma_port *port;
if (queue) {
/*
@@ -1431,7 +1439,7 @@ static int nvmet_rdma_device_removal(struct
rdma_cm_id *cm_id,
* cm_id destroy. use atomic xchg to make sure
* we don't compete with remove_port.
*/
- if (xchg(&port->priv, NULL) != cm_id)
+ if (xchg(&port->cm_id, NULL) != cm_id)
return 0;
/*
@@ -1462,6 +1470,13 @@ static int nvmet_rdma_cm_handler(struct
rdma_cm_id *cm_id,
nvmet_rdma_queue_established(queue);
break;
case RDMA_CM_EVENT_ADDR_CHANGE:
+ if (!queue) {
+ struct nvmet_rdma_port *port = cm_id->context;
+
+ schedule_delayed_work(&port->repair_work, 0);
+ break;
+ }
+ /* FALLTHROUGH */
case RDMA_CM_EVENT_DISCONNECTED:
case RDMA_CM_EVENT_TIMEWAIT_EXIT:
nvmet_rdma_queue_disconnect(queue);
@@ -1504,42 +1519,19 @@ static void nvmet_rdma_delete_ctrl(struct
nvmet_ctrl *ctrl)
mutex_unlock(&nvmet_rdma_queue_mutex);
}
-static int nvmet_rdma_add_port(struct nvmet_port *port)
+static void nvmet_rdma_disable_port(struct nvmet_rdma_port *port)
{
- struct rdma_cm_id *cm_id;
- struct sockaddr_storage addr = { };
- __kernel_sa_family_t af;
- int ret;
+ struct rdma_cm_id *cm_id = xchg(&port->cm_id, NULL);
- switch (port->disc_addr.adrfam) {
- case NVMF_ADDR_FAMILY_IP4:
- af = AF_INET;
- break;
- case NVMF_ADDR_FAMILY_IP6:
- af = AF_INET6;
- break;
- default:
- pr_err("address family %d not supported\n",
- port->disc_addr.adrfam);
- return -EINVAL;
- }
-
- if (port->inline_data_size < 0) {
- port->inline_data_size =
NVMET_RDMA_DEFAULT_INLINE_DATA_SIZE;
- } else if (port->inline_data_size >
NVMET_RDMA_MAX_INLINE_DATA_SIZE) {
- pr_warn("inline_data_size %u is too large, reducing to
%u\n",
- port->inline_data_size,
- NVMET_RDMA_MAX_INLINE_DATA_SIZE);
- port->inline_data_size = NVMET_RDMA_MAX_INLINE_DATA_SIZE;
- }
+ if (cm_id)
+ rdma_destroy_id(cm_id);
+}
- ret = inet_pton_with_scope(&init_net, af, port->disc_addr.traddr,
- port->disc_addr.trsvcid, &addr);
- if (ret) {
- pr_err("malformed ip/port passed: %s:%s\n",
- port->disc_addr.traddr, port->disc_addr.trsvcid);
- return ret;
- }
+static int nvmet_rdma_enable_port(struct nvmet_rdma_port *port)
+{
+ struct sockaddr *addr = (struct sockaddr *)&port->addr;
+ struct rdma_cm_id *cm_id;
+ int ret;
cm_id = rdma_create_id(&init_net, nvmet_rdma_cm_handler, port,
RDMA_PS_TCP, IB_QPT_RC);
@@ -1558,23 +1550,19 @@ static int nvmet_rdma_add_port(struct nvmet_port
*port)
goto out_destroy_id;
}
- ret = rdma_bind_addr(cm_id, (struct sockaddr *)&addr);
+ ret = rdma_bind_addr(cm_id, addr);
if (ret) {
- pr_err("binding CM ID to %pISpcs failed (%d)\n",
- (struct sockaddr *)&addr, ret);
+ pr_err("binding CM ID to %pISpcs failed (%d)\n", addr, ret);
goto out_destroy_id;
}
ret = rdma_listen(cm_id, 128);
if (ret) {
- pr_err("listening to %pISpcs failed (%d)\n",
- (struct sockaddr *)&addr, ret);
+ pr_err("listening to %pISpcs failed (%d)\n", addr, ret);
goto out_destroy_id;
}
- pr_info("enabling port %d (%pISpcs)\n",
- le16_to_cpu(port->disc_addr.portid), (struct sockaddr
*)&addr);
- port->priv = cm_id;
+ port->cm_id = cm_id;
return 0;
out_destroy_id:
@@ -1582,18 +1570,92 @@ static int nvmet_rdma_add_port(struct nvmet_port
*port)
return ret;
}
-static void nvmet_rdma_remove_port(struct nvmet_port *port)
+static void nvmet_rdma_repair_port_work(struct work_struct *w)
{
- struct rdma_cm_id *cm_id = xchg(&port->priv, NULL);
+ struct nvmet_rdma_port *port = container_of(to_delayed_work(w),
+ struct nvmet_rdma_port, repair_work);
+ int ret;
- if (cm_id)
- rdma_destroy_id(cm_id);
+ nvmet_rdma_disable_port(port);
+ ret = nvmet_rdma_enable_port(port);
+ if (ret)
+ schedule_delayed_work(&port->repair_work, 5 * HZ);
+}
+
+static int nvmet_rdma_add_port(struct nvmet_port *nport)
+{
+ struct nvmet_rdma_port *port;
+ __kernel_sa_family_t af;
+ int ret;
+
+ port = kzalloc(sizeof(*port), GFP_KERNEL);
+ if (!port)
+ return -ENOMEM;
+
+ nport->priv = port;
+ port->nport = nport;
+ INIT_DELAYED_WORK(&port->repair_work, nvmet_rdma_repair_port_work);
+
+ switch (nport->disc_addr.adrfam) {
+ case NVMF_ADDR_FAMILY_IP4:
+ af = AF_INET;
+ break;
+ case NVMF_ADDR_FAMILY_IP6:
+ af = AF_INET6;
+ break;
+ default:
+ pr_err("address family %d not supported\n",
+ nport->disc_addr.adrfam);
+ ret = -EINVAL;
+ goto out_free_port;
+ }
+
+ if (nport->inline_data_size < 0) {
+ nport->inline_data_size =
NVMET_RDMA_DEFAULT_INLINE_DATA_SIZE;
+ } else if (nport->inline_data_size >
NVMET_RDMA_MAX_INLINE_DATA_SIZE) {
+ pr_warn("inline_data_size %u is too large, reducing to
%u\n",
+ nport->inline_data_size,
+ NVMET_RDMA_MAX_INLINE_DATA_SIZE);
+ nport->inline_data_size = NVMET_RDMA_MAX_INLINE_DATA_SIZE;
+ }
+
+ ret = inet_pton_with_scope(&init_net, af, nport->disc_addr.traddr,
+ nport->disc_addr.trsvcid, &port->addr);
+ if (ret) {
+ pr_err("malformed ip/port passed: %s:%s\n",
+ nport->disc_addr.traddr, nport->disc_addr.trsvcid);
+ goto out_free_port;
+ }
+
+ ret = nvmet_rdma_enable_port(port);
+ if(ret)
+ goto out_free_port;
+
+ pr_info("enabling port %d (%pISpcs)\n",
+ le16_to_cpu(nport->disc_addr.portid),
+ (struct sockaddr *)&port->addr);
+
+ return 0;
+
+out_free_port:
+ kfree(port);
+ return ret;
+}
+
+static void nvmet_rdma_remove_port(struct nvmet_port *nport)
+{
+ struct nvmet_rdma_port *port = nport->priv;
+
+ cancel_delayed_work_sync(&port->repair_work);
+ nvmet_rdma_disable_port(port);
+ kfree(port);
}
static void nvmet_rdma_disc_port_addr(struct nvmet_req *req,
- struct nvmet_port *port, char *traddr)
+ struct nvmet_port *nport, char *traddr)
{
- struct rdma_cm_id *cm_id = port->priv;
+ struct nvmet_rdma_port *port = nport->priv;
+ struct rdma_cm_id *cm_id = port->cm_id;
if (inet_addr_is_any((struct sockaddr
*)&cm_id->route.addr.src_addr)) {
struct nvmet_rdma_rsp *rsp =
@@ -1603,7 +1665,7 @@ static void nvmet_rdma_disc_port_addr(struct
nvmet_req *req,
sprintf(traddr, "%pISc", addr);
} else {
- memcpy(traddr, port->disc_addr.traddr, NVMF_TRADDR_SIZE);
+ memcpy(traddr, nport->disc_addr.traddr, NVMF_TRADDR_SIZE);
}
}
--
_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: NULL pointer dereference in nvmet_rdma_queue_disconnect during bond failover
2020-03-31 0:21 ` Sagi Grimberg
2020-03-31 8:16 ` Shlomi Nimrodi
@ 2020-04-02 9:13 ` Alex Lyakas
2020-04-02 15:08 ` Max Gurtovoy
1 sibling, 1 reply; 19+ messages in thread
From: Alex Lyakas @ 2020-04-02 9:13 UTC (permalink / raw)
To: Sagi Grimberg
Cc: Shlomi Nimrodi, tomwu, Israel Rukshin, linux-nvme, Max Gurtovoy
[-- Attachment #1: Type: text/plain, Size: 12348 bytes --]
Hi Sagi,
On Tue, Mar 31, 2020 at 3:21 AM Sagi Grimberg <sagi@grimberg.me> wrote:
>
> Hey Alex,
>
> >>> Alex, Max? did you retest this?
> >>
> >> Raising this from the ashes...
> >>
> >> Alex, did you test this patch?
> >
> > Raising from the ashes!
> >
> > In short: this patch fixes the issue!
>
> Thanks for following up..
>
> >
> > More details:
> >
> > This patch doesn't apply on kernel 5.2. Moreover, I believe this patch
> > is incomplete, because nvmet_rdma_find_get_device() needs to be fixed
> > to treat cm_id->context as "struct nvmet_rdma_port" and not as "struct
> > nvmet_port".
>
> Does patch [1] apply on kernel 5.2?
>
> > However, since we are working with kernel modules from Mellanox OFED,
> > I tried applying this patch on OFED 4.7. I discovered that it already
> > has almost everything this patch introduces. Like "struct
> > nvmet_rdma_port" and the refactoring of nvmet_rdma_add_port into
> > nvmet_rdma_enable_port, and nvmet_rdma_remove_port to
> > nvmet_rdma_disable_port. I ended up with this patch [1].
> >
> > Tested bond failover, and cm_id is destroyed and re-created as expected [2]
> >
> > Israel, Max and other Mellanox folks: can we have this fix in OFED 4.9?
> >
>
> For MOFED issues you can follow-up with Max and Israel offline. If you
> can test upstream or even 5.2 stable that would be beneficial as I can
> add your Tested-by tag.
>
This patch did not apply to latest 5.2 (5.2.21). All of 10 hunks
failed. I applied it manually, and also handled cm_id->context in
nvmet_rdma_find_get_device as I mentioned earlier. I am attaching the
patch that I tested on kernel 5.2.21 (target side). I confirm that
this patch fixes the bond failover issue.
Tested-by: Alex Lyakas <alex@zadara.com>
Max, will this help to deliver this fix upstream, so that we can get
it in MOFED 4.9?
Thanks,
Alex.
> Thanks.
>
> [1]:
> --
> Author: Sagi Grimberg <sagi@grimberg.me>
> Date: Wed Jul 3 15:33:01 2019 -0700
>
> nvmet-rdma: fix bonding failover possible NULL deref
>
> RDMA_CM_EVENT_ADDR_CHANGE event occur in the case of bonding failover
> on normal as well as on listening cm_ids. Hence this event will
> immediately trigger a NULL dereference trying to disconnect a queue
> for a cm_id that actually belongs to the port.
>
> To fix this we provide a different handler for the listener cm_ids
> that will defer a work to disable+(re)enable the port which essentially
> destroys and setups another listener cm_id
>
> Reported-by: Alex Lyakas <alex@zadara.com>
> Signed-off-by: Sagi Grimberg <sagi@grimberg.me>
>
> diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
> index 9e1b8c61f54e..8dac89b7aa12 100644
> --- a/drivers/nvme/target/rdma.c
> +++ b/drivers/nvme/target/rdma.c
> @@ -105,6 +105,13 @@ struct nvmet_rdma_queue {
> struct list_head queue_list;
> };
>
> +struct nvmet_rdma_port {
> + struct nvmet_port *nport;
> + struct sockaddr_storage addr;
> + struct rdma_cm_id *cm_id;
> + struct delayed_work repair_work;
> +};
> +
> struct nvmet_rdma_device {
> struct ib_device *device;
> struct ib_pd *pd;
> @@ -1272,6 +1279,7 @@ static int nvmet_rdma_cm_accept(struct rdma_cm_id
> *cm_id,
> static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id,
> struct rdma_cm_event *event)
> {
> + struct nvmet_rdma_port *port = cm_id->context;
> struct nvmet_rdma_device *ndev;
> struct nvmet_rdma_queue *queue;
> int ret = -EINVAL;
> @@ -1287,7 +1295,7 @@ static int nvmet_rdma_queue_connect(struct
> rdma_cm_id *cm_id,
> ret = -ENOMEM;
> goto put_device;
> }
> - queue->port = cm_id->context;
> + queue->port = port->nport;
>
> if (queue->host_qid == 0) {
> /* Let inflight controller teardown complete */
> @@ -1412,7 +1420,7 @@ static void nvmet_rdma_queue_connect_fail(struct
> rdma_cm_id *cm_id,
> static int nvmet_rdma_device_removal(struct rdma_cm_id *cm_id,
> struct nvmet_rdma_queue *queue)
> {
> - struct nvmet_port *port;
> + struct nvmet_rdma_port *port;
>
> if (queue) {
> /*
> @@ -1431,7 +1439,7 @@ static int nvmet_rdma_device_removal(struct
> rdma_cm_id *cm_id,
> * cm_id destroy. use atomic xchg to make sure
> * we don't compete with remove_port.
> */
> - if (xchg(&port->priv, NULL) != cm_id)
> + if (xchg(&port->cm_id, NULL) != cm_id)
> return 0;
>
> /*
> @@ -1462,6 +1470,13 @@ static int nvmet_rdma_cm_handler(struct
> rdma_cm_id *cm_id,
> nvmet_rdma_queue_established(queue);
> break;
> case RDMA_CM_EVENT_ADDR_CHANGE:
> + if (!queue) {
> + struct nvmet_rdma_port *port = cm_id->context;
> +
> + schedule_delayed_work(&port->repair_work, 0);
> + break;
> + }
> + /* FALLTHROUGH */
> case RDMA_CM_EVENT_DISCONNECTED:
> case RDMA_CM_EVENT_TIMEWAIT_EXIT:
> nvmet_rdma_queue_disconnect(queue);
> @@ -1504,42 +1519,19 @@ static void nvmet_rdma_delete_ctrl(struct
> nvmet_ctrl *ctrl)
> mutex_unlock(&nvmet_rdma_queue_mutex);
> }
>
> -static int nvmet_rdma_add_port(struct nvmet_port *port)
> +static void nvmet_rdma_disable_port(struct nvmet_rdma_port *port)
> {
> - struct rdma_cm_id *cm_id;
> - struct sockaddr_storage addr = { };
> - __kernel_sa_family_t af;
> - int ret;
> + struct rdma_cm_id *cm_id = xchg(&port->cm_id, NULL);
>
> - switch (port->disc_addr.adrfam) {
> - case NVMF_ADDR_FAMILY_IP4:
> - af = AF_INET;
> - break;
> - case NVMF_ADDR_FAMILY_IP6:
> - af = AF_INET6;
> - break;
> - default:
> - pr_err("address family %d not supported\n",
> - port->disc_addr.adrfam);
> - return -EINVAL;
> - }
> -
> - if (port->inline_data_size < 0) {
> - port->inline_data_size =
> NVMET_RDMA_DEFAULT_INLINE_DATA_SIZE;
> - } else if (port->inline_data_size >
> NVMET_RDMA_MAX_INLINE_DATA_SIZE) {
> - pr_warn("inline_data_size %u is too large, reducing to
> %u\n",
> - port->inline_data_size,
> - NVMET_RDMA_MAX_INLINE_DATA_SIZE);
> - port->inline_data_size = NVMET_RDMA_MAX_INLINE_DATA_SIZE;
> - }
> + if (cm_id)
> + rdma_destroy_id(cm_id);
> +}
>
> - ret = inet_pton_with_scope(&init_net, af, port->disc_addr.traddr,
> - port->disc_addr.trsvcid, &addr);
> - if (ret) {
> - pr_err("malformed ip/port passed: %s:%s\n",
> - port->disc_addr.traddr, port->disc_addr.trsvcid);
> - return ret;
> - }
> +static int nvmet_rdma_enable_port(struct nvmet_rdma_port *port)
> +{
> + struct sockaddr *addr = (struct sockaddr *)&port->addr;
> + struct rdma_cm_id *cm_id;
> + int ret;
>
> cm_id = rdma_create_id(&init_net, nvmet_rdma_cm_handler, port,
> RDMA_PS_TCP, IB_QPT_RC);
> @@ -1558,23 +1550,19 @@ static int nvmet_rdma_add_port(struct nvmet_port
> *port)
> goto out_destroy_id;
> }
>
> - ret = rdma_bind_addr(cm_id, (struct sockaddr *)&addr);
> + ret = rdma_bind_addr(cm_id, addr);
> if (ret) {
> - pr_err("binding CM ID to %pISpcs failed (%d)\n",
> - (struct sockaddr *)&addr, ret);
> + pr_err("binding CM ID to %pISpcs failed (%d)\n", addr, ret);
> goto out_destroy_id;
> }
>
> ret = rdma_listen(cm_id, 128);
> if (ret) {
> - pr_err("listening to %pISpcs failed (%d)\n",
> - (struct sockaddr *)&addr, ret);
> + pr_err("listening to %pISpcs failed (%d)\n", addr, ret);
> goto out_destroy_id;
> }
>
> - pr_info("enabling port %d (%pISpcs)\n",
> - le16_to_cpu(port->disc_addr.portid), (struct sockaddr
> *)&addr);
> - port->priv = cm_id;
> + port->cm_id = cm_id;
> return 0;
>
> out_destroy_id:
> @@ -1582,18 +1570,92 @@ static int nvmet_rdma_add_port(struct nvmet_port
> *port)
> return ret;
> }
>
> -static void nvmet_rdma_remove_port(struct nvmet_port *port)
> +static void nvmet_rdma_repair_port_work(struct work_struct *w)
> {
> - struct rdma_cm_id *cm_id = xchg(&port->priv, NULL);
> + struct nvmet_rdma_port *port = container_of(to_delayed_work(w),
> + struct nvmet_rdma_port, repair_work);
> + int ret;
>
> - if (cm_id)
> - rdma_destroy_id(cm_id);
> + nvmet_rdma_disable_port(port);
> + ret = nvmet_rdma_enable_port(port);
> + if (ret)
> + schedule_delayed_work(&port->repair_work, 5 * HZ);
> +}
> +
> +static int nvmet_rdma_add_port(struct nvmet_port *nport)
> +{
> + struct nvmet_rdma_port *port;
> + __kernel_sa_family_t af;
> + int ret;
> +
> + port = kzalloc(sizeof(*port), GFP_KERNEL);
> + if (!port)
> + return -ENOMEM;
> +
> + nport->priv = port;
> + port->nport = nport;
> + INIT_DELAYED_WORK(&port->repair_work, nvmet_rdma_repair_port_work);
> +
> + switch (nport->disc_addr.adrfam) {
> + case NVMF_ADDR_FAMILY_IP4:
> + af = AF_INET;
> + break;
> + case NVMF_ADDR_FAMILY_IP6:
> + af = AF_INET6;
> + break;
> + default:
> + pr_err("address family %d not supported\n",
> + nport->disc_addr.adrfam);
> + ret = -EINVAL;
> + goto out_free_port;
> + }
> +
> + if (nport->inline_data_size < 0) {
> + nport->inline_data_size =
> NVMET_RDMA_DEFAULT_INLINE_DATA_SIZE;
> + } else if (nport->inline_data_size >
> NVMET_RDMA_MAX_INLINE_DATA_SIZE) {
> + pr_warn("inline_data_size %u is too large, reducing to
> %u\n",
> + nport->inline_data_size,
> + NVMET_RDMA_MAX_INLINE_DATA_SIZE);
> + nport->inline_data_size = NVMET_RDMA_MAX_INLINE_DATA_SIZE;
> + }
> +
> + ret = inet_pton_with_scope(&init_net, af, nport->disc_addr.traddr,
> + nport->disc_addr.trsvcid, &port->addr);
> + if (ret) {
> + pr_err("malformed ip/port passed: %s:%s\n",
> + nport->disc_addr.traddr, nport->disc_addr.trsvcid);
> + goto out_free_port;
> + }
> +
> + ret = nvmet_rdma_enable_port(port);
> + if(ret)
> + goto out_free_port;
> +
> + pr_info("enabling port %d (%pISpcs)\n",
> + le16_to_cpu(nport->disc_addr.portid),
> + (struct sockaddr *)&port->addr);
> +
> + return 0;
> +
> +out_free_port:
> + kfree(port);
> + return ret;
> +}
> +
> +static void nvmet_rdma_remove_port(struct nvmet_port *nport)
> +{
> + struct nvmet_rdma_port *port = nport->priv;
> +
> + cancel_delayed_work_sync(&port->repair_work);
> + nvmet_rdma_disable_port(port);
> + kfree(port);
> }
>
> static void nvmet_rdma_disc_port_addr(struct nvmet_req *req,
> - struct nvmet_port *port, char *traddr)
> + struct nvmet_port *nport, char *traddr)
> {
> - struct rdma_cm_id *cm_id = port->priv;
> + struct nvmet_rdma_port *port = nport->priv;
> + struct rdma_cm_id *cm_id = port->cm_id;
>
> if (inet_addr_is_any((struct sockaddr
> *)&cm_id->route.addr.src_addr)) {
> struct nvmet_rdma_rsp *rsp =
> @@ -1603,7 +1665,7 @@ static void nvmet_rdma_disc_port_addr(struct
> nvmet_req *req,
>
> sprintf(traddr, "%pISc", addr);
> } else {
> - memcpy(traddr, port->disc_addr.traddr, NVMF_TRADDR_SIZE);
> + memcpy(traddr, nport->disc_addr.traddr, NVMF_TRADDR_SIZE);
> }
> }
> --
[-- Attachment #2: 0001-nvmet-rdma-fix-bonding-failover-possible-NULL-deref.5.2.patch --]
[-- Type: application/octet-stream, Size: 8809 bytes --]
diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index 36d906a..ebe7c43 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -102,6 +102,13 @@ struct nvmet_rdma_queue {
struct list_head queue_list;
};
+struct nvmet_rdma_port {
+ struct nvmet_port *nport;
+ struct sockaddr_storage addr;
+ struct rdma_cm_id *cm_id;
+ struct delayed_work repair_work;
+};
+
struct nvmet_rdma_device {
struct ib_device *device;
struct ib_pd *pd;
@@ -914,7 +921,8 @@ static void nvmet_rdma_free_dev(struct kref *ref)
static struct nvmet_rdma_device *
nvmet_rdma_find_get_device(struct rdma_cm_id *cm_id)
{
- struct nvmet_port *port = cm_id->context;
+ struct nvmet_rdma_port *port = cm_id->context;
+ struct nvmet_port *nport = port->nport;
struct nvmet_rdma_device *ndev;
int inline_page_count;
int inline_sge_count;
@@ -931,17 +939,17 @@ static void nvmet_rdma_free_dev(struct kref *ref)
if (!ndev)
goto out_err;
- inline_page_count = num_pages(port->inline_data_size);
+ inline_page_count = num_pages(nport->inline_data_size);
inline_sge_count = max(cm_id->device->attrs.max_sge_rd,
cm_id->device->attrs.max_recv_sge) - 1;
if (inline_page_count > inline_sge_count) {
pr_warn("inline_data_size %d cannot be supported by device %s. Reducing to %lu.\n",
- port->inline_data_size, cm_id->device->name,
+ nport->inline_data_size, cm_id->device->name,
inline_sge_count * PAGE_SIZE);
- port->inline_data_size = inline_sge_count * PAGE_SIZE;
+ nport->inline_data_size = inline_sge_count * PAGE_SIZE;
inline_page_count = inline_sge_count;
}
- ndev->inline_data_size = port->inline_data_size;
+ ndev->inline_data_size = nport->inline_data_size;
ndev->inline_page_count = inline_page_count;
ndev->device = cm_id->device;
kref_init(&ndev->ref);
@@ -1267,6 +1275,7 @@ static int nvmet_rdma_cm_accept(struct rdma_cm_id *cm_id,
static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id,
struct rdma_cm_event *event)
{
+ struct nvmet_rdma_port *port = cm_id->context;
struct nvmet_rdma_device *ndev;
struct nvmet_rdma_queue *queue;
int ret = -EINVAL;
@@ -1282,7 +1291,7 @@ static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id,
ret = -ENOMEM;
goto put_device;
}
- queue->port = cm_id->context;
+ queue->port = port->nport;
if (queue->host_qid == 0) {
/* Let inflight controller teardown complete */
@@ -1407,7 +1416,7 @@ static void nvmet_rdma_queue_connect_fail(struct rdma_cm_id *cm_id,
static int nvmet_rdma_device_removal(struct rdma_cm_id *cm_id,
struct nvmet_rdma_queue *queue)
{
- struct nvmet_port *port;
+ struct nvmet_rdma_port *port;
if (queue) {
/*
@@ -1426,7 +1435,7 @@ static int nvmet_rdma_device_removal(struct rdma_cm_id *cm_id,
* cm_id destroy. use atomic xchg to make sure
* we don't compete with remove_port.
*/
- if (xchg(&port->priv, NULL) != cm_id)
+ if (xchg(&port->cm_id, NULL) != cm_id)
return 0;
/*
@@ -1457,6 +1466,13 @@ static int nvmet_rdma_cm_handler(struct rdma_cm_id *cm_id,
nvmet_rdma_queue_established(queue);
break;
case RDMA_CM_EVENT_ADDR_CHANGE:
+ if (!queue) {
+ struct nvmet_rdma_port *port = cm_id->context;
+
+ schedule_delayed_work(&port->repair_work, 0);
+ break;
+ }
+ /* FALLTHROUGH */
case RDMA_CM_EVENT_DISCONNECTED:
case RDMA_CM_EVENT_TIMEWAIT_EXIT:
nvmet_rdma_queue_disconnect(queue);
@@ -1499,42 +1515,19 @@ static void nvmet_rdma_delete_ctrl(struct nvmet_ctrl *ctrl)
mutex_unlock(&nvmet_rdma_queue_mutex);
}
-static int nvmet_rdma_add_port(struct nvmet_port *port)
+static void nvmet_rdma_disable_port(struct nvmet_rdma_port *port)
{
- struct rdma_cm_id *cm_id;
- struct sockaddr_storage addr = { };
- __kernel_sa_family_t af;
- int ret;
-
- switch (port->disc_addr.adrfam) {
- case NVMF_ADDR_FAMILY_IP4:
- af = AF_INET;
- break;
- case NVMF_ADDR_FAMILY_IP6:
- af = AF_INET6;
- break;
- default:
- pr_err("address family %d not supported\n",
- port->disc_addr.adrfam);
- return -EINVAL;
- }
+ struct rdma_cm_id *cm_id = xchg(&port->cm_id, NULL);
- if (port->inline_data_size < 0) {
- port->inline_data_size = NVMET_RDMA_DEFAULT_INLINE_DATA_SIZE;
- } else if (port->inline_data_size > NVMET_RDMA_MAX_INLINE_DATA_SIZE) {
- pr_warn("inline_data_size %u is too large, reducing to %u\n",
- port->inline_data_size,
- NVMET_RDMA_MAX_INLINE_DATA_SIZE);
- port->inline_data_size = NVMET_RDMA_MAX_INLINE_DATA_SIZE;
- }
+ if (cm_id)
+ rdma_destroy_id(cm_id);
+}
- ret = inet_pton_with_scope(&init_net, af, port->disc_addr.traddr,
- port->disc_addr.trsvcid, &addr);
- if (ret) {
- pr_err("malformed ip/port passed: %s:%s\n",
- port->disc_addr.traddr, port->disc_addr.trsvcid);
- return ret;
- }
+static int nvmet_rdma_enable_port(struct nvmet_rdma_port *port)
+{
+ struct sockaddr *addr = (struct sockaddr *)&port->addr;
+ struct rdma_cm_id *cm_id;
+ int ret;
cm_id = rdma_create_id(&init_net, nvmet_rdma_cm_handler, port,
RDMA_PS_TCP, IB_QPT_RC);
@@ -1553,23 +1546,19 @@ static int nvmet_rdma_add_port(struct nvmet_port *port)
goto out_destroy_id;
}
- ret = rdma_bind_addr(cm_id, (struct sockaddr *)&addr);
+ ret = rdma_bind_addr(cm_id, addr);
if (ret) {
- pr_err("binding CM ID to %pISpcs failed (%d)\n",
- (struct sockaddr *)&addr, ret);
+ pr_err("binding CM ID to %pISpcs failed (%d)\n", addr, ret);
goto out_destroy_id;
}
ret = rdma_listen(cm_id, 128);
if (ret) {
- pr_err("listening to %pISpcs failed (%d)\n",
- (struct sockaddr *)&addr, ret);
+ pr_err("listening to %pISpcs failed (%d)\n", addr, ret);
goto out_destroy_id;
}
- pr_info("enabling port %d (%pISpcs)\n",
- le16_to_cpu(port->disc_addr.portid), (struct sockaddr *)&addr);
- port->priv = cm_id;
+ port->cm_id = cm_id;
return 0;
out_destroy_id:
@@ -1577,18 +1566,92 @@ static int nvmet_rdma_add_port(struct nvmet_port *port)
return ret;
}
-static void nvmet_rdma_remove_port(struct nvmet_port *port)
+static void nvmet_rdma_repair_port_work(struct work_struct *w)
{
- struct rdma_cm_id *cm_id = xchg(&port->priv, NULL);
+ struct nvmet_rdma_port *port = container_of(to_delayed_work(w),
+ struct nvmet_rdma_port, repair_work);
+ int ret;
- if (cm_id)
- rdma_destroy_id(cm_id);
+ nvmet_rdma_disable_port(port);
+ ret = nvmet_rdma_enable_port(port);
+ if (ret)
+ schedule_delayed_work(&port->repair_work, 5 * HZ);
+}
+
+static int nvmet_rdma_add_port(struct nvmet_port *nport)
+{
+ struct nvmet_rdma_port *port;
+ __kernel_sa_family_t af;
+ int ret;
+
+ port = kzalloc(sizeof(*port), GFP_KERNEL);
+ if (!port)
+ return -ENOMEM;
+
+ nport->priv = port;
+ port->nport = nport;
+ INIT_DELAYED_WORK(&port->repair_work, nvmet_rdma_repair_port_work);
+
+ switch (nport->disc_addr.adrfam) {
+ case NVMF_ADDR_FAMILY_IP4:
+ af = AF_INET;
+ break;
+ case NVMF_ADDR_FAMILY_IP6:
+ af = AF_INET6;
+ break;
+ default:
+ pr_err("address family %d not supported\n",
+ nport->disc_addr.adrfam);
+ ret = -EINVAL;
+ goto out_free_port;
+ }
+
+ if (nport->inline_data_size < 0) {
+ nport->inline_data_size = NVMET_RDMA_DEFAULT_INLINE_DATA_SIZE;
+ } else if (nport->inline_data_size > NVMET_RDMA_MAX_INLINE_DATA_SIZE) {
+ pr_warn("inline_data_size %u is too large, reducing to %u\n",
+ nport->inline_data_size,
+ NVMET_RDMA_MAX_INLINE_DATA_SIZE);
+ nport->inline_data_size = NVMET_RDMA_MAX_INLINE_DATA_SIZE;
+ }
+
+ ret = inet_pton_with_scope(&init_net, af, nport->disc_addr.traddr,
+ nport->disc_addr.trsvcid, &port->addr);
+ if (ret) {
+ pr_err("malformed ip/port passed: %s:%s\n",
+ nport->disc_addr.traddr, nport->disc_addr.trsvcid);
+ goto out_free_port;
+ }
+
+ ret = nvmet_rdma_enable_port(port);
+ if(ret)
+ goto out_free_port;
+
+ pr_info("enabling port %d (%pISpcs)\n",
+ le16_to_cpu(nport->disc_addr.portid),
+ (struct sockaddr *)&port->addr);
+
+ return 0;
+
+out_free_port:
+ kfree(port);
+ return ret;
+}
+
+static void nvmet_rdma_remove_port(struct nvmet_port *nport)
+{
+ struct nvmet_rdma_port *port = nport->priv;
+
+ cancel_delayed_work_sync(&port->repair_work);
+ nvmet_rdma_disable_port(port);
+ kfree(port);
}
static void nvmet_rdma_disc_port_addr(struct nvmet_req *req,
- struct nvmet_port *port, char *traddr)
+ struct nvmet_port *nport, char *traddr)
{
- struct rdma_cm_id *cm_id = port->priv;
+ struct nvmet_rdma_port *port = nport->priv;
+ struct rdma_cm_id *cm_id = port->cm_id;
if (inet_addr_is_any((struct sockaddr *)&cm_id->route.addr.src_addr)) {
struct nvmet_rdma_rsp *rsp =
@@ -1598,7 +1661,7 @@ static void nvmet_rdma_disc_port_addr(struct nvmet_req *req,
sprintf(traddr, "%pISc", addr);
} else {
- memcpy(traddr, port->disc_addr.traddr, NVMF_TRADDR_SIZE);
+ memcpy(traddr, nport->disc_addr.traddr, NVMF_TRADDR_SIZE);
}
}
[-- Attachment #3: Type: text/plain, Size: 158 bytes --]
_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: NULL pointer dereference in nvmet_rdma_queue_disconnect during bond failover
2020-04-02 9:13 ` Alex Lyakas
@ 2020-04-02 15:08 ` Max Gurtovoy
2020-04-02 15:26 ` Sagi Grimberg
0 siblings, 1 reply; 19+ messages in thread
From: Max Gurtovoy @ 2020-04-02 15:08 UTC (permalink / raw)
To: Alex Lyakas, Sagi Grimberg
Cc: Shlomi Nimrodi, tomwu, linux-nvme, Israel Rukshin
On 4/2/2020 12:13 PM, Alex Lyakas wrote:
> Hi Sagi,
>
> On Tue, Mar 31, 2020 at 3:21 AM Sagi Grimberg <sagi@grimberg.me> wrote:
>> Hey Alex,
>>
>>>>> Alex, Max? did you retest this?
>>>> Raising this from the ashes...
>>>>
>>>> Alex, did you test this patch?
>>> Raising from the ashes!
>>>
>>> In short: this patch fixes the issue!
>> Thanks for following up..
>>
>>> More details:
>>>
>>> This patch doesn't apply on kernel 5.2. Moreover, I believe this patch
>>> is incomplete, because nvmet_rdma_find_get_device() needs to be fixed
>>> to treat cm_id->context as "struct nvmet_rdma_port" and not as "struct
>>> nvmet_port".
>> Does patch [1] apply on kernel 5.2?
>>
>>> However, since we are working with kernel modules from Mellanox OFED,
>>> I tried applying this patch on OFED 4.7. I discovered that it already
>>> has almost everything this patch introduces. Like "struct
>>> nvmet_rdma_port" and the refactoring of nvmet_rdma_add_port into
>>> nvmet_rdma_enable_port, and nvmet_rdma_remove_port to
>>> nvmet_rdma_disable_port. I ended up with this patch [1].
>>>
>>> Tested bond failover, and cm_id is destroyed and re-created as expected [2]
>>>
>>> Israel, Max and other Mellanox folks: can we have this fix in OFED 4.9?
>>>
>> For MOFED issues you can follow-up with Max and Israel offline. If you
>> can test upstream or even 5.2 stable that would be beneficial as I can
>> add your Tested-by tag.
>>
> This patch did not apply to latest 5.2 (5.2.21). All of 10 hunks
> failed. I applied it manually, and also handled cm_id->context in
> nvmet_rdma_find_get_device as I mentioned earlier. I am attaching the
> patch that I tested on kernel 5.2.21 (target side). I confirm that
> this patch fixes the bond failover issue.
>
> Tested-by: Alex Lyakas <alex@zadara.com>
>
> Max, will this help to deliver this fix upstream, so that we can get
> it in MOFED 4.9?
>
> Thanks,
> Alex.
Alex,
Thanks for testing this.
Waiting for Sagi's official rebased version for doing full review.
We can take it on us to send the rebased, reviewed and validated version
(Sagi - let me know what you prefer).
For mofed - this is not the forum to discuss Mellanox SW release (let's
take it offline).
>
>
>> Thanks.
>>
>> [1]:
>> --
>> Author: Sagi Grimberg <sagi@grimberg.me>
>> Date: Wed Jul 3 15:33:01 2019 -0700
>>
>> nvmet-rdma: fix bonding failover possible NULL deref
>>
>> RDMA_CM_EVENT_ADDR_CHANGE event occur in the case of bonding failover
>> on normal as well as on listening cm_ids. Hence this event will
>> immediately trigger a NULL dereference trying to disconnect a queue
>> for a cm_id that actually belongs to the port.
>>
>> To fix this we provide a different handler for the listener cm_ids
>> that will defer a work to disable+(re)enable the port which essentially
>> destroys and setups another listener cm_id
>>
>> Reported-by: Alex Lyakas <alex@zadara.com>
>> Signed-off-by: Sagi Grimberg <sagi@grimberg.me>
>>
>> diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
>> index 9e1b8c61f54e..8dac89b7aa12 100644
>> --- a/drivers/nvme/target/rdma.c
>> +++ b/drivers/nvme/target/rdma.c
>> @@ -105,6 +105,13 @@ struct nvmet_rdma_queue {
>> struct list_head queue_list;
>> };
>>
>> +struct nvmet_rdma_port {
>> + struct nvmet_port *nport;
>> + struct sockaddr_storage addr;
>> + struct rdma_cm_id *cm_id;
>> + struct delayed_work repair_work;
>> +};
>> +
>> struct nvmet_rdma_device {
>> struct ib_device *device;
>> struct ib_pd *pd;
>> @@ -1272,6 +1279,7 @@ static int nvmet_rdma_cm_accept(struct rdma_cm_id
>> *cm_id,
>> static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id,
>> struct rdma_cm_event *event)
>> {
>> + struct nvmet_rdma_port *port = cm_id->context;
>> struct nvmet_rdma_device *ndev;
>> struct nvmet_rdma_queue *queue;
>> int ret = -EINVAL;
>> @@ -1287,7 +1295,7 @@ static int nvmet_rdma_queue_connect(struct
>> rdma_cm_id *cm_id,
>> ret = -ENOMEM;
>> goto put_device;
>> }
>> - queue->port = cm_id->context;
>> + queue->port = port->nport;
>>
>> if (queue->host_qid == 0) {
>> /* Let inflight controller teardown complete */
>> @@ -1412,7 +1420,7 @@ static void nvmet_rdma_queue_connect_fail(struct
>> rdma_cm_id *cm_id,
>> static int nvmet_rdma_device_removal(struct rdma_cm_id *cm_id,
>> struct nvmet_rdma_queue *queue)
>> {
>> - struct nvmet_port *port;
>> + struct nvmet_rdma_port *port;
>>
>> if (queue) {
>> /*
>> @@ -1431,7 +1439,7 @@ static int nvmet_rdma_device_removal(struct
>> rdma_cm_id *cm_id,
>> * cm_id destroy. use atomic xchg to make sure
>> * we don't compete with remove_port.
>> */
>> - if (xchg(&port->priv, NULL) != cm_id)
>> + if (xchg(&port->cm_id, NULL) != cm_id)
>> return 0;
>>
>> /*
>> @@ -1462,6 +1470,13 @@ static int nvmet_rdma_cm_handler(struct
>> rdma_cm_id *cm_id,
>> nvmet_rdma_queue_established(queue);
>> break;
>> case RDMA_CM_EVENT_ADDR_CHANGE:
>> + if (!queue) {
>> + struct nvmet_rdma_port *port = cm_id->context;
>> +
>> + schedule_delayed_work(&port->repair_work, 0);
>> + break;
>> + }
>> + /* FALLTHROUGH */
>> case RDMA_CM_EVENT_DISCONNECTED:
>> case RDMA_CM_EVENT_TIMEWAIT_EXIT:
>> nvmet_rdma_queue_disconnect(queue);
>> @@ -1504,42 +1519,19 @@ static void nvmet_rdma_delete_ctrl(struct
>> nvmet_ctrl *ctrl)
>> mutex_unlock(&nvmet_rdma_queue_mutex);
>> }
>>
>> -static int nvmet_rdma_add_port(struct nvmet_port *port)
>> +static void nvmet_rdma_disable_port(struct nvmet_rdma_port *port)
>> {
>> - struct rdma_cm_id *cm_id;
>> - struct sockaddr_storage addr = { };
>> - __kernel_sa_family_t af;
>> - int ret;
>> + struct rdma_cm_id *cm_id = xchg(&port->cm_id, NULL);
>>
>> - switch (port->disc_addr.adrfam) {
>> - case NVMF_ADDR_FAMILY_IP4:
>> - af = AF_INET;
>> - break;
>> - case NVMF_ADDR_FAMILY_IP6:
>> - af = AF_INET6;
>> - break;
>> - default:
>> - pr_err("address family %d not supported\n",
>> - port->disc_addr.adrfam);
>> - return -EINVAL;
>> - }
>> -
>> - if (port->inline_data_size < 0) {
>> - port->inline_data_size =
>> NVMET_RDMA_DEFAULT_INLINE_DATA_SIZE;
>> - } else if (port->inline_data_size >
>> NVMET_RDMA_MAX_INLINE_DATA_SIZE) {
>> - pr_warn("inline_data_size %u is too large, reducing to
>> %u\n",
>> - port->inline_data_size,
>> - NVMET_RDMA_MAX_INLINE_DATA_SIZE);
>> - port->inline_data_size = NVMET_RDMA_MAX_INLINE_DATA_SIZE;
>> - }
>> + if (cm_id)
>> + rdma_destroy_id(cm_id);
>> +}
>>
>> - ret = inet_pton_with_scope(&init_net, af, port->disc_addr.traddr,
>> - port->disc_addr.trsvcid, &addr);
>> - if (ret) {
>> - pr_err("malformed ip/port passed: %s:%s\n",
>> - port->disc_addr.traddr, port->disc_addr.trsvcid);
>> - return ret;
>> - }
>> +static int nvmet_rdma_enable_port(struct nvmet_rdma_port *port)
>> +{
>> + struct sockaddr *addr = (struct sockaddr *)&port->addr;
>> + struct rdma_cm_id *cm_id;
>> + int ret;
>>
>> cm_id = rdma_create_id(&init_net, nvmet_rdma_cm_handler, port,
>> RDMA_PS_TCP, IB_QPT_RC);
>> @@ -1558,23 +1550,19 @@ static int nvmet_rdma_add_port(struct nvmet_port
>> *port)
>> goto out_destroy_id;
>> }
>>
>> - ret = rdma_bind_addr(cm_id, (struct sockaddr *)&addr);
>> + ret = rdma_bind_addr(cm_id, addr);
>> if (ret) {
>> - pr_err("binding CM ID to %pISpcs failed (%d)\n",
>> - (struct sockaddr *)&addr, ret);
>> + pr_err("binding CM ID to %pISpcs failed (%d)\n", addr, ret);
>> goto out_destroy_id;
>> }
>>
>> ret = rdma_listen(cm_id, 128);
>> if (ret) {
>> - pr_err("listening to %pISpcs failed (%d)\n",
>> - (struct sockaddr *)&addr, ret);
>> + pr_err("listening to %pISpcs failed (%d)\n", addr, ret);
>> goto out_destroy_id;
>> }
>>
>> - pr_info("enabling port %d (%pISpcs)\n",
>> - le16_to_cpu(port->disc_addr.portid), (struct sockaddr
>> *)&addr);
>> - port->priv = cm_id;
>> + port->cm_id = cm_id;
>> return 0;
>>
>> out_destroy_id:
>> @@ -1582,18 +1570,92 @@ static int nvmet_rdma_add_port(struct nvmet_port
>> *port)
>> return ret;
>> }
>>
>> -static void nvmet_rdma_remove_port(struct nvmet_port *port)
>> +static void nvmet_rdma_repair_port_work(struct work_struct *w)
>> {
>> - struct rdma_cm_id *cm_id = xchg(&port->priv, NULL);
>> + struct nvmet_rdma_port *port = container_of(to_delayed_work(w),
>> + struct nvmet_rdma_port, repair_work);
>> + int ret;
>>
>> - if (cm_id)
>> - rdma_destroy_id(cm_id);
>> + nvmet_rdma_disable_port(port);
>> + ret = nvmet_rdma_enable_port(port);
>> + if (ret)
>> + schedule_delayed_work(&port->repair_work, 5 * HZ);
>> +}
>> +
>> +static int nvmet_rdma_add_port(struct nvmet_port *nport)
>> +{
>> + struct nvmet_rdma_port *port;
>> + __kernel_sa_family_t af;
>> + int ret;
>> +
>> + port = kzalloc(sizeof(*port), GFP_KERNEL);
>> + if (!port)
>> + return -ENOMEM;
>> +
>> + nport->priv = port;
>> + port->nport = nport;
>> + INIT_DELAYED_WORK(&port->repair_work, nvmet_rdma_repair_port_work);
>> +
>> + switch (nport->disc_addr.adrfam) {
>> + case NVMF_ADDR_FAMILY_IP4:
>> + af = AF_INET;
>> + break;
>> + case NVMF_ADDR_FAMILY_IP6:
>> + af = AF_INET6;
>> + break;
>> + default:
>> + pr_err("address family %d not supported\n",
>> + nport->disc_addr.adrfam);
>> + ret = -EINVAL;
>> + goto out_free_port;
>> + }
>> +
>> + if (nport->inline_data_size < 0) {
>> + nport->inline_data_size =
>> NVMET_RDMA_DEFAULT_INLINE_DATA_SIZE;
>> + } else if (nport->inline_data_size >
>> NVMET_RDMA_MAX_INLINE_DATA_SIZE) {
>> + pr_warn("inline_data_size %u is too large, reducing to
>> %u\n",
>> + nport->inline_data_size,
>> + NVMET_RDMA_MAX_INLINE_DATA_SIZE);
>> + nport->inline_data_size = NVMET_RDMA_MAX_INLINE_DATA_SIZE;
>> + }
>> +
>> + ret = inet_pton_with_scope(&init_net, af, nport->disc_addr.traddr,
>> + nport->disc_addr.trsvcid, &port->addr);
>> + if (ret) {
>> + pr_err("malformed ip/port passed: %s:%s\n",
>> + nport->disc_addr.traddr, nport->disc_addr.trsvcid);
>> + goto out_free_port;
>> + }
>> +
>> + ret = nvmet_rdma_enable_port(port);
>> + if(ret)
>> + goto out_free_port;
>> +
>> + pr_info("enabling port %d (%pISpcs)\n",
>> + le16_to_cpu(nport->disc_addr.portid),
>> + (struct sockaddr *)&port->addr);
>> +
>> + return 0;
>> +
>> +out_free_port:
>> + kfree(port);
>> + return ret;
>> +}
>> +
>> +static void nvmet_rdma_remove_port(struct nvmet_port *nport)
>> +{
>> + struct nvmet_rdma_port *port = nport->priv;
>> +
>> + cancel_delayed_work_sync(&port->repair_work);
>> + nvmet_rdma_disable_port(port);
>> + kfree(port);
>> }
>>
>> static void nvmet_rdma_disc_port_addr(struct nvmet_req *req,
>> - struct nvmet_port *port, char *traddr)
>> + struct nvmet_port *nport, char *traddr)
>> {
>> - struct rdma_cm_id *cm_id = port->priv;
>> + struct nvmet_rdma_port *port = nport->priv;
>> + struct rdma_cm_id *cm_id = port->cm_id;
>>
>> if (inet_addr_is_any((struct sockaddr
>> *)&cm_id->route.addr.src_addr)) {
>> struct nvmet_rdma_rsp *rsp =
>> @@ -1603,7 +1665,7 @@ static void nvmet_rdma_disc_port_addr(struct
>> nvmet_req *req,
>>
>> sprintf(traddr, "%pISc", addr);
>> } else {
>> - memcpy(traddr, port->disc_addr.traddr, NVMF_TRADDR_SIZE);
>> + memcpy(traddr, nport->disc_addr.traddr, NVMF_TRADDR_SIZE);
>> }
>> }
>> --
_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: NULL pointer dereference in nvmet_rdma_queue_disconnect during bond failover
2020-04-02 15:08 ` Max Gurtovoy
@ 2020-04-02 15:26 ` Sagi Grimberg
2020-04-05 11:11 ` Shlomi Nimrodi
0 siblings, 1 reply; 19+ messages in thread
From: Sagi Grimberg @ 2020-04-02 15:26 UTC (permalink / raw)
To: Max Gurtovoy, Alex Lyakas
Cc: Shlomi Nimrodi, tomwu, linux-nvme, Israel Rukshin
>> This patch did not apply to latest 5.2 (5.2.21). All of 10 hunks
>> failed.
It applies on branch linux-5.2.y in the stable tree. What are you
using?
>> I applied it manually, and also handled cm_id->context in
>> nvmet_rdma_find_get_device as I mentioned earlier. I am attaching the
>> patch that I tested on kernel 5.2.21 (target side). I confirm that
>> this patch fixes the bond failover issue.
>>
>> Tested-by: Alex Lyakas <alex@zadara.com>
Cool.
>>
>> Max, will this help to deliver this fix upstream, so that we can get
>> it in MOFED 4.9?
>>
>> Thanks,
>> Alex.
>
> Alex,
>
> Thanks for testing this.
>
> Waiting for Sagi's official rebased version for doing full review.
I'll send a patch...
_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
^ permalink raw reply [flat|nested] 19+ messages in thread
* RE: NULL pointer dereference in nvmet_rdma_queue_disconnect during bond failover
2020-04-02 15:26 ` Sagi Grimberg
@ 2020-04-05 11:11 ` Shlomi Nimrodi
0 siblings, 0 replies; 19+ messages in thread
From: Shlomi Nimrodi @ 2020-04-05 11:11 UTC (permalink / raw)
To: Sagi Grimberg, Max Gurtovoy, Alex Lyakas
Cc: Tom Wu, Nitzan Carmi, linux-nvme, Israel Rukshin
++
-----Original Message-----
From: Sagi Grimberg <sagi@grimberg.me>
Sent: Thursday, April 2, 2020 18:27
To: Max Gurtovoy <maxg@mellanox.com>; Alex Lyakas <alex@zadara.com>
Cc: Tom Wu <tomwu@mellanox.com>; Israel Rukshin <israelr@mellanox.com>; linux-nvme <linux-nvme@lists.infradead.org>; Shlomi Nimrodi <shlomin@mellanox.com>
Subject: Re: NULL pointer dereference in nvmet_rdma_queue_disconnect during bond failover
>> This patch did not apply to latest 5.2 (5.2.21). All of 10 hunks
>> failed.
It applies on branch linux-5.2.y in the stable tree. What are you using?
>> I applied it manually, and also handled cm_id->context in
>> nvmet_rdma_find_get_device as I mentioned earlier. I am attaching the
>> patch that I tested on kernel 5.2.21 (target side). I confirm that
>> this patch fixes the bond failover issue.
>>
>> Tested-by: Alex Lyakas <alex@zadara.com>
Cool.
>>
>> Max, will this help to deliver this fix upstream, so that we can get
>> it in MOFED 4.9?
>>
>> Thanks,
>> Alex.
>
> Alex,
>
> Thanks for testing this.
>
> Waiting for Sagi's official rebased version for doing full review.
I'll send a patch...
_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2020-04-05 11:11 UTC | newest]
Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-06-05 18:03 NULL pointer dereference in nvmet_rdma_queue_disconnect during bond failover Alex Lyakas
2019-06-06 0:05 ` Sagi Grimberg
2019-06-06 7:31 ` Max Gurtovoy
2019-07-03 9:28 ` Alex Lyakas
2019-07-03 12:56 ` Max Gurtovoy
2019-07-03 22:42 ` Sagi Grimberg
2019-07-12 19:38 ` Sagi Grimberg
2019-07-13 19:44 ` Alex Lyakas
2019-07-14 7:27 ` Sagi Grimberg
2019-08-01 1:08 ` Sagi Grimberg
2019-09-13 18:44 ` Sagi Grimberg
2020-03-30 19:02 ` Alex Lyakas
2020-03-30 21:06 ` Max Gurtovoy
2020-03-31 0:21 ` Sagi Grimberg
2020-03-31 8:16 ` Shlomi Nimrodi
2020-04-02 9:13 ` Alex Lyakas
2020-04-02 15:08 ` Max Gurtovoy
2020-04-02 15:26 ` Sagi Grimberg
2020-04-05 11:11 ` Shlomi Nimrodi
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).