* [PATCH] nvme-rdma: Avoid double freeing of async event data
@ 2020-03-09 21:07 ` Prabhath Sajeepa
0 siblings, 0 replies; 8+ messages in thread
From: Prabhath Sajeepa @ 2020-03-09 21:07 UTC (permalink / raw)
To: kbusch, axboe, hch, sagi, linux-nvme, linux-kernel; +Cc: roland, psajeepa
The timeout of identify cmd, which is invoked as part of admin queue
creation, can result in freeing of async event data both in
nvme_rdma_timeout handler and error handling path of
nvme_rdma_configure_admin queue thus causing NULL pointer reference.
Call Trace:
? nvme_rdma_setup_ctrl+0x223/0x800 [nvme_rdma]
nvme_rdma_create_ctrl+0x2ba/0x3f7 [nvme_rdma]
nvmf_dev_write+0xa54/0xcc6 [nvme_fabrics]
__vfs_write+0x1b/0x40
vfs_write+0xb2/0x1b0
ksys_write+0x61/0xd0
__x64_sys_write+0x1a/0x20
do_syscall_64+0x60/0x1e0
entry_SYSCALL_64_after_hwframe+0x44/0xa9
Signed-off-by: Prabhath Sajeepa <psajeepa@purestorage.com>
Reviewed-by: Roland Dreier <roland@purestorage.com>
---
drivers/nvme/host/rdma.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 3e85c5c..0fe08c4 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -850,9 +850,11 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,
if (new)
blk_mq_free_tag_set(ctrl->ctrl.admin_tagset);
out_free_async_qe:
- nvme_rdma_free_qe(ctrl->device->dev, &ctrl->async_event_sqe,
- sizeof(struct nvme_command), DMA_TO_DEVICE);
- ctrl->async_event_sqe.data = NULL;
+ if (ctrl->async_event_sqe.data) {
+ nvme_rdma_free_qe(ctrl->device->dev, &ctrl->async_event_sqe,
+ sizeof(struct nvme_command), DMA_TO_DEVICE);
+ ctrl->async_event_sqe.data = NULL;
+ }
out_free_queue:
nvme_rdma_free_queue(&ctrl->queues[0]);
return error;
--
2.7.4
_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
^ permalink raw reply related [flat|nested] 8+ messages in thread* Re: [PATCH] nvme-rdma: Avoid double freeing of async event data
2020-03-09 21:07 ` Prabhath Sajeepa
@ 2020-03-09 21:53 ` Max Gurtovoy
-1 siblings, 0 replies; 8+ messages in thread
From: Max Gurtovoy @ 2020-03-09 21:53 UTC (permalink / raw)
To: Prabhath Sajeepa, kbusch, axboe, hch, sagi, linux-nvme, linux-kernel
Cc: roland
On 3/9/2020 11:07 PM, Prabhath Sajeepa wrote:
> The timeout of identify cmd, which is invoked as part of admin queue
> creation, can result in freeing of async event data both in
> nvme_rdma_timeout handler and error handling path of
> nvme_rdma_configure_admin queue thus causing NULL pointer reference.
> Call Trace:
> ? nvme_rdma_setup_ctrl+0x223/0x800 [nvme_rdma]
> nvme_rdma_create_ctrl+0x2ba/0x3f7 [nvme_rdma]
> nvmf_dev_write+0xa54/0xcc6 [nvme_fabrics]
> __vfs_write+0x1b/0x40
> vfs_write+0xb2/0x1b0
> ksys_write+0x61/0xd0
> __x64_sys_write+0x1a/0x20
> do_syscall_64+0x60/0x1e0
> entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
> Signed-off-by: Prabhath Sajeepa <psajeepa@purestorage.com>
> Reviewed-by: Roland Dreier <roland@purestorage.com>
> ---
> drivers/nvme/host/rdma.c | 8 +++++---
> 1 file changed, 5 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
> index 3e85c5c..0fe08c4 100644
> --- a/drivers/nvme/host/rdma.c
> +++ b/drivers/nvme/host/rdma.c
> @@ -850,9 +850,11 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,
> if (new)
> blk_mq_free_tag_set(ctrl->ctrl.admin_tagset);
> out_free_async_qe:
> - nvme_rdma_free_qe(ctrl->device->dev, &ctrl->async_event_sqe,
> - sizeof(struct nvme_command), DMA_TO_DEVICE);
> - ctrl->async_event_sqe.data = NULL;
> + if (ctrl->async_event_sqe.data) {
> + nvme_rdma_free_qe(ctrl->device->dev, &ctrl->async_event_sqe,
> + sizeof(struct nvme_command), DMA_TO_DEVICE);
> + ctrl->async_event_sqe.data = NULL;
> + }
> out_free_queue:
> nvme_rdma_free_queue(&ctrl->queues[0]);
> return error;
Looks good,
Reviewed-by: Max Gurtovoy <maxg@mellanox.com>
We did the same fix in-house yesterday and planed to send it tomorrow :)
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [PATCH] nvme-rdma: Avoid double freeing of async event data
@ 2020-03-09 21:53 ` Max Gurtovoy
0 siblings, 0 replies; 8+ messages in thread
From: Max Gurtovoy @ 2020-03-09 21:53 UTC (permalink / raw)
To: Prabhath Sajeepa, kbusch, axboe, hch, sagi, linux-nvme, linux-kernel
Cc: roland
On 3/9/2020 11:07 PM, Prabhath Sajeepa wrote:
> The timeout of identify cmd, which is invoked as part of admin queue
> creation, can result in freeing of async event data both in
> nvme_rdma_timeout handler and error handling path of
> nvme_rdma_configure_admin queue thus causing NULL pointer reference.
> Call Trace:
> ? nvme_rdma_setup_ctrl+0x223/0x800 [nvme_rdma]
> nvme_rdma_create_ctrl+0x2ba/0x3f7 [nvme_rdma]
> nvmf_dev_write+0xa54/0xcc6 [nvme_fabrics]
> __vfs_write+0x1b/0x40
> vfs_write+0xb2/0x1b0
> ksys_write+0x61/0xd0
> __x64_sys_write+0x1a/0x20
> do_syscall_64+0x60/0x1e0
> entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
> Signed-off-by: Prabhath Sajeepa <psajeepa@purestorage.com>
> Reviewed-by: Roland Dreier <roland@purestorage.com>
> ---
> drivers/nvme/host/rdma.c | 8 +++++---
> 1 file changed, 5 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
> index 3e85c5c..0fe08c4 100644
> --- a/drivers/nvme/host/rdma.c
> +++ b/drivers/nvme/host/rdma.c
> @@ -850,9 +850,11 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,
> if (new)
> blk_mq_free_tag_set(ctrl->ctrl.admin_tagset);
> out_free_async_qe:
> - nvme_rdma_free_qe(ctrl->device->dev, &ctrl->async_event_sqe,
> - sizeof(struct nvme_command), DMA_TO_DEVICE);
> - ctrl->async_event_sqe.data = NULL;
> + if (ctrl->async_event_sqe.data) {
> + nvme_rdma_free_qe(ctrl->device->dev, &ctrl->async_event_sqe,
> + sizeof(struct nvme_command), DMA_TO_DEVICE);
> + ctrl->async_event_sqe.data = NULL;
> + }
> out_free_queue:
> nvme_rdma_free_queue(&ctrl->queues[0]);
> return error;
Looks good,
Reviewed-by: Max Gurtovoy <maxg@mellanox.com>
We did the same fix in-house yesterday and planed to send it tomorrow :)
_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] nvme-rdma: Avoid double freeing of async event data
2020-03-09 21:07 ` Prabhath Sajeepa
@ 2020-03-10 17:53 ` Christoph Hellwig
-1 siblings, 0 replies; 8+ messages in thread
From: Christoph Hellwig @ 2020-03-10 17:53 UTC (permalink / raw)
To: Prabhath Sajeepa
Cc: kbusch, axboe, hch, sagi, linux-nvme, linux-kernel, roland
On Mon, Mar 09, 2020 at 03:07:53PM -0600, Prabhath Sajeepa wrote:
> The timeout of identify cmd, which is invoked as part of admin queue
> creation, can result in freeing of async event data both in
> nvme_rdma_timeout handler and error handling path of
> nvme_rdma_configure_admin queue thus causing NULL pointer reference.
> Call Trace:
> ? nvme_rdma_setup_ctrl+0x223/0x800 [nvme_rdma]
> nvme_rdma_create_ctrl+0x2ba/0x3f7 [nvme_rdma]
> nvmf_dev_write+0xa54/0xcc6 [nvme_fabrics]
> __vfs_write+0x1b/0x40
> vfs_write+0xb2/0x1b0
> ksys_write+0x61/0xd0
> __x64_sys_write+0x1a/0x20
> do_syscall_64+0x60/0x1e0
> entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
> Signed-off-by: Prabhath Sajeepa <psajeepa@purestorage.com>
> Reviewed-by: Roland Dreier <roland@purestorage.com>
This looks good as a hot fix:
Reviewed-by: Christoph Hellwig <hch@lst.de>
But I really think we need to do something about init vs timeout
in the state machine. Otherwise we're going to deal with racing
resource deallocation forever.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] nvme-rdma: Avoid double freeing of async event data
@ 2020-03-10 17:53 ` Christoph Hellwig
0 siblings, 0 replies; 8+ messages in thread
From: Christoph Hellwig @ 2020-03-10 17:53 UTC (permalink / raw)
To: Prabhath Sajeepa
Cc: roland, sagi, linux-kernel, linux-nvme, axboe, kbusch, hch
On Mon, Mar 09, 2020 at 03:07:53PM -0600, Prabhath Sajeepa wrote:
> The timeout of identify cmd, which is invoked as part of admin queue
> creation, can result in freeing of async event data both in
> nvme_rdma_timeout handler and error handling path of
> nvme_rdma_configure_admin queue thus causing NULL pointer reference.
> Call Trace:
> ? nvme_rdma_setup_ctrl+0x223/0x800 [nvme_rdma]
> nvme_rdma_create_ctrl+0x2ba/0x3f7 [nvme_rdma]
> nvmf_dev_write+0xa54/0xcc6 [nvme_fabrics]
> __vfs_write+0x1b/0x40
> vfs_write+0xb2/0x1b0
> ksys_write+0x61/0xd0
> __x64_sys_write+0x1a/0x20
> do_syscall_64+0x60/0x1e0
> entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
> Signed-off-by: Prabhath Sajeepa <psajeepa@purestorage.com>
> Reviewed-by: Roland Dreier <roland@purestorage.com>
This looks good as a hot fix:
Reviewed-by: Christoph Hellwig <hch@lst.de>
But I really think we need to do something about init vs timeout
in the state machine. Otherwise we're going to deal with racing
resource deallocation forever.
_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] nvme-rdma: Avoid double freeing of async event data
2020-03-09 21:07 ` Prabhath Sajeepa
@ 2020-03-10 20:59 ` Keith Busch
-1 siblings, 0 replies; 8+ messages in thread
From: Keith Busch @ 2020-03-10 20:59 UTC (permalink / raw)
To: Prabhath Sajeepa; +Cc: axboe, hch, sagi, linux-nvme, linux-kernel, roland
On Mon, Mar 09, 2020 at 03:07:53PM -0600, Prabhath Sajeepa wrote:
> The timeout of identify cmd, which is invoked as part of admin queue
> creation, can result in freeing of async event data both in
> nvme_rdma_timeout handler and error handling path of
> nvme_rdma_configure_admin queue thus causing NULL pointer reference.
> Call Trace:
> ? nvme_rdma_setup_ctrl+0x223/0x800 [nvme_rdma]
> nvme_rdma_create_ctrl+0x2ba/0x3f7 [nvme_rdma]
> nvmf_dev_write+0xa54/0xcc6 [nvme_fabrics]
> __vfs_write+0x1b/0x40
> vfs_write+0xb2/0x1b0
> ksys_write+0x61/0xd0
> __x64_sys_write+0x1a/0x20
> do_syscall_64+0x60/0x1e0
> entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
> Signed-off-by: Prabhath Sajeepa <psajeepa@purestorage.com>
> Reviewed-by: Roland Dreier <roland@purestorage.com>
Thanks, applied to new branch for 5.6-rc6.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] nvme-rdma: Avoid double freeing of async event data
@ 2020-03-10 20:59 ` Keith Busch
0 siblings, 0 replies; 8+ messages in thread
From: Keith Busch @ 2020-03-10 20:59 UTC (permalink / raw)
To: Prabhath Sajeepa; +Cc: roland, sagi, linux-kernel, linux-nvme, axboe, hch
On Mon, Mar 09, 2020 at 03:07:53PM -0600, Prabhath Sajeepa wrote:
> The timeout of identify cmd, which is invoked as part of admin queue
> creation, can result in freeing of async event data both in
> nvme_rdma_timeout handler and error handling path of
> nvme_rdma_configure_admin queue thus causing NULL pointer reference.
> Call Trace:
> ? nvme_rdma_setup_ctrl+0x223/0x800 [nvme_rdma]
> nvme_rdma_create_ctrl+0x2ba/0x3f7 [nvme_rdma]
> nvmf_dev_write+0xa54/0xcc6 [nvme_fabrics]
> __vfs_write+0x1b/0x40
> vfs_write+0xb2/0x1b0
> ksys_write+0x61/0xd0
> __x64_sys_write+0x1a/0x20
> do_syscall_64+0x60/0x1e0
> entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
> Signed-off-by: Prabhath Sajeepa <psajeepa@purestorage.com>
> Reviewed-by: Roland Dreier <roland@purestorage.com>
Thanks, applied to new branch for 5.6-rc6.
_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
^ permalink raw reply [flat|nested] 8+ messages in thread