From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
stable@vger.kernel.org, Anton Eidelman <anton@lightbitslabs.com>,
Sagi Grimberg <sagi@grimberg.me>, Christoph Hellwig <hch@lst.de>,
Sasha Levin <sashal@kernel.org>
Subject: [PATCH 5.4 14/65] nvme: fix possible deadlock when I/O is blocked
Date: Tue, 7 Jul 2020 17:16:53 +0200 [thread overview]
Message-ID: <20200707145753.145545420@linuxfoundation.org> (raw)
In-Reply-To: <20200707145752.417212219@linuxfoundation.org>
From: Sagi Grimberg <sagi@grimberg.me>
[ Upstream commit 3b4b19721ec652ad2c4fe51dfbe5124212b5f581 ]
Revert fab7772bfbcf ("nvme-multipath: revalidate nvme_ns_head gendisk
in nvme_validate_ns")
When adding a new namespace to the head disk (via nvme_mpath_set_live)
we will see partition scan which triggers I/O on the mpath device node.
This process will usually be triggered from the scan_work which holds
the scan_lock. If I/O blocks (if we got ana change currently have only
available paths but none are accessible) this can deadlock on the head
disk bd_mutex as both partition scan I/O takes it, and head disk revalidation
takes it to check for resize (also triggered from scan_work on a different
path). See trace [1].
The mpath disk revalidation was originally added to detect online disk
size change, but this is no longer needed since commit cb224c3af4df
("nvme: Convert to use set_capacity_revalidate_and_notify") which already
updates resize info without unnecessarily revalidating the disk (the
mpath disk doesn't even implement .revalidate_disk fop).
[1]:
--
kernel: INFO: task kworker/u65:9:494 blocked for more than 241 seconds.
kernel: Tainted: G OE 5.3.5-050305-generic #201910071830
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: kworker/u65:9 D 0 494 2 0x80004000
kernel: Workqueue: nvme-wq nvme_scan_work [nvme_core]
kernel: Call Trace:
kernel: __schedule+0x2b9/0x6c0
kernel: schedule+0x42/0xb0
kernel: schedule_preempt_disabled+0xe/0x10
kernel: __mutex_lock.isra.0+0x182/0x4f0
kernel: __mutex_lock_slowpath+0x13/0x20
kernel: mutex_lock+0x2e/0x40
kernel: revalidate_disk+0x63/0xa0
kernel: __nvme_revalidate_disk+0xfe/0x110 [nvme_core]
kernel: nvme_revalidate_disk+0xa4/0x160 [nvme_core]
kernel: ? evict+0x14c/0x1b0
kernel: revalidate_disk+0x2b/0xa0
kernel: nvme_validate_ns+0x49/0x940 [nvme_core]
kernel: ? blk_mq_free_request+0xd2/0x100
kernel: ? __nvme_submit_sync_cmd+0xbe/0x1e0 [nvme_core]
kernel: nvme_scan_work+0x24f/0x380 [nvme_core]
kernel: process_one_work+0x1db/0x380
kernel: worker_thread+0x249/0x400
kernel: kthread+0x104/0x140
kernel: ? process_one_work+0x380/0x380
kernel: ? kthread_park+0x80/0x80
kernel: ret_from_fork+0x1f/0x40
...
kernel: INFO: task kworker/u65:1:2630 blocked for more than 241 seconds.
kernel: Tainted: G OE 5.3.5-050305-generic #201910071830
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: kworker/u65:1 D 0 2630 2 0x80004000
kernel: Workqueue: nvme-wq nvme_scan_work [nvme_core]
kernel: Call Trace:
kernel: __schedule+0x2b9/0x6c0
kernel: schedule+0x42/0xb0
kernel: io_schedule+0x16/0x40
kernel: do_read_cache_page+0x438/0x830
kernel: ? __switch_to_asm+0x34/0x70
kernel: ? file_fdatawait_range+0x30/0x30
kernel: read_cache_page+0x12/0x20
kernel: read_dev_sector+0x27/0xc0
kernel: read_lba+0xc1/0x220
kernel: ? kmem_cache_alloc_trace+0x19c/0x230
kernel: efi_partition+0x1e6/0x708
kernel: ? vsnprintf+0x39e/0x4e0
kernel: ? snprintf+0x49/0x60
kernel: check_partition+0x154/0x244
kernel: rescan_partitions+0xae/0x280
kernel: __blkdev_get+0x40f/0x560
kernel: blkdev_get+0x3d/0x140
kernel: __device_add_disk+0x388/0x480
kernel: device_add_disk+0x13/0x20
kernel: nvme_mpath_set_live+0x119/0x140 [nvme_core]
kernel: nvme_update_ns_ana_state+0x5c/0x60 [nvme_core]
kernel: nvme_set_ns_ana_state+0x1e/0x30 [nvme_core]
kernel: nvme_parse_ana_log+0xa1/0x180 [nvme_core]
kernel: ? nvme_update_ns_ana_state+0x60/0x60 [nvme_core]
kernel: nvme_mpath_add_disk+0x47/0x90 [nvme_core]
kernel: nvme_validate_ns+0x396/0x940 [nvme_core]
kernel: ? blk_mq_free_request+0xd2/0x100
kernel: nvme_scan_work+0x24f/0x380 [nvme_core]
kernel: process_one_work+0x1db/0x380
kernel: worker_thread+0x249/0x400
kernel: kthread+0x104/0x140
kernel: ? process_one_work+0x380/0x380
kernel: ? kthread_park+0x80/0x80
kernel: ret_from_fork+0x1f/0x40
--
Fixes: fab7772bfbcf ("nvme-multipath: revalidate nvme_ns_head gendisk
in nvme_validate_ns")
Signed-off-by: Anton Eidelman <anton@lightbitslabs.com>
Signed-off-by: Sagi Grimberg <sagi@grimberg.me>
Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
drivers/nvme/host/core.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index d4b388793f40d..c44c00b9e1d85 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1870,7 +1870,6 @@ static void __nvme_revalidate_disk(struct gendisk *disk, struct nvme_id_ns *id)
if (ns->head->disk) {
nvme_update_disk_info(ns->head->disk, ns, id);
blk_queue_stack_limits(ns->head->disk->queue, ns->queue);
- revalidate_disk(ns->head->disk);
}
#endif
}
--
2.25.1
next prev parent reply other threads:[~2020-07-07 15:20 UTC|newest]
Thread overview: 78+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-07-07 15:16 [PATCH 5.4 00/65] 5.4.51-rc1 review Greg Kroah-Hartman
2020-07-07 15:16 ` [PATCH 5.4 01/65] io_uring: make sure async workqueue is canceled on exit Greg Kroah-Hartman
2020-07-07 15:16 ` [PATCH 5.4 02/65] mm: fix swap cache node allocation mask Greg Kroah-Hartman
2020-07-07 15:16 ` [PATCH 5.4 03/65] EDAC/amd64: Read back the scrub rate PCI register on F15h Greg Kroah-Hartman
2020-07-07 15:16 ` [PATCH 5.4 04/65] usbnet: smsc95xx: Fix use-after-free after removal Greg Kroah-Hartman
2020-07-07 15:16 ` [PATCH 5.4 05/65] sched/debug: Make sd->flags sysctl read-only Greg Kroah-Hartman
2020-07-07 15:16 ` [PATCH 5.4 06/65] mm/slub.c: fix corrupted freechain in deactivate_slab() Greg Kroah-Hartman
2020-07-07 15:16 ` [PATCH 5.4 07/65] mm/slub: fix stack overruns with SLUB_STATS Greg Kroah-Hartman
2020-07-07 15:16 ` [PATCH 5.4 08/65] rxrpc: Fix race between incoming ACK parser and retransmitter Greg Kroah-Hartman
2020-07-07 15:16 ` [PATCH 5.4 09/65] usb: usbtest: fix missing kfree(dev->buf) in usbtest_disconnect Greg Kroah-Hartman
2020-07-07 15:16 ` [PATCH 5.4 10/65] tools lib traceevent: Add append() function helper for appending strings Greg Kroah-Hartman
2020-07-07 15:16 ` [PATCH 5.4 11/65] tools lib traceevent: Handle __attribute__((user)) in field names Greg Kroah-Hartman
2020-07-07 15:16 ` [PATCH 5.4 12/65] s390/debug: avoid kernel warning on too large number of pages Greg Kroah-Hartman
2020-07-07 15:16 ` [PATCH 5.4 13/65] nvme-multipath: set bdi capabilities once Greg Kroah-Hartman
2020-07-07 15:16 ` Greg Kroah-Hartman [this message]
2020-07-07 15:16 ` [PATCH 5.4 15/65] nvme-multipath: fix deadlock between ana_work and scan_work Greg Kroah-Hartman
2020-07-07 15:16 ` [PATCH 5.4 16/65] nvme-multipath: fix deadlock due to head->lock Greg Kroah-Hartman
2020-07-07 15:16 ` [PATCH 5.4 17/65] nvme-multipath: fix bogus request queue reference put Greg Kroah-Hartman
2020-07-07 15:16 ` [PATCH 5.4 18/65] kgdb: Avoid suspicious RCU usage warning Greg Kroah-Hartman
2020-07-07 15:16 ` [PATCH 5.4 19/65] selftests: tpm: Use /bin/sh instead of /bin/bash Greg Kroah-Hartman
2020-07-07 15:16 ` [PATCH 5.4 20/65] tpm: Fix TIS locality timeout problems Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 21/65] crypto: af_alg - fix use-after-free in af_alg_accept() due to bh_lock_sock() Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 22/65] drm/msm/dpu: fix error return code in dpu_encoder_init Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 23/65] rxrpc: Fix afs large storage transmission performance drop Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 24/65] RDMA/counter: Query a counter before release Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 25/65] cxgb4: use unaligned conversion for fetching timestamp Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 26/65] cxgb4: parse TC-U32 key values and masks natively Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 27/65] cxgb4: fix endian conversions for L4 ports in filters Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 28/65] cxgb4: use correct type for all-mask IP address comparison Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 29/65] cxgb4: fix SGE queue dump destination buffer context Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 30/65] hwmon: (max6697) Make sure the OVERT mask is set correctly Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 31/65] hwmon: (acpi_power_meter) Fix potential memory leak in acpi_power_meter_add() Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 32/65] thermal/drivers/mediatek: Fix bank number settings on mt8183 Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 33/65] thermal/drivers/rcar_gen3: Fix undefined temperature if negative Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 34/65] kthread: save thread function Greg Kroah-Hartman
2020-07-07 15:31 ` J. Bruce Fields
2020-07-07 15:17 ` [PATCH 5.4 35/65] nfsd: clients dont need to break their own delegations Greg Kroah-Hartman
2020-07-07 15:31 ` J. Bruce Fields
2020-07-07 17:20 ` Sasha Levin
2020-07-07 17:29 ` Sasha Levin
2020-07-07 17:31 ` J. Bruce Fields
2020-07-07 15:17 ` [PATCH 5.4 36/65] nfsd4: fix nfsdfs reference count loop Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 37/65] nfsd: fix nfsdfs inode reference count leak Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 38/65] drm: sun4i: hdmi: Remove extra HPD polling Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 39/65] virtio-blk: free vblk-vqs in error path of virtblk_probe() Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 40/65] SMB3: Honor posix flag for multiuser mounts Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 41/65] nvme: fix identify error status silent ignore Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 42/65] nvme: fix a crash in nvme_mpath_add_disk Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 43/65] samples/vfs: avoid warning in statx override Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 44/65] i2c: algo-pca: Add 0x78 as SCL stuck low status for PCA9665 Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 45/65] i2c: mlxcpld: check correct size of maximum RECV_LEN packet Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 46/65] spi: spi-fsl-dspi: Fix external abort on interrupt in resume or exit paths Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 47/65] nfsd: apply umask on fs without ACL support Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 48/65] Revert "ALSA: usb-audio: Improve frames size computation" Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 49/65] SMB3: Honor seal flag for multiuser mounts Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 50/65] SMB3: Honor persistent/resilient handle flags " Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 51/65] SMB3: Honor lease disabling " Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 52/65] SMB3: Honor handletimeout flag " Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 53/65] cifs: Fix the target file was deleted when rename failed Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 54/65] MIPS: lantiq: xway: sysctrl: fix the GPHY clock alias names Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 55/65] MIPS: Add missing EHB in mtc0 -> mfc0 sequence for DSPen Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 56/65] drm/amd/display: Only revalidate bandwidth on medium and fast updates Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 57/65] drm/amdgpu: use %u rather than %d for sclk/mclk Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 58/65] drm/amdgpu/atomfirmware: fix vram_info fetching for renoir Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 59/65] dma-buf: Move dma_buf_release() from fops to dentry_ops Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 60/65] irqchip/gic: Atomically update affinity Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 61/65] mm, compaction: fully assume capture is not NULL in compact_zone_order() Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 62/65] mm, compaction: make capture control handling safe wrt interrupts Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 63/65] x86/resctrl: Fix memory bandwidth counter width for AMD Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 64/65] dm zoned: assign max_io_len correctly Greg Kroah-Hartman
2020-07-07 15:17 ` [PATCH 5.4 65/65] efi: Make it possible to disable efivar_ssdt entirely Greg Kroah-Hartman
2020-07-08 5:30 ` [PATCH 5.4 00/65] 5.4.51-rc1 review Naresh Kamboju
2020-07-08 8:41 ` Jon Hunter
2020-07-08 15:15 ` Greg Kroah-Hartman
2020-07-08 16:58 ` Jon Hunter
2020-07-09 9:30 ` Greg Kroah-Hartman
2020-07-08 15:00 ` Shuah Khan
2020-07-08 17:53 ` Guenter Roeck
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200707145753.145545420@linuxfoundation.org \
--to=gregkh@linuxfoundation.org \
--cc=anton@lightbitslabs.com \
--cc=hch@lst.de \
--cc=linux-kernel@vger.kernel.org \
--cc=sagi@grimberg.me \
--cc=sashal@kernel.org \
--cc=stable@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).