From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
stable@vger.kernel.org, David Jeffery <djeffery@redhat.com>,
Xiao Ni <xni@redhat.com>, Song Liu <songliubraving@fb.com>,
Jack Wang <jinpu.wang@cloud.ionos.com>
Subject: [PATCH 4.19 36/38] md: Set prev_flush_start and flush_bio in an atomic way
Date: Mon, 8 Feb 2021 16:01:23 +0100 [thread overview]
Message-ID: <20210208145807.613541970@linuxfoundation.org> (raw)
In-Reply-To: <20210208145806.141056364@linuxfoundation.org>
From: Xiao Ni <xni@redhat.com>
commit dc5d17a3c39b06aef866afca19245a9cfb533a79 upstream.
One customer reports a crash problem which causes by flush request. It
triggers a warning before crash.
/* new request after previous flush is completed */
if (ktime_after(req_start, mddev->prev_flush_start)) {
WARN_ON(mddev->flush_bio);
mddev->flush_bio = bio;
bio = NULL;
}
The WARN_ON is triggered. We use spin lock to protect prev_flush_start and
flush_bio in md_flush_request. But there is no lock protection in
md_submit_flush_data. It can set flush_bio to NULL first because of
compiler reordering write instructions.
For example, flush bio1 sets flush bio to NULL first in
md_submit_flush_data. An interrupt or vmware causing an extended stall
happen between updating flush_bio and prev_flush_start. Because flush_bio
is NULL, flush bio2 can get the lock and submit to underlayer disks. Then
flush bio1 updates prev_flush_start after the interrupt or extended stall.
Then flush bio3 enters in md_flush_request. The start time req_start is
behind prev_flush_start. The flush_bio is not NULL(flush bio2 hasn't
finished). So it can trigger the WARN_ON now. Then it calls INIT_WORK
again. INIT_WORK() will re-initialize the list pointers in the
work_struct, which then can result in a corrupted work list and the
work_struct queued a second time. With the work list corrupted, it can
lead in invalid work items being used and cause a crash in
process_one_work.
We need to make sure only one flush bio can be handled at one same time.
So add spin lock in md_submit_flush_data to protect prev_flush_start and
flush_bio in an atomic way.
Reviewed-by: David Jeffery <djeffery@redhat.com>
Signed-off-by: Xiao Ni <xni@redhat.com>
Signed-off-by: Song Liu <songliubraving@fb.com>
Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
drivers/md/md.c | 2 ++
1 file changed, 2 insertions(+)
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -474,8 +474,10 @@ static void md_submit_flush_data(struct
* could wait for this and below md_handle_request could wait for those
* bios because of suspend check
*/
+ spin_lock_irq(&mddev->lock);
mddev->last_flush = mddev->start_flush;
mddev->flush_bio = NULL;
+ spin_unlock_irq(&mddev->lock);
wake_up(&mddev->sb_wait);
if (bio->bi_iter.bi_size == 0) {
next prev parent reply other threads:[~2021-02-08 16:19 UTC|newest]
Thread overview: 45+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-08 15:00 [PATCH 4.19 00/38] 4.19.175-rc1 review Greg Kroah-Hartman
2021-02-08 15:00 ` [PATCH 4.19 01/38] USB: serial: cp210x: add pid/vid for WSDA-200-USB Greg Kroah-Hartman
2021-02-08 15:00 ` [PATCH 4.19 02/38] USB: serial: cp210x: add new VID/PID for supporting Teraoka AD2000 Greg Kroah-Hartman
2021-02-08 15:00 ` [PATCH 4.19 03/38] USB: serial: option: Adding support for Cinterion MV31 Greg Kroah-Hartman
2021-02-08 15:00 ` [PATCH 4.19 04/38] elfcore: fix building with clang Greg Kroah-Hartman
2021-02-08 15:00 ` [PATCH 4.19 05/38] Input: i8042 - unbreak Pegatron C15B Greg Kroah-Hartman
2021-02-08 15:00 ` [PATCH 4.19 06/38] rxrpc: Fix deadlock around release of dst cached on udp tunnel Greg Kroah-Hartman
2021-02-08 15:00 ` [PATCH 4.19 07/38] arm64: dts: ls1046a: fix dcfg address range Greg Kroah-Hartman
2021-02-08 15:00 ` [PATCH 4.19 08/38] net: lapb: Copy the skb before sending a packet Greg Kroah-Hartman
2021-02-08 15:00 ` [PATCH 4.19 09/38] net: mvpp2: TCAM entry enable should be written after SRAM data Greg Kroah-Hartman
2021-02-08 15:00 ` [PATCH 4.19 10/38] memblock: do not start bottom-up allocations with kernel_end Greg Kroah-Hartman
2021-02-08 15:00 ` [PATCH 4.19 11/38] USB: gadget: legacy: fix an error code in eth_bind() Greg Kroah-Hartman
2021-02-08 15:00 ` [PATCH 4.19 12/38] USB: usblp: dont call usb_set_interface if theres a single alt Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 13/38] usb: renesas_usbhs: Clear pipe running flag in usbhs_pkt_pop() Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 14/38] usb: dwc2: Fix endpoint direction check in ep_from_windex Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 15/38] usb: dwc3: fix clock issue during resume in OTG mode Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 16/38] ovl: fix dentry leak in ovl_get_redirect Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 17/38] mac80211: fix station rate table updates on assoc Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 18/38] kretprobe: Avoid re-registration of the same kretprobe earlier Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 19/38] genirq/msi: Activate Multi-MSI early when MSI_FLAG_ACTIVATE_EARLY is set Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 20/38] xhci: fix bounce buffer usage for non-sg list case Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 21/38] cifs: report error instead of invalid when revalidating a dentry fails Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 22/38] smb3: Fix out-of-bounds bug in SMB2_negotiate() Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 23/38] mmc: core: Limit retries when analyse of SDIO tuples fails Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 24/38] nvme-pci: avoid the deepest sleep state on Kingston A2000 SSDs Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 25/38] KVM: SVM: Treat SVM as unsupported when running as an SEV guest Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 26/38] ARM: footbridge: fix dc21285 PCI configuration accessors Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 27/38] mm: hugetlbfs: fix cannot migrate the fallocated HugeTLB page Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 28/38] mm: hugetlb: fix a race between freeing and dissolving the page Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 29/38] mm: hugetlb: fix a race between isolating and freeing page Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 30/38] mm: hugetlb: remove VM_BUG_ON_PAGE from page_huge_active Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 31/38] mm: thp: fix MADV_REMOVE deadlock on shmem THP Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 32/38] x86/build: Disable CET instrumentation in the kernel Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 33/38] x86/apic: Add extra serialization for non-serializing MSRs Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 34/38] Input: xpad - sync supported devices with fork on GitHub Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 35/38] iommu/vt-d: Do not use flush-queue when caching-mode is on Greg Kroah-Hartman
2021-02-08 15:01 ` Greg Kroah-Hartman [this message]
2021-02-08 15:01 ` [PATCH 4.19 37/38] net: ip_tunnel: fix mtu calculation Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 38/38] net: dsa: mv88e6xxx: override existent unicast portvec in port_fdb_add Greg Kroah-Hartman
2021-02-08 18:34 ` [PATCH 4.19 00/38] 4.19.175-rc1 review Pavel Machek
2021-02-08 20:42 ` Shuah Khan
2021-02-09 11:00 ` Igor Torrente
2021-02-09 15:02 ` Naresh Kamboju
2021-02-09 18:11 ` Guenter Roeck
2021-02-10 1:16 ` Ross Schmidt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210208145807.613541970@linuxfoundation.org \
--to=gregkh@linuxfoundation.org \
--cc=djeffery@redhat.com \
--cc=jinpu.wang@cloud.ionos.com \
--cc=linux-kernel@vger.kernel.org \
--cc=songliubraving@fb.com \
--cc=stable@vger.kernel.org \
--cc=xni@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).