linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	stable@vger.kernel.org, David Jeffery <djeffery@redhat.com>,
	Xiao Ni <xni@redhat.com>, Song Liu <songliubraving@fb.com>,
	Jack Wang <jinpu.wang@cloud.ionos.com>
Subject: [PATCH 4.19 36/38] md: Set prev_flush_start and flush_bio in an atomic way
Date: Mon,  8 Feb 2021 16:01:23 +0100	[thread overview]
Message-ID: <20210208145807.613541970@linuxfoundation.org> (raw)
In-Reply-To: <20210208145806.141056364@linuxfoundation.org>

From: Xiao Ni <xni@redhat.com>

commit dc5d17a3c39b06aef866afca19245a9cfb533a79 upstream.

One customer reports a crash problem which causes by flush request. It
triggers a warning before crash.

        /* new request after previous flush is completed */
        if (ktime_after(req_start, mddev->prev_flush_start)) {
                WARN_ON(mddev->flush_bio);
                mddev->flush_bio = bio;
                bio = NULL;
        }

The WARN_ON is triggered. We use spin lock to protect prev_flush_start and
flush_bio in md_flush_request. But there is no lock protection in
md_submit_flush_data. It can set flush_bio to NULL first because of
compiler reordering write instructions.

For example, flush bio1 sets flush bio to NULL first in
md_submit_flush_data. An interrupt or vmware causing an extended stall
happen between updating flush_bio and prev_flush_start. Because flush_bio
is NULL, flush bio2 can get the lock and submit to underlayer disks. Then
flush bio1 updates prev_flush_start after the interrupt or extended stall.

Then flush bio3 enters in md_flush_request. The start time req_start is
behind prev_flush_start. The flush_bio is not NULL(flush bio2 hasn't
finished). So it can trigger the WARN_ON now. Then it calls INIT_WORK
again. INIT_WORK() will re-initialize the list pointers in the
work_struct, which then can result in a corrupted work list and the
work_struct queued a second time. With the work list corrupted, it can
lead in invalid work items being used and cause a crash in
process_one_work.

We need to make sure only one flush bio can be handled at one same time.
So add spin lock in md_submit_flush_data to protect prev_flush_start and
flush_bio in an atomic way.

Reviewed-by: David Jeffery <djeffery@redhat.com>
Signed-off-by: Xiao Ni <xni@redhat.com>
Signed-off-by: Song Liu <songliubraving@fb.com>
Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
 drivers/md/md.c |    2 ++
 1 file changed, 2 insertions(+)

--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -474,8 +474,10 @@ static void md_submit_flush_data(struct
 	 * could wait for this and below md_handle_request could wait for those
 	 * bios because of suspend check
 	 */
+	spin_lock_irq(&mddev->lock);
 	mddev->last_flush = mddev->start_flush;
 	mddev->flush_bio = NULL;
+	spin_unlock_irq(&mddev->lock);
 	wake_up(&mddev->sb_wait);
 
 	if (bio->bi_iter.bi_size == 0) {



  parent reply	other threads:[~2021-02-08 16:19 UTC|newest]

Thread overview: 45+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-02-08 15:00 [PATCH 4.19 00/38] 4.19.175-rc1 review Greg Kroah-Hartman
2021-02-08 15:00 ` [PATCH 4.19 01/38] USB: serial: cp210x: add pid/vid for WSDA-200-USB Greg Kroah-Hartman
2021-02-08 15:00 ` [PATCH 4.19 02/38] USB: serial: cp210x: add new VID/PID for supporting Teraoka AD2000 Greg Kroah-Hartman
2021-02-08 15:00 ` [PATCH 4.19 03/38] USB: serial: option: Adding support for Cinterion MV31 Greg Kroah-Hartman
2021-02-08 15:00 ` [PATCH 4.19 04/38] elfcore: fix building with clang Greg Kroah-Hartman
2021-02-08 15:00 ` [PATCH 4.19 05/38] Input: i8042 - unbreak Pegatron C15B Greg Kroah-Hartman
2021-02-08 15:00 ` [PATCH 4.19 06/38] rxrpc: Fix deadlock around release of dst cached on udp tunnel Greg Kroah-Hartman
2021-02-08 15:00 ` [PATCH 4.19 07/38] arm64: dts: ls1046a: fix dcfg address range Greg Kroah-Hartman
2021-02-08 15:00 ` [PATCH 4.19 08/38] net: lapb: Copy the skb before sending a packet Greg Kroah-Hartman
2021-02-08 15:00 ` [PATCH 4.19 09/38] net: mvpp2: TCAM entry enable should be written after SRAM data Greg Kroah-Hartman
2021-02-08 15:00 ` [PATCH 4.19 10/38] memblock: do not start bottom-up allocations with kernel_end Greg Kroah-Hartman
2021-02-08 15:00 ` [PATCH 4.19 11/38] USB: gadget: legacy: fix an error code in eth_bind() Greg Kroah-Hartman
2021-02-08 15:00 ` [PATCH 4.19 12/38] USB: usblp: dont call usb_set_interface if theres a single alt Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 13/38] usb: renesas_usbhs: Clear pipe running flag in usbhs_pkt_pop() Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 14/38] usb: dwc2: Fix endpoint direction check in ep_from_windex Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 15/38] usb: dwc3: fix clock issue during resume in OTG mode Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 16/38] ovl: fix dentry leak in ovl_get_redirect Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 17/38] mac80211: fix station rate table updates on assoc Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 18/38] kretprobe: Avoid re-registration of the same kretprobe earlier Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 19/38] genirq/msi: Activate Multi-MSI early when MSI_FLAG_ACTIVATE_EARLY is set Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 20/38] xhci: fix bounce buffer usage for non-sg list case Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 21/38] cifs: report error instead of invalid when revalidating a dentry fails Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 22/38] smb3: Fix out-of-bounds bug in SMB2_negotiate() Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 23/38] mmc: core: Limit retries when analyse of SDIO tuples fails Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 24/38] nvme-pci: avoid the deepest sleep state on Kingston A2000 SSDs Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 25/38] KVM: SVM: Treat SVM as unsupported when running as an SEV guest Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 26/38] ARM: footbridge: fix dc21285 PCI configuration accessors Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 27/38] mm: hugetlbfs: fix cannot migrate the fallocated HugeTLB page Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 28/38] mm: hugetlb: fix a race between freeing and dissolving the page Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 29/38] mm: hugetlb: fix a race between isolating and freeing page Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 30/38] mm: hugetlb: remove VM_BUG_ON_PAGE from page_huge_active Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 31/38] mm: thp: fix MADV_REMOVE deadlock on shmem THP Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 32/38] x86/build: Disable CET instrumentation in the kernel Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 33/38] x86/apic: Add extra serialization for non-serializing MSRs Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 34/38] Input: xpad - sync supported devices with fork on GitHub Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 35/38] iommu/vt-d: Do not use flush-queue when caching-mode is on Greg Kroah-Hartman
2021-02-08 15:01 ` Greg Kroah-Hartman [this message]
2021-02-08 15:01 ` [PATCH 4.19 37/38] net: ip_tunnel: fix mtu calculation Greg Kroah-Hartman
2021-02-08 15:01 ` [PATCH 4.19 38/38] net: dsa: mv88e6xxx: override existent unicast portvec in port_fdb_add Greg Kroah-Hartman
2021-02-08 18:34 ` [PATCH 4.19 00/38] 4.19.175-rc1 review Pavel Machek
2021-02-08 20:42 ` Shuah Khan
2021-02-09 11:00 ` Igor Torrente
2021-02-09 15:02 ` Naresh Kamboju
2021-02-09 18:11 ` Guenter Roeck
2021-02-10  1:16 ` Ross Schmidt

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210208145807.613541970@linuxfoundation.org \
    --to=gregkh@linuxfoundation.org \
    --cc=djeffery@redhat.com \
    --cc=jinpu.wang@cloud.ionos.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=songliubraving@fb.com \
    --cc=stable@vger.kernel.org \
    --cc=xni@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).