From: Greg KH <gregkh@linuxfoundation.org>
To: Jack Wang <jinpu.wang@cloud.ionos.com>
Cc: sashal@kernel.org, stable@vger.kernel.org,
Xiao Ni <xni@redhat.com>, David Jeffery <djeffery@redhat.com>,
Song Liu <songliubraving@fb.com>
Subject: Re: [stable-4.19 8/8] md: Set prev_flush_start and flush_bio in an atomic way
Date: Thu, 4 Feb 2021 16:12:44 +0100 [thread overview]
Message-ID: <YBwO7EsBfljWNliY@kroah.com> (raw)
In-Reply-To: <20210203132022.92406-9-jinpu.wang@cloud.ionos.com>
On Wed, Feb 03, 2021 at 02:20:22PM +0100, Jack Wang wrote:
> From: Xiao Ni <xni@redhat.com>
>
> One customer reports a crash problem which causes by flush request. It
> triggers a warning before crash.
>
> /* new request after previous flush is completed */
> if (ktime_after(req_start, mddev->prev_flush_start)) {
> WARN_ON(mddev->flush_bio);
> mddev->flush_bio = bio;
> bio = NULL;
> }
>
> The WARN_ON is triggered. We use spin lock to protect prev_flush_start and
> flush_bio in md_flush_request. But there is no lock protection in
> md_submit_flush_data. It can set flush_bio to NULL first because of
> compiler reordering write instructions.
>
> For example, flush bio1 sets flush bio to NULL first in
> md_submit_flush_data. An interrupt or vmware causing an extended stall
> happen between updating flush_bio and prev_flush_start. Because flush_bio
> is NULL, flush bio2 can get the lock and submit to underlayer disks. Then
> flush bio1 updates prev_flush_start after the interrupt or extended stall.
>
> Then flush bio3 enters in md_flush_request. The start time req_start is
> behind prev_flush_start. The flush_bio is not NULL(flush bio2 hasn't
> finished). So it can trigger the WARN_ON now. Then it calls INIT_WORK
> again. INIT_WORK() will re-initialize the list pointers in the
> work_struct, which then can result in a corrupted work list and the
> work_struct queued a second time. With the work list corrupted, it can
> lead in invalid work items being used and cause a crash in
> process_one_work.
>
> We need to make sure only one flush bio can be handled at one same time.
> So add spin lock in md_submit_flush_data to protect prev_flush_start and
> flush_bio in an atomic way.
>
> Reviewed-by: David Jeffery <djeffery@redhat.com>
> Signed-off-by: Xiao Ni <xni@redhat.com>
> Signed-off-by: Song Liu <songliubraving@fb.com>
> [jwang: backport dc5d17a3c39b06aef866afca19245a9cfb533a79 to 4.19]
I can not take patches backported to older kernels that "skip" kernel
releases.
For example, if I take this into 4.19.y, and then someone moves to 5.4
or 5.10, they will hit the same issue.
So please provide a backported series for all affected releases, back as
far as you want, but never skip releases.
I can't take this series, I'll drop it for now and wait for an updated
set of patches.
thanks,
greg k-h
next prev parent reply other threads:[~2021-02-04 15:14 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-03 13:20 [stable-4.19 0/8] Some misc fixes Jack Wang
2021-02-03 13:20 ` [stable-4.19 1/8] block: fix NULL pointer dereference in register_disk Jack Wang
2021-02-03 13:20 ` [stable-4.19 2/8] block: don't hold q->sysfs_lock in elevator_init_mq Jack Wang
2021-02-03 13:20 ` [stable-4.19 3/8] blk-mq: don't hold q->sysfs_lock in blk_mq_map_swqueue Jack Wang
2021-02-03 13:20 ` [stable-4.19 4/8] block: add helper for checking if queue is registered Jack Wang
2021-02-03 13:20 ` [stable-4.19 5/8] block: split .sysfs_lock into two locks Jack Wang
2021-02-03 13:20 ` [stable-4.19 6/8] block: fix race between switching elevator and removing queues Jack Wang
2021-02-03 13:20 ` [stable-4.19 7/8] block: don't release queue's sysfs lock during switching elevator Jack Wang
2021-02-03 13:20 ` [stable-4.19 8/8] md: Set prev_flush_start and flush_bio in an atomic way Jack Wang
2021-02-04 15:12 ` Greg KH [this message]
2021-02-05 6:57 ` Jinpu Wang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YBwO7EsBfljWNliY@kroah.com \
--to=gregkh@linuxfoundation.org \
--cc=djeffery@redhat.com \
--cc=jinpu.wang@cloud.ionos.com \
--cc=sashal@kernel.org \
--cc=songliubraving@fb.com \
--cc=stable@vger.kernel.org \
--cc=xni@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).