From: Song Liu <songliubraving@fb.com>
To: Guoqing Jiang <guoqing.jiang@cloud.ionos.com>
Cc: Vojtech Myslivec <vojtech@xmyslivec.cz>,
"linux-btrfs@vger.kernel.org" <linux-btrfs@vger.kernel.org>,
"linux-raid@vger.kernel.org" <linux-raid@vger.kernel.org>,
Michal Moravec <michal.moravec@logicworks.cz>
Subject: Re: Linux RAID with btrfs stuck and consume 100 % CPU
Date: Thu, 30 Jul 2020 06:45:04 +0000 [thread overview]
Message-ID: <D8373CAD-7BB0-4DB9-AB6C-7BF0BE035286@fb.com> (raw)
In-Reply-To: <a070c45a-0509-e900-e3f3-98d20267c8c9@cloud.ionos.com>
> On Jul 29, 2020, at 2:06 PM, Guoqing Jiang <guoqing.jiang@cloud.ionos.com> wrote:
>
> Hi,
>
> On 7/22/20 10:47 PM, Vojtech Myslivec wrote:
>> 1. What should be the cause of this problem?
>
> Just a quick glance based on the stacks which you attached, I guess it could be
> a deadlock issue of raid5 cache super write.
>
> Maybe the commit 8e018c21da3f ("raid5-cache: fix a deadlock in superblock
> write") didn't fix the problem completely. Cc Song.
>
> And I am curious why md thread is not waked if mddev_trylock fails, you can
> give it a try but I can't promise it helps ...
>
> --- a/drivers/md/raid5-cache.c
> +++ b/drivers/md/raid5-cache.c
> @@ -1337,8 +1337,10 @@ static void r5l_write_super_and_discard_space(struct r5l_log *log,
> */
> set_mask_bits(&mddev->sb_flags, 0,
> BIT(MD_SB_CHANGE_DEVS) | BIT(MD_SB_CHANGE_PENDING));
> - if (!mddev_trylock(mddev))
> + if (!mddev_trylock(mddev)) {
> + md_wakeup_thread(mddev->thread);
> return;
> + }
> md_update_sb(mddev, 1);
> mddev_unlock(mddev);
>
Thanks Guoqing!
I am not sure whether we hit the mddev_trylock() failure. Looks like the
md1_raid6 thread is already running at 100%.
A few questions:
1. I see wbt_wait in the stack trace. Are we using write back throttling here?
2. Could you please get the /proc/<pid>/stack for <pid> of md1_raid6? We may
want to sample it multiple times.
Thanks,
Song
next prev parent reply other threads:[~2020-07-30 6:45 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-07-22 20:47 Linux RAID with btrfs stuck and consume 100 % CPU Vojtech Myslivec
2020-07-22 22:00 ` antlists
2020-07-23 2:08 ` Chris Murphy
[not found] ` <29509e08-e373-b352-d696-fcb9f507a545@xmyslivec.cz>
2020-07-28 20:23 ` Chris Murphy
[not found] ` <695936b4-67a2-c862-9cb6-5545b4ab3c42@xmyslivec.cz>
2020-08-14 20:04 ` Chris Murphy
[not found] ` <2f2f1c21-c81b-55aa-6f77-e2d3f32d32cb@xmyslivec.cz>
2020-08-19 22:58 ` Chris Murphy
2020-08-19 23:11 ` Peter Grandi
2020-08-26 15:35 ` Vojtech Myslivec
2020-08-26 18:07 ` Chris Murphy
2020-09-16 9:42 ` Vojtech Myslivec
2020-09-17 17:08 ` Chris Murphy
2020-09-17 17:20 ` Chris Murphy
2020-09-17 17:43 ` Chris Murphy
2020-09-23 18:14 ` Vojtech Myslivec
[not found] ` <DBB07C8C-0D83-47DC-9B91-78AD385775E3@snapdragon.cc>
[not found] ` <D3026A55-A7F2-4432-87A8-3E9B2CACE4C2@snapdragon.cc>
[not found] ` <56AD80D0-6853-4E3A-A94C-AD1477D3FDA4@snapdragon.cc>
2021-03-17 15:55 ` Vojtech Myslivec
2020-07-29 21:06 ` Guoqing Jiang
2020-07-29 21:48 ` Chris Murphy
2020-08-12 14:19 ` Vojtech Myslivec
2020-07-30 6:45 ` Song Liu [this message]
2020-08-12 13:58 ` Vojtech Myslivec
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=D8373CAD-7BB0-4DB9-AB6C-7BF0BE035286@fb.com \
--to=songliubraving@fb.com \
--cc=guoqing.jiang@cloud.ionos.com \
--cc=linux-btrfs@vger.kernel.org \
--cc=linux-raid@vger.kernel.org \
--cc=michal.moravec@logicworks.cz \
--cc=vojtech@xmyslivec.cz \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).