From: Hannes Reinecke <hare@suse.de>
To: Yu Kuai <yukuai1@huaweicloud.com>,
Carlos Carvalho <carlos@fisica.ufpr.br>,
Song Liu <song@kernel.org>
Cc: Bagas Sanjaya <bagasdotme@gmail.com>,
Christoph Hellwig <hch@lst.de>, AceLan Kao <acelan@gmail.com>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
Linux Regressions <regressions@lists.linux.dev>,
Linux RAID <linux-raid@vger.kernel.org>,
"yukuai (C)" <yukuai3@huawei.com>,
"yangerkun@huawei.com" <yangerkun@huawei.com>
Subject: Re: Infiniate systemd loop when power off the machine with multiple MD RAIDs
Date: Wed, 23 Aug 2023 08:04:10 +0200 [thread overview]
Message-ID: <c7454021-5559-6556-e9ec-de856e00f1fb@suse.de> (raw)
In-Reply-To: <909a6669-01af-dbe7-b64b-84a5f3bc75a3@huaweicloud.com>
On 8/23/23 03:28, Yu Kuai wrote:
> Hi,
>
> 在 2023/08/23 3:13, Carlos Carvalho 写道:
>> Song Liu (song@kernel.org) wrote on Tue, Aug 22, 2023 at 03:56:04PM -03:
>>> >From systemd code, i.e. function delete_md(), this error:
>>>
>>> [ 205.957004] systemd-shutdown[1]: Stopping MD /dev/md124p1 (259:6).
>>> [ 205.964177] systemd-shutdown[1]: Could not stop MD /dev/md124p1:
>>> Device or resource busy
>>>
>>> is most likely triggered by ioctl(STOP_ARRAY).
>>>
>>> And based on the code, I think the ioctl fails here:
>>>
>>> if (cmd == STOP_ARRAY || cmd == STOP_ARRAY_RO) {
>>> /* Need to flush page cache, and ensure no-one else
>>> opens
>>> * and writes
>>> */
>>> mutex_lock(&mddev->open_mutex);
>>> if (mddev->pers && atomic_read(&mddev->openers) > 1) {
>>> mutex_unlock(&mddev->open_mutex);
>>> err = -EBUSY;
>>> goto out; ////////////////////// HERE
>
> Yes, I suspect here as well, but I do some test with error injection to
> gurantee that ioctl(STOP_ARRAY) always return -EBUSY, but I found that
> system reboot didn't hang, it'll try a few times but eventually reboot
> finished.
>
>>> }
>>> if (test_and_set_bit(MD_CLOSING, &mddev->flags)) {
>>> mutex_unlock(&mddev->open_mutex);
>>> err = -EBUSY;
>>> goto out;
>>> }
>>> did_set_md_closing = true;
>>> mutex_unlock(&mddev->open_mutex);
>>> sync_blockdev(bdev);
>>> }
>>
>> Probably. The problem is why doesn't it manage to flush the page
>> cache? I find
>> strange that the problem appears only when trying to stop the array, I
>> get it
>> when trying to umount the filesystem, where it also hangs because of
>> the same
>> reason. The kworker thread runs continuously using 100% cpu of only 1
>> core.
>
> The kworker do you mean the daemon thread or the sync thread? runs
> continuously using 100% cpu doen't sounds correct to me.
>
That sounds more like kworker waiting on a mutex to be released.
Can you trigger a kdump while in that state?
Or do a sysrq-T to get a stack trace of all running processes?
That should tell you where exactly the 100% cpu time is being
wasted.
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
hare@suse.de +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman
prev parent reply other threads:[~2023-08-23 6:04 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-08-16 9:37 Fwd: Infiniate systemd loop when power off the machine with multiple MD RAIDs Bagas Sanjaya
2023-08-18 8:16 ` Mariusz Tkaczyk
2023-08-18 9:21 ` Hannes Reinecke
2023-08-21 3:23 ` AceLan Kao
2023-08-22 3:51 ` Guoqing Jiang
2023-08-22 6:17 ` Song Liu
2023-08-22 6:39 ` Mariusz Tkaczyk
2023-08-22 8:13 ` AceLan Kao
2023-08-22 12:41 ` Guoqing Jiang
2023-08-23 8:02 ` AceLan Kao
2023-08-23 13:25 ` Song Liu
2023-08-26 4:31 ` AceLan Kao
2023-08-28 5:20 ` Song Liu
2023-08-28 10:48 ` AceLan Kao
2023-08-29 3:12 ` AceLan Kao
2023-08-28 13:50 ` Yu Kuai
2023-08-31 2:28 ` Yu Kuai
2023-08-31 6:50 ` Mariusz Tkaczyk
2023-09-06 6:26 ` AceLan Kao
2023-09-06 10:27 ` Mariusz Tkaczyk
2023-09-07 2:04 ` Yu Kuai
2023-09-07 10:18 ` Mariusz Tkaczyk
[not found] ` <cffca94f-5729-622d-9327-632b3ff2891a@huaweicloud.com>
[not found] ` <3e7edf0c-cadd-59b0-4e10-dffdb86b93b7@huaweicloud.com>
2023-09-07 12:41 ` Mariusz Tkaczyk
2023-09-07 12:53 ` Yu Kuai
2023-09-07 15:09 ` Mariusz Tkaczyk
2023-09-08 20:25 ` Song Liu
2023-08-21 13:18 ` Fwd: " Yu Kuai
2023-08-22 1:39 ` AceLan Kao
2023-08-22 18:56 ` Song Liu
2023-08-22 19:13 ` Carlos Carvalho
2023-08-23 1:28 ` Yu Kuai
2023-08-23 6:04 ` Hannes Reinecke [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=c7454021-5559-6556-e9ec-de856e00f1fb@suse.de \
--to=hare@suse.de \
--cc=acelan@gmail.com \
--cc=bagasdotme@gmail.com \
--cc=carlos@fisica.ufpr.br \
--cc=hch@lst.de \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-raid@vger.kernel.org \
--cc=regressions@lists.linux.dev \
--cc=song@kernel.org \
--cc=yangerkun@huawei.com \
--cc=yukuai1@huaweicloud.com \
--cc=yukuai3@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).