From: Jens Axboe <axboe@kernel.dk>
To: Josef Bacik <josef@toxicpanda.com>,
Chris Murphy <lists@colorremedies.com>
Cc: Paolo Valente <paolo.valente@linaro.org>,
Btrfs BTRFS <linux-btrfs@vger.kernel.org>,
Linux-RAID <linux-raid@vger.kernel.org>,
linux-block <linux-block@vger.kernel.org>,
linux-kernel <linux-kernel@vger.kernel.org>,
Jan Kara <jack@suse.cz>
Subject: Re: stalling IO regression since linux 5.12, through 5.18
Date: Fri, 12 Aug 2022 12:02:55 -0600 [thread overview]
Message-ID: <d48c7e95-e21e-dcdc-a776-8ae7bed566cb@kernel.dk> (raw)
In-Reply-To: <CAEzrpqe3rRTvH=s+-aXTtupn-XaCxe0=KUe_iQfEyHWp-pXb5w@mail.gmail.com>
On 8/12/22 11:59 AM, Josef Bacik wrote:
> On Fri, Aug 12, 2022 at 12:05 PM Chris Murphy <lists@colorremedies.com> wrote:
>>
>>
>>
>> On Wed, Aug 10, 2022, at 3:34 PM, Chris Murphy wrote:
>>> Booted with cgroup_disable=io, and confirmed cat
>>> /sys/fs/cgroup/cgroup.controllers does not list io.
>>
>> The problem still reproduces with the cgroup IO controller disabled.
>>
>> On a whim, I decided to switch the IO scheduler from Fedora's default bfq for rotating drives to mq-deadline. The problem does not reproduce for 15+ hours, which is not 100% conclusive but probably 99% conclusive. I then switched live while running the workload to bfq on all eight drives, and within 10 minutes the system cratered, all new commands just hang. Load average goes to triple digits, i/o wait increasing, i/o pressure for the workload tasks to 100%, and IO completely stalls to zero. I was able to switch only two of the drive queues back to mq-deadline and then lost responsivness in that shell and had to issue sysrq+b...
>>
>> Before that I was able to extra sysrq+w and sysrq+t.
>> https://drive.google.com/file/d/16hdQjyBnuzzQIhiQT6fQdE0nkRQJj7EI/view?usp=sharing
>>
>> I can't tell if this is a bfq bug, or if there's some negative interaction between bfq and scsi or megaraid_sas. Obviously it's rare because otherwise people would have been falling over this much sooner. But at this point there's strong correlation that it's bfq related and is a kernel regression that's been around since 5.12.0 through 5.18.0, and I suspect also 5.19.0 but it's being partly masked by other improvements.
>
> This matches observations we've had internally (inside Facebook) as
> well as my continual integration performance testing. It should
> probably be looked into by the BFQ guys as it was working previously.
> Thanks,
5.12 has a few BFQ changes:
Jan Kara:
bfq: Avoid false bfq queue merging
bfq: Use 'ttime' local variable
bfq: Use only idle IO periods for think time calculations
Jia Cheng Hu
block, bfq: set next_rq to waker_bfqq->next_rq in waker injection
Paolo Valente
block, bfq: use half slice_idle as a threshold to check short ttime
block, bfq: increase time window for waker detection
block, bfq: do not raise non-default weights
block, bfq: avoid spurious switches to soft_rt of interactive queues
block, bfq: do not expire a queue when it is the only busy one
block, bfq: replace mechanism for evaluating I/O intensity
block, bfq: re-evaluate convenience of I/O plugging on rq arrivals
block, bfq: fix switch back from soft-rt weitgh-raising
block, bfq: save also weight-raised service on queue merging
block, bfq: save also injection state on queue merging
block, bfq: make waker-queue detection more robust
Might be worth trying to revert those from 5.12 to see if they are
causing the issue? Jan, Paolo - does this ring any bells?
--
Jens Axboe
next prev parent reply other threads:[~2022-08-12 18:03 UTC|newest]
Thread overview: 58+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-08-10 16:35 stalling IO regression in linux 5.12 Chris Murphy
2022-08-10 17:48 ` Josef Bacik
2022-08-10 18:33 ` Chris Murphy
2022-08-10 18:42 ` Chris Murphy
2022-08-10 19:31 ` Josef Bacik
2022-08-10 19:34 ` Chris Murphy
2022-08-12 16:05 ` stalling IO regression since linux 5.12, through 5.18 Chris Murphy
2022-08-12 17:59 ` Josef Bacik
2022-08-12 18:02 ` Jens Axboe [this message]
2022-08-14 20:28 ` Chris Murphy
2022-08-16 14:22 ` Chris Murphy
2022-08-16 15:25 ` Nikolay Borisov
2022-08-16 15:34 ` Chris Murphy
2022-08-17 9:52 ` Holger Hoffstätte
2022-08-17 11:49 ` Jan Kara
2022-08-17 14:37 ` Chris Murphy
2022-08-17 15:09 ` Chris Murphy
2022-08-17 16:30 ` Jan Kara
2022-08-17 16:47 ` Chris Murphy
2022-08-17 17:57 ` Chris Murphy
2022-08-17 18:15 ` Jan Kara
2022-08-17 18:18 ` Chris Murphy
2022-08-17 18:33 ` Jan Kara
2022-08-17 18:54 ` Chris Murphy
2022-08-17 19:23 ` Chris Murphy
2022-08-18 2:31 ` Chris Murphy
2022-08-17 18:21 ` Holger Hoffstätte
2022-08-17 11:57 ` Chris Murphy
2022-08-17 12:31 ` Holger Hoffstätte
2022-08-17 18:16 ` Chris Murphy
2022-08-17 18:38 ` Holger Hoffstätte
2022-08-17 12:06 ` Ming Lei
2022-08-17 14:34 ` Chris Murphy
2022-08-17 14:53 ` Ming Lei
2022-08-17 15:02 ` Chris Murphy
2022-08-17 15:34 ` Ming Lei
2022-08-17 16:34 ` Chris Murphy
2022-08-18 1:03 ` Ming Lei
2022-08-18 2:30 ` Chris Murphy
2022-08-18 3:24 ` Ming Lei
2022-08-18 4:12 ` Chris Murphy
2022-08-18 4:18 ` Chris Murphy
2022-08-18 4:27 ` Chris Murphy
2022-08-18 4:32 ` Chris Murphy
2022-08-18 5:15 ` Ming Lei
2022-08-18 18:52 ` Chris Murphy
2022-08-18 5:24 ` Ming Lei
2022-08-18 13:50 ` Chris Murphy
2022-08-18 15:10 ` Ming Lei
2022-08-19 19:20 ` Chris Murphy
2022-08-20 7:00 ` Ming Lei
2022-09-01 7:02 ` Yu Kuai
2022-09-01 8:03 ` Jan Kara
2022-09-01 8:19 ` Yu Kuai
2022-09-06 9:49 ` Paolo Valente
2022-09-02 16:53 ` Chris Murphy
2022-09-06 9:45 ` Paolo Valente
2022-08-15 11:25 ` stalling IO regression in linux 5.12 Thorsten Leemhuis
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=d48c7e95-e21e-dcdc-a776-8ae7bed566cb@kernel.dk \
--to=axboe@kernel.dk \
--cc=jack@suse.cz \
--cc=josef@toxicpanda.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-btrfs@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-raid@vger.kernel.org \
--cc=lists@colorremedies.com \
--cc=paolo.valente@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).