From: Yu Kuai <yukuai1@huaweicloud.com>
To: Jan Kara <jack@suse.cz>, Yu Kuai <yukuai1@huaweicloud.com>
Cc: Ming Lei <ming.lei@redhat.com>,
Chris Murphy <lists@colorremedies.com>,
Nikolay Borisov <nborisov@suse.com>, Jens Axboe <axboe@kernel.dk>,
Paolo Valente <paolo.valente@linaro.org>,
Btrfs BTRFS <linux-btrfs@vger.kernel.org>,
Linux-RAID <linux-raid@vger.kernel.org>,
linux-block <linux-block@vger.kernel.org>,
linux-kernel <linux-kernel@vger.kernel.org>,
Josef Bacik <josef@toxicpanda.com>,
"yukuai (C)" <yukuai3@huawei.com>
Subject: Re: stalling IO regression since linux 5.12, through 5.18
Date: Thu, 1 Sep 2022 16:19:16 +0800 [thread overview]
Message-ID: <ee48bdc1-1020-78ca-a90e-ef958171a05f@huaweicloud.com> (raw)
In-Reply-To: <20220901080336.glpv4i3hyae2zkpk@quack3>
在 2022/09/01 16:03, Jan Kara 写道:
> On Thu 01-09-22 15:02:03, Yu Kuai wrote:
>> Hi, Chris
>>
>> 在 2022/08/20 15:00, Ming Lei 写道:
>>> On Fri, Aug 19, 2022 at 03:20:25PM -0400, Chris Murphy wrote:
>>>>
>>>>
>>>> On Thu, Aug 18, 2022, at 1:24 AM, Ming Lei wrote:
>>>>> On Thu, Aug 18, 2022 at 12:27:04AM -0400, Chris Murphy wrote:
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 18, 2022, at 12:18 AM, Chris Murphy wrote:
>>>>>>> On Thu, Aug 18, 2022, at 12:12 AM, Chris Murphy wrote:
>>>>>>>> On Wed, Aug 17, 2022, at 11:41 PM, Ming Lei wrote:
>>>>>>>>
>>>>>>>>> OK, can you post the blk-mq debugfs log after you trigger it on v5.17?
>>>>>>
>>>>>> Same boot, 3rd log. But the load is above 300 so I kinda need to sysrq+b soon.
>>>>>>
>>>>>> https://drive.google.com/file/d/1375H558kqPTdng439rvG6LuXXWPXLToo/view?usp=sharing
>>>>>>
>>>>>
>>>>> Also please test the following one too:
>>>>>
>>>>>
>>>>> diff --git a/block/blk-mq.c b/block/blk-mq.c
>>>>> index 5ee62b95f3e5..d01c64be08e2 100644
>>>>> --- a/block/blk-mq.c
>>>>> +++ b/block/blk-mq.c
>>>>> @@ -1991,7 +1991,8 @@ bool blk_mq_dispatch_rq_list(struct blk_mq_hw_ctx
>>>>> *hctx, struct list_head *list,
>>>>> if (!needs_restart ||
>>>>> (no_tag && list_empty_careful(&hctx->dispatch_wait.entry)))
>>>>> blk_mq_run_hw_queue(hctx, true);
>>>>> - else if (needs_restart && needs_resource)
>>>>> + else if (needs_restart && (needs_resource ||
>>>>> + blk_mq_is_shared_tags(hctx->flags)))
>>>>> blk_mq_delay_run_hw_queue(hctx, BLK_MQ_RESOURCE_DELAY);
>>>>>
>>>>> blk_mq_update_dispatch_busy(hctx, true);
>>>>>
>>>>
>>>>
>>>> With just this patch on top of 5.17.0, it still hangs. I've captured block debugfs log:
>>>> https://drive.google.com/file/d/1ic4YHxoL9RrCdy_5FNdGfh_q_J3d_Ft0/view?usp=sharing
>>>
>>> The log is similar with before, and the only difference is RESTART not
>>> set.
>>>
>>> Also follows another patch merged to v5.18 and it fixes io stall too, feel free to test it:
>>>
>>> 8f5fea65b06d blk-mq: avoid extending delays of active hctx from blk_mq_delay_run_hw_queues
>>
>> Have you tried this patch?
>>
>> We meet a similar problem in our test, and I'm pretty sure about the
>> situation at the scene,
>>
>> Our test environment:nvme with bfq ioscheduler,
>>
>> How io is stalled:
>>
>> 1. hctx1 dispatch rq from bfq in service queue, bfqq becomes empty,
>> dispatch somehow fails and rq is inserted to hctx1->dispatch, new run
>> work is queued.
>>
>> 2. other hctx tries to dispatch rq, however, in service bfqq is
>> empty, bfq_dispatch_request return NULL, thus
>> blk_mq_delay_run_hw_queues is called.
>>
>> 3. for the problem described in above patch,run work from "hctx1"
>> can be stalled.
>>
>> Above patch should fix this io stall, however, it seems to me bfq do
>> have some problems that in service bfqq doesn't expire under following
>> situation:
>>
>> 1. dispatched rqs don't complete
>> 2. no new rq is issued to bfq
>
> And I guess:
> 3. there are requests queued in other bfqqs
> ?
Yes, of course, other bfqqs still have requests, but current
implementation have flaws that even if other bfqqs doesn't have
requests, bfq_asymmetric_scenario() can still return true because
num_groups_with_pending_reqs > 0. We tried to fix this, however, there
seems to be some misunderstanding with Paolo, and it's not applied to
mainline yet...
Thanks,
Kuai
>
> Otherwise I don't see a point in expiring current bfqq because there's
> nothing bfq could do anyway. But under normal circumstances the request
> completion should not take so long so I don't think it would be really
> worth it to implement some special mechanism for this in bfq.
>
> Honza
>
next prev parent reply other threads:[~2022-09-01 8:19 UTC|newest]
Thread overview: 58+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-08-10 16:35 stalling IO regression in linux 5.12 Chris Murphy
2022-08-10 17:48 ` Josef Bacik
2022-08-10 18:33 ` Chris Murphy
2022-08-10 18:42 ` Chris Murphy
2022-08-10 19:31 ` Josef Bacik
2022-08-10 19:34 ` Chris Murphy
2022-08-12 16:05 ` stalling IO regression since linux 5.12, through 5.18 Chris Murphy
2022-08-12 17:59 ` Josef Bacik
2022-08-12 18:02 ` Jens Axboe
2022-08-14 20:28 ` Chris Murphy
2022-08-16 14:22 ` Chris Murphy
2022-08-16 15:25 ` Nikolay Borisov
2022-08-16 15:34 ` Chris Murphy
2022-08-17 9:52 ` Holger Hoffstätte
2022-08-17 11:49 ` Jan Kara
2022-08-17 14:37 ` Chris Murphy
2022-08-17 15:09 ` Chris Murphy
2022-08-17 16:30 ` Jan Kara
2022-08-17 16:47 ` Chris Murphy
2022-08-17 17:57 ` Chris Murphy
2022-08-17 18:15 ` Jan Kara
2022-08-17 18:18 ` Chris Murphy
2022-08-17 18:33 ` Jan Kara
2022-08-17 18:54 ` Chris Murphy
2022-08-17 19:23 ` Chris Murphy
2022-08-18 2:31 ` Chris Murphy
2022-08-17 18:21 ` Holger Hoffstätte
2022-08-17 11:57 ` Chris Murphy
2022-08-17 12:31 ` Holger Hoffstätte
2022-08-17 18:16 ` Chris Murphy
2022-08-17 18:38 ` Holger Hoffstätte
2022-08-17 12:06 ` Ming Lei
2022-08-17 14:34 ` Chris Murphy
2022-08-17 14:53 ` Ming Lei
2022-08-17 15:02 ` Chris Murphy
2022-08-17 15:34 ` Ming Lei
2022-08-17 16:34 ` Chris Murphy
2022-08-18 1:03 ` Ming Lei
2022-08-18 2:30 ` Chris Murphy
2022-08-18 3:24 ` Ming Lei
2022-08-18 4:12 ` Chris Murphy
2022-08-18 4:18 ` Chris Murphy
2022-08-18 4:27 ` Chris Murphy
2022-08-18 4:32 ` Chris Murphy
2022-08-18 5:15 ` Ming Lei
2022-08-18 18:52 ` Chris Murphy
2022-08-18 5:24 ` Ming Lei
2022-08-18 13:50 ` Chris Murphy
2022-08-18 15:10 ` Ming Lei
2022-08-19 19:20 ` Chris Murphy
2022-08-20 7:00 ` Ming Lei
2022-09-01 7:02 ` Yu Kuai
2022-09-01 8:03 ` Jan Kara
2022-09-01 8:19 ` Yu Kuai [this message]
2022-09-06 9:49 ` Paolo Valente
2022-09-02 16:53 ` Chris Murphy
2022-09-06 9:45 ` Paolo Valente
2022-08-15 11:25 ` stalling IO regression in linux 5.12 Thorsten Leemhuis
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ee48bdc1-1020-78ca-a90e-ef958171a05f@huaweicloud.com \
--to=yukuai1@huaweicloud.com \
--cc=axboe@kernel.dk \
--cc=jack@suse.cz \
--cc=josef@toxicpanda.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-btrfs@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-raid@vger.kernel.org \
--cc=lists@colorremedies.com \
--cc=ming.lei@redhat.com \
--cc=nborisov@suse.com \
--cc=paolo.valente@linaro.org \
--cc=yukuai3@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).