From: Roman Penyaev <roman.penyaev@profitbricks.com>
To: Tejun Heo <tj@kernel.org>
Cc: Akinobu Mita <akinobu.mita@gmail.com>,
Jens Axboe <axboe@kernel.dk>, Christoph Hellwig <hch@lst.de>,
linux-block@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2 1/1] blk-mq: fix hang caused by freeze/unfreeze sequence
Date: Wed, 10 Aug 2016 13:36:58 +0200 [thread overview]
Message-ID: <CAJrWOzCUzPiFUOS5HbToB1AdsMKFGG=UQHAr5=ZL5KYGyf2GLg@mail.gmail.com> (raw)
In-Reply-To: <CAJrWOzCWfMN7qKipgWn2P0dfycu4y5DQysD+7Q1UANgXBgw0+g@mail.gmail.com>
On Wed, Aug 10, 2016 at 10:42 AM, Roman Penyaev
<roman.penyaev@profitbricks.com> wrote:
> Hi,
>
> On Wed, Aug 10, 2016 at 5:55 AM, Tejun Heo <tj@kernel.org> wrote:
>> Hello,
>>
>> On Mon, Aug 08, 2016 at 01:39:08PM +0200, Roman Pen wrote:
>>> Long time ago there was a similar fix proposed by Akinobu Mita[1],
>>> but it seems that time everyone decided to fix this subtle race in
>>> percpu-refcount and Tejun Heo[2] did an attempt (as I can see that
>>> patchset was not applied).
>>
>> So, I probably forgot about it while waiting for confirmation of fix.
>> Can you please verify that the patchset fixes the issue? I can apply
>> the patchset right away.
>
> I have not checked your patchset but according to my understanding
> it should not fix *this* issue.
So, your patchset does not help (but for sure it helps for keeping
internal percpu-refcount members consistent, but that is not related
to this issue). That's the backtrace which I observe:
Call Trace:
[<ffffffff810ba8df>] ? vprintk_default+0x1f/0x30
[<ffffffff816a47f5>] schedule+0x35/0x80
[<ffffffff81336154>] blk_mq_freeze_queue_wait+0x124/0x1a0
[<ffffffff810a3f70>] ? wake_atomic_t_function+0x60/0x60
[<ffffffff8133821a>] blk_mq_freeze_queue+0x1a/0x20
[<ffffffff8133822e>] blk_freeze_queue+0xe/0x10
[<ffffffff81329cc2>] blk_cleanup_queue+0xe2/0x280
To ease reproduction I do the following:
-------------------------------------------
static int thread_fn(void *data)
{
struct blk_mq_tag_set *tags = data;
struct request_queue *q;
while (!kthread_should_stop()) {
q = blk_mq_init_queue(tags);
BUG_ON(q == NULL);
/*
* That is done by blk_register_queue(), but here
* we are reproducing blk-mq bug and do not require
* gendisk and friends. Just silently switch to percpu.
*/
percpu_ref_switch_to_percpu(&q->q_usage_counter);
msleep(prandom_u32_max(10));
blk_cleanup_queue(q);
}
return 0;
}
-------------------------------------------
o Start 2 threads (exactly 2, we need 2 queues for 1 shared tags)
o Pass same shared tags pointer for each thread
o Wait
o PROFIT
To make immediate reproduction this hunk can be applied:
@@ -129,6 +142,7 @@ void blk_mq_unfreeze_queue(struct request_queue *q)
freeze_depth = atomic_dec_return(&q->mq_freeze_depth);
WARN_ON_ONCE(freeze_depth < 0);
if (!freeze_depth) {
+ msleep(1000);
percpu_ref_reinit(&q->q_usage_counter);
wake_up_all(&q->mq_freeze_wq);
}
--
Roman
next prev parent reply other threads:[~2016-08-10 19:34 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-08-08 11:39 [PATCH v2 1/1] blk-mq: fix hang caused by freeze/unfreeze sequence Roman Pen
2016-08-10 3:55 ` Tejun Heo
2016-08-10 8:42 ` Roman Penyaev
2016-08-10 11:36 ` Roman Penyaev [this message]
2016-08-30 9:37 ` 王金浦
2016-08-10 18:54 ` Tejun Heo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAJrWOzCUzPiFUOS5HbToB1AdsMKFGG=UQHAr5=ZL5KYGyf2GLg@mail.gmail.com' \
--to=roman.penyaev@profitbricks.com \
--cc=akinobu.mita@gmail.com \
--cc=axboe@kernel.dk \
--cc=hch@lst.de \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).