From: Roman Penyaev <rpenyaev@suse.de>
To: Bart Van Assche <bvanassche@acm.org>
Cc: Bob Liu <bob.liu@oracle.com>,
linux-block@vger.kernel.org, shirley.ma@oracle.com,
martin.petersen@oracle.com,
Roman Pen <roman.penyaev@profitbricks.com>,
Akinobu Mita <akinobu.mita@gmail.com>, Tejun Heo <tj@kernel.org>,
Jens Axboe <axboe@kernel.dk>, Christoph Hellwig <hch@lst.de>,
linux-kernel@vger.kernel.org, linux-block-owner@vger.kernel.org
Subject: Re: [RESEND PATCH] blk-mq: fix hang caused by freeze/unfreeze sequence
Date: Mon, 15 Apr 2019 11:46:04 +0200 [thread overview]
Message-ID: <84cf04edf8f3f3b87b78383a1837aff3@suse.de> (raw)
In-Reply-To: <0763cb5a-5598-69e3-e5ac-765989aab5b1@acm.org>
On 2019-04-13 05:42, Bart Van Assche wrote:
> On 4/9/19 2:08 AM, Bob Liu wrote:
>> void blk_freeze_queue_start(struct request_queue *q)
>> {
>> - int freeze_depth;
>> -
>> - freeze_depth = atomic_inc_return(&q->mq_freeze_depth);
>> - if (freeze_depth == 1) {
>> + mutex_lock(&q->mq_freeze_lock);
>> + if (++q->mq_freeze_depth == 1) {
>> percpu_ref_kill(&q->q_usage_counter);
>> + mutex_unlock(&q->mq_freeze_lock);
>> if (queue_is_mq(q))
>> blk_mq_run_hw_queues(q, false);
>> + } else {
>> + mutex_unlock(&q->mq_freeze_lock);
>> }
>> }
> Have you considered to move the mutex_unlock() call to the end of the
> function
> such that there is only one mutex_unlock() call instead of two? In case
> you
> would be worried about holding the mutex around the code that runs the
> queue,
> how about changing the blk_mq_run_hw_queues() call such that the queues
> are
> run async?
Hi Bart,
The only purpose of 'mq_freeze_lock' is to avoid race between
mq_freeze_depth
variable and the following usage of q_usage_counter percpu ref. I admit
that
my original comment is quite unclear, but locked section should be as
short
as possible, so returning to your question: better to have two unlock
calls
instead of expanding locked critical section.
Unfortunately I do not have hardware to play again with the issue, but I
see
there is a nice candidate for a quick reproduction: null_blk queues
with
shared tags. Having several queues with shared tags and a script, which
powers on/off (I mean 'power' entry of configfs of the null_blk)
different
null devices from different cpus it is quite possible to trigger the
issue.
Random short msdelay() in correct places can help to increase
probability to
hit the issue quite fast.
But Bob, what is the backtrace of the issue you hit? What is the
device?
Conditions to reproduce the issue are quite specific and frankly I did
not
find any "naked" (without any locks) calls of blk_mq_freeze/unfreeze
sequence,
the only candidate which I found, seems, null_blk (not 100% sure, but
worth to
try).
--
Roman
next prev parent reply other threads:[~2019-04-15 9:46 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-04-09 9:08 [RESEND PATCH] blk-mq: fix hang caused by freeze/unfreeze sequence Bob Liu
2019-04-09 9:29 ` Jinpu Wang
2019-04-13 0:36 ` Bob Liu
2019-04-09 11:27 ` Dongli Zhang
2019-04-13 3:42 ` Bart Van Assche
2019-04-14 13:09 ` Bob Liu
2019-04-15 9:46 ` Roman Penyaev [this message]
2019-04-17 4:06 ` Bob Liu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=84cf04edf8f3f3b87b78383a1837aff3@suse.de \
--to=rpenyaev@suse.de \
--cc=akinobu.mita@gmail.com \
--cc=axboe@kernel.dk \
--cc=bob.liu@oracle.com \
--cc=bvanassche@acm.org \
--cc=hch@lst.de \
--cc=linux-block-owner@vger.kernel.org \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=martin.petersen@oracle.com \
--cc=roman.penyaev@profitbricks.com \
--cc=shirley.ma@oracle.com \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).