* [PATCH v2 1/1] blk-mq: fix hang caused by freeze/unfreeze sequence
@ 2016-08-08 11:39 Roman Pen
2016-08-10 3:55 ` Tejun Heo
0 siblings, 1 reply; 6+ messages in thread
From: Roman Pen @ 2016-08-08 11:39 UTC (permalink / raw)
Cc: Roman Pen, Akinobu Mita, Tejun Heo, Jens Axboe,
Christoph Hellwig, linux-block, linux-kernel
Long time ago there was a similar fix proposed by Akinobu Mita[1],
but it seems that time everyone decided to fix this subtle race in
percpu-refcount and Tejun Heo[2] did an attempt (as I can see that
patchset was not applied).
The following is a description of a hang in blk_mq_freeze_queue_wait() -
same fix but a bug from another angle.
The hang happens on attempt to freeze a queue while another task does
queue unfreeze.
The root cause is an incorrect sequence of percpu_ref_reinit() and
percpu_ref_kill() and as a result those two can be swapped:
CPU#0 CPU#1
---------------- -----------------
percpu_ref_kill()
percpu_ref_kill() << atomic reference does
percpu_ref_reinit() << not guarantee the order
blk_mq_freeze_queue_wait() << HANG HERE
percpu_ref_reinit()
Firstly this wrong sequence raises two kernel warnings:
1st. WARNING at lib/percpu-recount.c:309
percpu_ref_kill_and_confirm called more than once
2nd. WARNING at lib/percpu-refcount.c:331
But the most unpleasant effect is a hang of a blk_mq_freeze_queue_wait(),
which waits for a zero of a q_usage_counter, which never happens
because percpu-ref was reinited (instead of being killed) and stays in
PERCPU state forever.
The simplified sequence above can be reproduced on shared tags, when
queue A is going to die meanwhile another queue B is in init state and
is trying to freeze the queue A, which shares the same tags set:
CPU#0 CPU#1
------------------------------- ------------------------------------
q1 = blk_mq_init_queue(shared_tags)
q2 = blk_mq_init_queue(shared_tags):
blk_mq_add_queue_tag_set(shared_tags):
blk_mq_update_tag_set_depth(shared_tags):
blk_mq_freeze_queue(q1)
blk_cleanup_queue(q1) ...
blk_mq_freeze_queue(q1) <<<->>> blk_mq_unfreeze_queue(q1)
[1] Message id: 1443287365-4244-7-git-send-email-akinobu.mita@gmail.com
[2] Message id: 1443563240-29306-6-git-send-email-tj@kernel.org
Signed-off-by: Roman Pen <roman.penyaev@profitbricks.com>
Cc: Akinobu Mita <akinobu.mita@gmail.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Christoph Hellwig <hch@lst.de>
Cc: linux-block@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
v2:
- forgotten hunk from local repo
- minor tweaks in the commit message
block/blk-core.c | 3 ++-
block/blk-mq.c | 22 +++++++++++-----------
include/linux/blkdev.h | 7 ++++++-
3 files changed, 19 insertions(+), 13 deletions(-)
diff --git a/block/blk-core.c b/block/blk-core.c
index ef78848..4fd27e9 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -658,7 +658,7 @@ int blk_queue_enter(struct request_queue *q, gfp_t gfp)
return -EBUSY;
ret = wait_event_interruptible(q->mq_freeze_wq,
- !atomic_read(&q->mq_freeze_depth) ||
+ !q->mq_freeze_depth ||
blk_queue_dying(q));
if (blk_queue_dying(q))
return -ENODEV;
@@ -740,6 +740,7 @@ struct request_queue *blk_alloc_queue_node(gfp_t gfp_mask, int node_id)
__set_bit(QUEUE_FLAG_BYPASS, &q->queue_flags);
init_waitqueue_head(&q->mq_freeze_wq);
+ mutex_init(&q->mq_freeze_lock);
/*
* Init percpu_ref in atomic mode so that it's faster to shutdown.
diff --git a/block/blk-mq.c b/block/blk-mq.c
index 6d6f8fe..1f3e81b 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -80,13 +80,13 @@ static void blk_mq_hctx_clear_pending(struct blk_mq_hw_ctx *hctx,
void blk_mq_freeze_queue_start(struct request_queue *q)
{
- int freeze_depth;
-
- freeze_depth = atomic_inc_return(&q->mq_freeze_depth);
- if (freeze_depth == 1) {
+ mutex_lock(&q->mq_freeze_lock);
+ if (++q->mq_freeze_depth == 1) {
percpu_ref_kill(&q->q_usage_counter);
+ mutex_unlock(&q->mq_freeze_lock);
blk_mq_run_hw_queues(q, false);
- }
+ } else
+ mutex_unlock(&q->mq_freeze_lock);
}
EXPORT_SYMBOL_GPL(blk_mq_freeze_queue_start);
@@ -124,14 +124,14 @@ EXPORT_SYMBOL_GPL(blk_mq_freeze_queue);
void blk_mq_unfreeze_queue(struct request_queue *q)
{
- int freeze_depth;
-
- freeze_depth = atomic_dec_return(&q->mq_freeze_depth);
- WARN_ON_ONCE(freeze_depth < 0);
- if (!freeze_depth) {
+ mutex_lock(&q->mq_freeze_lock);
+ q->mq_freeze_depth--;
+ WARN_ON_ONCE(q->mq_freeze_depth < 0);
+ if (!q->mq_freeze_depth) {
percpu_ref_reinit(&q->q_usage_counter);
wake_up_all(&q->mq_freeze_wq);
}
+ mutex_unlock(&q->mq_freeze_lock);
}
EXPORT_SYMBOL_GPL(blk_mq_unfreeze_queue);
@@ -2105,7 +2105,7 @@ void blk_mq_free_queue(struct request_queue *q)
static void blk_mq_queue_reinit(struct request_queue *q,
const struct cpumask *online_mask)
{
- WARN_ON_ONCE(!atomic_read(&q->mq_freeze_depth));
+ WARN_ON_ONCE(!q->mq_freeze_depth);
blk_mq_sysfs_unregister(q);
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index f6ff9d1..d692c16 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -445,7 +445,7 @@ struct request_queue {
struct mutex sysfs_lock;
int bypass_depth;
- atomic_t mq_freeze_depth;
+ int mq_freeze_depth;
#if defined(CONFIG_BLK_DEV_BSG)
bsg_job_fn *bsg_job_fn;
@@ -459,6 +459,11 @@ struct request_queue {
#endif
struct rcu_head rcu_head;
wait_queue_head_t mq_freeze_wq;
+ /*
+ * Protect concurrent access to q_usage_counter by
+ * percpu_ref_kill() and percpu_ref_reinit().
+ */
+ struct mutex mq_freeze_lock;
struct percpu_ref q_usage_counter;
struct list_head all_q_node;
--
2.9.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH v2 1/1] blk-mq: fix hang caused by freeze/unfreeze sequence
2016-08-08 11:39 [PATCH v2 1/1] blk-mq: fix hang caused by freeze/unfreeze sequence Roman Pen
@ 2016-08-10 3:55 ` Tejun Heo
2016-08-10 8:42 ` Roman Penyaev
0 siblings, 1 reply; 6+ messages in thread
From: Tejun Heo @ 2016-08-10 3:55 UTC (permalink / raw)
To: Roman Pen
Cc: Akinobu Mita, Jens Axboe, Christoph Hellwig, linux-block, linux-kernel
Hello,
On Mon, Aug 08, 2016 at 01:39:08PM +0200, Roman Pen wrote:
> Long time ago there was a similar fix proposed by Akinobu Mita[1],
> but it seems that time everyone decided to fix this subtle race in
> percpu-refcount and Tejun Heo[2] did an attempt (as I can see that
> patchset was not applied).
So, I probably forgot about it while waiting for confirmation of fix.
Can you please verify that the patchset fixes the issue? I can apply
the patchset right away.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v2 1/1] blk-mq: fix hang caused by freeze/unfreeze sequence
2016-08-10 3:55 ` Tejun Heo
@ 2016-08-10 8:42 ` Roman Penyaev
2016-08-10 11:36 ` Roman Penyaev
2016-08-10 18:54 ` Tejun Heo
0 siblings, 2 replies; 6+ messages in thread
From: Roman Penyaev @ 2016-08-10 8:42 UTC (permalink / raw)
To: Tejun Heo
Cc: Akinobu Mita, Jens Axboe, Christoph Hellwig, linux-block, linux-kernel
Hi,
On Wed, Aug 10, 2016 at 5:55 AM, Tejun Heo <tj@kernel.org> wrote:
> Hello,
>
> On Mon, Aug 08, 2016 at 01:39:08PM +0200, Roman Pen wrote:
>> Long time ago there was a similar fix proposed by Akinobu Mita[1],
>> but it seems that time everyone decided to fix this subtle race in
>> percpu-refcount and Tejun Heo[2] did an attempt (as I can see that
>> patchset was not applied).
>
> So, I probably forgot about it while waiting for confirmation of fix.
> Can you please verify that the patchset fixes the issue? I can apply
> the patchset right away.
I have not checked your patchset but according to my understanding
it should not fix *this* issue. What happens here is a wrong order
of invocation of percpu_ref_reinit() and percpu_ref_kill(). So what
was observed is the following:
CPU#0 CPU#1
---------------- -----------------
percpu_ref_kill()
percpu_ref_kill() << atomic reference does
percpu_ref_reinit() << not guarantee the order
blk_mq_freeze_queue_wait() !! HANG HERE
percpu_ref_reinit()
blk_mq_freeze_queue_wait() on CPU#1 expects percpu-refcount to be
switched to ATOMIC mode (killed), but that does not happen, because
CPU#2 was faster and has been switched percpu-refcount to PERCPU
mode.
This race happens inside blk-mq, because invocation of kill/reinit
is controlled by the reference counter, which does not guarantee the
order of the following functions calls (kill/reinit).
So the fix is the same as originally proposed by Akinobu Mita, but
the issue is different.
But of course I can run tests on top of your series, just to verify
that everything goes smoothly and internally percpu-refcount members
are consistent.
--
Roman
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v2 1/1] blk-mq: fix hang caused by freeze/unfreeze sequence
2016-08-10 8:42 ` Roman Penyaev
@ 2016-08-10 11:36 ` Roman Penyaev
2016-08-30 9:37 ` 王金浦
2016-08-10 18:54 ` Tejun Heo
1 sibling, 1 reply; 6+ messages in thread
From: Roman Penyaev @ 2016-08-10 11:36 UTC (permalink / raw)
To: Tejun Heo
Cc: Akinobu Mita, Jens Axboe, Christoph Hellwig, linux-block, linux-kernel
On Wed, Aug 10, 2016 at 10:42 AM, Roman Penyaev
<roman.penyaev@profitbricks.com> wrote:
> Hi,
>
> On Wed, Aug 10, 2016 at 5:55 AM, Tejun Heo <tj@kernel.org> wrote:
>> Hello,
>>
>> On Mon, Aug 08, 2016 at 01:39:08PM +0200, Roman Pen wrote:
>>> Long time ago there was a similar fix proposed by Akinobu Mita[1],
>>> but it seems that time everyone decided to fix this subtle race in
>>> percpu-refcount and Tejun Heo[2] did an attempt (as I can see that
>>> patchset was not applied).
>>
>> So, I probably forgot about it while waiting for confirmation of fix.
>> Can you please verify that the patchset fixes the issue? I can apply
>> the patchset right away.
>
> I have not checked your patchset but according to my understanding
> it should not fix *this* issue.
So, your patchset does not help (but for sure it helps for keeping
internal percpu-refcount members consistent, but that is not related
to this issue). That's the backtrace which I observe:
Call Trace:
[<ffffffff810ba8df>] ? vprintk_default+0x1f/0x30
[<ffffffff816a47f5>] schedule+0x35/0x80
[<ffffffff81336154>] blk_mq_freeze_queue_wait+0x124/0x1a0
[<ffffffff810a3f70>] ? wake_atomic_t_function+0x60/0x60
[<ffffffff8133821a>] blk_mq_freeze_queue+0x1a/0x20
[<ffffffff8133822e>] blk_freeze_queue+0xe/0x10
[<ffffffff81329cc2>] blk_cleanup_queue+0xe2/0x280
To ease reproduction I do the following:
-------------------------------------------
static int thread_fn(void *data)
{
struct blk_mq_tag_set *tags = data;
struct request_queue *q;
while (!kthread_should_stop()) {
q = blk_mq_init_queue(tags);
BUG_ON(q == NULL);
/*
* That is done by blk_register_queue(), but here
* we are reproducing blk-mq bug and do not require
* gendisk and friends. Just silently switch to percpu.
*/
percpu_ref_switch_to_percpu(&q->q_usage_counter);
msleep(prandom_u32_max(10));
blk_cleanup_queue(q);
}
return 0;
}
-------------------------------------------
o Start 2 threads (exactly 2, we need 2 queues for 1 shared tags)
o Pass same shared tags pointer for each thread
o Wait
o PROFIT
To make immediate reproduction this hunk can be applied:
@@ -129,6 +142,7 @@ void blk_mq_unfreeze_queue(struct request_queue *q)
freeze_depth = atomic_dec_return(&q->mq_freeze_depth);
WARN_ON_ONCE(freeze_depth < 0);
if (!freeze_depth) {
+ msleep(1000);
percpu_ref_reinit(&q->q_usage_counter);
wake_up_all(&q->mq_freeze_wq);
}
--
Roman
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v2 1/1] blk-mq: fix hang caused by freeze/unfreeze sequence
2016-08-10 11:36 ` Roman Penyaev
@ 2016-08-30 9:37 ` 王金浦
0 siblings, 0 replies; 6+ messages in thread
From: 王金浦 @ 2016-08-30 9:37 UTC (permalink / raw)
To: Jens Axboe
Cc: Tejun Heo, Akinobu Mita, Roman Penyaev, Christoph Hellwig,
linux-block, LKML
2016-08-10 13:36 GMT+02:00 Roman Penyaev <roman.penyaev@profitbricks.com>:
> On Wed, Aug 10, 2016 at 10:42 AM, Roman Penyaev
> <roman.penyaev@profitbricks.com> wrote:
>> Hi,
>>
>> On Wed, Aug 10, 2016 at 5:55 AM, Tejun Heo <tj@kernel.org> wrote:
>>> Hello,
>>>
>>> On Mon, Aug 08, 2016 at 01:39:08PM +0200, Roman Pen wrote:
>>>> Long time ago there was a similar fix proposed by Akinobu Mita[1],
>>>> but it seems that time everyone decided to fix this subtle race in
>>>> percpu-refcount and Tejun Heo[2] did an attempt (as I can see that
>>>> patchset was not applied).
>>>
>>> So, I probably forgot about it while waiting for confirmation of fix.
>>> Can you please verify that the patchset fixes the issue? I can apply
>>> the patchset right away.
>>
>> I have not checked your patchset but according to my understanding
>> it should not fix *this* issue.
>
> So, your patchset does not help (but for sure it helps for keeping
> internal percpu-refcount members consistent, but that is not related
> to this issue). That's the backtrace which I observe:
>
> Call Trace:
> [<ffffffff810ba8df>] ? vprintk_default+0x1f/0x30
> [<ffffffff816a47f5>] schedule+0x35/0x80
> [<ffffffff81336154>] blk_mq_freeze_queue_wait+0x124/0x1a0
> [<ffffffff810a3f70>] ? wake_atomic_t_function+0x60/0x60
> [<ffffffff8133821a>] blk_mq_freeze_queue+0x1a/0x20
> [<ffffffff8133822e>] blk_freeze_queue+0xe/0x10
> [<ffffffff81329cc2>] blk_cleanup_queue+0xe2/0x280
>
> To ease reproduction I do the following:
>
> -------------------------------------------
> static int thread_fn(void *data)
> {
> struct blk_mq_tag_set *tags = data;
> struct request_queue *q;
>
> while (!kthread_should_stop()) {
> q = blk_mq_init_queue(tags);
> BUG_ON(q == NULL);
> /*
> * That is done by blk_register_queue(), but here
> * we are reproducing blk-mq bug and do not require
> * gendisk and friends. Just silently switch to percpu.
> */
> percpu_ref_switch_to_percpu(&q->q_usage_counter);
>
> msleep(prandom_u32_max(10));
> blk_cleanup_queue(q);
> }
>
> return 0;
> }
> -------------------------------------------
>
> o Start 2 threads (exactly 2, we need 2 queues for 1 shared tags)
> o Pass same shared tags pointer for each thread
> o Wait
> o PROFIT
>
> To make immediate reproduction this hunk can be applied:
>
> @@ -129,6 +142,7 @@ void blk_mq_unfreeze_queue(struct request_queue *q)
> freeze_depth = atomic_dec_return(&q->mq_freeze_depth);
> WARN_ON_ONCE(freeze_depth < 0);
> if (!freeze_depth) {
> + msleep(1000);
> percpu_ref_reinit(&q->q_usage_counter);
> wake_up_all(&q->mq_freeze_wq);
> }
>
> --
> Roman
Hi Jens,
I didn't see this patch in you tree, what's the blocker?
Thanks,
Jinpu
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v2 1/1] blk-mq: fix hang caused by freeze/unfreeze sequence
2016-08-10 8:42 ` Roman Penyaev
2016-08-10 11:36 ` Roman Penyaev
@ 2016-08-10 18:54 ` Tejun Heo
1 sibling, 0 replies; 6+ messages in thread
From: Tejun Heo @ 2016-08-10 18:54 UTC (permalink / raw)
To: Roman Penyaev
Cc: Akinobu Mita, Jens Axboe, Christoph Hellwig, linux-block, linux-kernel
On Wed, Aug 10, 2016 at 10:42:09AM +0200, Roman Penyaev wrote:
> Hi,
>
> On Wed, Aug 10, 2016 at 5:55 AM, Tejun Heo <tj@kernel.org> wrote:
> > Hello,
> >
> > On Mon, Aug 08, 2016 at 01:39:08PM +0200, Roman Pen wrote:
> >> Long time ago there was a similar fix proposed by Akinobu Mita[1],
> >> but it seems that time everyone decided to fix this subtle race in
> >> percpu-refcount and Tejun Heo[2] did an attempt (as I can see that
> >> patchset was not applied).
> >
> > So, I probably forgot about it while waiting for confirmation of fix.
> > Can you please verify that the patchset fixes the issue? I can apply
> > the patchset right away.
>
> I have not checked your patchset but according to my understanding
> it should not fix *this* issue. What happens here is a wrong order
> of invocation of percpu_ref_reinit() and percpu_ref_kill(). So what
> was observed is the following:
Ah, understood.
Acked-by: Tejun Heo <tj@kernel.org>
I'll commit the percpu_refcnt patches too. While they don't fix the
problem on their own, the changes are generally useful for all mode
switching use cases.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-08-30 9:37 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-08-08 11:39 [PATCH v2 1/1] blk-mq: fix hang caused by freeze/unfreeze sequence Roman Pen
2016-08-10 3:55 ` Tejun Heo
2016-08-10 8:42 ` Roman Penyaev
2016-08-10 11:36 ` Roman Penyaev
2016-08-30 9:37 ` 王金浦
2016-08-10 18:54 ` Tejun Heo
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).