* [PATCH] blk-mq: fix a hung issue when fsync
@ 2019-01-30 9:01 Jianchao Wang
2019-01-30 15:54 ` Jens Axboe
0 siblings, 1 reply; 4+ messages in thread
From: Jianchao Wang @ 2019-01-30 9:01 UTC (permalink / raw)
To: axboe; +Cc: m19, linux-block, linux-kernel
Florian reported a io hung issue when fsync(). It should be
triggered by following race condition.
data + post flush a flush
blk_flush_complete_seq
case REQ_FSEQ_DATA
blk_flush_queue_rq
issued to driver blk_mq_dispatch_rq_list
try to issue a flush req
failed due to NON-NCQ command
.queue_rq return BLK_STS_DEV_RESOURCE
request completion
req->end_io // doesn't check RESTART
mq_flush_data_end_io
case REQ_FSEQ_POSTFLUSH
blk_kick_flush
do nothing because previous flush
has not been completed
blk_mq_run_hw_queue
insert rq to hctx->dispatch
due to RESTART is still set, do nothing
To fix this, replace the blk_mq_run_hw_queue in mq_flush_data_end_io
with blk_mq_sched_restart to check and clear the RESTART flag.
Fixes: bd166ef1 (blk-mq-sched: add framework for MQ capable IO schedulers)
Reported-by: Florian Stecker <m19@florianstecker.de>
Tested-by: Florian Stecker <m19@florianstecker.de>
Signed-off-by: Jianchao Wang <jianchao.w.wang@oracle.com>
---
block/blk-flush.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/block/blk-flush.c b/block/blk-flush.c
index a3fc7191..6e0f2d9 100644
--- a/block/blk-flush.c
+++ b/block/blk-flush.c
@@ -335,7 +335,7 @@ static void mq_flush_data_end_io(struct request *rq, blk_status_t error)
blk_flush_complete_seq(rq, fq, REQ_FSEQ_DATA, error);
spin_unlock_irqrestore(&fq->mq_flush_lock, flags);
- blk_mq_run_hw_queue(hctx, true);
+ blk_mq_sched_restart(hctx);
}
/**
--
2.7.4
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH] blk-mq: fix a hung issue when fsync
2019-01-30 9:01 [PATCH] blk-mq: fix a hung issue when fsync Jianchao Wang
@ 2019-01-30 15:54 ` Jens Axboe
2019-02-17 15:37 ` Thibaut Sautereau
0 siblings, 1 reply; 4+ messages in thread
From: Jens Axboe @ 2019-01-30 15:54 UTC (permalink / raw)
To: Jianchao Wang; +Cc: m19, linux-block, linux-kernel
On 1/30/19 2:01 AM, Jianchao Wang wrote:
> Florian reported a io hung issue when fsync(). It should be
> triggered by following race condition.
>
> data + post flush a flush
>
> blk_flush_complete_seq
> case REQ_FSEQ_DATA
> blk_flush_queue_rq
> issued to driver blk_mq_dispatch_rq_list
> try to issue a flush req
> failed due to NON-NCQ command
> .queue_rq return BLK_STS_DEV_RESOURCE
>
> request completion
> req->end_io // doesn't check RESTART
> mq_flush_data_end_io
> case REQ_FSEQ_POSTFLUSH
> blk_kick_flush
> do nothing because previous flush
> has not been completed
> blk_mq_run_hw_queue
> insert rq to hctx->dispatch
> due to RESTART is still set, do nothing
>
> To fix this, replace the blk_mq_run_hw_queue in mq_flush_data_end_io
> with blk_mq_sched_restart to check and clear the RESTART flag.
Applied, thanks.
--
Jens Axboe
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] blk-mq: fix a hung issue when fsync
2019-01-30 15:54 ` Jens Axboe
@ 2019-02-17 15:37 ` Thibaut Sautereau
2019-02-17 17:28 ` Sasha Levin
0 siblings, 1 reply; 4+ messages in thread
From: Thibaut Sautereau @ 2019-02-17 15:37 UTC (permalink / raw)
To: stable; +Cc: Jianchao Wang, m19, Jens Axboe, linux-block, linux-kernel
On Wed, Jan 30, 2019 at 08:54:09AM -0700, Jens Axboe wrote:
> On 1/30/19 2:01 AM, Jianchao Wang wrote:
> > Florian reported a io hung issue when fsync(). It should be
> > triggered by following race condition.
> >
> > data + post flush a flush
> >
> > blk_flush_complete_seq
> > case REQ_FSEQ_DATA
> > blk_flush_queue_rq
> > issued to driver blk_mq_dispatch_rq_list
> > try to issue a flush req
> > failed due to NON-NCQ command
> > .queue_rq return BLK_STS_DEV_RESOURCE
> >
> > request completion
> > req->end_io // doesn't check RESTART
> > mq_flush_data_end_io
> > case REQ_FSEQ_POSTFLUSH
> > blk_kick_flush
> > do nothing because previous flush
> > has not been completed
> > blk_mq_run_hw_queue
> > insert rq to hctx->dispatch
> > due to RESTART is still set, do nothing
> >
> > To fix this, replace the blk_mq_run_hw_queue in mq_flush_data_end_io
> > with blk_mq_sched_restart to check and clear the RESTART flag.
>
> Applied, thanks.
>
> --
> Jens Axboe
Can this be applied to stable kernels please?
It's commit 85bd6e61f34dffa8ec2dc75ff3c02ee7b2f1cbce upstream.
Thanks,
--
Thibaut
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] blk-mq: fix a hung issue when fsync
2019-02-17 15:37 ` Thibaut Sautereau
@ 2019-02-17 17:28 ` Sasha Levin
0 siblings, 0 replies; 4+ messages in thread
From: Sasha Levin @ 2019-02-17 17:28 UTC (permalink / raw)
To: Thibaut Sautereau
Cc: stable, Jianchao Wang, m19, Jens Axboe, linux-block, linux-kernel
On Sun, Feb 17, 2019 at 04:37:29PM +0100, Thibaut Sautereau wrote:
>On Wed, Jan 30, 2019 at 08:54:09AM -0700, Jens Axboe wrote:
>> On 1/30/19 2:01 AM, Jianchao Wang wrote:
>> > Florian reported a io hung issue when fsync(). It should be
>> > triggered by following race condition.
>> >
>> > data + post flush a flush
>> >
>> > blk_flush_complete_seq
>> > case REQ_FSEQ_DATA
>> > blk_flush_queue_rq
>> > issued to driver blk_mq_dispatch_rq_list
>> > try to issue a flush req
>> > failed due to NON-NCQ command
>> > .queue_rq return BLK_STS_DEV_RESOURCE
>> >
>> > request completion
>> > req->end_io // doesn't check RESTART
>> > mq_flush_data_end_io
>> > case REQ_FSEQ_POSTFLUSH
>> > blk_kick_flush
>> > do nothing because previous flush
>> > has not been completed
>> > blk_mq_run_hw_queue
>> > insert rq to hctx->dispatch
>> > due to RESTART is still set, do nothing
>> >
>> > To fix this, replace the blk_mq_run_hw_queue in mq_flush_data_end_io
>> > with blk_mq_sched_restart to check and clear the RESTART flag.
>>
>> Applied, thanks.
>>
>> --
>> Jens Axboe
>
>Can this be applied to stable kernels please?
>
>It's commit 85bd6e61f34dffa8ec2dc75ff3c02ee7b2f1cbce upstream.
I've queued it for 4.20, 4.19 and 4.14.
--
Thanks,
Sasha
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2019-02-17 17:28 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-01-30 9:01 [PATCH] blk-mq: fix a hung issue when fsync Jianchao Wang
2019-01-30 15:54 ` Jens Axboe
2019-02-17 15:37 ` Thibaut Sautereau
2019-02-17 17:28 ` Sasha Levin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).