From: Benjamin Block <bblock@linux.ibm.com>
To: Ming Lei <ming.lei@redhat.com>
Cc: Jens Axboe <axboe@kernel.dk>,
linux-block@vger.kernel.org, Bart Van Assche <bvanassche@acm.org>,
Christoph Hellwig <hch@lst.de>,
kernel test robot <rong.a.chen@intel.com>,
Jens Remus <jremus@linux.ibm.com>
Subject: Re: [PATCH] block: free sched's request pool in blk_cleanup_queue
Date: Thu, 6 Jun 2019 16:47:14 +0200 [thread overview]
Message-ID: <20190606144714.GA6549@t480-pf1aa2c2> (raw)
In-Reply-To: <20190604130802.17076-1-ming.lei@redhat.com>
On Tue, Jun 04, 2019 at 09:08:02PM +0800, Ming Lei wrote:
> In theory, IO scheduler belongs to request queue, and the request pool
> of sched tags belongs to the request queue too.
>
> However, the current tags allocation interfaces are re-used for both
> driver tags and sched tags, and driver tags is definitely host wide,
> and doesn't belong to any request queue, same with its request pool.
> So we need tagset instance for freeing request of sched tags.
>
> Meantime, blk_mq_free_tag_set() often follows blk_cleanup_queue() in case
> of non-BLK_MQ_F_TAG_SHARED, this way requires that request pool of sched
> tags to be freed before calling blk_mq_free_tag_set().
>
> Commit 47cdee29ef9d94e ("block: move blk_exit_queue into __blk_release_queue")
> moves blk_exit_queue into __blk_release_queue for simplying the fast
> path in generic_make_request(), then causes oops during freeing requests
> of sched tags in __blk_release_queue().
>
> Fix the above issue by move freeing request pool of sched tags into
> blk_cleanup_queue(), this way is safe becasue queue has been frozen and no any
> in-queue requests at that time. Freeing sched tags has to be kept in queue's
> release handler becasue there might be un-completed dispatch activity
> which might refer to sched tags.
>
> Cc: Bart Van Assche <bvanassche@acm.org>
> Cc: Christoph Hellwig <hch@lst.de>
> Fixes: 47cdee29ef9d94e485eb08f962c74943023a5271 ("block: move blk_exit_queue into __blk_release_queue")
> Reported-by: kernel test robot <rong.a.chen@intel.com>
> Signed-off-by: Ming Lei <ming.lei@redhat.com>
Our CI meanwhile also crashes regularly because of this:
run blktests block/002 at 2019-06-06 14:44:55
Unable to handle kernel pointer dereference in virtual kernel address space, Failing address: 6b6b6b6b6b6b6000 TEID: 6b6b6b6b6b6b6803
Fault in home space mode while using kernel ASCE.
AS:0000000057290007 R3:0000000000000024
Oops: 0038 ilc:3 [#1] PREEMPT SMP
Modules linked in: ...
CPU: 4 PID: 139 Comm: kworker/4:2 Kdump: loaded Not tainted 5.2.0-rc3-master-05489-g55f909514069 #3
Hardware name: IBM 3906 M03 703 (LPAR)
Workqueue: events __blk_release_queue
Krnl PSW : 0704e00180000000 000000005657db18 (blk_mq_free_rqs+0x48/0x128)
R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 RI:0 EA:3
Krnl GPRS: 00000000a8309db5 6b6b6b6b6b6b6b6b 000000008beb3858 00000000a2befbc8
0000000000000000 0000000000000001 0000000056bb16c8 00000000b4070aa8
000000008beb3858 000000008bc46b38 00000000a2befbc8 0000000000000000
00000000bafb8100 00000000568e8040 000003e0092b3c30 000003e0092b3be0
Krnl Code: 000000005657db0a: a7f4006e brc 15,5657dbe6
000000005657db0e: e31020380004 lg %r1,56(%r2)
#000000005657db14: b9040082 lgr %r8,%r2
>000000005657db18: e31010500002 ltg %r1,80(%r1)
000000005657db1e: a784ffee brc 8,5657dafa
000000005657db22: e32030000012 lt %r2,0(%r3)
000000005657db28: a784ffe9 brc 8,5657dafa
000000005657db2c: b9040074 lgr %r7,%r4
Call Trace:
([<000000008ff8ed00>] 0x8ff8ed00)
[<0000000056582958>] blk_mq_sched_tags_teardown+0x68/0x98
[<0000000056583396>] blk_mq_exit_sched+0xc6/0xd8
[<0000000056569324>] elevator_exit+0x54/0x70
[<0000000056570644>] __blk_release_queue+0x84/0x110
[<0000000055f416c6>] process_one_work+0x3a6/0x6b8
[<0000000055f41c50>] worker_thread+0x278/0x478
[<0000000055f49e08>] kthread+0x160/0x178
[<00000000568d83e8>] ret_from_fork+0x34/0x38
INFO: lockdep is turned off.
Last Breaking-Event-Address:
[<000000005657daf6>] blk_mq_free_rqs+0x26/0x128
Kernel panic - not syncing: Fatal exception: panic_on_oops
run blktests block/003 at 2019-06-06 14:44:56
When I tried to reproduced this with this patch, it went away (at least all of
blktest/block ran w/o crash).
I don't feel competent enough to review this patch right now, but it would be
good if we get something upstream for this.
--
With Best Regards, Benjamin Block / Linux on IBM Z Kernel Development
IBM Systems & Technology Group / IBM Deutschland Research & Development GmbH
Vorsitz. AufsR.: Matthias Hartmann / Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: AmtsG Stuttgart, HRB 243294
next prev parent reply other threads:[~2019-06-06 14:47 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-06-04 13:08 [PATCH] block: free sched's request pool in blk_cleanup_queue Ming Lei
2019-06-05 1:27 ` Yi Zhang
2019-06-06 14:47 ` Benjamin Block [this message]
2019-06-06 22:43 ` Ming Lei
2019-06-07 4:39 ` Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190606144714.GA6549@t480-pf1aa2c2 \
--to=bblock@linux.ibm.com \
--cc=axboe@kernel.dk \
--cc=bvanassche@acm.org \
--cc=hch@lst.de \
--cc=jremus@linux.ibm.com \
--cc=linux-block@vger.kernel.org \
--cc=ming.lei@redhat.com \
--cc=rong.a.chen@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).