* blk: accessing invalid memory with "blk-mq: dynamic h/w context count"
@ 2016-02-12 5:41 Sasha Levin
2016-02-12 8:24 ` Ming Lei
0 siblings, 1 reply; 3+ messages in thread
From: Sasha Levin @ 2016-02-12 5:41 UTC (permalink / raw)
To: keith.busch, Jens Axboe, Christoph Hellwig, jonathan.derrick
Cc: LKML, linux-block
Hi all,
I've started seeing the following errors on boot:
[6035791.296570] ==================================================================
[6035791.297467] BUG: KASAN: slab-out-of-bounds in loop_init_request+0x19c/0x1c0 at addr ffff880052e5c190
[6035791.298355] Write of size 8 by task swapper/0/1
[6035791.298842] =============================================================================
[6035791.299751] BUG kmalloc-512 (Tainted: G W ): kasan: bad access detected
[6035791.300736] -----------------------------------------------------------------------------
[6035791.300736]
[6035791.301696] Disabling lock debugging due to kernel taint
[6035791.302220] INFO: Slab 0xffffea00014b9700 objects=32 used=32 fp=0x (null) flags=0x1fffff80004080
[6035791.303218] INFO: Object 0xffff880052e5c000 @offset=0 fp=0x (null)
[6035791.303218]
[6035791.304047] Object ffff880052e5c000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.304955] Object ffff880052e5c010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.305970] Object ffff880052e5c020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.306916] Object ffff880052e5c030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.307908] Object ffff880052e5c040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.308903] Object ffff880052e5c050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.309959] Object ffff880052e5c060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.310896] Object ffff880052e5c070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.311849] Object ffff880052e5c080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.312784] Object ffff880052e5c090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.313734] Object ffff880052e5c0a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.314646] Object ffff880052e5c0b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.315567] Object ffff880052e5c0c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.316519] Object ffff880052e5c0d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.317475] Object ffff880052e5c0e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.318461] Object ffff880052e5c0f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.319428] Object ffff880052e5c100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.320548] Object ffff880052e5c110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.321680] Object ffff880052e5c120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.322585] Object ffff880052e5c130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.323587] Object ffff880052e5c140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.324574] Object ffff880052e5c150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.325505] Object ffff880052e5c160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.326449] Object ffff880052e5c170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.327412] Object ffff880052e5c180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.328329] Object ffff880052e5c190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.329200] Object ffff880052e5c1a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.330117] Object ffff880052e5c1b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.331000] Object ffff880052e5c1c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.331949] Object ffff880052e5c1d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.332888] Object ffff880052e5c1e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.333886] Object ffff880052e5c1f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[6035791.334813] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G B W 4.5.0-rc3-next-20160211-sasha-00028-g542d18e-dirty #2898
[6035791.335884] 1ffff1000a714ed2 00000000534d57fe ffff8800538a7718 ffffffffa34d4a15
[6035791.336796] ffffffff00000000 fffffbfff5eec534 0000000041b58ab3 ffffffffaefba520
[6035791.337631] ffffffffa34d489f 00000000534d57fe ffff880184220000 ffffffffaefd813f
[6035791.338458] Call Trace:
[6035791.338756] dump_stack (lib/dump_stack.c:53)
[6035791.340573] print_trailer (mm/slub.c:661)
[6035791.341117] object_err (mm/slub.c:668)
[6035791.341738] kasan_report_error (include/linux/kasan.h:28 mm/kasan/report.c:170 mm/kasan/report.c:237)
[6035791.344327] __asan_report_store8_noabort (mm/kasan/report.c:259 mm/kasan/report.c:285)
[6035791.345775] loop_init_request (drivers/block/loop.c:1699)
[6035791.347753] blk_mq_realloc_hw_ctxs (block/blk-mq.c:1722 block/blk-mq.c:1981)
[6035791.351966] blk_mq_init_allocated_queue (block/blk-mq.c:2027)
[6035791.355528] blk_mq_init_queue (block/blk-mq.c:1944)
[6035791.356081] loop_add (drivers/block/loop.c:1749)
[6035791.358663] loop_init (drivers/block/loop.c:2006 (discriminator 3))
[6035791.362708] do_one_initcall (init/main.c:788)
[6035791.363968] kernel_init_freeable (init/main.c:853 init/main.c:861 init/main.c:879 init/main.c:1004)
[6035791.366040] kernel_init (init/main.c:932)
[6035791.366573] ret_from_fork (arch/x86/entry/entry_64.S:383)
[6035791.367782] Memory state around the buggy address:
[6035791.368247] ffff880052e5c080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[6035791.368968] ffff880052e5c100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fc fc
[6035791.369852] >ffff880052e5c180: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[6035791.370635] ^
[6035791.371015] ffff880052e5c200: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[6035791.371816] ffff880052e5c280: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
Bisection pointed to:
commit 868f2f0b72068a097508b6e8870a8950fd8eb7ef
Author: Keith Busch <keith.busch@intel.com>
Date: Thu Dec 17 17:08:14 2015 -0700
blk-mq: dynamic h/w context count
Thanks,
Sasha
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: blk: accessing invalid memory with "blk-mq: dynamic h/w context count"
2016-02-12 5:41 blk: accessing invalid memory with "blk-mq: dynamic h/w context count" Sasha Levin
@ 2016-02-12 8:24 ` Ming Lei
2016-02-12 18:50 ` Keith Busch
0 siblings, 1 reply; 3+ messages in thread
From: Ming Lei @ 2016-02-12 8:24 UTC (permalink / raw)
To: Sasha Levin
Cc: keith.busch, Jens Axboe, Christoph Hellwig, jonathan.derrick,
LKML, linux-block, tom.leiming
On Fri, 12 Feb 2016 00:41:28 -0500
Sasha Levin <sasha.levin@oracle.com> wrote:
> Hi all,
>
> I've started seeing the following errors on boot:
>
> [6035791.296570] ==================================================================
> [6035791.297467] BUG: KASAN: slab-out-of-bounds in loop_init_request+0x19c/0x1c0 at addr ffff880052e5c190
> [6035791.298355] Write of size 8 by task swapper/0/1
> [6035791.298842] =============================================================================
> [6035791.299751] BUG kmalloc-512 (Tainted: G W ): kasan: bad access detected
> [6035791.300736] -----------------------------------------------------------------------------
> [6035791.300736]
> [6035791.301696] Disabling lock debugging due to kernel taint
> [6035791.302220] INFO: Slab 0xffffea00014b9700 objects=32 used=32 fp=0x (null) flags=0x1fffff80004080
> [6035791.303218] INFO: Object 0xffff880052e5c000 @offset=0 fp=0x (null)
> [6035791.303218]
> [6035791.304047] Object ffff880052e5c000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.304955] Object ffff880052e5c010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.305970] Object ffff880052e5c020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.306916] Object ffff880052e5c030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.307908] Object ffff880052e5c040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.308903] Object ffff880052e5c050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.309959] Object ffff880052e5c060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.310896] Object ffff880052e5c070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.311849] Object ffff880052e5c080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.312784] Object ffff880052e5c090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.313734] Object ffff880052e5c0a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.314646] Object ffff880052e5c0b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.315567] Object ffff880052e5c0c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.316519] Object ffff880052e5c0d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.317475] Object ffff880052e5c0e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.318461] Object ffff880052e5c0f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.319428] Object ffff880052e5c100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.320548] Object ffff880052e5c110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.321680] Object ffff880052e5c120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.322585] Object ffff880052e5c130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.323587] Object ffff880052e5c140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.324574] Object ffff880052e5c150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.325505] Object ffff880052e5c160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.326449] Object ffff880052e5c170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.327412] Object ffff880052e5c180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.328329] Object ffff880052e5c190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.329200] Object ffff880052e5c1a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.330117] Object ffff880052e5c1b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.331000] Object ffff880052e5c1c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.331949] Object ffff880052e5c1d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.332888] Object ffff880052e5c1e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.333886] Object ffff880052e5c1f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [6035791.334813] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G B W 4.5.0-rc3-next-20160211-sasha-00028-g542d18e-dirty #2898
> [6035791.335884] 1ffff1000a714ed2 00000000534d57fe ffff8800538a7718 ffffffffa34d4a15
> [6035791.336796] ffffffff00000000 fffffbfff5eec534 0000000041b58ab3 ffffffffaefba520
> [6035791.337631] ffffffffa34d489f 00000000534d57fe ffff880184220000 ffffffffaefd813f
> [6035791.338458] Call Trace:
> [6035791.338756] dump_stack (lib/dump_stack.c:53)
> [6035791.340573] print_trailer (mm/slub.c:661)
> [6035791.341117] object_err (mm/slub.c:668)
> [6035791.341738] kasan_report_error (include/linux/kasan.h:28 mm/kasan/report.c:170 mm/kasan/report.c:237)
> [6035791.344327] __asan_report_store8_noabort (mm/kasan/report.c:259 mm/kasan/report.c:285)
> [6035791.345775] loop_init_request (drivers/block/loop.c:1699)
> [6035791.347753] blk_mq_realloc_hw_ctxs (block/blk-mq.c:1722 block/blk-mq.c:1981)
> [6035791.351966] blk_mq_init_allocated_queue (block/blk-mq.c:2027)
> [6035791.355528] blk_mq_init_queue (block/blk-mq.c:1944)
> [6035791.356081] loop_add (drivers/block/loop.c:1749)
> [6035791.358663] loop_init (drivers/block/loop.c:2006 (discriminator 3))
> [6035791.362708] do_one_initcall (init/main.c:788)
> [6035791.363968] kernel_init_freeable (init/main.c:853 init/main.c:861 init/main.c:879 init/main.c:1004)
> [6035791.366040] kernel_init (init/main.c:932)
> [6035791.366573] ret_from_fork (arch/x86/entry/entry_64.S:383)
> [6035791.367782] Memory state around the buggy address:
> [6035791.368247] ffff880052e5c080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> [6035791.368968] ffff880052e5c100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fc fc
> [6035791.369852] >ffff880052e5c180: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> [6035791.370635] ^
> [6035791.371015] ffff880052e5c200: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> [6035791.371816] ffff880052e5c280: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
>
> Bisection pointed to:
>
> commit 868f2f0b72068a097508b6e8870a8950fd8eb7ef
> Author: Keith Busch <keith.busch@intel.com>
> Date: Thu Dec 17 17:08:14 2015 -0700
>
> blk-mq: dynamic h/w context count
Hi Sasha,
It should be about timing of setting q->mq_ops, and
I believe the following patch may fix the issue, could
you give a test?
Thanks,
---
>From 299dfbd27a4ede53104608b07669041d202afe1f Mon Sep 17 00:00:00 2001
From: Ming Lei <tom.leiming@gmail.com>
Date: Fri, 12 Feb 2016 15:27:00 +0800
Subject: [PATCH] blk-mq: mark request queue as mq asap
Currently q->mq_ops is used widely to decide if the queue
is mq or not, so we should set the 'flag' asap so that both
block core and drivers can get the correct mq info.
For example, commit 868f2f0b720(blk-mq: dynamic h/w context count)
moves the hctx's initialization before setting q->mq_ops in
blk_mq_init_allocated_queue(), then cause blk_alloc_flush_queue()
to think the queue is non-mq and don't allocate command size
for the per-hctx flush rq.
This patches should fix the problem reported by Sasha.
Cc: Keith Busch <keith.busch@intel.com>
Reported-by: Sasha Levin <sasha.levin@oracle.com>
Signed-off-by: Ming Lei <tom.leiming@gmail.com>
---
block/blk-mq.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/block/blk-mq.c b/block/blk-mq.c
index 645eb9e..f539a53 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -2010,6 +2010,9 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
struct request_queue *q)
{
+ /* mark the queue as mq asap */
+ q->mq_ops = set->ops;
+
q->queue_ctx = alloc_percpu(struct blk_mq_ctx);
if (!q->queue_ctx)
return ERR_PTR(-ENOMEM);
@@ -2032,7 +2035,6 @@ struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
q->nr_queues = nr_cpu_ids;
- q->mq_ops = set->ops;
q->queue_flags |= QUEUE_FLAG_MQ_DEFAULT;
if (!(set->flags & BLK_MQ_F_SG_MERGE))
--
1.9.1
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: blk: accessing invalid memory with "blk-mq: dynamic h/w context count"
2016-02-12 8:24 ` Ming Lei
@ 2016-02-12 18:50 ` Keith Busch
0 siblings, 0 replies; 3+ messages in thread
From: Keith Busch @ 2016-02-12 18:50 UTC (permalink / raw)
To: Ming Lei
Cc: Sasha Levin, Jens Axboe, Christoph Hellwig, jonathan.derrick,
LKML, linux-block
On Fri, Feb 12, 2016 at 04:24:35PM +0800, Ming Lei wrote:
> On Fri, 12 Feb 2016 00:41:28 -0500
> Hi Sasha,
>
> It should be about timing of setting q->mq_ops, and
> I believe the following patch may fix the issue, could
> you give a test?
Thanks, Ming, that looks better and looks like the same as the 0-day
failure from when this was posted a couple months ago. I thought this
was potentially risky looking, but haven't had time to make changes.
I also didn't see that this was applied yet either. I've a broken filter
moving important emails to the junk... On the plus side, the exposure
led to a potential fix.
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2016-02-12 18:50 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-02-12 5:41 blk: accessing invalid memory with "blk-mq: dynamic h/w context count" Sasha Levin
2016-02-12 8:24 ` Ming Lei
2016-02-12 18:50 ` Keith Busch
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).