* Slab BUG with DEBUG_* options
@ 2013-11-30 11:42 Meelis Roos
2013-11-30 22:31 ` David Rientjes
` (3 more replies)
0 siblings, 4 replies; 15+ messages in thread
From: Meelis Roos @ 2013-11-30 11:42 UTC (permalink / raw)
To: Pekka Enberg, Christoph Lameter, Matt Mackall; +Cc: Linux Kernel list, linux-mm
I am debugging a reboot problem on Sun Ultra 5 (sparc64) with 512M RAM
and turned on DEBUG_PAGEALLOC DEBUG_SLAB and DEBUG_SLAB_LEAK (and most
other debug options) and got the following BUG and hang on startup. This
happened originally with 3.11-rc2-00058 where my bisection of
another problem lead, but I retested 3.12 to have the same BUG in the
same place.
kernel BUG at mm/slab.c:2391!
\|/ ____ \|/
"@'/ .. \`@"
/_| \__/ |_\
\__U_/
swapper(0): Kernel bad sw trap 5 [#1]
CPU: 0 PID: 0 Comm: swapper Not tainted 3.11.0-rc2-00058-g20bafb3-dirty #127
task: 00000000008ac468 ti: 000000000089c000 task.ti: 000000000089c000
TSTATE: 0000004480e01606 TPC: 00000000004f57d4 TNPC: 00000000004f57d8 Y: 00000000 Not tainted
TPC: <__kmem_cache_create+0x374/0x480>
g0: 00000000000000f8 g1: 00000000008bb400 g2: 000000000002780b g3: 00000000008b5120
g4: 00000000008ac468 g5: 0000000000000000 g6: 000000000089c000 g7: 0000000000000000
o0: 0000000000845f08 o1: 0000000000000957 o2: ffffffffffffffe0 o3: 0000000000000000
o4: 0000000000002004 o5: 0000000000000000 sp: 000000000089f301 ret_pc: 00000000004f57cc
RPC: <__kmem_cache_create+0x36c/0x480>
l0: fffff8001e812040 l1: fffff8001e819f80 l2: fffff8001e819fb8 l3: fffff8001e819fd8
l4: 0000000000000001 l5: fffff8001e819fc8 l6: 0000000000845f08 l7: fffff8001e8300a0
i0: fffff8001e831fa0 i1: 0000000080002800 i2: 0000000080000000 i3: 0000000000000034
i4: 0000000000000000 i5: 0000000000002000 i6: 000000000089f3b1 i7: 0000000000907464
I7: <create_boot_cache+0x4c/0x84>
Call Trace:
[0000000000907464] create_boot_cache+0x4c/0x84
[00000000009074d0] create_kmalloc_cache+0x34/0x60
[0000000000907540] create_kmalloc_caches+0x44/0x168
[0000000000908dfc] kmem_cache_init+0x1d0/0x1e0
[00000000008fc658] start_kernel+0x18c/0x370
[0000000000761df4] tlb_fixup_done+0x88/0x94
[0000000000000000] (null)
Disabling lock debugging due to kernel taint
Caller[0000000000907464]: create_boot_cache+0x4c/0x84
Caller[00000000009074d0]: create_kmalloc_cache+0x34/0x60
Caller[0000000000907540]: create_kmalloc_caches+0x44/0x168
Caller[0000000000908dfc]: kmem_cache_init+0x1d0/0x1e0
Caller[00000000008fc658]: start_kernel+0x18c/0x370
Caller[0000000000761df4]: tlb_fixup_done+0x88/0x94
Caller[0000000000000000]: (null)
Instruction DUMP: 92102957 7ffccb35 90122308 <91d02005> 90100018 4009b371 920f20d0 ba922000 02480006
Kernel panic - not syncing: Attempted to kill the idle task!
Press Stop-A (L1-A) to return to the boot prom
The line shows that __kmem_cache_create gets a NULL from kmalloc_slab().
I instrumented the code and found the following:
__kmem_cache_create: starting, size=248, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 248 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: starting, size=96, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 96 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: starting, size=192, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 192 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: starting, size=32, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 32 because of redzoning
__kmem_cache_create: aligned size to 32
__kmem_cache_create: num=226, slab_size=960
__kmem_cache_create: starting, size=64, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 64 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=8192, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
With slab size 64, it turns on CFLGS_OFF_SLAB and off slab allocation
with this size fails. I do not know slab internals so I can not tell if
this just happens because of the debug paths, or is it a real problem
without the debug options too.
--
Meelis Roos (mroos@linux.ee)
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Slab BUG with DEBUG_* options
2013-11-30 11:42 Slab BUG with DEBUG_* options Meelis Roos
@ 2013-11-30 22:31 ` David Rientjes
2013-12-02 19:02 ` Christoph Lameter
2013-12-03 9:17 ` Meelis Roos
2013-12-03 11:53 ` Pekka Enberg
` (2 subsequent siblings)
3 siblings, 2 replies; 15+ messages in thread
From: David Rientjes @ 2013-11-30 22:31 UTC (permalink / raw)
To: Meelis Roos
Cc: Pekka Enberg, Christoph Lameter, Matt Mackall, Linux Kernel list,
linux-mm
On Sat, 30 Nov 2013, Meelis Roos wrote:
> I am debugging a reboot problem on Sun Ultra 5 (sparc64) with 512M RAM
> and turned on DEBUG_PAGEALLOC DEBUG_SLAB and DEBUG_SLAB_LEAK (and most
> other debug options) and got the following BUG and hang on startup. This
> happened originally with 3.11-rc2-00058 where my bisection of
> another problem lead, but I retested 3.12 to have the same BUG in the
> same place.
>
> kernel BUG at mm/slab.c:2391!
> \|/ ____ \|/
> "@'/ .. \`@"
> /_| \__/ |_\
> \__U_/
> swapper(0): Kernel bad sw trap 5 [#1]
> CPU: 0 PID: 0 Comm: swapper Not tainted 3.11.0-rc2-00058-g20bafb3-dirty #127
> task: 00000000008ac468 ti: 000000000089c000 task.ti: 000000000089c000
> TSTATE: 0000004480e01606 TPC: 00000000004f57d4 TNPC: 00000000004f57d8 Y: 00000000 Not tainted
> TPC: <__kmem_cache_create+0x374/0x480>
> g0: 00000000000000f8 g1: 00000000008bb400 g2: 000000000002780b g3: 00000000008b5120
> g4: 00000000008ac468 g5: 0000000000000000 g6: 000000000089c000 g7: 0000000000000000
> o0: 0000000000845f08 o1: 0000000000000957 o2: ffffffffffffffe0 o3: 0000000000000000
> o4: 0000000000002004 o5: 0000000000000000 sp: 000000000089f301 ret_pc: 00000000004f57cc
> RPC: <__kmem_cache_create+0x36c/0x480>
> l0: fffff8001e812040 l1: fffff8001e819f80 l2: fffff8001e819fb8 l3: fffff8001e819fd8
> l4: 0000000000000001 l5: fffff8001e819fc8 l6: 0000000000845f08 l7: fffff8001e8300a0
> i0: fffff8001e831fa0 i1: 0000000080002800 i2: 0000000080000000 i3: 0000000000000034
> i4: 0000000000000000 i5: 0000000000002000 i6: 000000000089f3b1 i7: 0000000000907464
> I7: <create_boot_cache+0x4c/0x84>
> Call Trace:
> [0000000000907464] create_boot_cache+0x4c/0x84
> [00000000009074d0] create_kmalloc_cache+0x34/0x60
> [0000000000907540] create_kmalloc_caches+0x44/0x168
> [0000000000908dfc] kmem_cache_init+0x1d0/0x1e0
> [00000000008fc658] start_kernel+0x18c/0x370
> [0000000000761df4] tlb_fixup_done+0x88/0x94
> [0000000000000000] (null)
> Disabling lock debugging due to kernel taint
> Caller[0000000000907464]: create_boot_cache+0x4c/0x84
> Caller[00000000009074d0]: create_kmalloc_cache+0x34/0x60
> Caller[0000000000907540]: create_kmalloc_caches+0x44/0x168
> Caller[0000000000908dfc]: kmem_cache_init+0x1d0/0x1e0
> Caller[00000000008fc658]: start_kernel+0x18c/0x370
> Caller[0000000000761df4]: tlb_fixup_done+0x88/0x94
> Caller[0000000000000000]: (null)
> Instruction DUMP: 92102957 7ffccb35 90122308 <91d02005> 90100018 4009b371 920f20d0 ba922000 02480006
> Kernel panic - not syncing: Attempted to kill the idle task!
> Press Stop-A (L1-A) to return to the boot prom
>
> The line shows that __kmem_cache_create gets a NULL from kmalloc_slab().
>
> I instrumented the code and found the following:
>
> __kmem_cache_create: starting, size=248, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 248 because of redzoning
> __kmem_cache_create: pagealloc debug, setting size to 8192
> __kmem_cache_create: aligned size to 8192
> __kmem_cache_create: num=1, slab_size=64
> __kmem_cache_create: starting, size=96, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 96 because of redzoning
> __kmem_cache_create: pagealloc debug, setting size to 8192
> __kmem_cache_create: aligned size to 8192
> __kmem_cache_create: num=1, slab_size=64
> __kmem_cache_create: starting, size=192, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 192 because of redzoning
> __kmem_cache_create: pagealloc debug, setting size to 8192
> __kmem_cache_create: aligned size to 8192
> __kmem_cache_create: num=1, slab_size=64
> __kmem_cache_create: starting, size=32, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 32 because of redzoning
> __kmem_cache_create: aligned size to 32
> __kmem_cache_create: num=226, slab_size=960
> __kmem_cache_create: starting, size=64, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 64 because of redzoning
> __kmem_cache_create: pagealloc debug, setting size to 8192
> __kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
> __kmem_cache_create: aligned size to 8192
> __kmem_cache_create: num=1, slab_size=64
> __kmem_cache_create: CFLGS_OFF_SLAB, size=8192, slab_size=52
> __kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
>
> With slab size 64, it turns on CFLGS_OFF_SLAB and off slab allocation
> with this size fails. I do not know slab internals so I can not tell if
> this just happens because of the debug paths, or is it a real problem
> without the debug options too.
>
Sounds like a problem with create_kmalloc_caches(), what are the values
for KMALLOC_SHIFT_LOW and KMALLOC_SHIFT_HIGH?
It's interesting you report this as starting with 3.11-rc2 because this
changed in 3.9, what kernels were tested before 3.11-rc2 if any?
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Slab BUG with DEBUG_* options
2013-11-30 22:31 ` David Rientjes
@ 2013-12-02 19:02 ` Christoph Lameter
2013-12-03 8:46 ` Meelis Roos
2013-12-03 9:17 ` Meelis Roos
1 sibling, 1 reply; 15+ messages in thread
From: Christoph Lameter @ 2013-12-02 19:02 UTC (permalink / raw)
To: David Rientjes
Cc: Meelis Roos, Pekka Enberg, Matt Mackall, Linux Kernel list, linux-mm
Does this patch from 3.13 fix it?
commit c6f58d9b362b45c52afebe4342c9137d0dabe47f
Author: Christoph Lameter <cl@linux.com>
Date: Thu Nov 7 16:29:15 2013 +0000
slub: Handle NULL parameter in kmem_cache_flags
Andreas Herrmann writes:
When I've used slub_debug kernel option (e.g.
"slub_debug=,skbuff_fclone_cache" or similar) on a debug session I've
seen a panic like:
Highbank #setenv bootargs console=ttyAMA0 root=/dev/sda2 kgdboc.kgdboc=ttyAMA0,115200 slub_debug=,kmalloc-4096 earlypr
...
Unable to handle kernel NULL pointer dereference at virtual address 00000000
pgd = c0004000
[00000000] *pgd=00000000
Internal error: Oops: 5 [#1] SMP ARM
Modules linked in:
CPU: 0 PID: 0 Comm: swapper Tainted: G W 3.12.0-00048-gbe408cd #314
task: c0898360 ti: c088a000 task.ti: c088a000
PC is at strncmp+0x1c/0x84
LR is at kmem_cache_flags.isra.46.part.47+0x44/0x60
pc : [<c02c6da0>] lr : [<c0110a3c>] psr: 200001d3
sp : c088bea8 ip : c088beb8 fp : c088beb4
r10: 00000000 r9 : 413fc090 r8 : 00000001
r7 : 00000000 r6 : c2984a08 r5 : c0966e78 r4 : 00000000
r3 : 0000006b r2 : 0000000c r1 : 00000000 r0 : c2984a08
Flags: nzCv IRQs off FIQs off Mode SVC_32 ISA ARM Segment kernel
Control: 10c5387d Table: 0000404a DAC: 00000015
Process swapper (pid: 0, stack limit = 0xc088a248)
Stack: (0xc088bea8 to 0xc088c000)
bea0: c088bed4 c088beb8 c0110a3c c02c6d90 c0966e78 00000040
bec0: ef001f00 00000040 c088bf14 c088bed8 c0112070 c0110a04 00000005 c010fac8
bee0: c088bf5c c088bef0 c010fac8 ef001f00 00000040 00000000 00000040 00000001
bf00: 413fc090 00000000 c088bf34 c088bf18 c0839190 c0112040 00000000 ef001f00
bf20: 00000000 00000000 c088bf54 c088bf38 c0839200 c083914c 00000006 c0961c4c
bf40: c0961c28 00000000 c088bf7c c088bf58 c08392ac c08391c0 c08a2ed8 c0966e78
bf60: c086b874 c08a3f50 c0961c28 00000001 c088bfb4 c088bf80 c083b258 c0839248
bf80: 2f800000 0f000000 c08935b4 ffffffff c08cd400 ffffffff c08cd400 c0868408
bfa0: c29849c0 00000000 c088bff4 c088bfb8 c0824974 c083b1e4 ffffffff ffffffff
bfc0: c08245c0 00000000 00000000 c0868408 00000000 10c5387d c0892bcc c0868404
bfe0: c0899440 0000406a 00000000 c088bff8 00008074 c0824824 00000000 00000000
[<c02c6da0>] (strncmp+0x1c/0x84) from [<c0110a3c>] (kmem_cache_flags.isra.46.part.47+0x44/0x60)
[<c0110a3c>] (kmem_cache_flags.isra.46.part.47+0x44/0x60) from [<c0112070>] (__kmem_cache_create+0x3c/0x410)
[<c0112070>] (__kmem_cache_create+0x3c/0x410) from [<c0839190>] (create_boot_cache+0x50/0x74)
[<c0839190>] (create_boot_cache+0x50/0x74) from [<c0839200>] (create_kmalloc_cache+0x4c/0x88)
[<c0839200>] (create_kmalloc_cache+0x4c/0x88) from [<c08392ac>] (create_kmalloc_caches+0x70/0x114)
[<c08392ac>] (create_kmalloc_caches+0x70/0x114) from [<c083b258>] (kmem_cache_init+0x80/0xe0)
[<c083b258>] (kmem_cache_init+0x80/0xe0) from [<c0824974>] (start_kernel+0x15c/0x318)
[<c0824974>] (start_kernel+0x15c/0x318) from [<00008074>] (0x8074)
Code: e3520000 01a00002 089da800 e5d03000 (e5d1c000)
---[ end trace 1b75b31a2719ed1d ]---
Kernel panic - not syncing: Fatal exception
Problem is that slub_debug option is not parsed before
create_boot_cache is called. Solve this by changing slub_debug to
early_param.
Kernels 3.11, 3.10 are also affected. I am not sure about older
kernels.
Christoph Lameter explains:
kmem_cache_flags may be called with NULL parameter during early boot.
Skip the test in that case.
Cc: stable@vger.kernel.org # 3.10 and 3.11
Reported-by: Andreas Herrmann <andreas.herrmann@calxeda.com>
Signed-off-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Slab BUG with DEBUG_* options
2013-12-02 19:02 ` Christoph Lameter
@ 2013-12-03 8:46 ` Meelis Roos
0 siblings, 0 replies; 15+ messages in thread
From: Meelis Roos @ 2013-12-03 8:46 UTC (permalink / raw)
To: Christoph Lameter
Cc: David Rientjes, Pekka Enberg, Matt Mackall, Linux Kernel list, linux-mm
> Does this patch from 3.13 fix it?
>
> commit c6f58d9b362b45c52afebe4342c9137d0dabe47f
> Author: Christoph Lameter <cl@linux.com>
> Date: Thu Nov 7 16:29:15 2013 +0000
>
> slub: Handle NULL parameter in kmem_cache_flags
I do not think so - it is for slub but this machine uses slab.
--
Meelis Roos (mroos@linux.ee)
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Slab BUG with DEBUG_* options
2013-11-30 22:31 ` David Rientjes
2013-12-02 19:02 ` Christoph Lameter
@ 2013-12-03 9:17 ` Meelis Roos
1 sibling, 0 replies; 15+ messages in thread
From: Meelis Roos @ 2013-12-03 9:17 UTC (permalink / raw)
To: David Rientjes
Cc: Pekka Enberg, Christoph Lameter, Matt Mackall, Linux Kernel list,
linux-mm
> > I am debugging a reboot problem on Sun Ultra 5 (sparc64) with 512M RAM
> > and turned on DEBUG_PAGEALLOC DEBUG_SLAB and DEBUG_SLAB_LEAK (and most
> > other debug options) and got the following BUG and hang on startup. This
> > happened originally with 3.11-rc2-00058 where my bisection of
> > another problem lead, but I retested 3.12 to have the same BUG in the
> > same place.
> >
> > kernel BUG at mm/slab.c:2391!
[...
> > The line shows that __kmem_cache_create gets a NULL from kmalloc_slab().
> >
> > I instrumented the code and found the following:
> >
> > __kmem_cache_create: starting, size=248, flags=8192
> > __kmem_cache_create: now flags=76800
> > __kmem_cache_create: aligned size to 248 because of redzoning
> > __kmem_cache_create: pagealloc debug, setting size to 8192
> > __kmem_cache_create: aligned size to 8192
> > __kmem_cache_create: num=1, slab_size=64
> > __kmem_cache_create: starting, size=96, flags=8192
> > __kmem_cache_create: now flags=76800
> > __kmem_cache_create: aligned size to 96 because of redzoning
> > __kmem_cache_create: pagealloc debug, setting size to 8192
> > __kmem_cache_create: aligned size to 8192
> > __kmem_cache_create: num=1, slab_size=64
> > __kmem_cache_create: starting, size=192, flags=8192
> > __kmem_cache_create: now flags=76800
> > __kmem_cache_create: aligned size to 192 because of redzoning
> > __kmem_cache_create: pagealloc debug, setting size to 8192
> > __kmem_cache_create: aligned size to 8192
> > __kmem_cache_create: num=1, slab_size=64
> > __kmem_cache_create: starting, size=32, flags=8192
> > __kmem_cache_create: now flags=76800
> > __kmem_cache_create: aligned size to 32 because of redzoning
> > __kmem_cache_create: aligned size to 32
> > __kmem_cache_create: num=226, slab_size=960
> > __kmem_cache_create: starting, size=64, flags=8192
> > __kmem_cache_create: now flags=76800
> > __kmem_cache_create: aligned size to 64 because of redzoning
> > __kmem_cache_create: pagealloc debug, setting size to 8192
> > __kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
> > __kmem_cache_create: aligned size to 8192
> > __kmem_cache_create: num=1, slab_size=64
> > __kmem_cache_create: CFLGS_OFF_SLAB, size=8192, slab_size=52
> > __kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
> >
> > With slab size 64, it turns on CFLGS_OFF_SLAB and off slab allocation
> > with this size fails. I do not know slab internals so I can not tell if
> > this just happens because of the debug paths, or is it a real problem
> > without the debug options too.
> >
>
> Sounds like a problem with create_kmalloc_caches(), what are the values
> for KMALLOC_SHIFT_LOW and KMALLOC_SHIFT_HIGH?
KMALLOC_SHIFT_LOW=5, KMALLOC_SHIFT_HIGH=23
> It's interesting you report this as starting with 3.11-rc2 because this
> changed in 3.9, what kernels were tested before 3.11-rc2 if any?
No other kernels were tested with slab debug - but all relsese kernels
and most rc-s starting with rc2 or rc3 eacg time were tested without
slab debug.
--
Meelis Roos (mroos@linux.ee)
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Slab BUG with DEBUG_* options
2013-11-30 11:42 Slab BUG with DEBUG_* options Meelis Roos
2013-11-30 22:31 ` David Rientjes
@ 2013-12-03 11:53 ` Pekka Enberg
2013-12-03 12:25 ` Joonsoo Kim
2013-12-03 15:46 ` Christoph Lameter
[not found] ` <alpine.DEB.2.02.1312030930450.4115@gentwo.org>
3 siblings, 1 reply; 15+ messages in thread
From: Pekka Enberg @ 2013-12-03 11:53 UTC (permalink / raw)
To: Meelis Roos, Pekka Enberg, Christoph Lameter, Matt Mackall
Cc: Linux Kernel list, linux-mm, Joonsoo Kim
On 11/30/2013 01:42 PM, Meelis Roos wrote:
> I am debugging a reboot problem on Sun Ultra 5 (sparc64) with 512M RAM
> and turned on DEBUG_PAGEALLOC DEBUG_SLAB and DEBUG_SLAB_LEAK (and most
> other debug options) and got the following BUG and hang on startup. This
> happened originally with 3.11-rc2-00058 where my bisection of
> another problem lead, but I retested 3.12 to have the same BUG in the
> same place.
>
> kernel BUG at mm/slab.c:2391!
> \|/ ____ \|/
> "@'/ .. \`@"
> /_| \__/ |_\
> \__U_/
> swapper(0): Kernel bad sw trap 5 [#1]
> CPU: 0 PID: 0 Comm: swapper Not tainted 3.11.0-rc2-00058-g20bafb3-dirty #127
> task: 00000000008ac468 ti: 000000000089c000 task.ti: 000000000089c000
> TSTATE: 0000004480e01606 TPC: 00000000004f57d4 TNPC: 00000000004f57d8 Y: 00000000 Not tainted
> TPC: <__kmem_cache_create+0x374/0x480>
> g0: 00000000000000f8 g1: 00000000008bb400 g2: 000000000002780b g3: 00000000008b5120
> g4: 00000000008ac468 g5: 0000000000000000 g6: 000000000089c000 g7: 0000000000000000
> o0: 0000000000845f08 o1: 0000000000000957 o2: ffffffffffffffe0 o3: 0000000000000000
> o4: 0000000000002004 o5: 0000000000000000 sp: 000000000089f301 ret_pc: 00000000004f57cc
> RPC: <__kmem_cache_create+0x36c/0x480>
> l0: fffff8001e812040 l1: fffff8001e819f80 l2: fffff8001e819fb8 l3: fffff8001e819fd8
> l4: 0000000000000001 l5: fffff8001e819fc8 l6: 0000000000845f08 l7: fffff8001e8300a0
> i0: fffff8001e831fa0 i1: 0000000080002800 i2: 0000000080000000 i3: 0000000000000034
> i4: 0000000000000000 i5: 0000000000002000 i6: 000000000089f3b1 i7: 0000000000907464
> I7: <create_boot_cache+0x4c/0x84>
> Call Trace:
> [0000000000907464] create_boot_cache+0x4c/0x84
> [00000000009074d0] create_kmalloc_cache+0x34/0x60
> [0000000000907540] create_kmalloc_caches+0x44/0x168
> [0000000000908dfc] kmem_cache_init+0x1d0/0x1e0
> [00000000008fc658] start_kernel+0x18c/0x370
> [0000000000761df4] tlb_fixup_done+0x88/0x94
> [0000000000000000] (null)
> Disabling lock debugging due to kernel taint
> Caller[0000000000907464]: create_boot_cache+0x4c/0x84
> Caller[00000000009074d0]: create_kmalloc_cache+0x34/0x60
> Caller[0000000000907540]: create_kmalloc_caches+0x44/0x168
> Caller[0000000000908dfc]: kmem_cache_init+0x1d0/0x1e0
> Caller[00000000008fc658]: start_kernel+0x18c/0x370
> Caller[0000000000761df4]: tlb_fixup_done+0x88/0x94
> Caller[0000000000000000]: (null)
> Instruction DUMP: 92102957 7ffccb35 90122308 <91d02005> 90100018 4009b371 920f20d0 ba922000 02480006
> Kernel panic - not syncing: Attempted to kill the idle task!
> Press Stop-A (L1-A) to return to the boot prom
>
> The line shows that __kmem_cache_create gets a NULL from kmalloc_slab().
>
> I instrumented the code and found the following:
>
> __kmem_cache_create: starting, size=248, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 248 because of redzoning
> __kmem_cache_create: pagealloc debug, setting size to 8192
> __kmem_cache_create: aligned size to 8192
> __kmem_cache_create: num=1, slab_size=64
> __kmem_cache_create: starting, size=96, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 96 because of redzoning
> __kmem_cache_create: pagealloc debug, setting size to 8192
> __kmem_cache_create: aligned size to 8192
> __kmem_cache_create: num=1, slab_size=64
> __kmem_cache_create: starting, size=192, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 192 because of redzoning
> __kmem_cache_create: pagealloc debug, setting size to 8192
> __kmem_cache_create: aligned size to 8192
> __kmem_cache_create: num=1, slab_size=64
> __kmem_cache_create: starting, size=32, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 32 because of redzoning
> __kmem_cache_create: aligned size to 32
> __kmem_cache_create: num=226, slab_size=960
> __kmem_cache_create: starting, size=64, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 64 because of redzoning
> __kmem_cache_create: pagealloc debug, setting size to 8192
> __kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
> __kmem_cache_create: aligned size to 8192
> __kmem_cache_create: num=1, slab_size=64
> __kmem_cache_create: CFLGS_OFF_SLAB, size=8192, slab_size=52
> __kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
>
> With slab size 64, it turns on CFLGS_OFF_SLAB and off slab allocation
> with this size fails. I do not know slab internals so I can not tell if
> this just happens because of the debug paths, or is it a real problem
> without the debug options too.
>
There was a rather large change to mm/slab.c that changed it to use
'struct page' like SLUB. Perhaps slab debugging was broken in the
process. Joonsoo, does the problem Meelis describes ring a bell?
Pekka
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Slab BUG with DEBUG_* options
2013-12-03 11:53 ` Pekka Enberg
@ 2013-12-03 12:25 ` Joonsoo Kim
2013-12-03 12:44 ` Pekka Enberg
0 siblings, 1 reply; 15+ messages in thread
From: Joonsoo Kim @ 2013-12-03 12:25 UTC (permalink / raw)
To: Pekka Enberg
Cc: Meelis Roos, Pekka Enberg, Christoph Lameter, Matt Mackall,
Linux Kernel list, Linux Memory Management List, Joonsoo Kim
2013/12/3 Pekka Enberg <penberg@iki.fi>:
> On 11/30/2013 01:42 PM, Meelis Roos wrote:
>>
>> I am debugging a reboot problem on Sun Ultra 5 (sparc64) with 512M RAM
>> and turned on DEBUG_PAGEALLOC DEBUG_SLAB and DEBUG_SLAB_LEAK (and most
>> other debug options) and got the following BUG and hang on startup. This
>> happened originally with 3.11-rc2-00058 where my bisection of
>> another problem lead, but I retested 3.12 to have the same BUG in the
>> same place.
>>
>> kernel BUG at mm/slab.c:2391!
>> \|/ ____ \|/
>> "@'/ .. \`@"
>> /_| \__/ |_\
>> \__U_/
>> swapper(0): Kernel bad sw trap 5 [#1]
>> CPU: 0 PID: 0 Comm: swapper Not tainted 3.11.0-rc2-00058-g20bafb3-dirty
>> #127
>> task: 00000000008ac468 ti: 000000000089c000 task.ti: 000000000089c000
>> TSTATE: 0000004480e01606 TPC: 00000000004f57d4 TNPC: 00000000004f57d8 Y:
>> 00000000 Not tainted
>> TPC: <__kmem_cache_create+0x374/0x480>
>> g0: 00000000000000f8 g1: 00000000008bb400 g2: 000000000002780b g3:
>> 00000000008b5120
>> g4: 00000000008ac468 g5: 0000000000000000 g6: 000000000089c000 g7:
>> 0000000000000000
>> o0: 0000000000845f08 o1: 0000000000000957 o2: ffffffffffffffe0 o3:
>> 0000000000000000
>> o4: 0000000000002004 o5: 0000000000000000 sp: 000000000089f301 ret_pc:
>> 00000000004f57cc
>> RPC: <__kmem_cache_create+0x36c/0x480>
>> l0: fffff8001e812040 l1: fffff8001e819f80 l2: fffff8001e819fb8 l3:
>> fffff8001e819fd8
>> l4: 0000000000000001 l5: fffff8001e819fc8 l6: 0000000000845f08 l7:
>> fffff8001e8300a0
>> i0: fffff8001e831fa0 i1: 0000000080002800 i2: 0000000080000000 i3:
>> 0000000000000034
>> i4: 0000000000000000 i5: 0000000000002000 i6: 000000000089f3b1 i7:
>> 0000000000907464
>> I7: <create_boot_cache+0x4c/0x84>
>> Call Trace:
>> [0000000000907464] create_boot_cache+0x4c/0x84
>> [00000000009074d0] create_kmalloc_cache+0x34/0x60
>> [0000000000907540] create_kmalloc_caches+0x44/0x168
>> [0000000000908dfc] kmem_cache_init+0x1d0/0x1e0
>> [00000000008fc658] start_kernel+0x18c/0x370
>> [0000000000761df4] tlb_fixup_done+0x88/0x94
>> [0000000000000000] (null)
>> Disabling lock debugging due to kernel taint
>> Caller[0000000000907464]: create_boot_cache+0x4c/0x84
>> Caller[00000000009074d0]: create_kmalloc_cache+0x34/0x60
>> Caller[0000000000907540]: create_kmalloc_caches+0x44/0x168
>> Caller[0000000000908dfc]: kmem_cache_init+0x1d0/0x1e0
>> Caller[00000000008fc658]: start_kernel+0x18c/0x370
>> Caller[0000000000761df4]: tlb_fixup_done+0x88/0x94
>> Caller[0000000000000000]: (null)
>> Instruction DUMP: 92102957 7ffccb35 90122308 <91d02005> 90100018
>> 4009b371 920f20d0 ba922000 02480006
>> Kernel panic - not syncing: Attempted to kill the idle task!
>> Press Stop-A (L1-A) to return to the boot prom
>>
>> The line shows that __kmem_cache_create gets a NULL from kmalloc_slab().
>>
>> I instrumented the code and found the following:
>>
>> __kmem_cache_create: starting, size=248, flags=8192
>> __kmem_cache_create: now flags=76800
>> __kmem_cache_create: aligned size to 248 because of redzoning
>> __kmem_cache_create: pagealloc debug, setting size to 8192
>> __kmem_cache_create: aligned size to 8192
>> __kmem_cache_create: num=1, slab_size=64
>> __kmem_cache_create: starting, size=96, flags=8192
>> __kmem_cache_create: now flags=76800
>> __kmem_cache_create: aligned size to 96 because of redzoning
>> __kmem_cache_create: pagealloc debug, setting size to 8192
>> __kmem_cache_create: aligned size to 8192
>> __kmem_cache_create: num=1, slab_size=64
>> __kmem_cache_create: starting, size=192, flags=8192
>> __kmem_cache_create: now flags=76800
>> __kmem_cache_create: aligned size to 192 because of redzoning
>> __kmem_cache_create: pagealloc debug, setting size to 8192
>> __kmem_cache_create: aligned size to 8192
>> __kmem_cache_create: num=1, slab_size=64
>> __kmem_cache_create: starting, size=32, flags=8192
>> __kmem_cache_create: now flags=76800
>> __kmem_cache_create: aligned size to 32 because of redzoning
>> __kmem_cache_create: aligned size to 32
>> __kmem_cache_create: num=226, slab_size=960
>> __kmem_cache_create: starting, size=64, flags=8192
>> __kmem_cache_create: now flags=76800
>> __kmem_cache_create: aligned size to 64 because of redzoning
>> __kmem_cache_create: pagealloc debug, setting size to 8192
>> __kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
>> __kmem_cache_create: aligned size to 8192
>> __kmem_cache_create: num=1, slab_size=64
>> __kmem_cache_create: CFLGS_OFF_SLAB, size=8192, slab_size=52
>> __kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
>>
>> With slab size 64, it turns on CFLGS_OFF_SLAB and off slab allocation
>> with this size fails. I do not know slab internals so I can not tell if
>> this just happens because of the debug paths, or is it a real problem
>> without the debug options too.
>>
>
> There was a rather large change to mm/slab.c that changed it to use 'struct
> page' like SLUB. Perhaps slab debugging was broken in the process. Joonsoo,
> does the problem Meelis describes ring a bell?
Hello, Pekka.
No. He report that BUG() is triggered on v3.11-rc2 and v3.12.
And my recent change is merged into v3.13-rc1 as you know. :)
Thanks.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Slab BUG with DEBUG_* options
2013-12-03 12:25 ` Joonsoo Kim
@ 2013-12-03 12:44 ` Pekka Enberg
0 siblings, 0 replies; 15+ messages in thread
From: Pekka Enberg @ 2013-12-03 12:44 UTC (permalink / raw)
To: Joonsoo Kim
Cc: Meelis Roos, Pekka Enberg, Christoph Lameter, Matt Mackall,
Linux Kernel list, Linux Memory Management List, Joonsoo Kim
On 12/03/2013 02:25 PM, Joonsoo Kim wrote:
> No. He report that BUG() is triggered on v3.11-rc2 and v3.12.
> And my recent change is merged into v3.13-rc1 as you know. :)
Hah, I guess my eyesight isn't what it used to be, I could have sworn it
said v3.13-rc2... Thanks anyway, Joonsoo!
Pekka
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Slab BUG with DEBUG_* options
2013-11-30 11:42 Slab BUG with DEBUG_* options Meelis Roos
2013-11-30 22:31 ` David Rientjes
2013-12-03 11:53 ` Pekka Enberg
@ 2013-12-03 15:46 ` Christoph Lameter
2013-12-03 20:33 ` Meelis Roos
[not found] ` <alpine.DEB.2.02.1312030930450.4115@gentwo.org>
3 siblings, 1 reply; 15+ messages in thread
From: Christoph Lameter @ 2013-12-03 15:46 UTC (permalink / raw)
To: Meelis Roos
Cc: Pekka Enberg, Matt Mackall, Linux Kernel list, linux-mm, Joonsoo Kim
On Sat, 30 Nov 2013, Meelis Roos wrote:
On Sat, 30 Nov 2013, Meelis Roos wrote:
> I am debugging a reboot problem on Sun Ultra 5 (sparc64) with 512M RAM
> and turned on DEBUG_PAGEALLOC DEBUG_SLAB and DEBUG_SLAB_LEAK (and most
> other debug options) and got the following BUG and hang on startup. This
> happened originally with 3.11-rc2-00058 where my bisection of
> another problem lead, but I retested 3.12 to have the same BUG in the
> same place.
Hmmm. With CONFIG_DEBUG_PAGEALLOC *and* DEBUG_SLAB you would get a pretty
strange configuration with massive sizes of slabs.
> kernel BUG at mm/slab.c:2391!
Ok so this means that we are trying to create a cache with off slab
management during bootstrap which should not happen.
> __kmem_cache_create: starting, size=248, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 248 because of redzoning
> __kmem_cache_create: pagealloc debug, setting size to 8192
> __kmem_cache_create: aligned size to 8192
> __kmem_cache_create: num=1, slab_size=64
> __kmem_cache_create: starting, size=96, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 96 because of redzoning
> __kmem_cache_create: pagealloc debug, setting size to 8192
> __kmem_cache_create: aligned size to 8192
> __kmem_cache_create: num=1, slab_size=64
> __kmem_cache_create: starting, size=192, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 192 because of redzoning
> __kmem_cache_create: pagealloc debug, setting size to 8192
> __kmem_cache_create: aligned size to 8192
> __kmem_cache_create: num=1, slab_size=64
> __kmem_cache_create: starting, size=32, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 32 because of redzoning
> __kmem_cache_create: aligned size to 32
> __kmem_cache_create: num=226, slab_size=960
> __kmem_cache_create: starting, size=64, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 64 because of redzoning
> __kmem_cache_create: pagealloc debug, setting size to 8192
> __kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
We should not be switching on CFLGS_OFF_SLAB here because the
kmalloc array does not contain the necessary entries yet.
Does this fix it? We may need a more sophisticated fix from someone who
knows how handle CONFIG_DEBUG_PAGEALLOC.
Subject: slab: Do not use off slab metadata for CONFIG_DEBUG_PAGEALLOC
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/mm/slab.c
===================================================================
--- linux.orig/mm/slab.c 2013-12-03 09:44:20.463144282 -0600
+++ linux/mm/slab.c 2013-12-03 09:45:09.321786608 -0600
@@ -2243,13 +2243,15 @@ __kmem_cache_create (struct kmem_cache *
* it too early on. Always use on-slab management when
* SLAB_NOLEAKTRACE to avoid recursive calls into kmemleak)
*/
+#ifndef CONFIG_DEBUG_PAGEALLOC
if ((size >= (PAGE_SIZE >> 3)) && !slab_early_init &&
- !(flags & SLAB_NOLEAKTRACE))
+ !(flags & SLAB_NOLEAKTRACE) )
/*
* Size is large, assume best to place the slab management obj
* off-slab (should allow better packing of objs).
*/
flags |= CFLGS_OFF_SLAB;
+#endif
size = ALIGN(size, cachep->align);
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Slab BUG with DEBUG_* options
[not found] ` <alpine.DEB.2.02.1312030930450.4115@gentwo.org>
@ 2013-12-03 20:25 ` Christoph Lameter
2013-12-03 21:18 ` Meelis Roos
0 siblings, 1 reply; 15+ messages in thread
From: Christoph Lameter @ 2013-12-03 20:25 UTC (permalink / raw)
To: Meelis Roos
Cc: Pekka Enberg, Matt Mackall, Linux Kernel list, linux-mm, Joonsoo Kim
On Tue, 3 Dec 2013, Christoph Lameter wrote:
> Subject: slab: Do not use off slab metadata for CONFIG_DEBUG_PAGEALLOC
>
> Signed-off-by: Christoph Lameter <cl@linux.com>
The patch fails here at the largest slab which requires off slab
management because otherwise the order becomes too high.
The fundamental problem is that debugging increases the size of the slab
significantly so that even small slabs require slab management. But
the kmalloc slabs used for the sizes required for management objects have not
been created yet.
Slab bootstrap relies on smaller slabs not requiring slab management
slabs. Once it gets to larger sizes then smaller slab management objects
may be allocated.
However, in this case the slab sizes are required for management at early
boot after slab_early_init has been set to 0 are not available.
If the slab management sizes required match other slab caches sizes that
were already created at boot then bootstrap will succeed regardless.
I have another patch here (that I cannot test since I cannot run sparc
code) that simply changes the determination for switching slab management
to base the decision not on the final size of the slab (which is always
large in the debugging cases here) but on the initial object size. For
small objects < PAGESIZE/8 this should avoid the use of slab management
even in the debugging case.
Subject: slab: Do not use slab management for slabs with smaller objects
Use the object size to make the off slab decision instead of the final
size of the slab objects (which is large in case of
CONFIG_PAGEALLOC_DEBUG).
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/mm/slab.c
===================================================================
--- linux.orig/mm/slab.c 2013-12-03 10:48:42.625823157 -0600
+++ linux/mm/slab.c 2013-12-03 10:49:06.755152653 -0600
@@ -2243,8 +2243,8 @@ __kmem_cache_create (struct kmem_cache *
* it too early on. Always use on-slab management when
* SLAB_NOLEAKTRACE to avoid recursive calls into kmemleak)
*/
- if ((size >= (PAGE_SIZE >> 3)) && !slab_early_init &&
- !(flags & SLAB_NOLEAKTRACE))
+ if ((cachep->size >= (PAGE_SIZE >> 3)) && !slab_early_init &&
+ !(flags & SLAB_NOLEAKTRACE) )
/*
* Size is large, assume best to place the slab management obj
* off-slab (should allow better packing of objs).
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Slab BUG with DEBUG_* options
2013-12-03 15:46 ` Christoph Lameter
@ 2013-12-03 20:33 ` Meelis Roos
2013-12-03 20:59 ` Christoph Lameter
0 siblings, 1 reply; 15+ messages in thread
From: Meelis Roos @ 2013-12-03 20:33 UTC (permalink / raw)
To: Christoph Lameter
Cc: Pekka Enberg, Matt Mackall, Linux Kernel list, linux-mm, Joonsoo Kim
> > I am debugging a reboot problem on Sun Ultra 5 (sparc64) with 512M RAM
> > and turned on DEBUG_PAGEALLOC DEBUG_SLAB and DEBUG_SLAB_LEAK (and most
> > other debug options) and got the following BUG and hang on startup. This
> > happened originally with 3.11-rc2-00058 where my bisection of
> > another problem lead, but I retested 3.12 to have the same BUG in the
> > same place.
>
> Hmmm. With CONFIG_DEBUG_PAGEALLOC *and* DEBUG_SLAB you would get a pretty
> strange configuration with massive sizes of slabs.
>
> > kernel BUG at mm/slab.c:2391!
>
> Ok so this means that we are trying to create a cache with off slab
> management during bootstrap which should not happen.
[...]
> We should not be switching on CFLGS_OFF_SLAB here because the
> kmalloc array does not contain the necessary entries yet.
>
> Does this fix it? We may need a more sophisticated fix from someone who
> knows how handle CONFIG_DEBUG_PAGEALLOC.
No:
Kernel panic - not syncing: Creation of kmalloc slab (null) size=8388608
failed. Reason -7
CPU: 0 PID: 0 Comm: swapper Not tainted 3.11.0-rc2-00058-g20bafb3-dirty
#134
Call Trace:
[000000000076416c] panic+0xb4/0x22c
[0000000000907488] create_boot_cache+0x70/0x84
[00000000009074d0] create_kmalloc_cache+0x34/0x60
[0000000000907540] create_kmalloc_caches+0x44/0x168
[0000000000908dfc] kmem_cache_init+0x1d0/0x1e0
[00000000008fc658] start_kernel+0x18c/0x370
[0000000000761db4] tlb_fixup_done+0x88/0x94
[0000000000000000] (null)
Am I just running out of memory perhaps?
Will try the other patch soon.
--
Meelis Roos (mroos@linux.ee)
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Slab BUG with DEBUG_* options
2013-12-03 20:33 ` Meelis Roos
@ 2013-12-03 20:59 ` Christoph Lameter
0 siblings, 0 replies; 15+ messages in thread
From: Christoph Lameter @ 2013-12-03 20:59 UTC (permalink / raw)
To: Meelis Roos
Cc: Pekka Enberg, Matt Mackall, Linux Kernel list, linux-mm, Joonsoo Kim
On Tue, 3 Dec 2013, Meelis Roos wrote:
> Kernel panic - not syncing: Creation of kmalloc slab (null) size=8388608
> failed. Reason -7
Same here.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Slab BUG with DEBUG_* options
2013-12-03 20:25 ` Christoph Lameter
@ 2013-12-03 21:18 ` Meelis Roos
2013-12-03 21:58 ` Christoph Lameter
0 siblings, 1 reply; 15+ messages in thread
From: Meelis Roos @ 2013-12-03 21:18 UTC (permalink / raw)
To: Christoph Lameter
Cc: Pekka Enberg, Matt Mackall, Linux Kernel list, linux-mm, Joonsoo Kim
> I have another patch here (that I cannot test since I cannot run sparc
> code) that simply changes the determination for switching slab management
> to base the decision not on the final size of the slab (which is always
> large in the debugging cases here) but on the initial object size. For
> small objects < PAGESIZE/8 this should avoid the use of slab management
> even in the debugging case.
>
> Subject: slab: Do not use slab management for slabs with smaller objects
>
> Use the object size to make the off slab decision instead of the final
> size of the slab objects (which is large in case of
> CONFIG_PAGEALLOC_DEBUG).
Tested it. seems to hang after switching to another console. Before
that, slabs are initialized successfully, I verified it with my previous
debug printk sprinkle patch. Many allocations are still off slab - is
that OK?
Memory: 493968K/523456K available (3521K kernel code, 343K rwdata, 1176K rodata, 264K init, 9803K bss, 29488K reserved)
__kmem_cache_create: starting, size=248, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 248 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: starting, size=96, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 96 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: starting, size=192, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 192 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: starting, size=32, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 32 because of redzoning
__kmem_cache_create: aligned size to 32
__kmem_cache_create: num=226, slab_size=960
__kmem_cache_create: starting, size=64, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 64 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: starting, size=128, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 128 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: starting, size=256, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 256 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: starting, size=512, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 512 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: starting, size=1024, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 1024 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=8192, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=2048, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 2048 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=8192, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=4096, flags=8192
__kmem_cache_create: now flags=10240
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=8192, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=8192, flags=8192
__kmem_cache_create: now flags=10240
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=8192, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=16384, flags=8192
__kmem_cache_create: now flags=10240
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=16384
__kmem_cache_create: aligned size to 16384
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=16384, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=32768, flags=8192
__kmem_cache_create: now flags=10240
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=32768
__kmem_cache_create: aligned size to 32768
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=32768, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=65536, flags=8192
__kmem_cache_create: now flags=10240
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=65536
__kmem_cache_create: aligned size to 65536
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=65536, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=131072, flags=8192
__kmem_cache_create: now flags=10240
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=131072
__kmem_cache_create: aligned size to 131072
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=131072, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=262144, flags=8192
__kmem_cache_create: now flags=10240
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=262144
__kmem_cache_create: aligned size to 262144
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=262144, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=524288, flags=8192
__kmem_cache_create: now flags=10240
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=524288
__kmem_cache_create: aligned size to 524288
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=524288, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=1048576, flags=8192
__kmem_cache_create: now flags=10240
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=1048576
__kmem_cache_create: aligned size to 1048576
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=1048576, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=2097152, flags=8192
__kmem_cache_create: now flags=10240
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=2097152
__kmem_cache_create: aligned size to 2097152
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=2097152, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=4194304, flags=8192
__kmem_cache_create: now flags=10240
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=4194304
__kmem_cache_create: aligned size to 4194304
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=4194304, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=8388608, flags=8192
__kmem_cache_create: now flags=10240
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8388608
__kmem_cache_create: aligned size to 8388608
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=8388608, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=8192, flags=0
__kmem_cache_create: now flags=2048
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=8192
__kmem_cache_create: CFLGS_OFF_SLAB, size=8192, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=8192, flags=0
__kmem_cache_create: now flags=2048
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=8192
__kmem_cache_create: CFLGS_OFF_SLAB, size=8192, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=16384, flags=0
__kmem_cache_create: now flags=2048
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=16384
__kmem_cache_create: aligned size to 16384
__kmem_cache_create: num=1, slab_size=16384
__kmem_cache_create: CFLGS_OFF_SLAB, size=16384, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=32768, flags=0
__kmem_cache_create: now flags=2048
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=32768
__kmem_cache_create: aligned size to 32768
__kmem_cache_create: num=1, slab_size=32768
__kmem_cache_create: CFLGS_OFF_SLAB, size=32768, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=65536, flags=0
__kmem_cache_create: now flags=2048
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=65536
__kmem_cache_create: aligned size to 65536
__kmem_cache_create: num=1, slab_size=65536
__kmem_cache_create: CFLGS_OFF_SLAB, size=65536, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=131072, flags=0
__kmem_cache_create: now flags=2048
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=131072
__kmem_cache_create: aligned size to 131072
__kmem_cache_create: num=1, slab_size=131072
__kmem_cache_create: CFLGS_OFF_SLAB, size=131072, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=262144, flags=0
__kmem_cache_create: now flags=2048
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=262144
__kmem_cache_create: aligned size to 262144
__kmem_cache_create: num=1, slab_size=262144
__kmem_cache_create: CFLGS_OFF_SLAB, size=262144, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=524288, flags=0
__kmem_cache_create: now flags=2048
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=524288
__kmem_cache_create: aligned size to 524288
__kmem_cache_create: num=1, slab_size=524288
__kmem_cache_create: CFLGS_OFF_SLAB, size=524288, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=1048576, flags=0
__kmem_cache_create: now flags=2048
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=1048576
__kmem_cache_create: aligned size to 1048576
__kmem_cache_create: num=1, slab_size=1048576
__kmem_cache_create: CFLGS_OFF_SLAB, size=1048576, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=2112, flags=262144
__kmem_cache_create: now flags=330752
__kmem_cache_create: aligned size to 2112 because of redzoning
__kmem_cache_create: aligned size to 2128 because of redzoning, take 2
__kmem_cache_create: increased size to 2136 because user store and redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=56
__kmem_cache_create: CFLGS_OFF_SLAB, size=8192, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=560, flags=393216
__kmem_cache_create: now flags=461824
__kmem_cache_create: aligned size to 560 because of redzoning
__kmem_cache_create: aligned size to 576 because of redzoning, take 2
__kmem_cache_create: increased size to 584 because user store and redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=56
NR_IRQS:255
kmemleak: Kernel memory leak detector disabled
clocksource: mult[2c71c72] shift[24]
clockevent: mult[5c28f5c3] shift[32]
Console: colour dummy device 80x25
console [tty0] enabled, bootconsole disabled
--
Meelis Roos (mroos@linux.ee)
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Slab BUG with DEBUG_* options
2013-12-03 21:18 ` Meelis Roos
@ 2013-12-03 21:58 ` Christoph Lameter
2013-12-08 15:00 ` Meelis Roos
0 siblings, 1 reply; 15+ messages in thread
From: Christoph Lameter @ 2013-12-03 21:58 UTC (permalink / raw)
To: Meelis Roos
Cc: Pekka Enberg, Matt Mackall, Linux Kernel list, linux-mm, Joonsoo Kim
On Tue, 3 Dec 2013, Meelis Roos wrote:
> Tested it. seems to hang after switching to another console. Before
> that, slabs are initialized successfully, I verified it with my previous
> debug printk sprinkle patch. Many allocations are still off slab - is
> that OK?
Yes that was the intend. Only exempt the small ones.
> console [tty0] enabled, bootconsole disabled
Looks like the bootstrap worked.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Slab BUG with DEBUG_* options
2013-12-03 21:58 ` Christoph Lameter
@ 2013-12-08 15:00 ` Meelis Roos
0 siblings, 0 replies; 15+ messages in thread
From: Meelis Roos @ 2013-12-08 15:00 UTC (permalink / raw)
To: Christoph Lameter
Cc: Pekka Enberg, Matt Mackall, Linux Kernel list, linux-mm,
Joonsoo Kim, Peter Hurley, Greg Kroah-Hartman, David Miller,
sparclinux
(Added 3 addresses to CC from my RED state exception thread since this
is related)
> On Tue, 3 Dec 2013, Meelis Roos wrote:
>
> > Tested it. seems to hang after switching to another console. Before
> > that, slabs are initialized successfully, I verified it with my previous
> > debug printk sprinkle patch. Many allocations are still off slab - is
> > that OK?
>
> Yes that was the intend. Only exempt the small ones.
>
> > console [tty0] enabled, bootconsole disabled
>
> Looks like the bootstrap worked.
But the configuration should work fine with this console setup - with no
slab debug options, it booted fine... I decided to do more tests.
In short, tests about 3.11-rc2-00058:
clean kernel: boots OK, RED state on shutdown (the actual problem I am
chasing)
clean kernel, slab debug: mm crash
your second slab patch, slab debug: OK - this one shows that the RED
state problem went away too which is good but strange
clean kernel, your second slab patch: OK - no RED state
Following another lead I had discovered that I can make the RED state
problem go away if I switch tty ldata allocation from vmalloc to
kmalloc. Tests with that:
ldata alloc change only, no slab debug: OK (works around RED state
somehow)
ldata alloc change + slab debug: mm crash, can not test for RED state
ldata alloc change + your second slab patch + slab debug: hang on boot
after
console [tty0] enabled, bootconsole disabled
(after that, I should see all the dmesg again on serial but I do not).
ldata alloc change + your second slab patch + no slab debug: OK
So, in short:
your slab patch 2 seems to cure both slab debug startup crash and the
RED state problem in this specific version of the kernel. However, it is
still mystery to me why tty ldata alloc change vmalloc->kmalloc would
break but that may to be in the serial field - will do more tests with
this patch applied and newer kernels.
--
Meelis Roos (mroos@linux.ee)
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2013-12-08 15:00 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-11-30 11:42 Slab BUG with DEBUG_* options Meelis Roos
2013-11-30 22:31 ` David Rientjes
2013-12-02 19:02 ` Christoph Lameter
2013-12-03 8:46 ` Meelis Roos
2013-12-03 9:17 ` Meelis Roos
2013-12-03 11:53 ` Pekka Enberg
2013-12-03 12:25 ` Joonsoo Kim
2013-12-03 12:44 ` Pekka Enberg
2013-12-03 15:46 ` Christoph Lameter
2013-12-03 20:33 ` Meelis Roos
2013-12-03 20:59 ` Christoph Lameter
[not found] ` <alpine.DEB.2.02.1312030930450.4115@gentwo.org>
2013-12-03 20:25 ` Christoph Lameter
2013-12-03 21:18 ` Meelis Roos
2013-12-03 21:58 ` Christoph Lameter
2013-12-08 15:00 ` Meelis Roos
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).