From: "John Thomson" <lists@johnthomson.fastmail.com.au>
To: "Hyeonggon Yoo" <42.hyeyoo@gmail.com>
Cc: "Feng Tang" <feng.tang@intel.com>,
"Vlastimil Babka" <vbabka@suse.cz>,
"Andrew Morton" <akpm@linux-foundation.org>,
"Christoph Lameter" <cl@linux.com>,
"Pekka Enberg" <penberg@kernel.org>,
"David Rientjes" <rientjes@google.com>,
"Joonsoo Kim" <iamjoonsoo.kim@lge.com>,
"Roman Gushchin" <roman.gushchin@linux.dev>,
"Dmitry Vyukov" <dvyukov@google.com>,
"Jonathan Corbet" <corbet@lwn.net>,
"Andrey Konovalov" <andreyknvl@gmail.com>,
"Hansen, Dave" <dave.hansen@intel.com>,
"linux-mm@kvack.org" <linux-mm@kvack.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"kasan-dev@googlegroups.com" <kasan-dev@googlegroups.com>,
"Robin Murphy" <robin.murphy@arm.com>,
"John Garry" <john.garry@huawei.com>,
"Kefeng Wang" <wangkefeng.wang@huawei.com>,
"Thomas Bogendoerfer" <tsbogend@alpha.franken.de>,
linux-mips@vger.kernel.org
Subject: Re: [PATCH v6 1/4] mm/slub: enable debugging memory wasting of kmalloc
Date: Tue, 01 Nov 2022 10:33:32 +0000 [thread overview]
Message-ID: <29271a2b-cf19-4af9-bfe5-5bcff8a23fda@app.fastmail.com> (raw)
In-Reply-To: <Y2DngwUc7cLB0dG7@hyeyoo>
On Tue, 1 Nov 2022, at 09:31, Hyeonggon Yoo wrote:
> On Tue, Nov 01, 2022 at 09:20:21AM +0000, John Thomson wrote:
>> On Tue, 1 Nov 2022, at 07:57, Feng Tang wrote:
>> > Hi Thomson,
>> >
>> > Thanks for testing!
>> >
>> > + mips maintainer and mail list. The original report is here
>> >
>> > https://lore.kernel.org/lkml/becf2ac3-2a90-4f3a-96d9-a70f67c66e4a@app.fastmail.com/
>>
>> I am guessing my issue comes from __kmem_cache_alloc_lru accessing s->object_size when (kmem_cache) s is NULL?
>> If that is the case, this change is not to blame, it only exposes the issue?
>>
>> I get the following dmesg (note very early NULL kmem_cache) with the below change atop v6.1-rc3:
>>
>> transfer started ......................................... transfer ok, time=2.02s
>> setting up elf image... OK
>> jumping to kernel code
>> zimage at: 80B842A0 810B4EFC
>>
>> Uncompressing Linux at load address 80001000
>>
>> Copy device tree to address 80B80EE0
>>
>> Now, booting the kernel...
>>
>> [ 0.000000] Linux version 6.1.0-rc3+ (john@john) (mipsel-buildroot-linux-gnu-gcc.br_real (Buildroot 2021.11-4428-g6b6741b) 12.2.0, GNU ld (GNU Binutils) 2.39) #61 SMP Tue Nov 1 18:04:13 AEST 2022
>> [ 0.000000] slub: kmem_cache_alloc called with kmem_cache: 0x0
>> [ 0.000000] slub: __kmem_cache_alloc_lru called with kmem_cache: 0x0
>> [ 0.000000] SoC Type: MediaTek MT7621 ver:1 eco:3
>> [ 0.000000] printk: bootconsole [early0] enabled
>> [ 0.000000] CPU0 revision is: 0001992f (MIPS 1004Kc)
>> [ 0.000000] MIPS: machine is MikroTik RouterBOARD 760iGS
>>
>> normal boot
>>
>>
>> diff --git a/mm/slub.c b/mm/slub.c
>> index 157527d7101b..10fcdf2520d2 100644
>> --- a/mm/slub.c
>> +++ b/mm/slub.c
>> @@ -3410,7 +3410,13 @@ static __always_inline
>> void *__kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru,
>> gfp_t gfpflags)
>> {
>> - void *ret = slab_alloc(s, lru, gfpflags, _RET_IP_, s->object_size);
>> + void *ret;
>> + if (IS_ERR_OR_NULL(s)) {
>> + pr_warn("slub: __kmem_cache_alloc_lru called with kmem_cache: %pSR\n", s);
>> + ret = slab_alloc(s, lru, gfpflags, _RET_IP_, 0);
>> + } else {
>> + ret = slab_alloc(s, lru, gfpflags, _RET_IP_, s->object_size);
>> + }
>>
>> trace_kmem_cache_alloc(_RET_IP_, ret, s, gfpflags, NUMA_NO_NODE);
>>
>> @@ -3419,6 +3425,8 @@ void *__kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru,
>>
>> void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags)
>> {
>> + if (IS_ERR_OR_NULL(s))
>> + pr_warn("slub: kmem_cache_alloc called with kmem_cache: %pSR\n", s);
>> return __kmem_cache_alloc_lru(s, NULL, gfpflags);
>> }
>> EXPORT_SYMBOL(kmem_cache_alloc);
>> @@ -3426,6 +3434,8 @@ EXPORT_SYMBOL(kmem_cache_alloc);
>> void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru,
>> gfp_t gfpflags)
>> {
>> + if (IS_ERR_OR_NULL(s))
>> + pr_warn("slub: __kmem_cache_alloc_lru called with kmem_cache: %pSR\n", s);
>> return __kmem_cache_alloc_lru(s, lru, gfpflags);
>> }
>> EXPORT_SYMBOL(kmem_cache_alloc_lru);
>>
>>
>> Any hints on where kmem_cache_alloc would be being called from this early?
>> I will start looking from /init/main.c around pr_notice("%s", linux_banner);
>
> Great. Would you try calling dump_stack(); when we observed s == NULL?
> That would give more information about who passed s == NULL to these
> functions.
>
With the dump_stack() in place:
Now, booting the kernel...
[ 0.000000] Linux version 6.1.0-rc3+ (john@john) (mipsel-buildroot-linux-gnu-gcc.br_real (Buildroot 2021.11-4428-g6b6741b) 12.2.0, GNU ld (GNU Binutils) 2.39) #62 SMP Tue Nov 1 19:49:52 AEST 2022
[ 0.000000] slub: __kmem_cache_alloc_lru called with kmem_cache ptr: 0x0
[ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 6.1.0-rc3+ #62
[ 0.000000] Stack : 810fff78 80084d98 80889d00 00000004 00000000 00000000 80889d5c 80c90000
[ 0.000000] 80920000 807bd380 8089d368 80923bd3 00000000 00000001 80889d08 00000000
[ 0.000000] 00000000 00000000 807bd380 8084bd51 00000002 00000002 00000001 6d6f4320
[ 0.000000] 00000000 80c97ce9 80c97d14 fffffffc 807bd380 00000000 00000003 00000dc0
[ 0.000000] 00000000 a0000000 80910000 8110a0b4 00000000 00000020 80010000 80010000
[ 0.000000] ...
[ 0.000000] Call Trace:
[ 0.000000] [<80008260>] show_stack+0x28/0xf0
[ 0.000000] [<8070cdc0>] dump_stack_lvl+0x60/0x80
[ 0.000000] [<801c1428>] kmem_cache_alloc+0x5c0/0x740
[ 0.000000] [<8092856c>] prom_soc_init+0x1fc/0x2b4
[ 0.000000] [<80928060>] prom_init+0x44/0xf0
[ 0.000000] [<80929214>] setup_arch+0x4c/0x6a8
[ 0.000000] [<809257e0>] start_kernel+0x88/0x7c0
[ 0.000000]
[ 0.000000] SoC Type: MediaTek MT7621 ver:1 eco:3
Now, booting the kernel...
[ 0.000000] Linux version 6.1.0-rc3+ (john@john) (mipsel-buildroot-linux-gnu-gcc.br_real (Buildroot 2021.11-4428-g6b6741b) 12.2.0, GNU ld (GNU Binutils) 2.39) #62 SMP Tue Nov 1 19:49:52 AEST 2022
[ 0.000000] slub: __kmem_cache_alloc_lru called with kmem_cache ptr: 0x0
[ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 6.1.0-rc3+ #62
[ 0.000000] Stack : 810fff78 80084d98 80889d00 00000004 00000000 00000000 80889d5c 80c90000
[ 0.000000] 80920000 807bd380 8089d368 80923bd3 00000000 00000001 80889d08 00000000
[ 0.000000] 00000000 00000000 807bd380 8084bd51 00000002 00000002 00000001 6d6f4320
[ 0.000000] 00000000 80c97ce9 80c97d14 fffffffc 807bd380 00000000 00000003 00000dc0
[ 0.000000] 00000000 a0000000 80910000 8110a0b4 00000000 00000020 80010000 80010000
[ 0.000000] ...
[ 0.000000] Call Trace:
[ 0.000000] show_stack (/mnt/pool_ssd/code/linux/linux-stable-mt7621/./arch/mips/include/asm/stacktrace.h:43 /mnt/pool_ssd/code/linux/linux-stable-mt7621/arch/mips/kernel/traps.c:223)
[ 0.000000] dump_stack_lvl (/mnt/pool_ssd/code/linux/linux-stable-mt7621/lib/dump_stack.c:107 (discriminator 1))
[ 0.000000] kmem_cache_alloc (/mnt/pool_ssd/code/linux/linux-stable-mt7621/mm/slub.c:3318 /mnt/pool_ssd/code/linux/linux-stable-mt7621/mm/slub.c:3406 /mnt/pool_ssd/code/linux/linux-stable-mt7621/mm/slub.c:3418 /mnt/pool_ssd/code/linux/linux-stable-mt7621/mm/slub.c:3430)
[ 0.000000] prom_soc_init (/mnt/pool_ssd/code/linux/linux-stable-mt7621/arch/mips/ralink/mt7621.c:106 /mnt/pool_ssd/code/linux/linux-stable-mt7621/arch/mips/ralink/mt7621.c:177)
[ 0.000000] prom_init (/mnt/pool_ssd/code/linux/linux-stable-mt7621/arch/mips/ralink/prom.c:64)
[ 0.000000] setup_arch (/mnt/pool_ssd/code/linux/linux-stable-mt7621/arch/mips/kernel/setup.c:786)
[ 0.000000] start_kernel (/mnt/pool_ssd/code/linux/linux-stable-mt7621/init/main.c:279 /mnt/pool_ssd/code/linux/linux-stable-mt7621/init/main.c:477 /mnt/pool_ssd/code/linux/linux-stable-mt7621/init/main.c:960)
[ 0.000000]
[ 0.000000] SoC Type: MediaTek MT7621 ver:1 eco:3
I have not found it yet.
Cheers,
--
John Thomson
next prev parent reply other threads:[~2022-11-01 10:34 UTC|newest]
Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-09-13 6:54 [PATCH v6 0/4] mm/slub: some debug enhancements for kmalloc Feng Tang
2022-09-13 6:54 ` [PATCH v6 1/4] mm/slub: enable debugging memory wasting of kmalloc Feng Tang
2022-09-23 11:43 ` Vlastimil Babka
2022-09-24 7:08 ` Feng Tang
2022-10-30 19:23 ` John Thomson
2022-10-30 21:30 ` Vlastimil Babka
2022-10-31 2:36 ` Feng Tang
2022-10-31 10:05 ` John Thomson
2022-10-31 11:36 ` Hyeonggon Yoo
2022-10-31 11:42 ` Feng Tang
2022-11-01 0:18 ` John Thomson
2022-11-01 2:41 ` John Thomson
2022-11-01 7:57 ` Feng Tang
2022-11-01 9:20 ` John Thomson
2022-11-01 9:31 ` Hyeonggon Yoo
2022-11-01 10:33 ` John Thomson [this message]
2022-11-01 10:42 ` Hyeonggon Yoo
2022-11-01 13:55 ` Feng Tang
2022-11-01 19:39 ` John Thomson
2022-11-02 6:08 ` Feng Tang
2022-11-02 7:16 ` Hyeonggon Yoo
2022-11-03 7:18 ` Feng Tang
2022-11-03 7:45 ` John Thomson
2022-11-03 8:16 ` Feng Tang
2022-11-02 8:22 ` Vlastimil Babka
2022-11-03 5:54 ` Feng Tang
2022-11-03 8:33 ` Vlastimil Babka
2022-11-03 14:16 ` Feng Tang
2022-11-03 14:36 ` Hyeonggon Yoo
2022-11-03 16:57 ` Vlastimil Babka
2022-11-03 17:35 ` Vlastimil Babka
2022-11-04 3:52 ` Feng Tang
2022-09-13 6:54 ` [PATCH v6 2/4] mm/slub: only zero the requested size of buffer for kzalloc Feng Tang
2022-09-26 19:11 ` Andrey Konovalov
2022-09-26 20:15 ` Kees Cook
2022-09-27 1:22 ` Feng Tang
2022-09-27 2:42 ` Feng Tang
2022-10-13 14:00 ` Andrey Konovalov
2022-10-14 5:59 ` Feng Tang
2022-09-13 6:54 ` [PATCH v6 3/4] mm: kasan: Add free_meta size info in struct kasan_cache Feng Tang
2022-09-20 19:20 ` Andrey Konovalov
2022-09-21 12:02 ` Feng Tang
2022-09-24 18:05 ` Andrey Konovalov
2022-09-25 11:26 ` Feng Tang
2022-09-25 16:31 ` Andrey Konovalov
2022-09-27 3:03 ` Feng Tang
2022-09-13 6:54 ` [PATCH v6 4/4] mm/slub: extend redzone check to extra allocated kmalloc space than requested Feng Tang
2022-09-13 8:53 ` Hyeonggon Yoo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=29271a2b-cf19-4af9-bfe5-5bcff8a23fda@app.fastmail.com \
--to=lists@johnthomson.fastmail.com.au \
--cc=42.hyeyoo@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=andreyknvl@gmail.com \
--cc=cl@linux.com \
--cc=corbet@lwn.net \
--cc=dave.hansen@intel.com \
--cc=dvyukov@google.com \
--cc=feng.tang@intel.com \
--cc=iamjoonsoo.kim@lge.com \
--cc=john.garry@huawei.com \
--cc=kasan-dev@googlegroups.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mips@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=penberg@kernel.org \
--cc=rientjes@google.com \
--cc=robin.murphy@arm.com \
--cc=roman.gushchin@linux.dev \
--cc=tsbogend@alpha.franken.de \
--cc=vbabka@suse.cz \
--cc=wangkefeng.wang@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).