From: Feng Tang <feng.tang@intel.com> To: Dmitry Vyukov <dvyukov@google.com> Cc: "Sang, Oliver" <oliver.sang@intel.com>, Vlastimil Babka <vbabka@suse.cz>, lkp <lkp@intel.com>, LKML <linux-kernel@vger.kernel.org>, "linux-mm@kvack.org" <linux-mm@kvack.org>, "lkp@lists.01.org" <lkp@lists.01.org>, Andrew Morton <akpm@linux-foundation.org>, "Christoph Lameter" <cl@linux.com>, Pekka Enberg <penberg@kernel.org>, David Rientjes <rientjes@google.com>, Joonsoo Kim <iamjoonsoo.kim@lge.com>, Roman Gushchin <roman.gushchin@linux.dev>, Hyeonggon Yoo <42.hyeyoo@gmail.com>, "Hansen, Dave" <dave.hansen@intel.com>, Robin Murphy <robin.murphy@arm.com>, "John Garry" <john.garry@huawei.com>, Kefeng Wang <wangkefeng.wang@huawei.com>, Andrey Konovalov <andreyknvl@gmail.com>, Andrey Ryabinin <ryabinin.a.a@gmail.com>, Alexander Potapenko <glider@google.com>, "kasan-dev@googlegroups.com" <kasan-dev@googlegroups.com> Subject: Re: [mm/slub] 3616799128: BUG_kmalloc-#(Not_tainted):kmalloc_Redzone_overwritten Date: Mon, 1 Aug 2022 15:48:54 +0800 [thread overview] Message-ID: <YueFZm1JHDZOKVw/@feng-skl> (raw) In-Reply-To: <CACT4Y+Y5aTQMuUU3j60KbLrH_DoFWq1e7EEF5Ka0c1F9a3FniA@mail.gmail.com> On Mon, Aug 01, 2022 at 03:26:43PM +0800, Dmitry Vyukov wrote: > On Mon, 1 Aug 2022 at 08:22, Feng Tang <feng.tang@intel.com> wrote: > > > > On Sun, Jul 31, 2022 at 04:16:53PM +0800, Tang, Feng wrote: > > > Hi Oliver, > > > > > > On Sun, Jul 31, 2022 at 02:53:17PM +0800, Sang, Oliver wrote: > > > > > > > > > > > > Greeting, > > > > > > > > FYI, we noticed the following commit (built with gcc-11): > > > > > > > > commit: 3616799128612e04ed919579e2c7b0dccf6bcb00 ("[PATCH v3 3/3] mm/slub: extend redzone check to cover extra allocated kmalloc space than requested") > > > > url: https://github.com/intel-lab-lkp/linux/commits/Feng-Tang/mm-slub-some-debug-enhancements/20220727-151318 > > > > base: git://git.kernel.org/cgit/linux/kernel/git/vbabka/slab.git for-next > > > > patch link: https://lore.kernel.org/linux-mm/20220727071042.8796-4-feng.tang@intel.com > > > > > > > > in testcase: boot > > > > > > > > on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G > > > > > > > > caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): > > > > > > > > > > > > If you fix the issue, kindly add following tag > > > > Reported-by: kernel test robot <oliver.sang@intel.com> > > > > > > > > > > > > [ 50.637839][ T154] ============================================================================= > > > > [ 50.639937][ T154] BUG kmalloc-16 (Not tainted): kmalloc Redzone overwritten > > > > [ 50.641291][ T154] ----------------------------------------------------------------------------- > > > > [ 50.641291][ T154] > > > > [ 50.643617][ T154] 0xffff88810018464c-0xffff88810018464f @offset=1612. First byte 0x7 instead of 0xcc > > > > [ 50.645311][ T154] Allocated in __sdt_alloc+0x258/0x457 age=14287 cpu=0 pid=1 > > > > [ 50.646584][ T154] ___slab_alloc+0x52b/0x5b6 > > > > [ 50.647411][ T154] __slab_alloc+0x1a/0x22 > > > > [ 50.648374][ T154] __kmalloc_node+0x10c/0x1e1 > > > > [ 50.649237][ T154] __sdt_alloc+0x258/0x457 > > > > [ 50.650060][ T154] build_sched_domains+0xae/0x10e8 > > > > [ 50.650981][ T154] sched_init_smp+0x30/0xa5 > > > > [ 50.651805][ T154] kernel_init_freeable+0x1c6/0x23b > > > > [ 50.652767][ T154] kernel_init+0x14/0x127 > > > > [ 50.653594][ T154] ret_from_fork+0x1f/0x30 > > > > [ 50.654414][ T154] Slab 0xffffea0004006100 objects=28 used=28 fp=0x0000000000000000 flags=0x1fffc0000000201(locked|slab|node=0|zone=1|lastcpupid=0x3fff) > > > > [ 50.656866][ T154] Object 0xffff888100184640 @offset=1600 fp=0xffff888100184520 > > > > [ 50.656866][ T154] > > > > [ 50.658410][ T154] Redzone ffff888100184630: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................ > > > > [ 50.660047][ T154] Object ffff888100184640: 00 32 80 00 81 88 ff ff 01 00 00 00 07 00 80 8a .2.............. > > > > [ 50.661837][ T154] Redzone ffff888100184650: cc cc cc cc cc cc cc cc ........ > > > > [ 50.663454][ T154] Padding ffff8881001846b4: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZ > > > > [ 50.665225][ T154] CPU: 0 PID: 154 Comm: systemd-udevd Not tainted 5.19.0-rc5-00010-g361679912861 #1 > > > > [ 50.666861][ T154] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.0-debian-1.16.0-4 04/01/2014 > > > > [ 50.668694][ T154] Call Trace: > > > > [ 50.669331][ T154] <TASK> > > > > [ 50.669832][ T154] dump_stack_lvl+0x57/0x7d > > > > [ 50.670601][ T154] check_bytes_and_report+0xca/0xfe > > > > [ 50.671436][ T154] check_object+0xdc/0x24d > > > > [ 50.672163][ T154] free_debug_processing+0x98/0x210 > > > > [ 50.673904][ T154] __slab_free+0x46/0x198 > > > > [ 50.675746][ T154] qlist_free_all+0xae/0xde > > > > [ 50.676552][ T154] kasan_quarantine_reduce+0x10d/0x145 > > > > [ 50.677507][ T154] __kasan_slab_alloc+0x1c/0x5a > > > > [ 50.678327][ T154] slab_post_alloc_hook+0x5a/0xa2 > > > > [ 50.680069][ T154] kmem_cache_alloc+0x102/0x135 > > > > [ 50.680938][ T154] getname_flags+0x4b/0x314 > > > > [ 50.681781][ T154] do_sys_openat2+0x7a/0x15c > > > > [ 50.706848][ T154] Disabling lock debugging due to kernel taint > > > > [ 50.707913][ T154] FIX kmalloc-16: Restoring kmalloc Redzone 0xffff88810018464c-0xffff88810018464f=0xcc > > > > > > Thanks for the report! > > > > > > From the log it happened when kasan is enabled, and my first guess is > > > the data processing from kmalloc redzone handling had some conflict > > > with kasan's in allocation path (though I tested some kernel config > > > with KASAN enabled) > > > > > > Will study more about kasan and reproduce/debug this. thanks > > > > Cc kansan mail list. > > > > This is really related with KASAN debug, that in free path, some > > kmalloc redzone ([orig_size+1, object_size]) area is written by > > kasan to save free meta info. > > > > The callstack is: > > > > kfree > > slab_free > > slab_free_freelist_hook > > slab_free_hook > > __kasan_slab_free > > ____kasan_slab_free > > kasan_set_free_info > > kasan_set_track > > > > And this issue only happens with "kmalloc-16" slab. Kasan has 2 > > tracks: alloc_track and free_track, for x86_64 test platform, most > > of the slabs will reserve space for alloc_track, and reuse the > > 'object' area for free_track. The kasan free_track is 16 bytes > > large, that it will occupy the whole 'kmalloc-16's object area, > > so when kmalloc-redzone is enabled by this patch, the 'overwritten' > > error is triggered. > > > > But it won't hurt other kmalloc slabs, as kasan's free meta won't > > conflict with kmalloc-redzone which stay in the latter part of > > kmalloc area. > > > > So the solution I can think of is: > > * skip the kmalloc-redzone for kmalloc-16 only, or > > * skip kmalloc-redzone if kasan is enabled, or > > * let kasan reserve the free meta (16 bytes) outside of object > > just like for alloc meta > > /\/\/\ > > Please just not the last option. Memory consumption matters. > > I would do whatever is the simplest, e.g. skip the check for > kmalloc-16 when KASAN is enabled. Thanks for giving the suggestion. I'm fine with all these options, and will also wait for Vlastimil and other developers opinion. > Or does it make sense to prohibit KASAN+SLUB_DEBUG combination? Does > SLUB_DEBUG add anything on top of KASAN? I did a quick glance, seems the KASAN will select SLUB_DEBUG in many cases, as shown in the lib/Kconfig.kasan: config KASAN_GENERIC ... select SLUB_DEBUG if SLUB config KASAN_SW_TAGS ... select SLUB_DEBUG if SLUB Thanks, Feng > > > I don't have way to test kasan's SW/HW tag configuration, which > > is only enabled on arm64 now. And I don't know if there will > > also be some conflict. > > > > Thanks, > > Feng > > >
WARNING: multiple messages have this Message-ID (diff)
From: Feng Tang <feng.tang@intel.com> To: lkp@lists.01.org Subject: Re: [mm/slub] 3616799128: BUG_kmalloc-#(Not_tainted):kmalloc_Redzone_overwritten Date: Mon, 01 Aug 2022 15:48:54 +0800 [thread overview] Message-ID: <YueFZm1JHDZOKVw/@feng-skl> (raw) In-Reply-To: <CACT4Y+Y5aTQMuUU3j60KbLrH_DoFWq1e7EEF5Ka0c1F9a3FniA@mail.gmail.com> [-- Attachment #1: Type: text/plain, Size: 7021 bytes --] On Mon, Aug 01, 2022 at 03:26:43PM +0800, Dmitry Vyukov wrote: > On Mon, 1 Aug 2022 at 08:22, Feng Tang <feng.tang@intel.com> wrote: > > > > On Sun, Jul 31, 2022 at 04:16:53PM +0800, Tang, Feng wrote: > > > Hi Oliver, > > > > > > On Sun, Jul 31, 2022 at 02:53:17PM +0800, Sang, Oliver wrote: > > > > > > > > > > > > Greeting, > > > > > > > > FYI, we noticed the following commit (built with gcc-11): > > > > > > > > commit: 3616799128612e04ed919579e2c7b0dccf6bcb00 ("[PATCH v3 3/3] mm/slub: extend redzone check to cover extra allocated kmalloc space than requested") > > > > url: https://github.com/intel-lab-lkp/linux/commits/Feng-Tang/mm-slub-some-debug-enhancements/20220727-151318 > > > > base: git://git.kernel.org/cgit/linux/kernel/git/vbabka/slab.git for-next > > > > patch link: https://lore.kernel.org/linux-mm/20220727071042.8796-4-feng.tang(a)intel.com > > > > > > > > in testcase: boot > > > > > > > > on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G > > > > > > > > caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): > > > > > > > > > > > > If you fix the issue, kindly add following tag > > > > Reported-by: kernel test robot <oliver.sang@intel.com> > > > > > > > > > > > > [ 50.637839][ T154] ============================================================================= > > > > [ 50.639937][ T154] BUG kmalloc-16 (Not tainted): kmalloc Redzone overwritten > > > > [ 50.641291][ T154] ----------------------------------------------------------------------------- > > > > [ 50.641291][ T154] > > > > [ 50.643617][ T154] 0xffff88810018464c-0xffff88810018464f @offset=1612. First byte 0x7 instead of 0xcc > > > > [ 50.645311][ T154] Allocated in __sdt_alloc+0x258/0x457 age=14287 cpu=0 pid=1 > > > > [ 50.646584][ T154] ___slab_alloc+0x52b/0x5b6 > > > > [ 50.647411][ T154] __slab_alloc+0x1a/0x22 > > > > [ 50.648374][ T154] __kmalloc_node+0x10c/0x1e1 > > > > [ 50.649237][ T154] __sdt_alloc+0x258/0x457 > > > > [ 50.650060][ T154] build_sched_domains+0xae/0x10e8 > > > > [ 50.650981][ T154] sched_init_smp+0x30/0xa5 > > > > [ 50.651805][ T154] kernel_init_freeable+0x1c6/0x23b > > > > [ 50.652767][ T154] kernel_init+0x14/0x127 > > > > [ 50.653594][ T154] ret_from_fork+0x1f/0x30 > > > > [ 50.654414][ T154] Slab 0xffffea0004006100 objects=28 used=28 fp=0x0000000000000000 flags=0x1fffc0000000201(locked|slab|node=0|zone=1|lastcpupid=0x3fff) > > > > [ 50.656866][ T154] Object 0xffff888100184640 @offset=1600 fp=0xffff888100184520 > > > > [ 50.656866][ T154] > > > > [ 50.658410][ T154] Redzone ffff888100184630: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................ > > > > [ 50.660047][ T154] Object ffff888100184640: 00 32 80 00 81 88 ff ff 01 00 00 00 07 00 80 8a .2.............. > > > > [ 50.661837][ T154] Redzone ffff888100184650: cc cc cc cc cc cc cc cc ........ > > > > [ 50.663454][ T154] Padding ffff8881001846b4: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZ > > > > [ 50.665225][ T154] CPU: 0 PID: 154 Comm: systemd-udevd Not tainted 5.19.0-rc5-00010-g361679912861 #1 > > > > [ 50.666861][ T154] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.0-debian-1.16.0-4 04/01/2014 > > > > [ 50.668694][ T154] Call Trace: > > > > [ 50.669331][ T154] <TASK> > > > > [ 50.669832][ T154] dump_stack_lvl+0x57/0x7d > > > > [ 50.670601][ T154] check_bytes_and_report+0xca/0xfe > > > > [ 50.671436][ T154] check_object+0xdc/0x24d > > > > [ 50.672163][ T154] free_debug_processing+0x98/0x210 > > > > [ 50.673904][ T154] __slab_free+0x46/0x198 > > > > [ 50.675746][ T154] qlist_free_all+0xae/0xde > > > > [ 50.676552][ T154] kasan_quarantine_reduce+0x10d/0x145 > > > > [ 50.677507][ T154] __kasan_slab_alloc+0x1c/0x5a > > > > [ 50.678327][ T154] slab_post_alloc_hook+0x5a/0xa2 > > > > [ 50.680069][ T154] kmem_cache_alloc+0x102/0x135 > > > > [ 50.680938][ T154] getname_flags+0x4b/0x314 > > > > [ 50.681781][ T154] do_sys_openat2+0x7a/0x15c > > > > [ 50.706848][ T154] Disabling lock debugging due to kernel taint > > > > [ 50.707913][ T154] FIX kmalloc-16: Restoring kmalloc Redzone 0xffff88810018464c-0xffff88810018464f=0xcc > > > > > > Thanks for the report! > > > > > > From the log it happened when kasan is enabled, and my first guess is > > > the data processing from kmalloc redzone handling had some conflict > > > with kasan's in allocation path (though I tested some kernel config > > > with KASAN enabled) > > > > > > Will study more about kasan and reproduce/debug this. thanks > > > > Cc kansan mail list. > > > > This is really related with KASAN debug, that in free path, some > > kmalloc redzone ([orig_size+1, object_size]) area is written by > > kasan to save free meta info. > > > > The callstack is: > > > > kfree > > slab_free > > slab_free_freelist_hook > > slab_free_hook > > __kasan_slab_free > > ____kasan_slab_free > > kasan_set_free_info > > kasan_set_track > > > > And this issue only happens with "kmalloc-16" slab. Kasan has 2 > > tracks: alloc_track and free_track, for x86_64 test platform, most > > of the slabs will reserve space for alloc_track, and reuse the > > 'object' area for free_track. The kasan free_track is 16 bytes > > large, that it will occupy the whole 'kmalloc-16's object area, > > so when kmalloc-redzone is enabled by this patch, the 'overwritten' > > error is triggered. > > > > But it won't hurt other kmalloc slabs, as kasan's free meta won't > > conflict with kmalloc-redzone which stay in the latter part of > > kmalloc area. > > > > So the solution I can think of is: > > * skip the kmalloc-redzone for kmalloc-16 only, or > > * skip kmalloc-redzone if kasan is enabled, or > > * let kasan reserve the free meta (16 bytes) outside of object > > just like for alloc meta > > /\/\/\ > > Please just not the last option. Memory consumption matters. > > I would do whatever is the simplest, e.g. skip the check for > kmalloc-16 when KASAN is enabled. Thanks for giving the suggestion. I'm fine with all these options, and will also wait for Vlastimil and other developers opinion. > Or does it make sense to prohibit KASAN+SLUB_DEBUG combination? Does > SLUB_DEBUG add anything on top of KASAN? I did a quick glance, seems the KASAN will select SLUB_DEBUG in many cases, as shown in the lib/Kconfig.kasan: config KASAN_GENERIC ... select SLUB_DEBUG if SLUB config KASAN_SW_TAGS ... select SLUB_DEBUG if SLUB Thanks, Feng > > > I don't have way to test kasan's SW/HW tag configuration, which > > is only enabled on arm64 now. And I don't know if there will > > also be some conflict. > > > > Thanks, > > Feng > > >
next prev parent reply other threads:[~2022-08-01 7:49 UTC|newest] Thread overview: 55+ messages / expand[flat|nested] mbox.gz Atom feed top 2022-07-27 7:10 [PATCH v3 0/3] mm/slub: some debug enhancements Feng Tang 2022-07-27 7:10 ` [PATCH v3 1/3] mm/slub: enable debugging memory wasting of kmalloc Feng Tang 2022-07-27 10:20 ` Christoph Lameter 2022-07-27 12:59 ` Feng Tang 2022-07-27 14:12 ` Vlastimil Babka 2022-07-27 7:10 ` [PATCH v3 2/3] mm/slub: only zero the requested size of buffer for kzalloc Feng Tang 2022-07-27 7:10 ` [PATCH v3 3/3] mm/slub: extend redzone check to cover extra allocated kmalloc space than requested Feng Tang 2022-07-31 6:53 ` [mm/slub] 3616799128: BUG_kmalloc-#(Not_tainted):kmalloc_Redzone_overwritten kernel test robot 2022-07-31 6:53 ` kernel test robot 2022-07-31 8:16 ` Feng Tang 2022-07-31 8:16 ` Feng Tang 2022-08-01 6:21 ` Feng Tang 2022-08-01 6:21 ` Feng Tang 2022-08-01 7:26 ` Dmitry Vyukov 2022-08-01 7:26 ` Dmitry Vyukov 2022-08-01 7:48 ` Feng Tang [this message] 2022-08-01 7:48 ` Feng Tang 2022-08-01 8:13 ` Christoph Lameter 2022-08-01 8:13 ` Christoph Lameter 2022-08-01 14:23 ` Vlastimil Babka 2022-08-01 14:23 ` Vlastimil Babka 2022-08-02 6:54 ` Feng Tang 2022-08-02 6:54 ` Feng Tang 2022-08-02 7:06 ` Dmitry Vyukov 2022-08-02 7:06 ` Dmitry Vyukov 2022-08-02 7:46 ` Feng Tang 2022-08-02 7:46 ` Feng Tang 2022-08-02 7:59 ` Dmitry Vyukov 2022-08-02 7:59 ` Dmitry Vyukov 2022-08-02 8:44 ` Feng Tang 2022-08-02 8:44 ` Feng Tang 2022-08-02 9:43 ` Vlastimil Babka 2022-08-02 9:43 ` Vlastimil Babka 2022-08-02 10:30 ` Dmitry Vyukov 2022-08-02 10:30 ` Dmitry Vyukov 2022-08-02 13:36 ` Feng Tang 2022-08-02 13:36 ` Feng Tang 2022-08-02 14:38 ` Dmitry Vyukov 2022-08-02 14:38 ` Dmitry Vyukov 2022-08-04 6:28 ` Feng Tang 2022-08-04 6:28 ` Feng Tang 2022-08-04 10:47 ` Dmitry Vyukov 2022-08-04 10:47 ` Dmitry Vyukov 2022-08-04 12:22 ` Feng Tang 2022-08-04 12:22 ` Feng Tang 2022-08-15 7:27 ` Feng Tang 2022-08-15 7:27 ` Feng Tang 2022-08-16 13:27 ` Oliver Sang 2022-08-16 13:27 ` Oliver Sang 2022-08-16 14:12 ` Feng Tang 2022-08-16 14:12 ` Feng Tang 2022-08-02 10:31 ` Dmitry Vyukov 2022-08-02 10:31 ` Dmitry Vyukov 2022-08-02 6:59 ` Dmitry Vyukov 2022-08-02 6:59 ` Dmitry Vyukov
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=YueFZm1JHDZOKVw/@feng-skl \ --to=feng.tang@intel.com \ --cc=42.hyeyoo@gmail.com \ --cc=akpm@linux-foundation.org \ --cc=andreyknvl@gmail.com \ --cc=cl@linux.com \ --cc=dave.hansen@intel.com \ --cc=dvyukov@google.com \ --cc=glider@google.com \ --cc=iamjoonsoo.kim@lge.com \ --cc=john.garry@huawei.com \ --cc=kasan-dev@googlegroups.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=lkp@intel.com \ --cc=lkp@lists.01.org \ --cc=oliver.sang@intel.com \ --cc=penberg@kernel.org \ --cc=rientjes@google.com \ --cc=robin.murphy@arm.com \ --cc=roman.gushchin@linux.dev \ --cc=ryabinin.a.a@gmail.com \ --cc=vbabka@suse.cz \ --cc=wangkefeng.wang@huawei.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.