CONFIG_KASAN_SW_TAGS=y NULL pointer dereference at freelist_dereference()

CONFIG_KASAN_SW_TAGS=y NULL pointer dereference at freelist_dereference()

[Qian Cai]

The kernel was compiled by clang-7.0.1 on a ThunderX2 server, and it fails to
boot. CONFIG_KASAN_GENERIC=y works fine.

deactivate_slab+0x84/0x6ac:
freelist_dereference at mm/slub.c:262
(inlined by) get_freepointer at mm/slub.c:268
(inlined by) deactivate_slab at mm/slub.c:2056

/* Returns the freelist pointer recorded at location ptr_addr. */
static inline void *freelist_dereference(const struct kmem_cache *s,
                                         void *ptr_addr)
{
        return freelist_ptr(s, (void *)*(unsigned long *)(ptr_addr),
                            (unsigned long)ptr_addr);
}

[    0.000000] Memory: 3259968K/100594752K available (15548K kernel code, 12360K
rwdata, 4096K rodata, 25536K init, 27244K bss, 7444672K reserved, 0K cma-reserved)
[    0.000000] Unable to handle kernel NULL pointer dereference at virtual
address 0000000000000078
[    0.000000] Mem abort info:
[    0.000000]   ESR = 0x96000005
[    0.000000]   Exception class = DABT (current EL), IL = 32 bits
[    0.000000]   SET = 0, FnV = 0
[    0.000000]   EA = 0, S1PTW = 0
[    0.000000] Data abort info:
[    0.000000]   ISV = 0, ISS = 0x00000005
[    0.000000]   CM = 0, WnR = 0
[    0.000000] [0000000000000078] user address but active_mm is swapper
[    0.000000] Internal error: Oops: 96000005 [#1] SMP
[    0.000000] Modules linked in:
[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.0.0-rc5+ #6
[    0.000000] pstate: 60000089 (nZCv daIf -PAN -UAO)
[    0.000000] pc : deactivate_slab+0x84/0x6ac
[    0.000000] lr : deactivate_slab+0x1cc/0x6ac
[    0.000000] sp : ffff100012cf7be0
[    0.000000] x29: ffff100012cf7cc0 x28: ffff1000114e4f00
[    0.000000] x27: ffff1000114e4f20 x26: ffff1000114e4f08
[    0.000000] x25: ffff1000114e5078 x24: fb00000000000000
[    0.000000] x23: ffff7fe002080008 x22: ffff808abb5b72d0
[    0.000000] x21: ffff7fe002080020 x20: ffff7fe002080028
[    0.000000] x19: ffff7fe002080000 x18: ffff1000148a5538
[    0.000000] x17: 000000000000001b x16: 0000000000000000
[    0.000000] x15: 007ffffffc000201 x14: 04ff80082000fa80
[    0.000000] x13: 0000000080660002 x12: 0000000080660003
[    0.000000] x11: 4582a03bdc147ab9 x10: ffff100012d31c90
[    0.000000] x9 : fb00000000000078 x8 : ffff100012d31c80
[    0.000000] x7 : cccccccccccccccc x6 : ffff1000105d8db8
[    0.000000] x5 : 0000000000000000 x4 : 0000000000000000
[    0.000000] x3 : ffff808abb5b72d0 x2 : 04ff800820000580
[    0.000000] x1 : ffff7fe002080000 x0 : ffff1000114e4f00
[    0.000000] Process swapper (pid: 0, stack limit = 0x(____ptrval____))
[    0.000000] Call trace:
[    0.000000]  deactivate_slab+0x84/0x6ac
[    0.000000]  ___slab_alloc+0x648/0x6fc
[    0.000000]  kmem_cache_alloc_node+0x408/0x538
[    0.000000]  __kmem_cache_create+0x20c/0x6a8
[    0.000000]  create_boot_cache+0x68/0xac
[    0.000000]  kmem_cache_init+0xb0/0x19c
[    0.000000]  start_kernel+0x4b4/0xac4
[    0.000000] Code: 14000057 b9400369 f940032b 8b090309 (f940012a)
[    0.000000] ---[ end trace 54ad7e55e4749a96 ]---
[    0.000000] Kernel panic - not syncing: Fatal exception
[    0.000000] ---[ end Kernel panic - not syncing: Fatal exception ]---

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: CONFIG_KASAN_SW_TAGS=y NULL pointer dereference at freelist_dereference()

[Andrey Konovalov]

On Thu, Feb 7, 2019 at 5:04 AM Qian Cai <cai@lca.pw> wrote:
>
> The kernel was compiled by clang-7.0.1 on a ThunderX2 server, and it fails to
> boot. CONFIG_KASAN_GENERIC=y works fine.

Hi Qian,

Could you share the kernel commit id and .config that you use?

Thanks!

>
> deactivate_slab+0x84/0x6ac:
> freelist_dereference at mm/slub.c:262
> (inlined by) get_freepointer at mm/slub.c:268
> (inlined by) deactivate_slab at mm/slub.c:2056
>
> /* Returns the freelist pointer recorded at location ptr_addr. */
> static inline void *freelist_dereference(const struct kmem_cache *s,
>                                          void *ptr_addr)
> {
>         return freelist_ptr(s, (void *)*(unsigned long *)(ptr_addr),
>                             (unsigned long)ptr_addr);
> }
>
> [    0.000000] Memory: 3259968K/100594752K available (15548K kernel code, 12360K
> rwdata, 4096K rodata, 25536K init, 27244K bss, 7444672K reserved, 0K cma-reserved)
> [    0.000000] Unable to handle kernel NULL pointer dereference at virtual
> address 0000000000000078
> [    0.000000] Mem abort info:
> [    0.000000]   ESR = 0x96000005
> [    0.000000]   Exception class = DABT (current EL), IL = 32 bits
> [    0.000000]   SET = 0, FnV = 0
> [    0.000000]   EA = 0, S1PTW = 0
> [    0.000000] Data abort info:
> [    0.000000]   ISV = 0, ISS = 0x00000005
> [    0.000000]   CM = 0, WnR = 0
> [    0.000000] [0000000000000078] user address but active_mm is swapper
> [    0.000000] Internal error: Oops: 96000005 [#1] SMP
> [    0.000000] Modules linked in:
> [    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.0.0-rc5+ #6
> [    0.000000] pstate: 60000089 (nZCv daIf -PAN -UAO)
> [    0.000000] pc : deactivate_slab+0x84/0x6ac
> [    0.000000] lr : deactivate_slab+0x1cc/0x6ac
> [    0.000000] sp : ffff100012cf7be0
> [    0.000000] x29: ffff100012cf7cc0 x28: ffff1000114e4f00
> [    0.000000] x27: ffff1000114e4f20 x26: ffff1000114e4f08
> [    0.000000] x25: ffff1000114e5078 x24: fb00000000000000
> [    0.000000] x23: ffff7fe002080008 x22: ffff808abb5b72d0
> [    0.000000] x21: ffff7fe002080020 x20: ffff7fe002080028
> [    0.000000] x19: ffff7fe002080000 x18: ffff1000148a5538
> [    0.000000] x17: 000000000000001b x16: 0000000000000000
> [    0.000000] x15: 007ffffffc000201 x14: 04ff80082000fa80
> [    0.000000] x13: 0000000080660002 x12: 0000000080660003
> [    0.000000] x11: 4582a03bdc147ab9 x10: ffff100012d31c90
> [    0.000000] x9 : fb00000000000078 x8 : ffff100012d31c80
> [    0.000000] x7 : cccccccccccccccc x6 : ffff1000105d8db8
> [    0.000000] x5 : 0000000000000000 x4 : 0000000000000000
> [    0.000000] x3 : ffff808abb5b72d0 x2 : 04ff800820000580
> [    0.000000] x1 : ffff7fe002080000 x0 : ffff1000114e4f00
> [    0.000000] Process swapper (pid: 0, stack limit = 0x(____ptrval____))
> [    0.000000] Call trace:
> [    0.000000]  deactivate_slab+0x84/0x6ac
> [    0.000000]  ___slab_alloc+0x648/0x6fc
> [    0.000000]  kmem_cache_alloc_node+0x408/0x538
> [    0.000000]  __kmem_cache_create+0x20c/0x6a8
> [    0.000000]  create_boot_cache+0x68/0xac
> [    0.000000]  kmem_cache_init+0xb0/0x19c
> [    0.000000]  start_kernel+0x4b4/0xac4
> [    0.000000] Code: 14000057 b9400369 f940032b 8b090309 (f940012a)
> [    0.000000] ---[ end trace 54ad7e55e4749a96 ]---
> [    0.000000] Kernel panic - not syncing: Fatal exception
> [    0.000000] ---[ end Kernel panic - not syncing: Fatal exception ]---
>
> --
> You received this message because you are subscribed to the Google Groups "kasan-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kasan-dev+unsubscribe@googlegroups.com.
> To post to this group, send email to kasan-dev@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kasan-dev/b1d210ae-3fc9-c77a-4010-40fb74a61727%40lca.pw.
> For more options, visit https://groups.google.com/d/optout.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: CONFIG_KASAN_SW_TAGS=y NULL pointer dereference at freelist_dereference()

[Qian Cai]

On 2/7/19 7:58 AM, Andrey Konovalov wrote:
> On Thu, Feb 7, 2019 at 5:04 AM Qian Cai <cai@lca.pw> wrote:
>>
>> The kernel was compiled by clang-7.0.1 on a ThunderX2 server, and it fails to
>> boot. CONFIG_KASAN_GENERIC=y works fine.
> 
> Hi Qian,
> 
> Could you share the kernel commit id and .config that you use?

v5.0-rc5

https://git.sr.ht/~cai/linux-debug/tree/master/config

# cat /proc/cmdline
page_poison=on crashkernel=768M earlycon page_owner=on numa_balancing=enable
slub_debug=-

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: CONFIG_KASAN_SW_TAGS=y NULL pointer dereference at freelist_dereference()

[Andrey Konovalov]

[-- Attachment #1: Type: text/plain, Size: 691 bytes --]

On Thu, Feb 7, 2019 at 2:27 PM Qian Cai <cai@lca.pw> wrote:
>
>
>
> On 2/7/19 7:58 AM, Andrey Konovalov wrote:
> > On Thu, Feb 7, 2019 at 5:04 AM Qian Cai <cai@lca.pw> wrote:
> >>
> >> The kernel was compiled by clang-7.0.1 on a ThunderX2 server, and it fails to
> >> boot. CONFIG_KASAN_GENERIC=y works fine.
> >
> > Hi Qian,
> >
> > Could you share the kernel commit id and .config that you use?
>
> v5.0-rc5
>
> https://git.sr.ht/~cai/linux-debug/tree/master/config
>
> # cat /proc/cmdline
> page_poison=on crashkernel=768M earlycon page_owner=on numa_balancing=enable
> slub_debug=-

Reproduced, looks like a conflict with CONFIG_SLAB_FREELIST_HARDENED.
Could you try the attached patch?

[-- Attachment #2: kasan-hardened-freelist-fix.patch --]
[-- Type: text/x-patch, Size: 467 bytes --]

diff --git a/mm/slub.c b/mm/slub.c
index 1e3d0ec4e200..5fb7507eb8d1 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -249,7 +249,8 @@ static inline void *freelist_ptr(const struct kmem_cache *s, void *ptr,
 				 unsigned long ptr_addr)
 {
 #ifdef CONFIG_SLAB_FREELIST_HARDENED
-	return (void *)((unsigned long)ptr ^ s->random ^ ptr_addr);
+	return (void *)((unsigned long)ptr ^ s->random ^
+			(unsigned long)kasan_reset_tag((void *)ptr_addr));
 #else
 	return ptr;
 #endif

[-- Attachment #3: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: CONFIG_KASAN_SW_TAGS=y NULL pointer dereference at freelist_dereference()

[Qian Cai]

On 2/7/19 10:34 AM, Andrey Konovalov wrote:
> Reproduced, looks like a conflict with CONFIG_SLAB_FREELIST_HARDENED.
> Could you try the attached patch?
> 

You nailed it!

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: CONFIG_KASAN_SW_TAGS=y not play well with kmemleak

[Andrey Konovalov]

[-- Attachment #1: Type: text/plain, Size: 2904 bytes --]

On Fri, Feb 8, 2019 at 5:16 AM Qian Cai <cai@lca.pw> wrote:
>
> Kmemleak is totally busted with CONFIG_KASAN_SW_TAGS=y because most of tracking
> object pointers passed to create_object() have the upper bits set by KASAN.

Hi Qian,

Yeah, the issue is that kmemleak performs a bunch of pointer
comparisons that break when pointers are tagged. Try the attached
patch, it should fix the issue. I don't like the way this patch does
it though, I'll see if I can come up with something better.

Thanks for the report!

> However, even after applied this patch [1] to fix a few things, it still has
> many errors during boot.
>
> https://git.sr.ht/~cai/linux-debug/tree/master/dmesg
>
> What I don't understand is that even the patch did call kasan_reset_tag() in
> paint_ptr(), it still complained on objects with upper bits set which indicates
> that this line did not run.
>
> return (__s64)(value << shift) >> shift;
>
> [   42.462799] kmemleak: Trying to color unknown object at 0xffff80082df80000 as
> Grey
> [   42.470524] CPU: 128 PID: 1 Comm: swapper/0 Not tainted 5.0.0-rc5+ #17
> [   42.477153] Call trace:
> [   42.479639]  dump_backtrace+0x0/0x450
> [   42.483362]  show_stack+0x20/0x2c
> [   42.486733]  __dump_stack+0x20/0x28
> [   42.490276]  dump_stack+0xa0/0xfc
> [   42.493649]  paint_ptr+0xa8/0xf4
> [   42.496934]  kmemleak_not_leak+0xa4/0x15c
> [   42.501013]  init_section_page_ext+0x1bc/0x328
> [   42.505528]  page_ext_init+0x4dc/0x75c
> [   42.509336]  kernel_init_freeable+0x684/0x1104
> [   42.513857]  kernel_init+0x18/0x2a4
> [   42.517407]  ret_from_fork+0x10/0x18
>
> [1]
> diff --git a/mm/kmemleak.c b/mm/kmemleak.c
> index f9d9dc250428..70343d887f34 100644
> --- a/mm/kmemleak.c
> +++ b/mm/kmemleak.c
> @@ -588,7 +588,7 @@ static struct kmemleak_object *create_object(unsigned long
> ptr, size_t size,
>         spin_lock_init(&object->lock);
>         atomic_set(&object->use_count, 1);
>         object->flags = OBJECT_ALLOCATED;
> -       object->pointer = ptr;
> +       object->pointer = (unsigned long)kasan_reset_tag((void *)ptr);
>         object->size = size;
>         object->excess_ref = 0;
>         object->min_count = min_count;
> @@ -748,11 +748,12 @@ static void paint_it(struct kmemleak_object *object, int
> color)
>  static void paint_ptr(unsigned long ptr, int color)
>  {
>         struct kmemleak_object *object;
> +       unsigned long addr = (unsigned long)kasan_reset_tag((void *)ptr);
>
> -       object = find_and_get_object(ptr, 0);
> +       object = find_and_get_object(addr, 0);
>         if (!object) {
>                 kmemleak_warn("Trying to color unknown object at 0x%08lx as %s\n",
> -                             ptr,
> +                             addr,
>                               (color == KMEMLEAK_GREY) ? "Grey" :
>                               (color == KMEMLEAK_BLACK) ? "Black" : "Unknown");
>                 return;
>
>

[-- Attachment #2: kasan-kmemleak-fix.patch --]
[-- Type: text/x-patch, Size: 2079 bytes --]

diff --git a/mm/kmemleak.c b/mm/kmemleak.c
index f9d9dc250428..5354e74f0d19 100644
--- a/mm/kmemleak.c
+++ b/mm/kmemleak.c
@@ -437,6 +437,8 @@ static struct kmemleak_object *lookup_object(unsigned long ptr, int alias)
 {
 	struct rb_node *rb = object_tree_root.rb_node;
 
+	ptr = (unsigned long)kasan_reset_tag((void *)ptr);
+
 	while (rb) {
 		struct kmemleak_object *object =
 			rb_entry(rb, struct kmemleak_object, rb_node);
@@ -575,6 +577,8 @@ static struct kmemleak_object *create_object(unsigned long ptr, size_t size,
 	struct kmemleak_object *object, *parent;
 	struct rb_node **link, *rb_parent;
 
+	ptr = (unsigned long)kasan_reset_tag((void *)ptr);
+
 	object = kmem_cache_alloc(object_cache, gfp_kmemleak_mask(gfp));
 	if (!object) {
 		pr_warn("Cannot allocate a kmemleak_object structure\n");
@@ -701,6 +705,8 @@ static void delete_object_part(unsigned long ptr, size_t size)
 	struct kmemleak_object *object;
 	unsigned long start, end;
 
+	ptr = (unsigned long)kasan_reset_tag((void *)ptr);
+
 	object = find_and_remove_object(ptr, 1);
 	if (!object) {
 #ifdef DEBUG
@@ -789,6 +795,8 @@ static void add_scan_area(unsigned long ptr, size_t size, gfp_t gfp)
 	struct kmemleak_object *object;
 	struct kmemleak_scan_area *area;
 
+	ptr = (unsigned long)kasan_reset_tag((void *)ptr);
+
 	object = find_and_get_object(ptr, 1);
 	if (!object) {
 		kmemleak_warn("Adding scan area to unknown object at 0x%08lx\n",
@@ -1334,6 +1342,9 @@ static void scan_block(void *_start, void *_end,
 	unsigned long *end = _end - (BYTES_PER_POINTER - 1);
 	unsigned long flags;
 
+	start = (unsigned long *)kasan_reset_tag((void *)start);
+	end = (unsigned long *)kasan_reset_tag((void *)end);
+
 	read_lock_irqsave(&kmemleak_lock, flags);
 	for (ptr = start; ptr < end; ptr++) {
 		struct kmemleak_object *object;
@@ -1344,7 +1355,7 @@ static void scan_block(void *_start, void *_end,
 			break;
 
 		kasan_disable_current();
-		pointer = *ptr;
+		pointer = (unsigned long)kasan_reset_tag((void *)*ptr);
 		kasan_enable_current();
 
 		if (pointer < min_addr || pointer >= max_addr)

[-- Attachment #3: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: CONFIG_KASAN_SW_TAGS=y not play well with kmemleak

[Qian Cai]

On 2/8/19 12:15 PM, Andrey Konovalov wrote:
> Yeah, the issue is that kmemleak performs a bunch of pointer
> comparisons that break when pointers are tagged. Try the attached
> patch, it should fix the issue. I don't like the way this patch does
> it though, I'll see if I can come up with something better.

The patch works great!

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: CONFIG_KASAN_SW_TAGS=y not play well with kmemleak

[Catalin Marinas]

On Fri, Feb 08, 2019 at 06:15:02PM +0100, Andrey Konovalov wrote:
> On Fri, Feb 8, 2019 at 5:16 AM Qian Cai <cai@lca.pw> wrote:
> > Kmemleak is totally busted with CONFIG_KASAN_SW_TAGS=y because most of tracking
> > object pointers passed to create_object() have the upper bits set by KASAN.
> 
> Yeah, the issue is that kmemleak performs a bunch of pointer
> comparisons that break when pointers are tagged.

Does it mean that the kmemleak API receives pointer aliases (i.e. same
object tagged with different values or tagged/untagged)?

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: CONFIG_KASAN_SW_TAGS=y not play well with kmemleak

[Andrey Konovalov]

On Mon, Feb 11, 2019 at 1:16 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Fri, Feb 08, 2019 at 06:15:02PM +0100, Andrey Konovalov wrote:
> > On Fri, Feb 8, 2019 at 5:16 AM Qian Cai <cai@lca.pw> wrote:
> > > Kmemleak is totally busted with CONFIG_KASAN_SW_TAGS=y because most of tracking
> > > object pointers passed to create_object() have the upper bits set by KASAN.
> >
> > Yeah, the issue is that kmemleak performs a bunch of pointer
> > comparisons that break when pointers are tagged.
>
> Does it mean that the kmemleak API receives pointer aliases (i.e. same
> object tagged with different values or tagged/untagged)?

Please disregard that comment. This is a bug in KASAN, will send a fix soon.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel