> > [ 75.793150] RIP: 0010:rb_insert_color+0x189/0x1480
>
> What's in that line? Try,
>
> $ ./scripts/faddr2line vmlinux rb_insert_color+0x189/0x1480

rb_insert_color+0x189/0x1480:
__rb_insert at /home/files/git/linux/lib/rbtree.c:131
(inlined by) rb_insert_color at /home/files/git/linux/lib/rbtree.c:452

>
> What's steps to reproduce this?

The steps is the kernel config provided (proc.config) and I double checked the attached C code from the qemu image (attached here).  If the kernel does not immediately crash, a ^C will cause the fault to be noticed.  The report from earlier is the report from the same code, my assumption was that the possible pool/redzone corruption is making it a bit tricky to pin down.

If you would like alternative kernel settings please let me know, I can do that, also, my current test-bench has about 256 core's on x64, 64 of them are bare metal and 32 are arm64.  Any possible preferred configuration tweaks I'm all ears, I'll be including some of these steps you suggested to me in any/additional upcoming threads (Thank you for that so far and future suggestions).

Also, there is some occasionally varying stacks depending on the corruption, so this stack just now (another execution of test3.c);

./scripts/faddr2line vmlinux rcu_process_callbacks+0xd45/0x1650
rcu_process_callbacks+0xd45/0x1650:
rcu_lock_release at include/linux/rcupdate.h:228
(inlined by) __rcu_reclaim at kernel/rcu/rcu.h:234
(inlined by) rcu_do_batch at kernel/rcu/tree.c:2452
(inlined by) invoke_rcu_callbacks at kernel/rcu/tree.c:2773
(inlined by) rcu_process_callbacks at kernel/rcu/tree.c:2754

(stack from just now)


[12580.358392] ==================================================================
[12580.360076] BUG: KASAN: double-free or invalid-free in rcu_process_callbacks+0xd45/0x1650
[12580.361738]
[12580.362068] CPU: 1 PID: 16 Comm: ksoftirqd/1 Not tainted 5.0.0-rc1+ #5
[12580.363383] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.11.1-1ubuntu1 04/01/2014
[12580.365223] Call Trace:
[12580.365772]  dump_stack+0x1d3/0x2c2
[12580.366518]  ? dump_stack_print_info.cold.1+0x20/0x20
[12580.367608]  ? printk+0xad/0xd3
[12580.368278]  ? kmsg_dump_rewind_nolock+0xf0/0xf0
[12580.369261]  print_address_description.cold.5+0x9/0x208
[12580.370393]  ? rcu_process_callbacks+0xd45/0x1650
[12580.371376]  kasan_report_invalid_free+0x64/0xa0
[12580.372356]  ? rcu_process_callbacks+0xd45/0x1650
[12580.373358]  __kasan_slab_free+0x138/0x150
[12580.374196]  ? rcu_process_callbacks+0xd45/0x1650
[12580.375142]  kasan_slab_free+0xe/0x10
[12580.375905]  kfree+0xcf/0x220
[12580.376537]  rcu_process_callbacks+0xd45/0x1650
[12580.377464]  ? rcu_process_callbacks+0xcf8/0x1650
[12580.378431]  ? rcu_fwd_progress_check+0xf0/0xf0
[12580.379371]  ? compat_start_thread+0x80/0x80
[12580.380292]  ? kasan_check_write+0x14/0x20
[12580.381145]  ? finish_task_switch+0x2cb/0x880
[12580.382028]  ? finish_task_switch+0x189/0x880
[12580.382920]  ? preempt_notifier_register+0x210/0x210
[12580.383944]  ? lock_repin_lock+0x450/0x450
[12580.384808]  ? __do_softirq+0x27d/0xb6a
[12580.385618]  ? kasan_check_read+0x11/0x20
[12580.386461]  ? rcu_is_watching+0x9d/0x160
[12580.387341]  ? trace_hardirqs_on+0xce/0x310
[12580.388217]  ? rcu_pm_notify+0xd0/0xd0
[12580.389008]  __do_softirq+0x2eb/0xb6a
[12580.389816]  ? __irqentry_text_end+0x1f9d5b/0x1f9d5b
[12580.390838]  ? trace_hardirqs_off+0xc6/0x310
[12580.391729]  ? smpboot_thread_fn+0x419/0x900
[12580.392611]  ? trace_hardirqs_on+0x310/0x310
[12580.393503]  ? check_same_owner+0x350/0x350
[12580.394368]  ? takeover_tasklets+0xaa0/0xaa0
[12580.395268]  ? takeover_tasklets+0xaa0/0xaa0
[12580.396153]  run_ksoftirqd+0x8b/0x110
[12580.396922]  smpboot_thread_fn+0x419/0x900
[12580.397785]  ? sort_range+0x40/0x40
[12580.398513]  ? __sanitizer_cov_trace_const_cmp8+0x18/0x20
[12580.399697]  ? __kthread_parkme+0x106/0x1c0
[12580.400563]  ? sort_range+0x40/0x40
[12580.401270]  kthread+0x358/0x460
[12580.401956]  ? kthread_bind+0x40/0x40
[12580.402741]  ret_from_fork+0x24/0x30
[12580.403504]
[12580.403844] Allocated by task 0:
[12580.404524] (stack is not available)
[12580.405276]
[12580.405620] Freed by task 0:
[12580.406223] (stack is not available)
[12580.406955]
[12580.407273] The buggy address belongs to the object at ffff88805b13e4f8
[12580.407273]  which belongs to the cache kmemleak_object of size 360
[12580.409867] The buggy address is located 120 bytes inside of
[12580.409867]  360-byte region [ffff88805b13e4f8, ffff88805b13e660)
[12580.412182] The buggy address belongs to the page:
[12580.413163] page:ffffea00016c4f80 count:1 mapcount:0 mapping:ffff88800fc13e40 index:0x0
[12580.414798] flags: 0x1fffc0000000200(slab)
[12580.415653] raw: 01fffc0000000200 ffffea00016c7fc8 ffffea0001aba308 ffff88800fc13e40
[12580.417245] raw: 0000000000000000 ffff88805b13e000 0000000100000009 0000000000000000
[12580.418800] page dumped because: kasan: bad access detected
[12580.419969]
[12580.420300] Memory state around the buggy address:
[12580.421303]  ffff88805b13e400: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[12580.422788]  ffff88805b13e480: fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc 00
[12580.424235] >ffff88805b13e500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[12580.425665]                                                              ^
[12580.427037]  ffff88805b13e580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[12580.428479]  ffff88805b13e600: 00 00 00 00 00 00 00 00 00 00 00 00 fc fc fc fc
[12580.429970] ==================================================================