Re: KMSAN: uninit-value in rcu_accelerate_cbs / KMSAN: uninit-value in rcu_process_callbacks

[Alexander Potapenko <glider@google.com>]

On Thu, Nov 15, 2018 at 5:05 AM Kyungtae Kim <kt0755@gmail.com> wrote:
>
> Thank you for all your comments.
>
> Thanks,
> Kyungtae Kim
> On Wed, Nov 14, 2018 at 11:05 AM Paul E. McKenney <paulmck@linux.ibm.com> wrote:
> >
> > On Wed, Nov 14, 2018 at 04:31:11PM +0100, Alexander Potapenko wrote:
> > > On Wed, Nov 14, 2018 at 4:09 PM Paul E. McKenney <paulmck@linux.ibm.com> wrote:
> > > >
> > > > On Wed, Nov 14, 2018 at 04:03:33AM -0500, Kyungtae Kim wrote:
> > > > > We report two crashes in v4.19-rc8 (4.20-rc1 as well, I guess):
> > > > > (Unfortunately, there is no repro for those.)
> > > > >
> > > > > The two crashes seem to share the same issue.
> > > > > In both cases, (uninitialized) memory access violation occurs
> > > > > when "rdp->cblist" is about to be accessed (kernel/rcu/tree.c:2838,1728).
> > > > > I guess those are freed before the use, but I still haven't figured
> > > > > out the reason why.
> > > > > I'm looking forward to some help.
> > >
> > > First of all, I'd avoid reporting KMSAN bugs without clear reproducers.
> > > The tool is still in beta and may still give false positives due to
> > > either missed initialization or rare memory corruptions.
> >
> > OK, I will set this aside, then, thank you!
> >
> >                                                         Thanx, Paul
> >
> > > > You lost me on this one.  In both cases, rdp references a per-CPU
> > > > variable that is implicitly initialized to all zeroes, due to being
> > > > (sort of) a C-language global.
> > > >
> > > > If a callback is queued early, then the following lines in __call_rcu()
> > > > will make an honest list of that field because of the :
> > > >
> > > >                 if (rcu_segcblist_empty(&rdp->cblist))
> > > >                         rcu_segcblist_init(&rdp->cblist);
> > > >
> > > > Otherwise, when rcu_init() is invoked during early boot, we have this
> > > > in rcu_init_percpu_data(), which is called from rcutree_prepare_cpu()
> > > > which is called from rcu_init(), which is called from start_kernel():
> > > >
> > > >         if (rcu_segcblist_empty(&rdp->cblist) && /* No early-boot CBs? */
> > > >             !init_nocb_callback_list(rdp))
> > > >                 rcu_segcblist_init(&rdp->cblist);  /* Re-enable callbacks. */
> > > >
> > > > So either init_nocb_callback_list() initializes the alternative callback
> > > > lists for a no-CBs CPU or rcu_segcblist_init() again makes an honest
> > > > list of that field.
> > > >
> > > > My guess is that your tool is missing the
> > > >
> > > >         rdp = this_cpu_ptr(rsp->rda);
> > > >
> > > > in the __call_rcu() case, and also missing the
> > > >
> > > >         struct rcu_data *rdp = per_cpu_ptr(rsp->rda, cpu);
> > > >
> > > > Note that the ->rda field is explicitly compile-time initialized to
> > > > the base address of the per-CPU variable, which is rcu_preempt_data,
> > > > rcu_bh_data, or rcu_sched_data, depending on which RCU flavor is at hand.
> > > > (In v4.20-rc1, these are all merged into a single flavor to rule them all.)
> > > >
> > > > Alternatively, your tool might be missing the implicit initialization
> > > > of per-CPU variables.
> > > This used to be fine, but after rebasing to v4.20-rc2 I also started
> > > seeing strange reports on per-CPU variables. Taking a look.
No, this was a bug in kmsan_check_memory(), which isn't involved in
the bug reported by Kyungtae Kim.
Therefore I'm assuming per-CPU variables are innocent, but would still
love to see a reproducer for the bug.
> > > > Or maybe I am missing something.  If so, please let me know what it is.
> > > >
> > > >                                                         Thanx, Paul
> > > >
> > > > > Crash log 1
> > > > > =========================================
> > > > > BUG: KMSAN: uninit-value in __rcu_process_callbacks
> > > > > kernel/rcu/tree.c:2838 [inline]
> > > > > BUG: KMSAN: uninit-value in rcu_process_callbacks+0x5ac/0x1cb0
> > > > > kernel/rcu/tree.c:2864
> > > > > CPU: 0 PID: 20 Comm: kauditd Not tainted 4.19.0-rc8+ #18
> > > > > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> > > > > Call Trace:
> > > > >  <IRQ>
> > > > >  __dump_stack lib/dump_stack.c:77 [inline]
> > > > >  dump_stack+0x305/0x460 lib/dump_stack.c:113
> > > > >  kmsan_report+0x1a2/0x2e0 mm/kmsan/kmsan.c:917
> > > > >  __msan_warning+0x7d/0xe0 mm/kmsan/kmsan_instr.c:500
> > > > >  __rcu_process_callbacks kernel/rcu/tree.c:2838 [inline]
> > > > >  rcu_process_callbacks+0x5ac/0x1cb0 kernel/rcu/tree.c:2864
> > > > >  __do_softirq+0x5ff/0xa55 kernel/softirq.c:292
> > > > >  invoke_softirq kernel/softirq.c:373 [inline]
> > > > >  irq_exit+0x22d/0x270 kernel/softirq.c:414
> > > > >  exiting_irq+0xe/0x10 arch/x86/include/asm/apic.h:536
> > > > >  smp_apic_timer_interrupt+0x64/0x90 arch/x86/kernel/apic/apic.c:1059
> > > > >  apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:869
> > > > >  </IRQ>
> > > > > RIP: 0010:finish_lock_switch+0x2b/0x40 kernel/sched/core.c:2578
> > > > > Code: 48 89 e5 53 48 89 fb e8 e3 43 9a 00 8b b8 88 0c 00 00 48 8b 00
> > > > > 48 85 c0 75 12 48 89 df e8 7d 38 9a 00 c6 00 00 c6 03 00 fb 5b <5d> c3
> > > > > e8 de 42 9a 00 eb e7 66 66 66 2e 0f 1f 84 00 00 00 00 00 55
> > > > > RSP: 0018:ffff88010622fca0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
> > > > > RAX: ffff8801105bcc40 RBX: ffff8801061554c0 RCX: ffff8801105bdc40
> > > > > RDX: ffff8801105bdc40 RSI: aaaaaaaaaaaab000 RDI: ffffea00077ec560
> > > > > RBP: ffff88010622fca0 R08: ffffffff7fffffff R09: 0000000000000002
> > > > > R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800751cb880
> > > > > R13: 0000000000000000 R14: ffff880106155db8 R15: ffff88013fcb9c40
> > > > >  finish_task_switch+0xe3/0x270 kernel/sched/core.c:2679
> > > > >  context_switch kernel/sched/core.c:2832 [inline]
> > > > >  __schedule+0x78f/0x8f0 kernel/sched/core.c:3479
> > > > >  schedule+0x1cc/0x300 kernel/sched/core.c:3523
> > > > >  kauditd_thread+0xc64/0xee0 kernel/audit.c:889
> > > > >  kthread+0x5b1/0x5f0 kernel/kthread.c:247
> > > > >  ret_from_fork+0x35/0x40 arch/x86/entry/entry_64.S:416
> > > > >
> > > > > Uninit was created at:
> > > > >  kmsan_save_stack_with_flags mm/kmsan/kmsan.c:255 [inline]
> > > > >  kmsan_internal_alloc_meta_for_pages+0x157/0x730 mm/kmsan/kmsan.c:693
> > > > >  kmsan_alloc_page+0x80/0xe0 mm/kmsan/kmsan_hooks.c:320
> > > > >  __alloc_pages_nodemask+0x128c/0x69b0 mm/page_alloc.c:4416
> > > > >  alloc_pages_current+0x51f/0x760 mm/mempolicy.c:2093
> > > > >  alloc_pages include/linux/gfp.h:511 [inline]
> > > > >  alloc_slab_page mm/slub.c:1459 [inline]
> > > > >  allocate_slab mm/slub.c:1604 [inline]
> > > > >  new_slab+0x552/0x1f30 mm/slub.c:1675
> > > > >  new_slab_objects mm/slub.c:2438 [inline]
> > > > >  ___slab_alloc+0x1414/0x1dd0 mm/slub.c:2590
> > > > >  __slab_alloc mm/slub.c:2630 [inline]
> > > > >  slab_alloc_node mm/slub.c:2693 [inline]
> > > > >  slab_alloc mm/slub.c:2735 [inline]
> > > > >  kmem_cache_alloc+0xc9b/0xda0 mm/slub.c:2740
> > > > >  kmem_cache_zalloc include/linux/slab.h:697 [inline]
> > > > >  avc_alloc_node+0x109/0xb90 security/selinux/avc.c:572
> > > > >  avc_update_node+0x172/0x1ee0 security/selinux/avc.c:859
> > > > >  avc_denied+0x312/0x360 security/selinux/avc.c:1024
> > > > >  avc_has_perm_noaudit+0x733/0x770 security/selinux/avc.c:1155
> > > > >  avc_has_perm+0x172/0x480 security/selinux/avc.c:1184
> > > > >  sock_has_perm security/selinux/hooks.c:4539 [inline]
> > > > >  selinux_socket_sendmsg+0x297/0x360 security/selinux/hooks.c:4875
> > > > >  security_socket_sendmsg+0x127/0x200 security/security.c:1410
> > > > >  sock_sendmsg net/socket.c:628 [inline]
> > > > >  ___sys_sendmsg+0xd5f/0x1290 net/socket.c:2116
> > > > >  __sys_sendmsg net/socket.c:2154 [inline]
> > > > >  __do_sys_sendmsg net/socket.c:2163 [inline]
> > > > >  __se_sys_sendmsg+0x307/0x460 net/socket.c:2161
> > > > >  __x64_sys_sendmsg+0x4a/0x70 net/socket.c:2161
> > > > >  do_syscall_64+0xb8/0x100 arch/x86/entry/common.c:291
> > > > >  entry_SYSCALL_64_after_hwframe+0x63/0xe7
> > > > > =========================================================
> > > > >
> > > > > Crash log 2
> > > > > =========================================================
> > > > > BUG: KMSAN: uninit-value in rcu_accelerate_cbs+0x821/0x990
> > > > > kernel/rcu/tree.c:1728
> > > > > CPU: 0 PID: 10 Comm: rcu_sched Not tainted 4.19.0-rc8+ #18
> > > > > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> > > > > Call Trace:
> > > > >  __dump_stack lib/dump_stack.c:77 [inline]
> > > > >  dump_stack+0x305/0x460 lib/dump_stack.c:113
> > > > >  kmsan_report+0x1a2/0x2e0 mm/kmsan/kmsan.c:917
> > > > >  __msan_warning+0x7d/0xe0 mm/kmsan/kmsan_instr.c:500
> > > > >  rcu_accelerate_cbs+0x821/0x990 kernel/rcu/tree.c:1728
> > > > >  __note_gp_changes+0x2ac/0x9e0 kernel/rcu/tree.c:1807
> > > > >  rcu_gp_cleanup kernel/rcu/tree.c:2109 [inline]
> > > > >  rcu_gp_kthread+0x3019/0x3990 kernel/rcu/tree.c:2236
> > > > >  kthread+0x5b1/0x5f0 kernel/kthread.c:247
> > > > >  ret_from_fork+0x35/0x40 arch/x86/entry/entry_64.S:416
> > > > >
> > > > > Uninit was created at:
> > > > >  kmsan_save_stack_with_flags mm/kmsan/kmsan.c:255 [inline]
> > > > >  kmsan_internal_alloc_meta_for_pages+0x157/0x730 mm/kmsan/kmsan.c:693
> > > > >  kmsan_alloc_page+0x80/0xe0 mm/kmsan/kmsan_hooks.c:320
> > > > >  __alloc_pages_nodemask+0x128c/0x69b0 mm/page_alloc.c:4416
> > > > >  alloc_pages_current+0x51f/0x760 mm/mempolicy.c:2093
> > > > >  alloc_pages include/linux/gfp.h:511 [inline]
> > > > >  alloc_slab_page mm/slub.c:1459 [inline]
> > > > >  allocate_slab mm/slub.c:1604 [inline]
> > > > >  new_slab+0x552/0x1f30 mm/slub.c:1675
> > > > >  new_slab_objects mm/slub.c:2438 [inline]
> > > > >  ___slab_alloc+0x1414/0x1dd0 mm/slub.c:2590
> > > > >  __slab_alloc mm/slub.c:2630 [inline]
> > > > >  slab_alloc_node mm/slub.c:2693 [inline]
> > > > >  slab_alloc mm/slub.c:2735 [inline]
> > > > >  kmem_cache_alloc+0xc9b/0xda0 mm/slub.c:2740
> > > > >  kmem_cache_zalloc include/linux/slab.h:697 [inline]
> > > > >  avc_alloc_node+0x109/0xb90 security/selinux/avc.c:572
> > > > >  avc_insert security/selinux/avc.c:696 [inline]
> > > > >  avc_compute_av+0x31e/0x1050 security/selinux/avc.c:1008
> > > > >  avc_has_perm_noaudit+0x516/0x770 security/selinux/avc.c:1149
> > > > >  avc_has_perm+0x172/0x480 security/selinux/avc.c:1184
> > > > >  selinux_socket_create+0x248/0x3c0 security/selinux/hooks.c:4560
> > > > >  security_socket_create+0x146/0x210 security/security.c:1372
> > > > >  __sock_create+0x26b/0xf30 net/socket.c:1232
> > > > >  sock_create net/socket.c:1317 [inline]
> > > > >  __sys_socket+0x180/0x670 net/socket.c:1347
> > > > >  __do_sys_socket net/socket.c:1356 [inline]
> > > > >  __se_sys_socket+0x8d/0xb0 net/socket.c:1354
> > > > >  __x64_sys_socket+0x4a/0x70 net/socket.c:1354
> > > > >  do_syscall_64+0xb8/0x100 arch/x86/entry/common.c:291
> > > > >  entry_SYSCALL_64_after_hwframe+0x63/0xe7
> > > > > =========================================================
> > > > >
> > > > > Thanks,
> > > > > Kyungtae Kim
> > > > >
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google Groups "syzkaller" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller+unsubscribe@googlegroups.com.
> > > > For more options, visit https://groups.google.com/d/optout.
> > >
> > >
> > >
> > > --
> > > Alexander Potapenko
> > > Software Engineer
> > >
> > > Google Germany GmbH
> > > Erika-Mann-Straße, 33
> > > 80636 München
> > >
> > > Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
> > > Registergericht und -nummer: Hamburg, HRB 86891
> > > Sitz der Gesellschaft: Hamburg
> > >
> >
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg