Re: WARNING in __rcu_read_unlock

[Dmitry Vyukov <dvyukov@google.com>]

On Sun, Dec 16, 2018 at 8:04 PM Paul E. McKenney <paulmck@linux.ibm.com> wrote:
>
> On Sat, Dec 15, 2018 at 04:41:03AM -0800, syzbot wrote:
> > Hello,
> >
> > syzbot found the following crash on:
> >
> > HEAD commit:    2aa55dccf83d hns3: prevent building without CONFIG_INET
> > git tree:       net-next
> > console output: https://syzkaller.appspot.com/x/log.txt?x=15628f6d400000
> > kernel config:  https://syzkaller.appspot.com/x/.config?x=d9655b05acfc97ff
> > dashboard link: https://syzkaller.appspot.com/bug?extid=43f6755d1c2e62743468
> > compiler:       gcc (GCC) 8.0.1 20180413 (experimental)
> > syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=125fda8b400000
> > C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=135e54cd400000
> >
> > IMPORTANT: if you fix the bug, please add the following tag to the commit:
> > Reported-by: syzbot+43f6755d1c2e62743468@syzkaller.appspotmail.com
> >
> > IPv6: ADDRCONF(NETDEV_CHANGE): veth1: link becomes ready
> > IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
> > 8021q: adding VLAN 0 to HW filter on device team0
> > kasan: CONFIG_KASAN_INLINE enabled
> > kasan: GPF could be caused by NULL-ptr deref or user memory access
> > WARNING: CPU: 0 PID: -2035180937 at kernel/rcu/tree_plugin.h:438
> > __rcu_read_unlock+0x266/0x2e0 kernel/rcu/tree_plugin.h:432
> > Kernel panic - not syncing: panic_on_warn set ...
> > CPU: 0 PID: -2035180937 Comm:  L ����� Not tainted 4.20.0-rc6+ #344
> > Hardware name: Google Google Compute Engine/Google Compute Engine,
> > BIOS Google 01/01/2011
> > Call Trace:
> >  <IRQ>
> >  __dump_stack lib/dump_stack.c:77 [inline]
> >  dump_stack+0x244/0x39d lib/dump_stack.c:113
> >  panic+0x2ad/0x55c kernel/panic.c:188
> >  __warn.cold.8+0x20/0x45 kernel/panic.c:540
> >  report_bug+0x254/0x2d0 lib/bug.c:186
> >  fixup_bug arch/x86/kernel/traps.c:178 [inline]
> >  do_error_trap+0x11b/0x200 arch/x86/kernel/traps.c:271
> >  do_invalid_op+0x36/0x40 arch/x86/kernel/traps.c:290
> >  invalid_op+0x14/0x20 arch/x86/entry/entry_64.S:973
> > RIP: 0010:__rcu_read_unlock+0x266/0x2e0 kernel/rcu/tree_plugin.h:432
> > Code: 00 00 00 fc ff df 4c 89 f2 48 c1 ea 03 0f b6 04 02 84 c0 74 04
> > 3c 03 7e 6f 41 c7 84 24 70 03 00 00 00 00 00 00 e9 5a fe ff ff <0f>
> > 0b e9 da fe ff ff 4c 89 f7 e8 1b 14 59 00 e9 2a fe ff ff 4c 89
> > RSP: 0018:ffff8881dae075e8 EFLAGS: 00010006
> > RAX: dffffc0000000000 RBX: 1ffff1103b5c0ebe RCX: ffffffff8153f599
> > RDX: 1ffff1103b5c0eca RSI: ffffffff8153f5bb RDI: 0000000000000005
> > RBP: ffff8881dae076b8 R08: ffff8881bf1f4540 R09: ffffed103b5c3ef8
> > R10: ffffed103b5c3ef8 R11: ffff8881dae1f7c7 R12: 00000000fdb21501
> > R13: 1ffff1103b5c0eca R14: ffff8881bf1f48b0 R15: ffff8881dae07690
> >  rcu_read_unlock include/linux/rcupdate.h:660 [inline]
> >  __atomic_notifier_call_chain kernel/notifier.c:184 [inline]
> >  atomic_notifier_call_chain+0xd0/0x190 kernel/notifier.c:193
> >  notify_die+0x1bd/0x2d0 kernel/notifier.c:549
> >  do_general_protection+0x16d/0x2f0 arch/x86/kernel/traps.c:557
> >  general_protection+0x1e/0x30 arch/x86/entry/entry_64.S:1142
> > RIP: 0010:task_css include/linux/cgroup.h:477 [inline]
> > RIP: 0010:task_ca kernel/sched/cpuacct.c:43 [inline]
> > RIP: 0010:cpuacct_account_field+0x140/0x3d0 kernel/sched/cpuacct.c:365
> > Code: b6 97 08 00 85 c0 74 0d 80 3d 69 bb b2 08 00 0f 84 a4 01 00 00
> > 49 8d 7e 10 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 <80>
> > 3c 02 00 0f 85 4e 02 00 00 4d 8b 7e 10 49 81 ff 20 23 58 89 0f
> > RSP: 0018:ffff8881dae078a8 EFLAGS: 00010002
> > RAX: dffffc0000000000 RBX: ffff8881dae07918 RCX: 0000000000000000
> > RDX: 000000000000000e RSI: 00000000ffff8881 RDI: 0000000000000072
> > RBP: ffff8881dae07940 R08: 0000000000000000 R09: 0000000000000000
> > R10: ffffed1037304851 R11: 0000000000000007 R12: 0000000000982e14
> > R13: dffffc0000000000 R14: 0000000000000062 R15: ffff8881bf1f4540
> >  cgroup_account_cputime_field include/linux/cgroup.h:775 [inline]
> >  task_group_account_field kernel/sched/cputime.c:108 [inline]
> >  account_system_index_time+0x1e8/0x5d0 kernel/sched/cputime.c:171
> >  irqtime_account_process_tick.isra.6+0x35b/0x490 kernel/sched/cputime.c:388
> >  account_process_tick+0x282/0x350 kernel/sched/cputime.c:483
> >  update_process_times+0x21/0x70 kernel/time/timer.c:1634
> >  tick_sched_handle+0x9f/0x180 kernel/time/tick-sched.c:164
> >  tick_sched_timer+0x45/0x130 kernel/time/tick-sched.c:1274
> >  __run_hrtimer kernel/time/hrtimer.c:1398 [inline]
> >  __hrtimer_run_queues+0x41c/0x10d0 kernel/time/hrtimer.c:1460
> >  hrtimer_interrupt+0x313/0x780 kernel/time/hrtimer.c:1518
> >  local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1034 [inline]
> >  smp_apic_timer_interrupt+0x1a1/0x760 arch/x86/kernel/apic/apic.c:1059
> >  apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:807
> >  </IRQ>
> > Kernel Offset: disabled
> > Rebooting in 86400 seconds..
>
>
> Hmmm...  Line 432 is the "t->rcu_read_lock_nesting = 0;" below
> and Line 438 is the int "rrln = READ_ONCE(t->rcu_read_lock_nesting);"
> below.  Are you saying that the value of "current" is NULL?  If you
> do that, you have voided your RCU warranty.  ;-)
>
> Or should I be looking elsewhere than v4.20-rc5?

Hi Paul,

Git tree and commit are in the first lines of the report ;)

I think it points to:
    WARN_ON_ONCE(rrln < 0 && rrln > INT_MIN / 2);

The exact source line for the RIP: line maybe off-by-one, because
usually we have call return PCs, and then it's necessary to subtract 1
during symbolization, but not for RIP: lines, because they contain
exact faulting PC. This was fixed few days ago, but I think this
report was generated before this fix:
https://github.com/google/syzkaller/commit/7a944a0a666587f229291814b30644cc0859674c

But the kernel output contains the right line number:
[   51.239451] WARNING: CPU: 0 PID: -2035180937 at
kernel/rcu/tree_plugin.h:438 __rcu_read_unlock+0x266/0x2e0

That "PID: -2035180937" looks concerning.

Out of 3 syscalls in the reproducer, 2 operate on invalid fd's so
probably no-op. And the remaining one injects a network packet. If
this is caused by the incoming network packet, it may be pretty bad.
+netdev.



> ------------------------------------------------------------------------
>
> void __rcu_read_unlock(void)
> {
>         struct task_struct *t = current;
>
>         if (t->rcu_read_lock_nesting != 1) {
>                 --t->rcu_read_lock_nesting;
>         } else {
>                 barrier();  /* critical section before exit code. */
>                 t->rcu_read_lock_nesting = INT_MIN;
>                 barrier();  /* assign before ->rcu_read_unlock_special load */
>                 if (unlikely(READ_ONCE(t->rcu_read_unlock_special.s)))
>                         rcu_read_unlock_special(t);
>                 barrier();  /* ->rcu_read_unlock_special load before assign */
>                 t->rcu_read_lock_nesting = 0;
>         }
> #ifdef CONFIG_PROVE_LOCKING
>         {
>                 int rrln = READ_ONCE(t->rcu_read_lock_nesting);
>
>                 WARN_ON_ONCE(rrln < 0 && rrln > INT_MIN / 2);
>         }
> #endif /* #ifdef CONFIG_PROVE_LOCKING */
> }
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/20181216190412.GE4170%40linux.ibm.com.
> For more options, visit https://groups.google.com/d/optout.