Re: [syzbot] upstream boot error: WARNING in __context_tracking_enter

Re: [syzbot] upstream boot error: WARNING in __context_tracking_enter

[Dmitry Vyukov]

.On Fri, Mar 19, 2021 at 10:44 AM syzbot
<syzbot+f09a12b2c77bfbbf51bd@syzkaller.appspotmail.com> wrote:
>
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit:    8b12a62a Merge tag 'drm-fixes-2021-03-19' of git://anongit..
> git tree:       upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=17e815aed00000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=cfeed364fc353c32
> dashboard link: https://syzkaller.appspot.com/bug?extid=f09a12b2c77bfbbf51bd
> userspace arch: arm
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+f09a12b2c77bfbbf51bd@syzkaller.appspotmail.com


+Mark, arm
It did not get far with CONFIG_CONTEXT_TRACKING_FORCE (kernel doesn't boot).


> Run /sbin/init as init process
> ------------[ cut here ]------------
> WARNING: CPU: 0 PID: 1 at include/linux/seqlock.h:271 __seqprop_assert include/linux/seqlock.h:271 [inline]
> WARNING: CPU: 0 PID: 1 at include/linux/seqlock.h:271 __seqprop_assert.constprop.0+0xf0/0x11c include/linux/seqlock.h:269
> Modules linked in:
> Kernel panic - not syncing: panic_on_warn set ...
> CPU: 0 PID: 1 Comm: init Not tainted 5.12.0-rc3-syzkaller #0
> Hardware name: ARM-Versatile Express
> Backtrace:
> [<81804310>] (dump_backtrace) from [<81804584>] (show_stack+0x18/0x1c arch/arm/kernel/traps.c:252)
>  r7:00000080 r6:60000193 r5:00000000 r4:82b58444
> [<8180456c>] (show_stack) from [<8180bc58>] (__dump_stack lib/dump_stack.c:79 [inline])
> [<8180456c>] (show_stack) from [<8180bc58>] (dump_stack+0xb8/0xe8 lib/dump_stack.c:120)
> [<8180bba0>] (dump_stack) from [<81805118>] (panic+0x130/0x378 kernel/panic.c:231)
>  r7:81f487d4 r6:82a390e4 r5:00000000 r4:82c6b0d8
> [<81804fe8>] (panic) from [<802447b4>] (__warn+0xb0/0x164 kernel/panic.c:605)
>  r3:82a2248c r2:00000000 r1:5b71e000 r0:81f487d4
>  r7:0000010f
> [<80244704>] (__warn) from [<818053c8>] (warn_slowpath_fmt+0x68/0xd4 kernel/panic.c:628)
>  r7:8028c4b4 r6:0000010f r5:81f40c30 r4:00000000
> [<81805364>] (warn_slowpath_fmt) from [<8028c4b4>] (__seqprop_assert include/linux/seqlock.h:271 [inline])
> [<81805364>] (warn_slowpath_fmt) from [<8028c4b4>] (__seqprop_assert.constprop.0+0xf0/0x11c include/linux/seqlock.h:269)
>  r8:00000000 r7:82a22498 r6:828ac940 r5:8181fbb0 r4:00000001
> [<8028c3c4>] (__seqprop_assert.constprop.0) from [<8028ea54>] (vtime_user_enter+0x1c/0x90 kernel/sched/cputime.c:709)
>  r5:8181fbb0 r4:834f0000
> [<8028ea38>] (vtime_user_enter) from [<8181fbb0>] (__context_tracking_enter+0x148/0x334 kernel/context_tracking.c:82)
>  r7:82a22498 r6:828ac940 r5:ffffe000 r4:00000001
> [<8181fa68>] (__context_tracking_enter) from [<80417170>] (context_tracking_enter kernel/context_tracking.c:122 [inline])
> [<8181fa68>] (__context_tracking_enter) from [<80417170>] (context_tracking_enter+0x54/0x84 kernel/context_tracking.c:106)
>  r7:00000000 r6:00000000 r5:60000193 r4:00000001
> [<8041711c>] (context_tracking_enter) from [<804171bc>] (user_enter include/linux/context_tracking.h:28 [inline])
> [<8041711c>] (context_tracking_enter) from [<804171bc>] (context_tracking_user_enter+0x1c/0x20 kernel/context_tracking.c:130)
>  r5:81820218 r4:00000000
> [<804171a0>] (context_tracking_user_enter) from [<802000d4>] (no_work_pending+0x8/0x38)
> Exception stack(0x834dffb0 to 0x834dfff8)
> ffa0:                                     00000000 00000000 00000000 00000000
> ffc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> ffe0: 00000000 7ec68ef0 00000000 76ee5a00 00000010 00000000
> Rebooting in 86400 seconds..
>
>
> ---
> This report is generated by a bot. It may contain errors.
> See https://goo.gl/tpsmEJ for more information about syzbot.
> syzbot engineers can be reached at syzkaller@googlegroups.com.
>
> syzbot will keep track of this issue. See:
> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/0000000000004f14c105bde08f75%40google.com.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [syzbot] upstream boot error: WARNING in __context_tracking_enter

[Russell King - ARM Linux admin]

On Fri, Mar 19, 2021 at 10:54:48AM +0100, Dmitry Vyukov wrote:
> .On Fri, Mar 19, 2021 at 10:44 AM syzbot
> <syzbot+f09a12b2c77bfbbf51bd@syzkaller.appspotmail.com> wrote:
> > syzbot found the following issue on:
> >
> > HEAD commit:    8b12a62a Merge tag 'drm-fixes-2021-03-19' of git://anongit..
> > git tree:       upstream
> > console output: https://syzkaller.appspot.com/x/log.txt?x=17e815aed00000
> > kernel config:  https://syzkaller.appspot.com/x/.config?x=cfeed364fc353c32
> > dashboard link: https://syzkaller.appspot.com/bug?extid=f09a12b2c77bfbbf51bd
> > userspace arch: arm
> >
> > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > Reported-by: syzbot+f09a12b2c77bfbbf51bd@syzkaller.appspotmail.com
> 
> 
> +Mark, arm
> It did not get far with CONFIG_CONTEXT_TRACKING_FORCE (kernel doesn't boot).

It seems that the path:

context_tracking_user_enter()
user_enter()
context_tracking_enter()
__context_tracking_enter()
vtime_user_enter()

expects preemption to be disabled. It effectively is, because local
interrupts are disabled by context_tracking_enter().

However, the requirement for preemption to be disabled is not
documented... so shrug. Maybe someone can say what the real requirements
are here.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [syzbot] upstream boot error: WARNING in __context_tracking_enter

[Mark Rutland]

Hi Russell,

On Fri, Mar 19, 2021 at 10:10:43AM +0000, Russell King - ARM Linux admin wrote:
> On Fri, Mar 19, 2021 at 10:54:48AM +0100, Dmitry Vyukov wrote:
> > .On Fri, Mar 19, 2021 at 10:44 AM syzbot
> > <syzbot+f09a12b2c77bfbbf51bd@syzkaller.appspotmail.com> wrote:
> > > syzbot found the following issue on:
> > >
> > > HEAD commit:    8b12a62a Merge tag 'drm-fixes-2021-03-19' of git://anongit..
> > > git tree:       upstream
> > > console output: https://syzkaller.appspot.com/x/log.txt?x=17e815aed00000
> > > kernel config:  https://syzkaller.appspot.com/x/.config?x=cfeed364fc353c32
> > > dashboard link: https://syzkaller.appspot.com/bug?extid=f09a12b2c77bfbbf51bd
> > > userspace arch: arm
> > >
> > > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > > Reported-by: syzbot+f09a12b2c77bfbbf51bd@syzkaller.appspotmail.com
> > 
> > 
> > +Mark, arm
> > It did not get far with CONFIG_CONTEXT_TRACKING_FORCE (kernel doesn't boot).
> 
> It seems that the path:
> 
> context_tracking_user_enter()
> user_enter()
> context_tracking_enter()
> __context_tracking_enter()
> vtime_user_enter()
> 
> expects preemption to be disabled. It effectively is, because local
> interrupts are disabled by context_tracking_enter().
> 
> However, the requirement for preemption to be disabled is not
> documented... so shrug. Maybe someone can say what the real requirements
> are here.

From dealing with this recently on arm64, theis is a bit messy. To
handle this robustly we need to do a few things in sequence, including
using the *_irqoff() variants of the context_tracking_user_*()
functions.

I wrote down the constraints in commit:
  
  23529049c6842382 ("arm64: entry: fix non-NMI user<->kernel transitions")

For user->kernel transitions, the arch code needs the following sequence
before invoking arbitrary kernel C code:

	lockdep_hardirqs_off(CALLER_ADDR0);
	user_exit_irqoff();
	trace_hardirqs_off_finish();

For kernel->user transitions, the arch code needs the following sequence
once it will no longer invoke arbitrary kernel C code, just before
returning to userspace:

	trace_hardirqs_on_prepare();
	lockdep_hardirqs_on_prepare(CALLER_ADDR0);
	user_enter_irqoff();
	lockdep_hardirqs_on(CALLER_ADDR0);

Thanks,
Mark.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [syzbot] upstream boot error: WARNING in __context_tracking_enter

[Dmitry Vyukov]

On Mon, Mar 22, 2021 at 6:22 PM Mark Rutland <mark.rutland@arm.com> wrote:
>
> Hi Russell,
>
> On Fri, Mar 19, 2021 at 10:10:43AM +0000, Russell King - ARM Linux admin wrote:
> > On Fri, Mar 19, 2021 at 10:54:48AM +0100, Dmitry Vyukov wrote:
> > > .On Fri, Mar 19, 2021 at 10:44 AM syzbot
> > > <syzbot+f09a12b2c77bfbbf51bd@syzkaller.appspotmail.com> wrote:
> > > > syzbot found the following issue on:
> > > >
> > > > HEAD commit:    8b12a62a Merge tag 'drm-fixes-2021-03-19' of git://anongit..
> > > > git tree:       upstream
> > > > console output: https://syzkaller.appspot.com/x/log.txt?x=17e815aed00000
> > > > kernel config:  https://syzkaller.appspot.com/x/.config?x=cfeed364fc353c32
> > > > dashboard link: https://syzkaller.appspot.com/bug?extid=f09a12b2c77bfbbf51bd
> > > > userspace arch: arm
> > > >
> > > > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > > > Reported-by: syzbot+f09a12b2c77bfbbf51bd@syzkaller.appspotmail.com
> > >
> > >
> > > +Mark, arm
> > > It did not get far with CONFIG_CONTEXT_TRACKING_FORCE (kernel doesn't boot).
> >
> > It seems that the path:
> >
> > context_tracking_user_enter()
> > user_enter()
> > context_tracking_enter()
> > __context_tracking_enter()
> > vtime_user_enter()
> >
> > expects preemption to be disabled. It effectively is, because local
> > interrupts are disabled by context_tracking_enter().
> >
> > However, the requirement for preemption to be disabled is not
> > documented... so shrug. Maybe someone can say what the real requirements
> > are here.
>
> From dealing with this recently on arm64, theis is a bit messy. To
> handle this robustly we need to do a few things in sequence, including
> using the *_irqoff() variants of the context_tracking_user_*()
> functions.
>
> I wrote down the constraints in commit:
>
>   23529049c6842382 ("arm64: entry: fix non-NMI user<->kernel transitions")
>
> For user->kernel transitions, the arch code needs the following sequence
> before invoking arbitrary kernel C code:
>
>         lockdep_hardirqs_off(CALLER_ADDR0);
>         user_exit_irqoff();
>         trace_hardirqs_off_finish();
>
> For kernel->user transitions, the arch code needs the following sequence
> once it will no longer invoke arbitrary kernel C code, just before
> returning to userspace:
>
>         trace_hardirqs_on_prepare();
>         lockdep_hardirqs_on_prepare(CALLER_ADDR0);
>         user_enter_irqoff();
>         lockdep_hardirqs_on(CALLER_ADDR0);

Hi Russell,

Does Mark's comment make sense to you?
lockdep_assert_preemption_disabled() also checks "&&
this_cpu_read(hardirqs_enabled)", so is it that we also need hardirq's
disabled around user_enter/exit?
This issue currently prevents ARM boot on syzbot.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [syzbot] upstream boot error: WARNING in __context_tracking_enter

[Marco Elver]

On Tue, Apr 13, 2021 at 07:14AM +0200, Dmitry Vyukov wrote:
[...]
> Hi Russell,
> 
> Does Mark's comment make sense to you?
> lockdep_assert_preemption_disabled() also checks "&&
> this_cpu_read(hardirqs_enabled)", so is it that we also need hardirq's
> disabled around user_enter/exit?
> This issue currently prevents ARM boot on syzbot.

We've disabled lockdep on syzbot's arm32 instance now as that instance had
been dead since March:
https://github.com/google/syzkaller/commit/be386ae8800e02b4a9a3239c9565e9d40e253c84

It's running again. There may be an influx of new bugs.

Thanks,
-- Marco

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel