* INFO: rcu detected stall in sys_sendfile64 @ 2018-12-19 9:27 syzbot 2018-12-19 10:11 ` Tetsuo Handa 0 siblings, 1 reply; 6+ messages in thread From: syzbot @ 2018-12-19 9:27 UTC (permalink / raw) To: jmorris, linux-kernel, linux-security-module, serge, syzkaller-bugs Hello, syzbot found the following crash on: HEAD commit: ddfbab46539f Merge tag 'scsi-fixes' of git://git.kernel.or.. git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=15b87fa3400000 kernel config: https://syzkaller.appspot.com/x/.config?x=861a3573f4e78ba1 dashboard link: https://syzkaller.appspot.com/bug?extid=bcad772bbc241b4c6147 compiler: gcc (GCC) 8.0.1 20180413 (experimental) syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13912ccd400000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=145781db400000 IMPORTANT: if you fix the bug, please add the following tag to the commit: Reported-by: syzbot+bcad772bbc241b4c6147@syzkaller.appspotmail.com audit: type=1800 audit(1545177748.235:30): pid=5832 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=collect_data cause=failed(directio) comm="startpar" name="rmnologin" dev="sda1" ino=2423 res=0 rcu: INFO: rcu_preempt self-detected stall on CPU rcu: 1-....: (10500 ticks this GP) idle=37e/1/0x4000000000000002 softirq=6309/6309 fqs=5250 rcu: (t=10502 jiffies g=4321 q=29) NMI backtrace for cpu 1 CPU: 1 PID: 5987 Comm: syz-executor800 Not tainted 4.20.0-rc7+ #377 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: <IRQ> __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x244/0x39d lib/dump_stack.c:113 nmi_cpu_backtrace.cold.2+0x5c/0xa1 lib/nmi_backtrace.c:101 nmi_trigger_cpumask_backtrace+0x1e8/0x22a lib/nmi_backtrace.c:62 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline] rcu_dump_cpu_stacks+0x16f/0x1bc kernel/rcu/tree.c:1195 print_cpu_stall.cold.67+0x1f3/0x3c7 kernel/rcu/tree.c:1334 check_cpu_stall kernel/rcu/tree.c:1408 [inline] rcu_pending kernel/rcu/tree.c:2961 [inline] rcu_check_callbacks+0xf3b/0x13f0 kernel/rcu/tree.c:2506 update_process_times+0x2d/0x70 kernel/time/timer.c:1636 tick_sched_handle+0x9f/0x180 kernel/time/tick-sched.c:164 tick_sched_timer+0x45/0x130 kernel/time/tick-sched.c:1274 __run_hrtimer kernel/time/hrtimer.c:1398 [inline] __hrtimer_run_queues+0x41c/0x10d0 kernel/time/hrtimer.c:1460 hrtimer_interrupt+0x313/0x780 kernel/time/hrtimer.c:1518 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1034 [inline] smp_apic_timer_interrupt+0x1a1/0x760 arch/x86/kernel/apic/apic.c:1059 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:807 </IRQ> RIP: 0010:__sanitizer_cov_trace_pc+0x4b/0x50 kernel/kcov.c:111 Code: 90 d8 12 00 00 83 fa 02 75 20 48 8b 88 e0 12 00 00 8b 80 dc 12 00 00 48 8b 11 48 83 c2 01 48 39 d0 76 07 48 89 34 d1 48 89 11 <5d> c3 0f 1f 00 55 40 0f b6 d6 40 0f b6 f7 31 ff 48 89 e5 48 8b 4d RSP: 0018:ffff8881c1e17988 EFLAGS: 00000293 ORIG_RAX: ffffffffffffff13 RAX: ffff8881c2860200 RBX: 0000000000000004 RCX: ffffffff81ced656 RDX: 0000000000000000 RSI: ffffffff8344bfe8 RDI: ffff8881b4364cc0 RBP: ffff8881c1e17988 R08: ffff8881c2860200 R09: 0000000000000007 R10: 0000000000000000 R11: ffff8881c2860200 R12: ffff8881b4364cc0 R13: 0000000001237d36 R14: ffff8881b4364cc0 R15: 0000000000000004 security_file_permission+0x18/0x220 security/security.c:886 rw_verify_area+0x111/0x360 fs/read_write.c:391 do_splice_to+0x8b/0x190 fs/splice.c:868 splice_direct_to_actor+0x31c/0x9d0 fs/splice.c:957 do_splice_direct+0x2d4/0x420 fs/splice.c:1066 do_sendfile+0x62a/0xe20 fs/read_write.c:1439 __do_sys_sendfile64 fs/read_write.c:1494 [inline] __se_sys_sendfile64 fs/read_write.c:1486 [inline] __x64_sys_sendfile64+0x15d/0x250 fs/read_write.c:1486 do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x4400d9 Code: 18 89 d0 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 fb 13 fc ff c3 66 2e 0f 1f 84 00 00 00 00 RSP: 002b:00007ffdd3e00e58 EFLAGS: 00000203 ORIG_RAX: 0000000000000028 RAX: ffffffffffffffda RBX: 00000000004002c8 RCX: 00000000004400d9 RDX: 0000000020000000 RSI: 0000000000000003 RDI: 0000000000000003 RBP: 00000000006ca018 R08: 00000000004002c8 R09: 00000000004002c8 R10: 00008080fffffffe R11: 0000000000000203 R12: 0000000000401960 R13: 00000000004019f0 R14: 0000000000000000 R15: 0000000000000000 --- This bug is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at syzkaller@googlegroups.com. syzbot will keep track of this bug report. See: https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with syzbot. syzbot can test patches for this bug, for details see: https://goo.gl/tpsmEJ#testing-patches ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: INFO: rcu detected stall in sys_sendfile64 2018-12-19 9:27 INFO: rcu detected stall in sys_sendfile64 syzbot @ 2018-12-19 10:11 ` Tetsuo Handa 2018-12-19 18:42 ` Dmitry Vyukov 0 siblings, 1 reply; 6+ messages in thread From: Tetsuo Handa @ 2018-12-19 10:11 UTC (permalink / raw) To: syzbot, syzkaller-bugs, Ingo Molnar, Peter Zijlstra Cc: jmorris, linux-kernel, linux-security-module, serge On 2018/12/19 18:27, syzbot wrote: > HEAD commit: ddfbab46539f Merge tag 'scsi-fixes' of git://git.kernel.or.. > git tree: upstream > console output: https://syzkaller.appspot.com/x/log.txt?x=15b87fa3400000 > kernel config: https://syzkaller.appspot.com/x/.config?x=861a3573f4e78ba1 > dashboard link: https://syzkaller.appspot.com/bug?extid=bcad772bbc241b4c6147 > compiler: gcc (GCC) 8.0.1 20180413 (experimental) > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13912ccd400000 > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=145781db400000 This is not a LSM problem, for the reproducer is calling sched_setattr(SCHED_DEADLINE) with very large values. sched_setattr(0, {size=0, sched_policy=0x6 /* SCHED_??? */, sched_flags=0, sched_nice=0, sched_priority=0, sched_runtime=2251799813724439, sched_deadline=4611686018427453437, sched_period=0}, 0) = 0 I think that this problem is nothing but an insane sched_setattr() parameter. #syz invalid ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: INFO: rcu detected stall in sys_sendfile64 2018-12-19 10:11 ` Tetsuo Handa @ 2018-12-19 18:42 ` Dmitry Vyukov 2020-01-04 11:09 ` Tetsuo Handa 0 siblings, 1 reply; 6+ messages in thread From: Dmitry Vyukov @ 2018-12-19 18:42 UTC (permalink / raw) To: Tetsuo Handa Cc: syzbot, syzkaller-bugs, Ingo Molnar, Peter Zijlstra, James Morris, LKML, linux-security-module, Serge E. Hallyn On Wed, Dec 19, 2018 at 11:13 AM Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp> wrote: > > On 2018/12/19 18:27, syzbot wrote: > > HEAD commit: ddfbab46539f Merge tag 'scsi-fixes' of git://git.kernel.or.. > > git tree: upstream > > console output: https://syzkaller.appspot.com/x/log.txt?x=15b87fa3400000 > > kernel config: https://syzkaller.appspot.com/x/.config?x=861a3573f4e78ba1 > > dashboard link: https://syzkaller.appspot.com/bug?extid=bcad772bbc241b4c6147 > > compiler: gcc (GCC) 8.0.1 20180413 (experimental) > > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13912ccd400000 > > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=145781db400000 > > This is not a LSM problem, for the reproducer is calling > sched_setattr(SCHED_DEADLINE) with very large values. > > sched_setattr(0, {size=0, sched_policy=0x6 /* SCHED_??? */, sched_flags=0, sched_nice=0, sched_priority=0, sched_runtime=2251799813724439, sched_deadline=4611686018427453437, sched_period=0}, 0) = 0 > > I think that this problem is nothing but an insane sched_setattr() parameter. > > #syz invalid Note there was another one with sched_setattr, which turned out to be some serious problem in kernel (sched_setattr should not cause CPU stall for 3 minutes): INFO: rcu detected stall in do_idle https://syzkaller.appspot.com/bug?extid=385468161961cee80c31 https://groups.google.com/forum/#!msg/syzkaller-bugs/crrfvusGtwI/IoD_zus4BgAJ Maybe it another incarnation of the same bug, that one is still not fixed. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: INFO: rcu detected stall in sys_sendfile64 2018-12-19 18:42 ` Dmitry Vyukov @ 2020-01-04 11:09 ` Tetsuo Handa 2020-01-07 13:02 ` Dmitry Vyukov 0 siblings, 1 reply; 6+ messages in thread From: Tetsuo Handa @ 2020-01-04 11:09 UTC (permalink / raw) To: Dmitry Vyukov Cc: syzbot, syzkaller-bugs, Ingo Molnar, Peter Zijlstra, James Morris, LKML, linux-security-module, Serge E. Hallyn On 2018/12/20 3:42, Dmitry Vyukov wrote: > On Wed, Dec 19, 2018 at 11:13 AM Tetsuo Handa > <penguin-kernel@i-love.sakura.ne.jp> wrote: >> >> On 2018/12/19 18:27, syzbot wrote: >>> HEAD commit: ddfbab46539f Merge tag 'scsi-fixes' of git://git.kernel.or.. >>> git tree: upstream >>> console output: https://syzkaller.appspot.com/x/log.txt?x=15b87fa3400000 >>> kernel config: https://syzkaller.appspot.com/x/.config?x=861a3573f4e78ba1 >>> dashboard link: https://syzkaller.appspot.com/bug?extid=bcad772bbc241b4c6147 >>> compiler: gcc (GCC) 8.0.1 20180413 (experimental) >>> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13912ccd400000 >>> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=145781db400000 >> >> This is not a LSM problem, for the reproducer is calling >> sched_setattr(SCHED_DEADLINE) with very large values. >> >> sched_setattr(0, {size=0, sched_policy=0x6 /* SCHED_??? */, sched_flags=0, sched_nice=0, sched_priority=0, sched_runtime=2251799813724439, sched_deadline=4611686018427453437, sched_period=0}, 0) = 0 >> >> I think that this problem is nothing but an insane sched_setattr() parameter. >> >> #syz invalid > > Note there was another one with sched_setattr, which turned out to be > some serious problem in kernel (sched_setattr should not cause CPU > stall for 3 minutes): > INFO: rcu detected stall in do_idle > https://syzkaller.appspot.com/bug?extid=385468161961cee80c31 > https://groups.google.com/forum/#!msg/syzkaller-bugs/crrfvusGtwI/IoD_zus4BgAJ > > Maybe it another incarnation of the same bug, that one is still not fixed. > Can we let syzbot blacklist sched_setattr() for now? There are many stall reports doing sched_setattr(SCHED_RR) which makes it difficult to find stall reports not using sched_setattr(). ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: INFO: rcu detected stall in sys_sendfile64 2020-01-04 11:09 ` Tetsuo Handa @ 2020-01-07 13:02 ` Dmitry Vyukov 2020-01-07 13:11 ` Peter Zijlstra 0 siblings, 1 reply; 6+ messages in thread From: Dmitry Vyukov @ 2020-01-07 13:02 UTC (permalink / raw) To: Tetsuo Handa Cc: syzbot, syzkaller-bugs, Ingo Molnar, Peter Zijlstra, James Morris, LKML, linux-security-module, Serge E. Hallyn, syzkaller On Sat, Jan 4, 2020 at 12:09 PM Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp> wrote: > > On 2018/12/20 3:42, Dmitry Vyukov wrote: > > On Wed, Dec 19, 2018 at 11:13 AM Tetsuo Handa > > <penguin-kernel@i-love.sakura.ne.jp> wrote: > >> > >> On 2018/12/19 18:27, syzbot wrote: > >>> HEAD commit: ddfbab46539f Merge tag 'scsi-fixes' of git://git.kernel.or.. > >>> git tree: upstream > >>> console output: https://syzkaller.appspot.com/x/log.txt?x=15b87fa3400000 > >>> kernel config: https://syzkaller.appspot.com/x/.config?x=861a3573f4e78ba1 > >>> dashboard link: https://syzkaller.appspot.com/bug?extid=bcad772bbc241b4c6147 > >>> compiler: gcc (GCC) 8.0.1 20180413 (experimental) > >>> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13912ccd400000 > >>> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=145781db400000 > >> > >> This is not a LSM problem, for the reproducer is calling > >> sched_setattr(SCHED_DEADLINE) with very large values. > >> > >> sched_setattr(0, {size=0, sched_policy=0x6 /* SCHED_??? */, sched_flags=0, sched_nice=0, sched_priority=0, sched_runtime=2251799813724439, sched_deadline=4611686018427453437, sched_period=0}, 0) = 0 > >> > >> I think that this problem is nothing but an insane sched_setattr() parameter. > >> > >> #syz invalid > > > > Note there was another one with sched_setattr, which turned out to be > > some serious problem in kernel (sched_setattr should not cause CPU > > stall for 3 minutes): > > INFO: rcu detected stall in do_idle > > https://syzkaller.appspot.com/bug?extid=385468161961cee80c31 > > https://groups.google.com/forum/#!msg/syzkaller-bugs/crrfvusGtwI/IoD_zus4BgAJ > > > > Maybe it another incarnation of the same bug, that one is still not fixed. > > > > Can we let syzbot blacklist sched_setattr() for now? There are many stall reports > doing sched_setattr(SCHED_RR) which makes it difficult to find stall reports not > using sched_setattr(). Hi Tetsuo, If we start practice of disabling whole syscalls, I would really like "for now" to be very well defined. When will it end? How will it happen? Is the problem on the radar of relevant people? Will it stay on somebody's radar until it's fixed? Normal practise of project sheriffing is to file a P1 bug assigned to somebody when something gets disabled. But I am not sure how we implement this for kernel. Since the problem is there for a long time and we disable it without defining any criteria, I afraid we disable it forever (then more bugs will pile and re-enabling it will be painful). At the very least we need to acknowledge that we stopping testing schedler for foreseeable future and schedler maintainers need to be notified about this. Blacklisting it and un-blacklisting will cause some churn. Was the bug given at least some attention? Significant number of bugs are relatively easy to fix and fixing it would solve all of the problems in a much better way. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: INFO: rcu detected stall in sys_sendfile64 2020-01-07 13:02 ` Dmitry Vyukov @ 2020-01-07 13:11 ` Peter Zijlstra 0 siblings, 0 replies; 6+ messages in thread From: Peter Zijlstra @ 2020-01-07 13:11 UTC (permalink / raw) To: Dmitry Vyukov Cc: Tetsuo Handa, syzbot, syzkaller-bugs, Ingo Molnar, James Morris, LKML, linux-security-module, Serge E. Hallyn, syzkaller On Tue, Jan 07, 2020 at 02:02:47PM +0100, Dmitry Vyukov wrote: > > >> This is not a LSM problem, for the reproducer is calling > > >> sched_setattr(SCHED_DEADLINE) with very large values. > > >> > > >> sched_setattr(0, {size=0, sched_policy=0x6 /* SCHED_??? */, sched_flags=0, sched_nice=0, sched_priority=0, sched_runtime=2251799813724439, sched_deadline=4611686018427453437, sched_period=0}, 0) = 0 > > >> > > >> I think that this problem is nothing but an insane sched_setattr() parameter. Argh, I had a patch for that somewhere. Let me go dig that out. ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-01-07 13:12 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-12-19 9:27 INFO: rcu detected stall in sys_sendfile64 syzbot 2018-12-19 10:11 ` Tetsuo Handa 2018-12-19 18:42 ` Dmitry Vyukov 2020-01-04 11:09 ` Tetsuo Handa 2020-01-07 13:02 ` Dmitry Vyukov 2020-01-07 13:11 ` Peter Zijlstra
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.