* INFO: rcu detected stall in sys_sendfile64
@ 2018-12-19 9:27 syzbot
2018-12-19 10:11 ` Tetsuo Handa
0 siblings, 1 reply; 6+ messages in thread
From: syzbot @ 2018-12-19 9:27 UTC (permalink / raw)
To: jmorris, linux-kernel, linux-security-module, serge, syzkaller-bugs
Hello,
syzbot found the following crash on:
HEAD commit: ddfbab46539f Merge tag 'scsi-fixes' of git://git.kernel.or..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=15b87fa3400000
kernel config: https://syzkaller.appspot.com/x/.config?x=861a3573f4e78ba1
dashboard link: https://syzkaller.appspot.com/bug?extid=bcad772bbc241b4c6147
compiler: gcc (GCC) 8.0.1 20180413 (experimental)
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13912ccd400000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=145781db400000
IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+bcad772bbc241b4c6147@syzkaller.appspotmail.com
audit: type=1800 audit(1545177748.235:30): pid=5832 uid=0 auid=4294967295
ses=4294967295 subj==unconfined op=collect_data cause=failed(directio)
comm="startpar" name="rmnologin" dev="sda1" ino=2423 res=0
rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 1-....: (10500 ticks this GP) idle=37e/1/0x4000000000000002
softirq=6309/6309 fqs=5250
rcu: (t=10502 jiffies g=4321 q=29)
NMI backtrace for cpu 1
CPU: 1 PID: 5987 Comm: syz-executor800 Not tainted 4.20.0-rc7+ #377
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x244/0x39d lib/dump_stack.c:113
nmi_cpu_backtrace.cold.2+0x5c/0xa1 lib/nmi_backtrace.c:101
nmi_trigger_cpumask_backtrace+0x1e8/0x22a lib/nmi_backtrace.c:62
arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38
trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
rcu_dump_cpu_stacks+0x16f/0x1bc kernel/rcu/tree.c:1195
print_cpu_stall.cold.67+0x1f3/0x3c7 kernel/rcu/tree.c:1334
check_cpu_stall kernel/rcu/tree.c:1408 [inline]
rcu_pending kernel/rcu/tree.c:2961 [inline]
rcu_check_callbacks+0xf3b/0x13f0 kernel/rcu/tree.c:2506
update_process_times+0x2d/0x70 kernel/time/timer.c:1636
tick_sched_handle+0x9f/0x180 kernel/time/tick-sched.c:164
tick_sched_timer+0x45/0x130 kernel/time/tick-sched.c:1274
__run_hrtimer kernel/time/hrtimer.c:1398 [inline]
__hrtimer_run_queues+0x41c/0x10d0 kernel/time/hrtimer.c:1460
hrtimer_interrupt+0x313/0x780 kernel/time/hrtimer.c:1518
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1034 [inline]
smp_apic_timer_interrupt+0x1a1/0x760 arch/x86/kernel/apic/apic.c:1059
apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:807
</IRQ>
RIP: 0010:__sanitizer_cov_trace_pc+0x4b/0x50 kernel/kcov.c:111
Code: 90 d8 12 00 00 83 fa 02 75 20 48 8b 88 e0 12 00 00 8b 80 dc 12 00 00
48 8b 11 48 83 c2 01 48 39 d0 76 07 48 89 34 d1 48 89 11 <5d> c3 0f 1f 00
55 40 0f b6 d6 40 0f b6 f7 31 ff 48 89 e5 48 8b 4d
RSP: 0018:ffff8881c1e17988 EFLAGS: 00000293 ORIG_RAX: ffffffffffffff13
RAX: ffff8881c2860200 RBX: 0000000000000004 RCX: ffffffff81ced656
RDX: 0000000000000000 RSI: ffffffff8344bfe8 RDI: ffff8881b4364cc0
RBP: ffff8881c1e17988 R08: ffff8881c2860200 R09: 0000000000000007
R10: 0000000000000000 R11: ffff8881c2860200 R12: ffff8881b4364cc0
R13: 0000000001237d36 R14: ffff8881b4364cc0 R15: 0000000000000004
security_file_permission+0x18/0x220 security/security.c:886
rw_verify_area+0x111/0x360 fs/read_write.c:391
do_splice_to+0x8b/0x190 fs/splice.c:868
splice_direct_to_actor+0x31c/0x9d0 fs/splice.c:957
do_splice_direct+0x2d4/0x420 fs/splice.c:1066
do_sendfile+0x62a/0xe20 fs/read_write.c:1439
__do_sys_sendfile64 fs/read_write.c:1494 [inline]
__se_sys_sendfile64 fs/read_write.c:1486 [inline]
__x64_sys_sendfile64+0x15d/0x250 fs/read_write.c:1486
do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x4400d9
Code: 18 89 d0 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 48 89 f8 48 89 f7
48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff
ff 0f 83 fb 13 fc ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007ffdd3e00e58 EFLAGS: 00000203 ORIG_RAX: 0000000000000028
RAX: ffffffffffffffda RBX: 00000000004002c8 RCX: 00000000004400d9
RDX: 0000000020000000 RSI: 0000000000000003 RDI: 0000000000000003
RBP: 00000000006ca018 R08: 00000000004002c8 R09: 00000000004002c8
R10: 00008080fffffffe R11: 0000000000000203 R12: 0000000000401960
R13: 00000000004019f0 R14: 0000000000000000 R15: 0000000000000000
---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.
syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with
syzbot.
syzbot can test patches for this bug, for details see:
https://goo.gl/tpsmEJ#testing-patches
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: INFO: rcu detected stall in sys_sendfile64
2018-12-19 9:27 INFO: rcu detected stall in sys_sendfile64 syzbot
@ 2018-12-19 10:11 ` Tetsuo Handa
2018-12-19 18:42 ` Dmitry Vyukov
0 siblings, 1 reply; 6+ messages in thread
From: Tetsuo Handa @ 2018-12-19 10:11 UTC (permalink / raw)
To: syzbot, syzkaller-bugs, Ingo Molnar, Peter Zijlstra
Cc: jmorris, linux-kernel, linux-security-module, serge
On 2018/12/19 18:27, syzbot wrote:
> HEAD commit: ddfbab46539f Merge tag 'scsi-fixes' of git://git.kernel.or..
> git tree: upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=15b87fa3400000
> kernel config: https://syzkaller.appspot.com/x/.config?x=861a3573f4e78ba1
> dashboard link: https://syzkaller.appspot.com/bug?extid=bcad772bbc241b4c6147
> compiler: gcc (GCC) 8.0.1 20180413 (experimental)
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13912ccd400000
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=145781db400000
This is not a LSM problem, for the reproducer is calling
sched_setattr(SCHED_DEADLINE) with very large values.
sched_setattr(0, {size=0, sched_policy=0x6 /* SCHED_??? */, sched_flags=0, sched_nice=0, sched_priority=0, sched_runtime=2251799813724439, sched_deadline=4611686018427453437, sched_period=0}, 0) = 0
I think that this problem is nothing but an insane sched_setattr() parameter.
#syz invalid
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: INFO: rcu detected stall in sys_sendfile64
2018-12-19 10:11 ` Tetsuo Handa
@ 2018-12-19 18:42 ` Dmitry Vyukov
2020-01-04 11:09 ` Tetsuo Handa
0 siblings, 1 reply; 6+ messages in thread
From: Dmitry Vyukov @ 2018-12-19 18:42 UTC (permalink / raw)
To: Tetsuo Handa
Cc: syzbot, syzkaller-bugs, Ingo Molnar, Peter Zijlstra,
James Morris, LKML, linux-security-module, Serge E. Hallyn
On Wed, Dec 19, 2018 at 11:13 AM Tetsuo Handa
<penguin-kernel@i-love.sakura.ne.jp> wrote:
>
> On 2018/12/19 18:27, syzbot wrote:
> > HEAD commit: ddfbab46539f Merge tag 'scsi-fixes' of git://git.kernel.or..
> > git tree: upstream
> > console output: https://syzkaller.appspot.com/x/log.txt?x=15b87fa3400000
> > kernel config: https://syzkaller.appspot.com/x/.config?x=861a3573f4e78ba1
> > dashboard link: https://syzkaller.appspot.com/bug?extid=bcad772bbc241b4c6147
> > compiler: gcc (GCC) 8.0.1 20180413 (experimental)
> > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13912ccd400000
> > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=145781db400000
>
> This is not a LSM problem, for the reproducer is calling
> sched_setattr(SCHED_DEADLINE) with very large values.
>
> sched_setattr(0, {size=0, sched_policy=0x6 /* SCHED_??? */, sched_flags=0, sched_nice=0, sched_priority=0, sched_runtime=2251799813724439, sched_deadline=4611686018427453437, sched_period=0}, 0) = 0
>
> I think that this problem is nothing but an insane sched_setattr() parameter.
>
> #syz invalid
Note there was another one with sched_setattr, which turned out to be
some serious problem in kernel (sched_setattr should not cause CPU
stall for 3 minutes):
INFO: rcu detected stall in do_idle
https://syzkaller.appspot.com/bug?extid=385468161961cee80c31
https://groups.google.com/forum/#!msg/syzkaller-bugs/crrfvusGtwI/IoD_zus4BgAJ
Maybe it another incarnation of the same bug, that one is still not fixed.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: INFO: rcu detected stall in sys_sendfile64
2018-12-19 18:42 ` Dmitry Vyukov
@ 2020-01-04 11:09 ` Tetsuo Handa
2020-01-07 13:02 ` Dmitry Vyukov
0 siblings, 1 reply; 6+ messages in thread
From: Tetsuo Handa @ 2020-01-04 11:09 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: syzbot, syzkaller-bugs, Ingo Molnar, Peter Zijlstra,
James Morris, LKML, linux-security-module, Serge E. Hallyn
On 2018/12/20 3:42, Dmitry Vyukov wrote:
> On Wed, Dec 19, 2018 at 11:13 AM Tetsuo Handa
> <penguin-kernel@i-love.sakura.ne.jp> wrote:
>>
>> On 2018/12/19 18:27, syzbot wrote:
>>> HEAD commit: ddfbab46539f Merge tag 'scsi-fixes' of git://git.kernel.or..
>>> git tree: upstream
>>> console output: https://syzkaller.appspot.com/x/log.txt?x=15b87fa3400000
>>> kernel config: https://syzkaller.appspot.com/x/.config?x=861a3573f4e78ba1
>>> dashboard link: https://syzkaller.appspot.com/bug?extid=bcad772bbc241b4c6147
>>> compiler: gcc (GCC) 8.0.1 20180413 (experimental)
>>> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13912ccd400000
>>> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=145781db400000
>>
>> This is not a LSM problem, for the reproducer is calling
>> sched_setattr(SCHED_DEADLINE) with very large values.
>>
>> sched_setattr(0, {size=0, sched_policy=0x6 /* SCHED_??? */, sched_flags=0, sched_nice=0, sched_priority=0, sched_runtime=2251799813724439, sched_deadline=4611686018427453437, sched_period=0}, 0) = 0
>>
>> I think that this problem is nothing but an insane sched_setattr() parameter.
>>
>> #syz invalid
>
> Note there was another one with sched_setattr, which turned out to be
> some serious problem in kernel (sched_setattr should not cause CPU
> stall for 3 minutes):
> INFO: rcu detected stall in do_idle
> https://syzkaller.appspot.com/bug?extid=385468161961cee80c31
> https://groups.google.com/forum/#!msg/syzkaller-bugs/crrfvusGtwI/IoD_zus4BgAJ
>
> Maybe it another incarnation of the same bug, that one is still not fixed.
>
Can we let syzbot blacklist sched_setattr() for now? There are many stall reports
doing sched_setattr(SCHED_RR) which makes it difficult to find stall reports not
using sched_setattr().
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: INFO: rcu detected stall in sys_sendfile64
2020-01-04 11:09 ` Tetsuo Handa
@ 2020-01-07 13:02 ` Dmitry Vyukov
2020-01-07 13:11 ` Peter Zijlstra
0 siblings, 1 reply; 6+ messages in thread
From: Dmitry Vyukov @ 2020-01-07 13:02 UTC (permalink / raw)
To: Tetsuo Handa
Cc: syzbot, syzkaller-bugs, Ingo Molnar, Peter Zijlstra,
James Morris, LKML, linux-security-module, Serge E. Hallyn,
syzkaller
On Sat, Jan 4, 2020 at 12:09 PM Tetsuo Handa
<penguin-kernel@i-love.sakura.ne.jp> wrote:
>
> On 2018/12/20 3:42, Dmitry Vyukov wrote:
> > On Wed, Dec 19, 2018 at 11:13 AM Tetsuo Handa
> > <penguin-kernel@i-love.sakura.ne.jp> wrote:
> >>
> >> On 2018/12/19 18:27, syzbot wrote:
> >>> HEAD commit: ddfbab46539f Merge tag 'scsi-fixes' of git://git.kernel.or..
> >>> git tree: upstream
> >>> console output: https://syzkaller.appspot.com/x/log.txt?x=15b87fa3400000
> >>> kernel config: https://syzkaller.appspot.com/x/.config?x=861a3573f4e78ba1
> >>> dashboard link: https://syzkaller.appspot.com/bug?extid=bcad772bbc241b4c6147
> >>> compiler: gcc (GCC) 8.0.1 20180413 (experimental)
> >>> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13912ccd400000
> >>> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=145781db400000
> >>
> >> This is not a LSM problem, for the reproducer is calling
> >> sched_setattr(SCHED_DEADLINE) with very large values.
> >>
> >> sched_setattr(0, {size=0, sched_policy=0x6 /* SCHED_??? */, sched_flags=0, sched_nice=0, sched_priority=0, sched_runtime=2251799813724439, sched_deadline=4611686018427453437, sched_period=0}, 0) = 0
> >>
> >> I think that this problem is nothing but an insane sched_setattr() parameter.
> >>
> >> #syz invalid
> >
> > Note there was another one with sched_setattr, which turned out to be
> > some serious problem in kernel (sched_setattr should not cause CPU
> > stall for 3 minutes):
> > INFO: rcu detected stall in do_idle
> > https://syzkaller.appspot.com/bug?extid=385468161961cee80c31
> > https://groups.google.com/forum/#!msg/syzkaller-bugs/crrfvusGtwI/IoD_zus4BgAJ
> >
> > Maybe it another incarnation of the same bug, that one is still not fixed.
> >
>
> Can we let syzbot blacklist sched_setattr() for now? There are many stall reports
> doing sched_setattr(SCHED_RR) which makes it difficult to find stall reports not
> using sched_setattr().
Hi Tetsuo,
If we start practice of disabling whole syscalls, I would really like
"for now" to be very well defined. When will it end? How will it
happen? Is the problem on the radar of relevant people? Will it stay
on somebody's radar until it's fixed? Normal practise of project
sheriffing is to file a P1 bug assigned to somebody when something
gets disabled. But I am not sure how we implement this for kernel.
Since the problem is there for a long time and we disable it without
defining any criteria, I afraid we disable it forever (then more bugs
will pile and re-enabling it will be painful). At the very least we
need to acknowledge that we stopping testing schedler for foreseeable
future and schedler maintainers need to be notified about this.
Blacklisting it and un-blacklisting will cause some churn. Was the bug
given at least some attention? Significant number of bugs are
relatively easy to fix and fixing it would solve all of the problems
in a much better way.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: INFO: rcu detected stall in sys_sendfile64
2020-01-07 13:02 ` Dmitry Vyukov
@ 2020-01-07 13:11 ` Peter Zijlstra
0 siblings, 0 replies; 6+ messages in thread
From: Peter Zijlstra @ 2020-01-07 13:11 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: Tetsuo Handa, syzbot, syzkaller-bugs, Ingo Molnar, James Morris,
LKML, linux-security-module, Serge E. Hallyn, syzkaller
On Tue, Jan 07, 2020 at 02:02:47PM +0100, Dmitry Vyukov wrote:
> > >> This is not a LSM problem, for the reproducer is calling
> > >> sched_setattr(SCHED_DEADLINE) with very large values.
> > >>
> > >> sched_setattr(0, {size=0, sched_policy=0x6 /* SCHED_??? */, sched_flags=0, sched_nice=0, sched_priority=0, sched_runtime=2251799813724439, sched_deadline=4611686018427453437, sched_period=0}, 0) = 0
> > >>
> > >> I think that this problem is nothing but an insane sched_setattr() parameter.
Argh, I had a patch for that somewhere. Let me go dig that out.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-01-07 13:12 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-12-19 9:27 INFO: rcu detected stall in sys_sendfile64 syzbot
2018-12-19 10:11 ` Tetsuo Handa
2018-12-19 18:42 ` Dmitry Vyukov
2020-01-04 11:09 ` Tetsuo Handa
2020-01-07 13:02 ` Dmitry Vyukov
2020-01-07 13:11 ` Peter Zijlstra
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).