* [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-10 16:46 syzbot
2021-03-10 17:16 ` Dmitry Vyukov
0 siblings, 1 reply; 51+ messages in thread
From: syzbot @ 2021-03-10 16:46 UTC (permalink / raw)
To: bristot, bsegall, dietmar.eggemann, juri.lelli, linux-kernel,
mgorman, mingo, peterz, rostedt, syzkaller-bugs, vincent.guittot
Hello,
syzbot found the following issue on:
HEAD commit: 0d7588ab riscv: process: Fix no prototype for arch_dup_tas..
git tree: git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
console output: https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
kernel config: https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
dashboard link: https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
userspace arch: riscv64
Unfortunately, I don't have any reproducer for this issue yet.
IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
Unable to handle kernel access to user memory without uaccess routines at virtual address 000000002749f0d0
Oops [#1]
Modules linked in:
CPU: 1 PID: 4875 Comm: syz-executor.0 Not tainted 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0
Hardware name: riscv-virtio,qemu (DT)
epc : schedule_tail+0x72/0xb2 kernel/sched/core.c:4264
ra : task_pid_vnr include/linux/sched.h:1421 [inline]
ra : schedule_tail+0x70/0xb2 kernel/sched/core.c:4264
epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp : ffffffe025d17ec0
gp : ffffffe005d25378 tp : ffffffe00f0d0000 t0 : 0000000000000000
t1 : 0000000000000001 t2 : 00000000000f4240 s0 : ffffffe025d17ee0
s1 : 000000002749f0d0 a0 : 000000000000002a a1 : 0000000000000003
a2 : 1ffffffc0cfac500 a3 : ffffffe0000c80cc a4 : 5ae9db91c19bbe00
a5 : 0000000000000000 a6 : 0000000000f00000 a7 : ffffffe000082eba
s2 : 0000000000040000 s3 : ffffffe00eef96c0 s4 : ffffffe022c77fe0
s5 : 0000000000004000 s6 : ffffffe067d74e00 s7 : ffffffe067d74850
s8 : ffffffe067d73e18 s9 : ffffffe067d74e00 s10: ffffffe00eef96e8
s11: 000000ae6cdf8368 t3 : 5ae9db91c19bbe00 t4 : ffffffc4043cafb2
t5 : ffffffc4043cafba t6 : 0000000000040000
status: 0000000000000120 badaddr: 000000002749f0d0 cause: 000000000000000f
Call Trace:
[<ffffffe00008c8b0>] schedule_tail+0x72/0xb2 kernel/sched/core.c:4264
[<ffffffe000005570>] ret_from_exception+0x0/0x14
Dumping ftrace buffer:
(ftrace buffer empty)
---[ end trace b5f8f9231dc87dda ]---
---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.
syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-10 16:46 [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail syzbot
@ 2021-03-10 17:16 ` Dmitry Vyukov
0 siblings, 0 replies; 51+ messages in thread
From: Dmitry Vyukov @ 2021-03-10 17:16 UTC (permalink / raw)
To: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv
Cc: Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
On Wed, Mar 10, 2021 at 5:46 PM syzbot
<syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit: 0d7588ab riscv: process: Fix no prototype for arch_dup_tas..
> git tree: git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
> console output: https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
> kernel config: https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
> dashboard link: https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
> userspace arch: riscv64
>
> Unfortunately, I don't have any reproducer for this issue yet.
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
+riscv maintainers
This is riscv64-specific.
I've seen similar crashes in put_user in other places. It looks like
put_user crashes in the user address is not mapped/protected (?).
> Unable to handle kernel access to user memory without uaccess routines at virtual address 000000002749f0d0
> Oops [#1]
> Modules linked in:
> CPU: 1 PID: 4875 Comm: syz-executor.0 Not tainted 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0
> Hardware name: riscv-virtio,qemu (DT)
> epc : schedule_tail+0x72/0xb2 kernel/sched/core.c:4264
> ra : task_pid_vnr include/linux/sched.h:1421 [inline]
> ra : schedule_tail+0x70/0xb2 kernel/sched/core.c:4264
> epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp : ffffffe025d17ec0
> gp : ffffffe005d25378 tp : ffffffe00f0d0000 t0 : 0000000000000000
> t1 : 0000000000000001 t2 : 00000000000f4240 s0 : ffffffe025d17ee0
> s1 : 000000002749f0d0 a0 : 000000000000002a a1 : 0000000000000003
> a2 : 1ffffffc0cfac500 a3 : ffffffe0000c80cc a4 : 5ae9db91c19bbe00
> a5 : 0000000000000000 a6 : 0000000000f00000 a7 : ffffffe000082eba
> s2 : 0000000000040000 s3 : ffffffe00eef96c0 s4 : ffffffe022c77fe0
> s5 : 0000000000004000 s6 : ffffffe067d74e00 s7 : ffffffe067d74850
> s8 : ffffffe067d73e18 s9 : ffffffe067d74e00 s10: ffffffe00eef96e8
> s11: 000000ae6cdf8368 t3 : 5ae9db91c19bbe00 t4 : ffffffc4043cafb2
> t5 : ffffffc4043cafba t6 : 0000000000040000
> status: 0000000000000120 badaddr: 000000002749f0d0 cause: 000000000000000f
> Call Trace:
> [<ffffffe00008c8b0>] schedule_tail+0x72/0xb2 kernel/sched/core.c:4264
> [<ffffffe000005570>] ret_from_exception+0x0/0x14
> Dumping ftrace buffer:
> (ftrace buffer empty)
> ---[ end trace b5f8f9231dc87dda ]---
>
>
> ---
> This report is generated by a bot. It may contain errors.
> See https://goo.gl/tpsmEJ for more information about syzbot.
> syzbot engineers can be reached at syzkaller@googlegroups.com.
>
> syzbot will keep track of this issue. See:
> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/000000000000b74f1b05bd316729%40google.com.
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-10 17:16 ` Dmitry Vyukov
0 siblings, 0 replies; 51+ messages in thread
From: Dmitry Vyukov @ 2021-03-10 17:16 UTC (permalink / raw)
To: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv
Cc: Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
On Wed, Mar 10, 2021 at 5:46 PM syzbot
<syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit: 0d7588ab riscv: process: Fix no prototype for arch_dup_tas..
> git tree: git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
> console output: https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
> kernel config: https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
> dashboard link: https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
> userspace arch: riscv64
>
> Unfortunately, I don't have any reproducer for this issue yet.
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
+riscv maintainers
This is riscv64-specific.
I've seen similar crashes in put_user in other places. It looks like
put_user crashes in the user address is not mapped/protected (?).
> Unable to handle kernel access to user memory without uaccess routines at virtual address 000000002749f0d0
> Oops [#1]
> Modules linked in:
> CPU: 1 PID: 4875 Comm: syz-executor.0 Not tainted 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0
> Hardware name: riscv-virtio,qemu (DT)
> epc : schedule_tail+0x72/0xb2 kernel/sched/core.c:4264
> ra : task_pid_vnr include/linux/sched.h:1421 [inline]
> ra : schedule_tail+0x70/0xb2 kernel/sched/core.c:4264
> epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp : ffffffe025d17ec0
> gp : ffffffe005d25378 tp : ffffffe00f0d0000 t0 : 0000000000000000
> t1 : 0000000000000001 t2 : 00000000000f4240 s0 : ffffffe025d17ee0
> s1 : 000000002749f0d0 a0 : 000000000000002a a1 : 0000000000000003
> a2 : 1ffffffc0cfac500 a3 : ffffffe0000c80cc a4 : 5ae9db91c19bbe00
> a5 : 0000000000000000 a6 : 0000000000f00000 a7 : ffffffe000082eba
> s2 : 0000000000040000 s3 : ffffffe00eef96c0 s4 : ffffffe022c77fe0
> s5 : 0000000000004000 s6 : ffffffe067d74e00 s7 : ffffffe067d74850
> s8 : ffffffe067d73e18 s9 : ffffffe067d74e00 s10: ffffffe00eef96e8
> s11: 000000ae6cdf8368 t3 : 5ae9db91c19bbe00 t4 : ffffffc4043cafb2
> t5 : ffffffc4043cafba t6 : 0000000000040000
> status: 0000000000000120 badaddr: 000000002749f0d0 cause: 000000000000000f
> Call Trace:
> [<ffffffe00008c8b0>] schedule_tail+0x72/0xb2 kernel/sched/core.c:4264
> [<ffffffe000005570>] ret_from_exception+0x0/0x14
> Dumping ftrace buffer:
> (ftrace buffer empty)
> ---[ end trace b5f8f9231dc87dda ]---
>
>
> ---
> This report is generated by a bot. It may contain errors.
> See https://goo.gl/tpsmEJ for more information about syzbot.
> syzbot engineers can be reached at syzkaller@googlegroups.com.
>
> syzbot will keep track of this issue. See:
> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/000000000000b74f1b05bd316729%40google.com.
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-10 17:16 ` Dmitry Vyukov
@ 2021-03-10 22:24 ` Ben Dooks
-1 siblings, 0 replies; 51+ messages in thread
From: Ben Dooks @ 2021-03-10 22:24 UTC (permalink / raw)
To: Dmitry Vyukov, syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou,
linux-riscv
Cc: Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
On 10/03/2021 17:16, Dmitry Vyukov wrote:
> On Wed, Mar 10, 2021 at 5:46 PM syzbot
> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>
>> Hello,
>>
>> syzbot found the following issue on:
>>
>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for arch_dup_tas..
>> git tree: git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>> console output: https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>> kernel config: https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>> dashboard link: https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>> userspace arch: riscv64
>>
>> Unfortunately, I don't have any reproducer for this issue yet.
>>
>> IMPORTANT: if you fix the issue, please add the following tag to the commit:
>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>
> +riscv maintainers
>
> This is riscv64-specific.
> I've seen similar crashes in put_user in other places. It looks like
> put_user crashes in the user address is not mapped/protected (?).
The unmapped case should have been handled.
I think this issue is that the check for user-mode access added. From
what I read the code may be wrong in
+ if (!user_mode(regs) && addr < TASK_SIZE &&
+ unlikely(!(regs->status & SR_SUM)))
+ die_kernel_fault("access to user memory without uaccess routines",
+ addr, regs);
I think the SR_SUM check might be wrong, as I read the standard the
SR_SUM should be set to disable user-space access. So the check
should be unlikely(regs->status & SR_SUM) to say access without
having disabled the protection.
Without this, you can end up with an infinite loop in the fault handler.
>
>> Unable to handle kernel access to user memory without uaccess routines at virtual address 000000002749f0d0
>> Oops [#1]
>> Modules linked in:
>> CPU: 1 PID: 4875 Comm: syz-executor.0 Not tainted 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0
>> Hardware name: riscv-virtio,qemu (DT)
>> epc : schedule_tail+0x72/0xb2 kernel/sched/core.c:4264
>> ra : task_pid_vnr include/linux/sched.h:1421 [inline]
>> ra : schedule_tail+0x70/0xb2 kernel/sched/core.c:4264
>> epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp : ffffffe025d17ec0
>> gp : ffffffe005d25378 tp : ffffffe00f0d0000 t0 : 0000000000000000
>> t1 : 0000000000000001 t2 : 00000000000f4240 s0 : ffffffe025d17ee0
>> s1 : 000000002749f0d0 a0 : 000000000000002a a1 : 0000000000000003
>> a2 : 1ffffffc0cfac500 a3 : ffffffe0000c80cc a4 : 5ae9db91c19bbe00
>> a5 : 0000000000000000 a6 : 0000000000f00000 a7 : ffffffe000082eba
>> s2 : 0000000000040000 s3 : ffffffe00eef96c0 s4 : ffffffe022c77fe0
>> s5 : 0000000000004000 s6 : ffffffe067d74e00 s7 : ffffffe067d74850
>> s8 : ffffffe067d73e18 s9 : ffffffe067d74e00 s10: ffffffe00eef96e8
>> s11: 000000ae6cdf8368 t3 : 5ae9db91c19bbe00 t4 : ffffffc4043cafb2
>> t5 : ffffffc4043cafba t6 : 0000000000040000
>> status: 0000000000000120 badaddr: 000000002749f0d0 cause: 000000000000000f
>> Call Trace:
>> [<ffffffe00008c8b0>] schedule_tail+0x72/0xb2 kernel/sched/core.c:4264
>> [<ffffffe000005570>] ret_from_exception+0x0/0x14
>> Dumping ftrace buffer:
>> (ftrace buffer empty)
>> ---[ end trace b5f8f9231dc87dda ]---
>>
>>
>> ---
>> This report is generated by a bot. It may contain errors.
>> See https://goo.gl/tpsmEJ for more information about syzbot.
>> syzbot engineers can be reached at syzkaller@googlegroups.com.
>>
>> syzbot will keep track of this issue. See:
>> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
>>
>> --
>> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/000000000000b74f1b05bd316729%40google.com.
>
> _______________________________________________
> linux-riscv mailing list
> linux-riscv@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-riscv
>
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-10 22:24 ` Ben Dooks
0 siblings, 0 replies; 51+ messages in thread
From: Ben Dooks @ 2021-03-10 22:24 UTC (permalink / raw)
To: Dmitry Vyukov, syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou,
linux-riscv
Cc: Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
On 10/03/2021 17:16, Dmitry Vyukov wrote:
> On Wed, Mar 10, 2021 at 5:46 PM syzbot
> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>
>> Hello,
>>
>> syzbot found the following issue on:
>>
>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for arch_dup_tas..
>> git tree: git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>> console output: https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>> kernel config: https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>> dashboard link: https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>> userspace arch: riscv64
>>
>> Unfortunately, I don't have any reproducer for this issue yet.
>>
>> IMPORTANT: if you fix the issue, please add the following tag to the commit:
>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>
> +riscv maintainers
>
> This is riscv64-specific.
> I've seen similar crashes in put_user in other places. It looks like
> put_user crashes in the user address is not mapped/protected (?).
The unmapped case should have been handled.
I think this issue is that the check for user-mode access added. From
what I read the code may be wrong in
+ if (!user_mode(regs) && addr < TASK_SIZE &&
+ unlikely(!(regs->status & SR_SUM)))
+ die_kernel_fault("access to user memory without uaccess routines",
+ addr, regs);
I think the SR_SUM check might be wrong, as I read the standard the
SR_SUM should be set to disable user-space access. So the check
should be unlikely(regs->status & SR_SUM) to say access without
having disabled the protection.
Without this, you can end up with an infinite loop in the fault handler.
>
>> Unable to handle kernel access to user memory without uaccess routines at virtual address 000000002749f0d0
>> Oops [#1]
>> Modules linked in:
>> CPU: 1 PID: 4875 Comm: syz-executor.0 Not tainted 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0
>> Hardware name: riscv-virtio,qemu (DT)
>> epc : schedule_tail+0x72/0xb2 kernel/sched/core.c:4264
>> ra : task_pid_vnr include/linux/sched.h:1421 [inline]
>> ra : schedule_tail+0x70/0xb2 kernel/sched/core.c:4264
>> epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp : ffffffe025d17ec0
>> gp : ffffffe005d25378 tp : ffffffe00f0d0000 t0 : 0000000000000000
>> t1 : 0000000000000001 t2 : 00000000000f4240 s0 : ffffffe025d17ee0
>> s1 : 000000002749f0d0 a0 : 000000000000002a a1 : 0000000000000003
>> a2 : 1ffffffc0cfac500 a3 : ffffffe0000c80cc a4 : 5ae9db91c19bbe00
>> a5 : 0000000000000000 a6 : 0000000000f00000 a7 : ffffffe000082eba
>> s2 : 0000000000040000 s3 : ffffffe00eef96c0 s4 : ffffffe022c77fe0
>> s5 : 0000000000004000 s6 : ffffffe067d74e00 s7 : ffffffe067d74850
>> s8 : ffffffe067d73e18 s9 : ffffffe067d74e00 s10: ffffffe00eef96e8
>> s11: 000000ae6cdf8368 t3 : 5ae9db91c19bbe00 t4 : ffffffc4043cafb2
>> t5 : ffffffc4043cafba t6 : 0000000000040000
>> status: 0000000000000120 badaddr: 000000002749f0d0 cause: 000000000000000f
>> Call Trace:
>> [<ffffffe00008c8b0>] schedule_tail+0x72/0xb2 kernel/sched/core.c:4264
>> [<ffffffe000005570>] ret_from_exception+0x0/0x14
>> Dumping ftrace buffer:
>> (ftrace buffer empty)
>> ---[ end trace b5f8f9231dc87dda ]---
>>
>>
>> ---
>> This report is generated by a bot. It may contain errors.
>> See https://goo.gl/tpsmEJ for more information about syzbot.
>> syzbot engineers can be reached at syzkaller@googlegroups.com.
>>
>> syzbot will keep track of this issue. See:
>> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
>>
>> --
>> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/000000000000b74f1b05bd316729%40google.com.
>
> _______________________________________________
> linux-riscv mailing list
> linux-riscv@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-riscv
>
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-10 22:24 ` Ben Dooks
@ 2021-03-11 6:39 ` Alex Ghiti
-1 siblings, 0 replies; 51+ messages in thread
From: Alex Ghiti @ 2021-03-11 6:39 UTC (permalink / raw)
To: Ben Dooks, Dmitry Vyukov, syzbot, Paul Walmsley, Palmer Dabbelt,
Albert Ou, linux-riscv
Cc: Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
Hi Ben,
Le 3/10/21 à 5:24 PM, Ben Dooks a écrit :
> On 10/03/2021 17:16, Dmitry Vyukov wrote:
>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>>
>>> Hello,
>>>
>>> syzbot found the following issue on:
>>>
>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
>>> arch_dup_tas..
>>> git tree:
>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>>> console output: https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>>> kernel config:
>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>>> dashboard link:
>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>>> userspace arch: riscv64
>>>
>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>
>>> IMPORTANT: if you fix the issue, please add the following tag to the
>>> commit:
>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>>
>> +riscv maintainers
>>
>> This is riscv64-specific.
>> I've seen similar crashes in put_user in other places. It looks like
>> put_user crashes in the user address is not mapped/protected (?).
>
> The unmapped case should have been handled.
>
> I think this issue is that the check for user-mode access added. From
> what I read the code may be wrong in
>
> + if (!user_mode(regs) && addr < TASK_SIZE &&
> + unlikely(!(regs->status & SR_SUM)))
> + die_kernel_fault("access to user memory without uaccess routines",
> + addr, regs);
>
> I think the SR_SUM check might be wrong, as I read the standard the
> SR_SUM should be set to disable user-space access. So the check
> should be unlikely(regs->status & SR_SUM) to say access without
> having disabled the protection.
The check that is done seems correct to me: "The SUM (permit Supervisor
User Memory access) bit modifies the privilege with which S-mode loads
and stores access virtual memory. *When SUM=0, S-mode memory accesses
to pages that are accessible by U-mode (U=1 in Figure 4.15) will fault*.
When SUM=1, these accesses are permitted.SUM has no effect when
page-based virtual memory is not in effect".
I will try to reproduce the problem locally.
Thanks,
Alex
>
> Without this, you can end up with an infinite loop in the fault handler.
>
>>
>>> Unable to handle kernel access to user memory without uaccess
>>> routines at virtual address 000000002749f0d0
>>> Oops [#1]
>>> Modules linked in:
>>> CPU: 1 PID: 4875 Comm: syz-executor.0 Not tainted
>>> 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0
>>> Hardware name: riscv-virtio,qemu (DT)
>>> epc : schedule_tail+0x72/0xb2 kernel/sched/core.c:4264
>>> ra : task_pid_vnr include/linux/sched.h:1421 [inline]
>>> ra : schedule_tail+0x70/0xb2 kernel/sched/core.c:4264
>>> epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp : ffffffe025d17ec0
>>> gp : ffffffe005d25378 tp : ffffffe00f0d0000 t0 : 0000000000000000
>>> t1 : 0000000000000001 t2 : 00000000000f4240 s0 : ffffffe025d17ee0
>>> s1 : 000000002749f0d0 a0 : 000000000000002a a1 : 0000000000000003
>>> a2 : 1ffffffc0cfac500 a3 : ffffffe0000c80cc a4 : 5ae9db91c19bbe00
>>> a5 : 0000000000000000 a6 : 0000000000f00000 a7 : ffffffe000082eba
>>> s2 : 0000000000040000 s3 : ffffffe00eef96c0 s4 : ffffffe022c77fe0
>>> s5 : 0000000000004000 s6 : ffffffe067d74e00 s7 : ffffffe067d74850
>>> s8 : ffffffe067d73e18 s9 : ffffffe067d74e00 s10: ffffffe00eef96e8
>>> s11: 000000ae6cdf8368 t3 : 5ae9db91c19bbe00 t4 : ffffffc4043cafb2
>>> t5 : ffffffc4043cafba t6 : 0000000000040000
>>> status: 0000000000000120 badaddr: 000000002749f0d0 cause:
>>> 000000000000000f
>>> Call Trace:
>>> [<ffffffe00008c8b0>] schedule_tail+0x72/0xb2 kernel/sched/core.c:4264
>>> [<ffffffe000005570>] ret_from_exception+0x0/0x14
>>> Dumping ftrace buffer:
>>> (ftrace buffer empty)
>>> ---[ end trace b5f8f9231dc87dda ]---
>>>
>>>
>>> ---
>>> This report is generated by a bot. It may contain errors.
>>> See https://goo.gl/tpsmEJ for more information about syzbot.
>>> syzbot engineers can be reached at syzkaller@googlegroups.com.
>>>
>>> syzbot will keep track of this issue. See:
>>> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "syzkaller-bugs" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>>> send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/syzkaller-bugs/000000000000b74f1b05bd316729%40google.com.
>>>
>>
>> _______________________________________________
>> linux-riscv mailing list
>> linux-riscv@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-riscv
>>
>
>
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-11 6:39 ` Alex Ghiti
0 siblings, 0 replies; 51+ messages in thread
From: Alex Ghiti @ 2021-03-11 6:39 UTC (permalink / raw)
To: Ben Dooks, Dmitry Vyukov, syzbot, Paul Walmsley, Palmer Dabbelt,
Albert Ou, linux-riscv
Cc: Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
Hi Ben,
Le 3/10/21 à 5:24 PM, Ben Dooks a écrit :
> On 10/03/2021 17:16, Dmitry Vyukov wrote:
>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>>
>>> Hello,
>>>
>>> syzbot found the following issue on:
>>>
>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
>>> arch_dup_tas..
>>> git tree:
>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>>> console output: https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>>> kernel config:
>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>>> dashboard link:
>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>>> userspace arch: riscv64
>>>
>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>
>>> IMPORTANT: if you fix the issue, please add the following tag to the
>>> commit:
>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>>
>> +riscv maintainers
>>
>> This is riscv64-specific.
>> I've seen similar crashes in put_user in other places. It looks like
>> put_user crashes in the user address is not mapped/protected (?).
>
> The unmapped case should have been handled.
>
> I think this issue is that the check for user-mode access added. From
> what I read the code may be wrong in
>
> + if (!user_mode(regs) && addr < TASK_SIZE &&
> + unlikely(!(regs->status & SR_SUM)))
> + die_kernel_fault("access to user memory without uaccess routines",
> + addr, regs);
>
> I think the SR_SUM check might be wrong, as I read the standard the
> SR_SUM should be set to disable user-space access. So the check
> should be unlikely(regs->status & SR_SUM) to say access without
> having disabled the protection.
The check that is done seems correct to me: "The SUM (permit Supervisor
User Memory access) bit modifies the privilege with which S-mode loads
and stores access virtual memory. *When SUM=0, S-mode memory accesses
to pages that are accessible by U-mode (U=1 in Figure 4.15) will fault*.
When SUM=1, these accesses are permitted.SUM has no effect when
page-based virtual memory is not in effect".
I will try to reproduce the problem locally.
Thanks,
Alex
>
> Without this, you can end up with an infinite loop in the fault handler.
>
>>
>>> Unable to handle kernel access to user memory without uaccess
>>> routines at virtual address 000000002749f0d0
>>> Oops [#1]
>>> Modules linked in:
>>> CPU: 1 PID: 4875 Comm: syz-executor.0 Not tainted
>>> 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0
>>> Hardware name: riscv-virtio,qemu (DT)
>>> epc : schedule_tail+0x72/0xb2 kernel/sched/core.c:4264
>>> ra : task_pid_vnr include/linux/sched.h:1421 [inline]
>>> ra : schedule_tail+0x70/0xb2 kernel/sched/core.c:4264
>>> epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp : ffffffe025d17ec0
>>> gp : ffffffe005d25378 tp : ffffffe00f0d0000 t0 : 0000000000000000
>>> t1 : 0000000000000001 t2 : 00000000000f4240 s0 : ffffffe025d17ee0
>>> s1 : 000000002749f0d0 a0 : 000000000000002a a1 : 0000000000000003
>>> a2 : 1ffffffc0cfac500 a3 : ffffffe0000c80cc a4 : 5ae9db91c19bbe00
>>> a5 : 0000000000000000 a6 : 0000000000f00000 a7 : ffffffe000082eba
>>> s2 : 0000000000040000 s3 : ffffffe00eef96c0 s4 : ffffffe022c77fe0
>>> s5 : 0000000000004000 s6 : ffffffe067d74e00 s7 : ffffffe067d74850
>>> s8 : ffffffe067d73e18 s9 : ffffffe067d74e00 s10: ffffffe00eef96e8
>>> s11: 000000ae6cdf8368 t3 : 5ae9db91c19bbe00 t4 : ffffffc4043cafb2
>>> t5 : ffffffc4043cafba t6 : 0000000000040000
>>> status: 0000000000000120 badaddr: 000000002749f0d0 cause:
>>> 000000000000000f
>>> Call Trace:
>>> [<ffffffe00008c8b0>] schedule_tail+0x72/0xb2 kernel/sched/core.c:4264
>>> [<ffffffe000005570>] ret_from_exception+0x0/0x14
>>> Dumping ftrace buffer:
>>> (ftrace buffer empty)
>>> ---[ end trace b5f8f9231dc87dda ]---
>>>
>>>
>>> ---
>>> This report is generated by a bot. It may contain errors.
>>> See https://goo.gl/tpsmEJ for more information about syzbot.
>>> syzbot engineers can be reached at syzkaller@googlegroups.com.
>>>
>>> syzbot will keep track of this issue. See:
>>> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "syzkaller-bugs" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>>> send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/syzkaller-bugs/000000000000b74f1b05bd316729%40google.com.
>>>
>>
>> _______________________________________________
>> linux-riscv mailing list
>> linux-riscv@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-riscv
>>
>
>
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-11 6:39 ` Alex Ghiti
@ 2021-03-11 6:50 ` Dmitry Vyukov
-1 siblings, 0 replies; 51+ messages in thread
From: Dmitry Vyukov @ 2021-03-11 6:50 UTC (permalink / raw)
To: Alex Ghiti
Cc: Ben Dooks, syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou,
linux-riscv, Daniel Bristot de Oliveira, Benjamin Segall,
dietmar.eggemann, Juri Lelli, LKML, Mel Gorman, Ingo Molnar,
Peter Zijlstra, Steven Rostedt, syzkaller-bugs, Vincent Guittot
On Thu, Mar 11, 2021 at 7:40 AM Alex Ghiti <alex@ghiti.fr> wrote:
>
> Hi Ben,
>
> Le 3/10/21 à 5:24 PM, Ben Dooks a écrit :
> > On 10/03/2021 17:16, Dmitry Vyukov wrote:
> >> On Wed, Mar 10, 2021 at 5:46 PM syzbot
> >> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
> >>>
> >>> Hello,
> >>>
> >>> syzbot found the following issue on:
> >>>
> >>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
> >>> arch_dup_tas..
> >>> git tree:
> >>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
> >>> console output: https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
> >>> kernel config:
> >>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
> >>> dashboard link:
> >>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
> >>> userspace arch: riscv64
> >>>
> >>> Unfortunately, I don't have any reproducer for this issue yet.
> >>>
> >>> IMPORTANT: if you fix the issue, please add the following tag to the
> >>> commit:
> >>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
> >>
> >> +riscv maintainers
> >>
> >> This is riscv64-specific.
> >> I've seen similar crashes in put_user in other places. It looks like
> >> put_user crashes in the user address is not mapped/protected (?).
> >
> > The unmapped case should have been handled.
> >
> > I think this issue is that the check for user-mode access added. From
> > what I read the code may be wrong in
> >
> > + if (!user_mode(regs) && addr < TASK_SIZE &&
> > + unlikely(!(regs->status & SR_SUM)))
> > + die_kernel_fault("access to user memory without uaccess routines",
> > + addr, regs);
> >
> > I think the SR_SUM check might be wrong, as I read the standard the
> > SR_SUM should be set to disable user-space access. So the check
> > should be unlikely(regs->status & SR_SUM) to say access without
> > having disabled the protection.
>
> The check that is done seems correct to me: "The SUM (permit Supervisor
> User Memory access) bit modifies the privilege with which S-mode loads
> and stores access virtual memory. *When SUM=0, S-mode memory accesses
> to pages that are accessible by U-mode (U=1 in Figure 4.15) will fault*.
> When SUM=1, these accesses are permitted.SUM has no effect when
> page-based virtual memory is not in effect".
>
> I will try to reproduce the problem locally.
Weird. It crashes with this all the time:
https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
Even on trivial programs that almost don't do anything.
Maybe it's qemu bug? Do registers look sane in the dump? That SR_SUM, etc.
00:13:27 executing program 1:
openat$drirender128(0xffffffffffffff9c,
&(0x7f0000000040)='/dev/dri/renderD128\x00', 0x0, 0x0)
[ 812.318182][ T4833] Unable to handle kernel access to user memory
without uaccess routines at virtual address 00000000250b60d0
[ 812.322304][ T4833] Oops [#1]
[ 812.323196][ T4833] Modules linked in:
[ 812.324110][ T4833] CPU: 1 PID: 4833 Comm: syz-executor.1 Not
tainted 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0
[ 812.325862][ T4833] Hardware name: riscv-virtio,qemu (DT)
[ 812.327561][ T4833] epc : schedule_tail+0x72/0xb2
[ 812.328640][ T4833] ra : schedule_tail+0x70/0xb2
[ 812.330088][ T4833] epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp
: ffffffe0238bbec0
[ 812.331312][ T4833] gp : ffffffe005d25378 tp : ffffffe00a275b00 t0
: 0000000000000000
[ 812.333014][ T4833] t1 : 0000000000000001 t2 : 00000000000f4240 s0
: ffffffe0238bbee0
[ 812.334137][ T4833] s1 : 00000000250b60d0 a0 : 0000000000000036 a1
: 0000000000000003
[ 812.336063][ T4833] a2 : 1ffffffc0cfa8b00 a3 : ffffffe0000c80cc a4
: 7f467e72c6adf800
[ 812.337398][ T4833] a5 : 0000000000000000 a6 : 0000000000f00000 a7
: ffffffe0000f8c84
[ 812.339287][ T4833] s2 : 0000000000040000 s3 : ffffffe0077a96c0 s4
: ffffffe020e67fe0
[ 812.340658][ T4833] s5 : 0000000000004020 s6 : ffffffe0077a9b58 s7
: ffffffe067d74850
[ 812.342492][ T4833] s8 : ffffffe067d73e18 s9 : 0000000000000000
s10: ffffffe00bd72280
[ 812.343668][ T4833] s11: 000000bd067bf638 t3 : 7f467e72c6adf800 t4
: ffffffc403ee7fb2
[ 812.345510][ T4833] t5 : ffffffc403ee7fba t6 : 0000000000040000
[ 812.347004][ T4833] status: 0000000000000120 badaddr:
00000000250b60d0 cause: 000000000000000f
[ 812.348091][ T4833] Call Trace:
[ 812.349291][ T4833] [<ffffffe00008c8b0>] schedule_tail+0x72/0xb2
[ 812.350796][ T4833] [<ffffffe000005570>] ret_from_exception+0x0/0x14
[ 812.352799][ T4833] Dumping ftrace buffer:
[ 812.354328][ T4833] (ftrace buffer empty)
[ 812.428145][ T4833] ---[ end trace 94b077e4d677ee73 ]---
00:10:42 executing program 1:
bpf$ENABLE_STATS(0x20, 0x0, 0x0)
bpf$ENABLE_STATS(0x20, 0x0, 0x0)
[ 646.536862][ T5163] loop0: detected capacity change from 0 to 1
[ 646.566730][ T5165] Unable to handle kernel access to user memory
without uaccess routines at virtual address 00000000032f80d0
[ 646.586024][ T5165] Oops [#1]
[ 646.586640][ T5165] Modules linked in:
[ 646.587350][ T5165] CPU: 1 PID: 5165 Comm: syz-executor.1 Not
tainted 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0
[ 646.588209][ T5165] Hardware name: riscv-virtio,qemu (DT)
[ 646.589019][ T5165] epc : schedule_tail+0x72/0xb2
[ 646.589811][ T5165] ra : schedule_tail+0x70/0xb2
[ 646.590435][ T5165] epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp
: ffffffe008013ec0
[ 646.591142][ T5165] gp : ffffffe005d25378 tp : ffffffe007634440 t0
: 0000000000000000
[ 646.591836][ T5165] t1 : 0000000000000001 t2 : 0000000000000008 s0
: ffffffe008013ee0
[ 646.592509][ T5165] s1 : 00000000032f80d0 a0 : 0000000000000004 a1
: 0000000000000003
[ 646.593188][ T5165] a2 : 1ffffffc0cfac500 a3 : ffffffe0000c80cc a4
: 8d229faaffda9500
[ 646.593878][ T5165] a5 : 0000000000000000 a6 : 0000000000f00000 a7
: ffffffe000082eba
[ 646.594552][ T5165] s2 : 0000000000040000 s3 : ffffffe00c82c440 s4
: ffffffe00e61ffe0
[ 646.595253][ T5165] s5 : 0000000000004000 s6 : ffffffe067d57e00 s7
: ffffffe067d57850
[ 646.595938][ T5165] s8 : ffffffe067d56e18 s9 : ffffffe067d57e00
s10: ffffffe00c82c878
[ 646.596627][ T5165] s11: 000000967ba7a1cc t3 : 8d229faaffda9500 t4
: ffffffc4011bc79b
[ 646.597319][ T5165] t5 : ffffffc4011bc79d t6 : ffffffe008de3ce8
[ 646.597909][ T5165] status: 0000000000000120 badaddr:
00000000032f80d0 cause: 000000000000000f
[ 646.598682][ T5165] Call Trace:
[ 646.599294][ T5165] [<ffffffe00008c8b0>] schedule_tail+0x72/0xb2
[ 646.600115][ T5165] [<ffffffe000005570>] ret_from_exception+0x0/0x14
[ 646.601333][ T5165] Dumping ftrace buffer:
[ 646.602322][ T5165] (ftrace buffer empty)
[ 646.663691][ T5165] ---[ end trace e7b7847ce74cdfca ]---
> Thanks,
>
> Alex
>
> >
> > Without this, you can end up with an infinite loop in the fault handler.
> >
> >>
> >>> Unable to handle kernel access to user memory without uaccess
> >>> routines at virtual address 000000002749f0d0
> >>> Oops [#1]
> >>> Modules linked in:
> >>> CPU: 1 PID: 4875 Comm: syz-executor.0 Not tainted
> >>> 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0
> >>> Hardware name: riscv-virtio,qemu (DT)
> >>> epc : schedule_tail+0x72/0xb2 kernel/sched/core.c:4264
> >>> ra : task_pid_vnr include/linux/sched.h:1421 [inline]
> >>> ra : schedule_tail+0x70/0xb2 kernel/sched/core.c:4264
> >>> epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp : ffffffe025d17ec0
> >>> gp : ffffffe005d25378 tp : ffffffe00f0d0000 t0 : 0000000000000000
> >>> t1 : 0000000000000001 t2 : 00000000000f4240 s0 : ffffffe025d17ee0
> >>> s1 : 000000002749f0d0 a0 : 000000000000002a a1 : 0000000000000003
> >>> a2 : 1ffffffc0cfac500 a3 : ffffffe0000c80cc a4 : 5ae9db91c19bbe00
> >>> a5 : 0000000000000000 a6 : 0000000000f00000 a7 : ffffffe000082eba
> >>> s2 : 0000000000040000 s3 : ffffffe00eef96c0 s4 : ffffffe022c77fe0
> >>> s5 : 0000000000004000 s6 : ffffffe067d74e00 s7 : ffffffe067d74850
> >>> s8 : ffffffe067d73e18 s9 : ffffffe067d74e00 s10: ffffffe00eef96e8
> >>> s11: 000000ae6cdf8368 t3 : 5ae9db91c19bbe00 t4 : ffffffc4043cafb2
> >>> t5 : ffffffc4043cafba t6 : 0000000000040000
> >>> status: 0000000000000120 badaddr: 000000002749f0d0 cause:
> >>> 000000000000000f
> >>> Call Trace:
> >>> [<ffffffe00008c8b0>] schedule_tail+0x72/0xb2 kernel/sched/core.c:4264
> >>> [<ffffffe000005570>] ret_from_exception+0x0/0x14
> >>> Dumping ftrace buffer:
> >>> (ftrace buffer empty)
> >>> ---[ end trace b5f8f9231dc87dda ]---
> >>>
> >>>
> >>> ---
> >>> This report is generated by a bot. It may contain errors.
> >>> See https://goo.gl/tpsmEJ for more information about syzbot.
> >>> syzbot engineers can be reached at syzkaller@googlegroups.com.
> >>>
> >>> syzbot will keep track of this issue. See:
> >>> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
> >>>
> >>> --
> >>> You received this message because you are subscribed to the Google
> >>> Groups "syzkaller-bugs" group.
> >>> To unsubscribe from this group and stop receiving emails from it,
> >>> send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
> >>> To view this discussion on the web visit
> >>> https://groups.google.com/d/msgid/syzkaller-bugs/000000000000b74f1b05bd316729%40google.com.
> >>>
> >>
> >> _______________________________________________
> >> linux-riscv mailing list
> >> linux-riscv@lists.infradead.org
> >> http://lists.infradead.org/mailman/listinfo/linux-riscv
> >>
> >
> >
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-11 6:50 ` Dmitry Vyukov
0 siblings, 0 replies; 51+ messages in thread
From: Dmitry Vyukov @ 2021-03-11 6:50 UTC (permalink / raw)
To: Alex Ghiti
Cc: Ben Dooks, syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou,
linux-riscv, Daniel Bristot de Oliveira, Benjamin Segall,
dietmar.eggemann, Juri Lelli, LKML, Mel Gorman, Ingo Molnar,
Peter Zijlstra, Steven Rostedt, syzkaller-bugs, Vincent Guittot
On Thu, Mar 11, 2021 at 7:40 AM Alex Ghiti <alex@ghiti.fr> wrote:
>
> Hi Ben,
>
> Le 3/10/21 à 5:24 PM, Ben Dooks a écrit :
> > On 10/03/2021 17:16, Dmitry Vyukov wrote:
> >> On Wed, Mar 10, 2021 at 5:46 PM syzbot
> >> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
> >>>
> >>> Hello,
> >>>
> >>> syzbot found the following issue on:
> >>>
> >>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
> >>> arch_dup_tas..
> >>> git tree:
> >>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
> >>> console output: https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
> >>> kernel config:
> >>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
> >>> dashboard link:
> >>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
> >>> userspace arch: riscv64
> >>>
> >>> Unfortunately, I don't have any reproducer for this issue yet.
> >>>
> >>> IMPORTANT: if you fix the issue, please add the following tag to the
> >>> commit:
> >>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
> >>
> >> +riscv maintainers
> >>
> >> This is riscv64-specific.
> >> I've seen similar crashes in put_user in other places. It looks like
> >> put_user crashes in the user address is not mapped/protected (?).
> >
> > The unmapped case should have been handled.
> >
> > I think this issue is that the check for user-mode access added. From
> > what I read the code may be wrong in
> >
> > + if (!user_mode(regs) && addr < TASK_SIZE &&
> > + unlikely(!(regs->status & SR_SUM)))
> > + die_kernel_fault("access to user memory without uaccess routines",
> > + addr, regs);
> >
> > I think the SR_SUM check might be wrong, as I read the standard the
> > SR_SUM should be set to disable user-space access. So the check
> > should be unlikely(regs->status & SR_SUM) to say access without
> > having disabled the protection.
>
> The check that is done seems correct to me: "The SUM (permit Supervisor
> User Memory access) bit modifies the privilege with which S-mode loads
> and stores access virtual memory. *When SUM=0, S-mode memory accesses
> to pages that are accessible by U-mode (U=1 in Figure 4.15) will fault*.
> When SUM=1, these accesses are permitted.SUM has no effect when
> page-based virtual memory is not in effect".
>
> I will try to reproduce the problem locally.
Weird. It crashes with this all the time:
https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
Even on trivial programs that almost don't do anything.
Maybe it's qemu bug? Do registers look sane in the dump? That SR_SUM, etc.
00:13:27 executing program 1:
openat$drirender128(0xffffffffffffff9c,
&(0x7f0000000040)='/dev/dri/renderD128\x00', 0x0, 0x0)
[ 812.318182][ T4833] Unable to handle kernel access to user memory
without uaccess routines at virtual address 00000000250b60d0
[ 812.322304][ T4833] Oops [#1]
[ 812.323196][ T4833] Modules linked in:
[ 812.324110][ T4833] CPU: 1 PID: 4833 Comm: syz-executor.1 Not
tainted 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0
[ 812.325862][ T4833] Hardware name: riscv-virtio,qemu (DT)
[ 812.327561][ T4833] epc : schedule_tail+0x72/0xb2
[ 812.328640][ T4833] ra : schedule_tail+0x70/0xb2
[ 812.330088][ T4833] epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp
: ffffffe0238bbec0
[ 812.331312][ T4833] gp : ffffffe005d25378 tp : ffffffe00a275b00 t0
: 0000000000000000
[ 812.333014][ T4833] t1 : 0000000000000001 t2 : 00000000000f4240 s0
: ffffffe0238bbee0
[ 812.334137][ T4833] s1 : 00000000250b60d0 a0 : 0000000000000036 a1
: 0000000000000003
[ 812.336063][ T4833] a2 : 1ffffffc0cfa8b00 a3 : ffffffe0000c80cc a4
: 7f467e72c6adf800
[ 812.337398][ T4833] a5 : 0000000000000000 a6 : 0000000000f00000 a7
: ffffffe0000f8c84
[ 812.339287][ T4833] s2 : 0000000000040000 s3 : ffffffe0077a96c0 s4
: ffffffe020e67fe0
[ 812.340658][ T4833] s5 : 0000000000004020 s6 : ffffffe0077a9b58 s7
: ffffffe067d74850
[ 812.342492][ T4833] s8 : ffffffe067d73e18 s9 : 0000000000000000
s10: ffffffe00bd72280
[ 812.343668][ T4833] s11: 000000bd067bf638 t3 : 7f467e72c6adf800 t4
: ffffffc403ee7fb2
[ 812.345510][ T4833] t5 : ffffffc403ee7fba t6 : 0000000000040000
[ 812.347004][ T4833] status: 0000000000000120 badaddr:
00000000250b60d0 cause: 000000000000000f
[ 812.348091][ T4833] Call Trace:
[ 812.349291][ T4833] [<ffffffe00008c8b0>] schedule_tail+0x72/0xb2
[ 812.350796][ T4833] [<ffffffe000005570>] ret_from_exception+0x0/0x14
[ 812.352799][ T4833] Dumping ftrace buffer:
[ 812.354328][ T4833] (ftrace buffer empty)
[ 812.428145][ T4833] ---[ end trace 94b077e4d677ee73 ]---
00:10:42 executing program 1:
bpf$ENABLE_STATS(0x20, 0x0, 0x0)
bpf$ENABLE_STATS(0x20, 0x0, 0x0)
[ 646.536862][ T5163] loop0: detected capacity change from 0 to 1
[ 646.566730][ T5165] Unable to handle kernel access to user memory
without uaccess routines at virtual address 00000000032f80d0
[ 646.586024][ T5165] Oops [#1]
[ 646.586640][ T5165] Modules linked in:
[ 646.587350][ T5165] CPU: 1 PID: 5165 Comm: syz-executor.1 Not
tainted 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0
[ 646.588209][ T5165] Hardware name: riscv-virtio,qemu (DT)
[ 646.589019][ T5165] epc : schedule_tail+0x72/0xb2
[ 646.589811][ T5165] ra : schedule_tail+0x70/0xb2
[ 646.590435][ T5165] epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp
: ffffffe008013ec0
[ 646.591142][ T5165] gp : ffffffe005d25378 tp : ffffffe007634440 t0
: 0000000000000000
[ 646.591836][ T5165] t1 : 0000000000000001 t2 : 0000000000000008 s0
: ffffffe008013ee0
[ 646.592509][ T5165] s1 : 00000000032f80d0 a0 : 0000000000000004 a1
: 0000000000000003
[ 646.593188][ T5165] a2 : 1ffffffc0cfac500 a3 : ffffffe0000c80cc a4
: 8d229faaffda9500
[ 646.593878][ T5165] a5 : 0000000000000000 a6 : 0000000000f00000 a7
: ffffffe000082eba
[ 646.594552][ T5165] s2 : 0000000000040000 s3 : ffffffe00c82c440 s4
: ffffffe00e61ffe0
[ 646.595253][ T5165] s5 : 0000000000004000 s6 : ffffffe067d57e00 s7
: ffffffe067d57850
[ 646.595938][ T5165] s8 : ffffffe067d56e18 s9 : ffffffe067d57e00
s10: ffffffe00c82c878
[ 646.596627][ T5165] s11: 000000967ba7a1cc t3 : 8d229faaffda9500 t4
: ffffffc4011bc79b
[ 646.597319][ T5165] t5 : ffffffc4011bc79d t6 : ffffffe008de3ce8
[ 646.597909][ T5165] status: 0000000000000120 badaddr:
00000000032f80d0 cause: 000000000000000f
[ 646.598682][ T5165] Call Trace:
[ 646.599294][ T5165] [<ffffffe00008c8b0>] schedule_tail+0x72/0xb2
[ 646.600115][ T5165] [<ffffffe000005570>] ret_from_exception+0x0/0x14
[ 646.601333][ T5165] Dumping ftrace buffer:
[ 646.602322][ T5165] (ftrace buffer empty)
[ 646.663691][ T5165] ---[ end trace e7b7847ce74cdfca ]---
> Thanks,
>
> Alex
>
> >
> > Without this, you can end up with an infinite loop in the fault handler.
> >
> >>
> >>> Unable to handle kernel access to user memory without uaccess
> >>> routines at virtual address 000000002749f0d0
> >>> Oops [#1]
> >>> Modules linked in:
> >>> CPU: 1 PID: 4875 Comm: syz-executor.0 Not tainted
> >>> 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0
> >>> Hardware name: riscv-virtio,qemu (DT)
> >>> epc : schedule_tail+0x72/0xb2 kernel/sched/core.c:4264
> >>> ra : task_pid_vnr include/linux/sched.h:1421 [inline]
> >>> ra : schedule_tail+0x70/0xb2 kernel/sched/core.c:4264
> >>> epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp : ffffffe025d17ec0
> >>> gp : ffffffe005d25378 tp : ffffffe00f0d0000 t0 : 0000000000000000
> >>> t1 : 0000000000000001 t2 : 00000000000f4240 s0 : ffffffe025d17ee0
> >>> s1 : 000000002749f0d0 a0 : 000000000000002a a1 : 0000000000000003
> >>> a2 : 1ffffffc0cfac500 a3 : ffffffe0000c80cc a4 : 5ae9db91c19bbe00
> >>> a5 : 0000000000000000 a6 : 0000000000f00000 a7 : ffffffe000082eba
> >>> s2 : 0000000000040000 s3 : ffffffe00eef96c0 s4 : ffffffe022c77fe0
> >>> s5 : 0000000000004000 s6 : ffffffe067d74e00 s7 : ffffffe067d74850
> >>> s8 : ffffffe067d73e18 s9 : ffffffe067d74e00 s10: ffffffe00eef96e8
> >>> s11: 000000ae6cdf8368 t3 : 5ae9db91c19bbe00 t4 : ffffffc4043cafb2
> >>> t5 : ffffffc4043cafba t6 : 0000000000040000
> >>> status: 0000000000000120 badaddr: 000000002749f0d0 cause:
> >>> 000000000000000f
> >>> Call Trace:
> >>> [<ffffffe00008c8b0>] schedule_tail+0x72/0xb2 kernel/sched/core.c:4264
> >>> [<ffffffe000005570>] ret_from_exception+0x0/0x14
> >>> Dumping ftrace buffer:
> >>> (ftrace buffer empty)
> >>> ---[ end trace b5f8f9231dc87dda ]---
> >>>
> >>>
> >>> ---
> >>> This report is generated by a bot. It may contain errors.
> >>> See https://goo.gl/tpsmEJ for more information about syzbot.
> >>> syzbot engineers can be reached at syzkaller@googlegroups.com.
> >>>
> >>> syzbot will keep track of this issue. See:
> >>> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
> >>>
> >>> --
> >>> You received this message because you are subscribed to the Google
> >>> Groups "syzkaller-bugs" group.
> >>> To unsubscribe from this group and stop receiving emails from it,
> >>> send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
> >>> To view this discussion on the web visit
> >>> https://groups.google.com/d/msgid/syzkaller-bugs/000000000000b74f1b05bd316729%40google.com.
> >>>
> >>
> >> _______________________________________________
> >> linux-riscv mailing list
> >> linux-riscv@lists.infradead.org
> >> http://lists.infradead.org/mailman/listinfo/linux-riscv
> >>
> >
> >
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-11 6:50 ` Dmitry Vyukov
@ 2021-03-11 6:52 ` Dmitry Vyukov
-1 siblings, 0 replies; 51+ messages in thread
From: Dmitry Vyukov @ 2021-03-11 6:52 UTC (permalink / raw)
To: Alex Ghiti
Cc: Ben Dooks, syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou,
linux-riscv, Daniel Bristot de Oliveira, Benjamin Segall,
dietmar.eggemann, Juri Lelli, LKML, Mel Gorman, Ingo Molnar,
Peter Zijlstra, Steven Rostedt, syzkaller-bugs, Vincent Guittot
On Thu, Mar 11, 2021 at 7:50 AM Dmitry Vyukov <dvyukov@google.com> wrote:
>
> On Thu, Mar 11, 2021 at 7:40 AM Alex Ghiti <alex@ghiti.fr> wrote:
> >
> > Hi Ben,
> >
> > Le 3/10/21 à 5:24 PM, Ben Dooks a écrit :
> > > On 10/03/2021 17:16, Dmitry Vyukov wrote:
> > >> On Wed, Mar 10, 2021 at 5:46 PM syzbot
> > >> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
> > >>>
> > >>> Hello,
> > >>>
> > >>> syzbot found the following issue on:
> > >>>
> > >>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
> > >>> arch_dup_tas..
> > >>> git tree:
> > >>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
> > >>> console output: https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
> > >>> kernel config:
> > >>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
> > >>> dashboard link:
> > >>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
> > >>> userspace arch: riscv64
> > >>>
> > >>> Unfortunately, I don't have any reproducer for this issue yet.
> > >>>
> > >>> IMPORTANT: if you fix the issue, please add the following tag to the
> > >>> commit:
> > >>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
> > >>
> > >> +riscv maintainers
> > >>
> > >> This is riscv64-specific.
> > >> I've seen similar crashes in put_user in other places. It looks like
> > >> put_user crashes in the user address is not mapped/protected (?).
> > >
> > > The unmapped case should have been handled.
> > >
> > > I think this issue is that the check for user-mode access added. From
> > > what I read the code may be wrong in
> > >
> > > + if (!user_mode(regs) && addr < TASK_SIZE &&
> > > + unlikely(!(regs->status & SR_SUM)))
> > > + die_kernel_fault("access to user memory without uaccess routines",
> > > + addr, regs);
> > >
> > > I think the SR_SUM check might be wrong, as I read the standard the
> > > SR_SUM should be set to disable user-space access. So the check
> > > should be unlikely(regs->status & SR_SUM) to say access without
> > > having disabled the protection.
> >
> > The check that is done seems correct to me: "The SUM (permit Supervisor
> > User Memory access) bit modifies the privilege with which S-mode loads
> > and stores access virtual memory. *When SUM=0, S-mode memory accesses
> > to pages that are accessible by U-mode (U=1 in Figure 4.15) will fault*.
> > When SUM=1, these accesses are permitted.SUM has no effect when
> > page-based virtual memory is not in effect".
> >
> > I will try to reproduce the problem locally.
>
> Weird. It crashes with this all the time:
> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>
> Even on trivial programs that almost don't do anything.
> Maybe it's qemu bug? Do registers look sane in the dump? That SR_SUM, etc.
>
>
> 00:13:27 executing program 1:
> openat$drirender128(0xffffffffffffff9c,
> &(0x7f0000000040)='/dev/dri/renderD128\x00', 0x0, 0x0)
>
> [ 812.318182][ T4833] Unable to handle kernel access to user memory
> without uaccess routines at virtual address 00000000250b60d0
> [ 812.322304][ T4833] Oops [#1]
> [ 812.323196][ T4833] Modules linked in:
> [ 812.324110][ T4833] CPU: 1 PID: 4833 Comm: syz-executor.1 Not
> tainted 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0
> [ 812.325862][ T4833] Hardware name: riscv-virtio,qemu (DT)
> [ 812.327561][ T4833] epc : schedule_tail+0x72/0xb2
> [ 812.328640][ T4833] ra : schedule_tail+0x70/0xb2
> [ 812.330088][ T4833] epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp
> : ffffffe0238bbec0
> [ 812.331312][ T4833] gp : ffffffe005d25378 tp : ffffffe00a275b00 t0
> : 0000000000000000
> [ 812.333014][ T4833] t1 : 0000000000000001 t2 : 00000000000f4240 s0
> : ffffffe0238bbee0
> [ 812.334137][ T4833] s1 : 00000000250b60d0 a0 : 0000000000000036 a1
> : 0000000000000003
> [ 812.336063][ T4833] a2 : 1ffffffc0cfa8b00 a3 : ffffffe0000c80cc a4
> : 7f467e72c6adf800
> [ 812.337398][ T4833] a5 : 0000000000000000 a6 : 0000000000f00000 a7
> : ffffffe0000f8c84
> [ 812.339287][ T4833] s2 : 0000000000040000 s3 : ffffffe0077a96c0 s4
> : ffffffe020e67fe0
> [ 812.340658][ T4833] s5 : 0000000000004020 s6 : ffffffe0077a9b58 s7
> : ffffffe067d74850
> [ 812.342492][ T4833] s8 : ffffffe067d73e18 s9 : 0000000000000000
> s10: ffffffe00bd72280
> [ 812.343668][ T4833] s11: 000000bd067bf638 t3 : 7f467e72c6adf800 t4
> : ffffffc403ee7fb2
> [ 812.345510][ T4833] t5 : ffffffc403ee7fba t6 : 0000000000040000
> [ 812.347004][ T4833] status: 0000000000000120 badaddr:
> 00000000250b60d0 cause: 000000000000000f
> [ 812.348091][ T4833] Call Trace:
> [ 812.349291][ T4833] [<ffffffe00008c8b0>] schedule_tail+0x72/0xb2
> [ 812.350796][ T4833] [<ffffffe000005570>] ret_from_exception+0x0/0x14
> [ 812.352799][ T4833] Dumping ftrace buffer:
> [ 812.354328][ T4833] (ftrace buffer empty)
> [ 812.428145][ T4833] ---[ end trace 94b077e4d677ee73 ]---
>
>
> 00:10:42 executing program 1:
> bpf$ENABLE_STATS(0x20, 0x0, 0x0)
> bpf$ENABLE_STATS(0x20, 0x0, 0x0)
>
> [ 646.536862][ T5163] loop0: detected capacity change from 0 to 1
> [ 646.566730][ T5165] Unable to handle kernel access to user memory
> without uaccess routines at virtual address 00000000032f80d0
> [ 646.586024][ T5165] Oops [#1]
> [ 646.586640][ T5165] Modules linked in:
> [ 646.587350][ T5165] CPU: 1 PID: 5165 Comm: syz-executor.1 Not
> tainted 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0
> [ 646.588209][ T5165] Hardware name: riscv-virtio,qemu (DT)
> [ 646.589019][ T5165] epc : schedule_tail+0x72/0xb2
> [ 646.589811][ T5165] ra : schedule_tail+0x70/0xb2
> [ 646.590435][ T5165] epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp
> : ffffffe008013ec0
> [ 646.591142][ T5165] gp : ffffffe005d25378 tp : ffffffe007634440 t0
> : 0000000000000000
> [ 646.591836][ T5165] t1 : 0000000000000001 t2 : 0000000000000008 s0
> : ffffffe008013ee0
> [ 646.592509][ T5165] s1 : 00000000032f80d0 a0 : 0000000000000004 a1
> : 0000000000000003
> [ 646.593188][ T5165] a2 : 1ffffffc0cfac500 a3 : ffffffe0000c80cc a4
> : 8d229faaffda9500
> [ 646.593878][ T5165] a5 : 0000000000000000 a6 : 0000000000f00000 a7
> : ffffffe000082eba
> [ 646.594552][ T5165] s2 : 0000000000040000 s3 : ffffffe00c82c440 s4
> : ffffffe00e61ffe0
> [ 646.595253][ T5165] s5 : 0000000000004000 s6 : ffffffe067d57e00 s7
> : ffffffe067d57850
> [ 646.595938][ T5165] s8 : ffffffe067d56e18 s9 : ffffffe067d57e00
> s10: ffffffe00c82c878
> [ 646.596627][ T5165] s11: 000000967ba7a1cc t3 : 8d229faaffda9500 t4
> : ffffffc4011bc79b
> [ 646.597319][ T5165] t5 : ffffffc4011bc79d t6 : ffffffe008de3ce8
> [ 646.597909][ T5165] status: 0000000000000120 badaddr:
> 00000000032f80d0 cause: 000000000000000f
> [ 646.598682][ T5165] Call Trace:
> [ 646.599294][ T5165] [<ffffffe00008c8b0>] schedule_tail+0x72/0xb2
> [ 646.600115][ T5165] [<ffffffe000005570>] ret_from_exception+0x0/0x14
> [ 646.601333][ T5165] Dumping ftrace buffer:
> [ 646.602322][ T5165] (ftrace buffer empty)
> [ 646.663691][ T5165] ---[ end trace e7b7847ce74cdfca ]---
Is it reasonable that schedule_tail is called from ret_from_exception?
Maybe the issue is in ret_from_exception? I see it does something with
registers.
> > Thanks,
> >
> > Alex
> >
> > >
> > > Without this, you can end up with an infinite loop in the fault handler.
> > >
> > >>
> > >>> Unable to handle kernel access to user memory without uaccess
> > >>> routines at virtual address 000000002749f0d0
> > >>> Oops [#1]
> > >>> Modules linked in:
> > >>> CPU: 1 PID: 4875 Comm: syz-executor.0 Not tainted
> > >>> 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0
> > >>> Hardware name: riscv-virtio,qemu (DT)
> > >>> epc : schedule_tail+0x72/0xb2 kernel/sched/core.c:4264
> > >>> ra : task_pid_vnr include/linux/sched.h:1421 [inline]
> > >>> ra : schedule_tail+0x70/0xb2 kernel/sched/core.c:4264
> > >>> epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp : ffffffe025d17ec0
> > >>> gp : ffffffe005d25378 tp : ffffffe00f0d0000 t0 : 0000000000000000
> > >>> t1 : 0000000000000001 t2 : 00000000000f4240 s0 : ffffffe025d17ee0
> > >>> s1 : 000000002749f0d0 a0 : 000000000000002a a1 : 0000000000000003
> > >>> a2 : 1ffffffc0cfac500 a3 : ffffffe0000c80cc a4 : 5ae9db91c19bbe00
> > >>> a5 : 0000000000000000 a6 : 0000000000f00000 a7 : ffffffe000082eba
> > >>> s2 : 0000000000040000 s3 : ffffffe00eef96c0 s4 : ffffffe022c77fe0
> > >>> s5 : 0000000000004000 s6 : ffffffe067d74e00 s7 : ffffffe067d74850
> > >>> s8 : ffffffe067d73e18 s9 : ffffffe067d74e00 s10: ffffffe00eef96e8
> > >>> s11: 000000ae6cdf8368 t3 : 5ae9db91c19bbe00 t4 : ffffffc4043cafb2
> > >>> t5 : ffffffc4043cafba t6 : 0000000000040000
> > >>> status: 0000000000000120 badaddr: 000000002749f0d0 cause:
> > >>> 000000000000000f
> > >>> Call Trace:
> > >>> [<ffffffe00008c8b0>] schedule_tail+0x72/0xb2 kernel/sched/core.c:4264
> > >>> [<ffffffe000005570>] ret_from_exception+0x0/0x14
> > >>> Dumping ftrace buffer:
> > >>> (ftrace buffer empty)
> > >>> ---[ end trace b5f8f9231dc87dda ]---
> > >>>
> > >>>
> > >>> ---
> > >>> This report is generated by a bot. It may contain errors.
> > >>> See https://goo.gl/tpsmEJ for more information about syzbot.
> > >>> syzbot engineers can be reached at syzkaller@googlegroups.com.
> > >>>
> > >>> syzbot will keep track of this issue. See:
> > >>> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
> > >>>
> > >>> --
> > >>> You received this message because you are subscribed to the Google
> > >>> Groups "syzkaller-bugs" group.
> > >>> To unsubscribe from this group and stop receiving emails from it,
> > >>> send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
> > >>> To view this discussion on the web visit
> > >>> https://groups.google.com/d/msgid/syzkaller-bugs/000000000000b74f1b05bd316729%40google.com.
> > >>>
> > >>
> > >> _______________________________________________
> > >> linux-riscv mailing list
> > >> linux-riscv@lists.infradead.org
> > >> http://lists.infradead.org/mailman/listinfo/linux-riscv
> > >>
> > >
> > >
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-11 6:52 ` Dmitry Vyukov
0 siblings, 0 replies; 51+ messages in thread
From: Dmitry Vyukov @ 2021-03-11 6:52 UTC (permalink / raw)
To: Alex Ghiti
Cc: Ben Dooks, syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou,
linux-riscv, Daniel Bristot de Oliveira, Benjamin Segall,
dietmar.eggemann, Juri Lelli, LKML, Mel Gorman, Ingo Molnar,
Peter Zijlstra, Steven Rostedt, syzkaller-bugs, Vincent Guittot
On Thu, Mar 11, 2021 at 7:50 AM Dmitry Vyukov <dvyukov@google.com> wrote:
>
> On Thu, Mar 11, 2021 at 7:40 AM Alex Ghiti <alex@ghiti.fr> wrote:
> >
> > Hi Ben,
> >
> > Le 3/10/21 à 5:24 PM, Ben Dooks a écrit :
> > > On 10/03/2021 17:16, Dmitry Vyukov wrote:
> > >> On Wed, Mar 10, 2021 at 5:46 PM syzbot
> > >> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
> > >>>
> > >>> Hello,
> > >>>
> > >>> syzbot found the following issue on:
> > >>>
> > >>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
> > >>> arch_dup_tas..
> > >>> git tree:
> > >>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
> > >>> console output: https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
> > >>> kernel config:
> > >>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
> > >>> dashboard link:
> > >>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
> > >>> userspace arch: riscv64
> > >>>
> > >>> Unfortunately, I don't have any reproducer for this issue yet.
> > >>>
> > >>> IMPORTANT: if you fix the issue, please add the following tag to the
> > >>> commit:
> > >>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
> > >>
> > >> +riscv maintainers
> > >>
> > >> This is riscv64-specific.
> > >> I've seen similar crashes in put_user in other places. It looks like
> > >> put_user crashes in the user address is not mapped/protected (?).
> > >
> > > The unmapped case should have been handled.
> > >
> > > I think this issue is that the check for user-mode access added. From
> > > what I read the code may be wrong in
> > >
> > > + if (!user_mode(regs) && addr < TASK_SIZE &&
> > > + unlikely(!(regs->status & SR_SUM)))
> > > + die_kernel_fault("access to user memory without uaccess routines",
> > > + addr, regs);
> > >
> > > I think the SR_SUM check might be wrong, as I read the standard the
> > > SR_SUM should be set to disable user-space access. So the check
> > > should be unlikely(regs->status & SR_SUM) to say access without
> > > having disabled the protection.
> >
> > The check that is done seems correct to me: "The SUM (permit Supervisor
> > User Memory access) bit modifies the privilege with which S-mode loads
> > and stores access virtual memory. *When SUM=0, S-mode memory accesses
> > to pages that are accessible by U-mode (U=1 in Figure 4.15) will fault*.
> > When SUM=1, these accesses are permitted.SUM has no effect when
> > page-based virtual memory is not in effect".
> >
> > I will try to reproduce the problem locally.
>
> Weird. It crashes with this all the time:
> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>
> Even on trivial programs that almost don't do anything.
> Maybe it's qemu bug? Do registers look sane in the dump? That SR_SUM, etc.
>
>
> 00:13:27 executing program 1:
> openat$drirender128(0xffffffffffffff9c,
> &(0x7f0000000040)='/dev/dri/renderD128\x00', 0x0, 0x0)
>
> [ 812.318182][ T4833] Unable to handle kernel access to user memory
> without uaccess routines at virtual address 00000000250b60d0
> [ 812.322304][ T4833] Oops [#1]
> [ 812.323196][ T4833] Modules linked in:
> [ 812.324110][ T4833] CPU: 1 PID: 4833 Comm: syz-executor.1 Not
> tainted 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0
> [ 812.325862][ T4833] Hardware name: riscv-virtio,qemu (DT)
> [ 812.327561][ T4833] epc : schedule_tail+0x72/0xb2
> [ 812.328640][ T4833] ra : schedule_tail+0x70/0xb2
> [ 812.330088][ T4833] epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp
> : ffffffe0238bbec0
> [ 812.331312][ T4833] gp : ffffffe005d25378 tp : ffffffe00a275b00 t0
> : 0000000000000000
> [ 812.333014][ T4833] t1 : 0000000000000001 t2 : 00000000000f4240 s0
> : ffffffe0238bbee0
> [ 812.334137][ T4833] s1 : 00000000250b60d0 a0 : 0000000000000036 a1
> : 0000000000000003
> [ 812.336063][ T4833] a2 : 1ffffffc0cfa8b00 a3 : ffffffe0000c80cc a4
> : 7f467e72c6adf800
> [ 812.337398][ T4833] a5 : 0000000000000000 a6 : 0000000000f00000 a7
> : ffffffe0000f8c84
> [ 812.339287][ T4833] s2 : 0000000000040000 s3 : ffffffe0077a96c0 s4
> : ffffffe020e67fe0
> [ 812.340658][ T4833] s5 : 0000000000004020 s6 : ffffffe0077a9b58 s7
> : ffffffe067d74850
> [ 812.342492][ T4833] s8 : ffffffe067d73e18 s9 : 0000000000000000
> s10: ffffffe00bd72280
> [ 812.343668][ T4833] s11: 000000bd067bf638 t3 : 7f467e72c6adf800 t4
> : ffffffc403ee7fb2
> [ 812.345510][ T4833] t5 : ffffffc403ee7fba t6 : 0000000000040000
> [ 812.347004][ T4833] status: 0000000000000120 badaddr:
> 00000000250b60d0 cause: 000000000000000f
> [ 812.348091][ T4833] Call Trace:
> [ 812.349291][ T4833] [<ffffffe00008c8b0>] schedule_tail+0x72/0xb2
> [ 812.350796][ T4833] [<ffffffe000005570>] ret_from_exception+0x0/0x14
> [ 812.352799][ T4833] Dumping ftrace buffer:
> [ 812.354328][ T4833] (ftrace buffer empty)
> [ 812.428145][ T4833] ---[ end trace 94b077e4d677ee73 ]---
>
>
> 00:10:42 executing program 1:
> bpf$ENABLE_STATS(0x20, 0x0, 0x0)
> bpf$ENABLE_STATS(0x20, 0x0, 0x0)
>
> [ 646.536862][ T5163] loop0: detected capacity change from 0 to 1
> [ 646.566730][ T5165] Unable to handle kernel access to user memory
> without uaccess routines at virtual address 00000000032f80d0
> [ 646.586024][ T5165] Oops [#1]
> [ 646.586640][ T5165] Modules linked in:
> [ 646.587350][ T5165] CPU: 1 PID: 5165 Comm: syz-executor.1 Not
> tainted 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0
> [ 646.588209][ T5165] Hardware name: riscv-virtio,qemu (DT)
> [ 646.589019][ T5165] epc : schedule_tail+0x72/0xb2
> [ 646.589811][ T5165] ra : schedule_tail+0x70/0xb2
> [ 646.590435][ T5165] epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp
> : ffffffe008013ec0
> [ 646.591142][ T5165] gp : ffffffe005d25378 tp : ffffffe007634440 t0
> : 0000000000000000
> [ 646.591836][ T5165] t1 : 0000000000000001 t2 : 0000000000000008 s0
> : ffffffe008013ee0
> [ 646.592509][ T5165] s1 : 00000000032f80d0 a0 : 0000000000000004 a1
> : 0000000000000003
> [ 646.593188][ T5165] a2 : 1ffffffc0cfac500 a3 : ffffffe0000c80cc a4
> : 8d229faaffda9500
> [ 646.593878][ T5165] a5 : 0000000000000000 a6 : 0000000000f00000 a7
> : ffffffe000082eba
> [ 646.594552][ T5165] s2 : 0000000000040000 s3 : ffffffe00c82c440 s4
> : ffffffe00e61ffe0
> [ 646.595253][ T5165] s5 : 0000000000004000 s6 : ffffffe067d57e00 s7
> : ffffffe067d57850
> [ 646.595938][ T5165] s8 : ffffffe067d56e18 s9 : ffffffe067d57e00
> s10: ffffffe00c82c878
> [ 646.596627][ T5165] s11: 000000967ba7a1cc t3 : 8d229faaffda9500 t4
> : ffffffc4011bc79b
> [ 646.597319][ T5165] t5 : ffffffc4011bc79d t6 : ffffffe008de3ce8
> [ 646.597909][ T5165] status: 0000000000000120 badaddr:
> 00000000032f80d0 cause: 000000000000000f
> [ 646.598682][ T5165] Call Trace:
> [ 646.599294][ T5165] [<ffffffe00008c8b0>] schedule_tail+0x72/0xb2
> [ 646.600115][ T5165] [<ffffffe000005570>] ret_from_exception+0x0/0x14
> [ 646.601333][ T5165] Dumping ftrace buffer:
> [ 646.602322][ T5165] (ftrace buffer empty)
> [ 646.663691][ T5165] ---[ end trace e7b7847ce74cdfca ]---
Is it reasonable that schedule_tail is called from ret_from_exception?
Maybe the issue is in ret_from_exception? I see it does something with
registers.
> > Thanks,
> >
> > Alex
> >
> > >
> > > Without this, you can end up with an infinite loop in the fault handler.
> > >
> > >>
> > >>> Unable to handle kernel access to user memory without uaccess
> > >>> routines at virtual address 000000002749f0d0
> > >>> Oops [#1]
> > >>> Modules linked in:
> > >>> CPU: 1 PID: 4875 Comm: syz-executor.0 Not tainted
> > >>> 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0
> > >>> Hardware name: riscv-virtio,qemu (DT)
> > >>> epc : schedule_tail+0x72/0xb2 kernel/sched/core.c:4264
> > >>> ra : task_pid_vnr include/linux/sched.h:1421 [inline]
> > >>> ra : schedule_tail+0x70/0xb2 kernel/sched/core.c:4264
> > >>> epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp : ffffffe025d17ec0
> > >>> gp : ffffffe005d25378 tp : ffffffe00f0d0000 t0 : 0000000000000000
> > >>> t1 : 0000000000000001 t2 : 00000000000f4240 s0 : ffffffe025d17ee0
> > >>> s1 : 000000002749f0d0 a0 : 000000000000002a a1 : 0000000000000003
> > >>> a2 : 1ffffffc0cfac500 a3 : ffffffe0000c80cc a4 : 5ae9db91c19bbe00
> > >>> a5 : 0000000000000000 a6 : 0000000000f00000 a7 : ffffffe000082eba
> > >>> s2 : 0000000000040000 s3 : ffffffe00eef96c0 s4 : ffffffe022c77fe0
> > >>> s5 : 0000000000004000 s6 : ffffffe067d74e00 s7 : ffffffe067d74850
> > >>> s8 : ffffffe067d73e18 s9 : ffffffe067d74e00 s10: ffffffe00eef96e8
> > >>> s11: 000000ae6cdf8368 t3 : 5ae9db91c19bbe00 t4 : ffffffc4043cafb2
> > >>> t5 : ffffffc4043cafba t6 : 0000000000040000
> > >>> status: 0000000000000120 badaddr: 000000002749f0d0 cause:
> > >>> 000000000000000f
> > >>> Call Trace:
> > >>> [<ffffffe00008c8b0>] schedule_tail+0x72/0xb2 kernel/sched/core.c:4264
> > >>> [<ffffffe000005570>] ret_from_exception+0x0/0x14
> > >>> Dumping ftrace buffer:
> > >>> (ftrace buffer empty)
> > >>> ---[ end trace b5f8f9231dc87dda ]---
> > >>>
> > >>>
> > >>> ---
> > >>> This report is generated by a bot. It may contain errors.
> > >>> See https://goo.gl/tpsmEJ for more information about syzbot.
> > >>> syzbot engineers can be reached at syzkaller@googlegroups.com.
> > >>>
> > >>> syzbot will keep track of this issue. See:
> > >>> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
> > >>>
> > >>> --
> > >>> You received this message because you are subscribed to the Google
> > >>> Groups "syzkaller-bugs" group.
> > >>> To unsubscribe from this group and stop receiving emails from it,
> > >>> send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
> > >>> To view this discussion on the web visit
> > >>> https://groups.google.com/d/msgid/syzkaller-bugs/000000000000b74f1b05bd316729%40google.com.
> > >>>
> > >>
> > >> _______________________________________________
> > >> linux-riscv mailing list
> > >> linux-riscv@lists.infradead.org
> > >> http://lists.infradead.org/mailman/listinfo/linux-riscv
> > >>
> > >
> > >
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-11 6:52 ` Dmitry Vyukov
@ 2021-03-11 10:41 ` Ben Dooks
-1 siblings, 0 replies; 51+ messages in thread
From: Ben Dooks @ 2021-03-11 10:41 UTC (permalink / raw)
To: Dmitry Vyukov, Alex Ghiti
Cc: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv,
Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
On 11/03/2021 06:52, Dmitry Vyukov wrote:
> On Thu, Mar 11, 2021 at 7:50 AM Dmitry Vyukov <dvyukov@google.com> wrote:
>>
>> On Thu, Mar 11, 2021 at 7:40 AM Alex Ghiti <alex@ghiti.fr> wrote:
>>>
>>> Hi Ben,
>>>
>>> Le 3/10/21 à 5:24 PM, Ben Dooks a écrit :
>>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
>>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
>>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> syzbot found the following issue on:
>>>>>>
>>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
>>>>>> arch_dup_tas..
>>>>>> git tree:
>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>>>>>> console output: https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>>>>>> kernel config:
>>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>>>>>> dashboard link:
>>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>>>>>> userspace arch: riscv64
>>>>>>
>>>>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>>>>
>>>>>> IMPORTANT: if you fix the issue, please add the following tag to the
>>>>>> commit:
>>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>>>>>
>>>>> +riscv maintainers
>>>>>
>>>>> This is riscv64-specific.
>>>>> I've seen similar crashes in put_user in other places. It looks like
>>>>> put_user crashes in the user address is not mapped/protected (?).
>>>>
>>>> The unmapped case should have been handled.
>>>>
>>>> I think this issue is that the check for user-mode access added. From
>>>> what I read the code may be wrong in
>>>>
>>>> + if (!user_mode(regs) && addr < TASK_SIZE &&
>>>> + unlikely(!(regs->status & SR_SUM)))
>>>> + die_kernel_fault("access to user memory without uaccess routines",
>>>> + addr, regs);
>>>>
>>>> I think the SR_SUM check might be wrong, as I read the standard the
>>>> SR_SUM should be set to disable user-space access. So the check
>>>> should be unlikely(regs->status & SR_SUM) to say access without
>>>> having disabled the protection.
>>>
>>> The check that is done seems correct to me: "The SUM (permit Supervisor
>>> User Memory access) bit modifies the privilege with which S-mode loads
>>> and stores access virtual memory. *When SUM=0, S-mode memory accesses
>>> to pages that are accessible by U-mode (U=1 in Figure 4.15) will fault*.
>>> When SUM=1, these accesses are permitted.SUM has no effect when
>>> page-based virtual memory is not in effect".
>>>
>>> I will try to reproduce the problem locally.
>>
>> Weird. It crashes with this all the time:
>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>>
>> Even on trivial programs that almost don't do anything.
>> Maybe it's qemu bug? Do registers look sane in the dump? That SR_SUM, etc.
>>
>>
>> 00:13:27 executing program 1:
>> openat$drirender128(0xffffffffffffff9c,
>> &(0x7f0000000040)='/dev/dri/renderD128\x00', 0x0, 0x0)
>>
>> [ 812.318182][ T4833] Unable to handle kernel access to user memory
>> without uaccess routines at virtual address 00000000250b60d0
>> [ 812.322304][ T4833] Oops [#1]
>> [ 812.323196][ T4833] Modules linked in:
>> [ 812.324110][ T4833] CPU: 1 PID: 4833 Comm: syz-executor.1 Not
>> tainted 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0
>> [ 812.325862][ T4833] Hardware name: riscv-virtio,qemu (DT)
>> [ 812.327561][ T4833] epc : schedule_tail+0x72/0xb2
>> [ 812.328640][ T4833] ra : schedule_tail+0x70/0xb2
>> [ 812.330088][ T4833] epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp
>> : ffffffe0238bbec0
>> [ 812.331312][ T4833] gp : ffffffe005d25378 tp : ffffffe00a275b00 t0
>> : 0000000000000000
>> [ 812.333014][ T4833] t1 : 0000000000000001 t2 : 00000000000f4240 s0
>> : ffffffe0238bbee0
>> [ 812.334137][ T4833] s1 : 00000000250b60d0 a0 : 0000000000000036 a1
>> : 0000000000000003
>> [ 812.336063][ T4833] a2 : 1ffffffc0cfa8b00 a3 : ffffffe0000c80cc a4
>> : 7f467e72c6adf800
>> [ 812.337398][ T4833] a5 : 0000000000000000 a6 : 0000000000f00000 a7
>> : ffffffe0000f8c84
>> [ 812.339287][ T4833] s2 : 0000000000040000 s3 : ffffffe0077a96c0 s4
>> : ffffffe020e67fe0
>> [ 812.340658][ T4833] s5 : 0000000000004020 s6 : ffffffe0077a9b58 s7
>> : ffffffe067d74850
>> [ 812.342492][ T4833] s8 : ffffffe067d73e18 s9 : 0000000000000000
>> s10: ffffffe00bd72280
>> [ 812.343668][ T4833] s11: 000000bd067bf638 t3 : 7f467e72c6adf800 t4
>> : ffffffc403ee7fb2
>> [ 812.345510][ T4833] t5 : ffffffc403ee7fba t6 : 0000000000040000
>> [ 812.347004][ T4833] status: 0000000000000120 badaddr:
>> 00000000250b60d0 cause: 000000000000000f
>> [ 812.348091][ T4833] Call Trace:
>> [ 812.349291][ T4833] [<ffffffe00008c8b0>] schedule_tail+0x72/0xb2
>> [ 812.350796][ T4833] [<ffffffe000005570>] ret_from_exception+0x0/0x14
>> [ 812.352799][ T4833] Dumping ftrace buffer:
>> [ 812.354328][ T4833] (ftrace buffer empty)
>> [ 812.428145][ T4833] ---[ end trace 94b077e4d677ee73 ]---
>>
>>
>> 00:10:42 executing program 1:
>> bpf$ENABLE_STATS(0x20, 0x0, 0x0)
>> bpf$ENABLE_STATS(0x20, 0x0, 0x0)
>>
>> [ 646.536862][ T5163] loop0: detected capacity change from 0 to 1
>> [ 646.566730][ T5165] Unable to handle kernel access to user memory
>> without uaccess routines at virtual address 00000000032f80d0
>> [ 646.586024][ T5165] Oops [#1]
>> [ 646.586640][ T5165] Modules linked in:
>> [ 646.587350][ T5165] CPU: 1 PID: 5165 Comm: syz-executor.1 Not
>> tainted 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0
>> [ 646.588209][ T5165] Hardware name: riscv-virtio,qemu (DT)
>> [ 646.589019][ T5165] epc : schedule_tail+0x72/0xb2
>> [ 646.589811][ T5165] ra : schedule_tail+0x70/0xb2
>> [ 646.590435][ T5165] epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp
>> : ffffffe008013ec0
>> [ 646.591142][ T5165] gp : ffffffe005d25378 tp : ffffffe007634440 t0
>> : 0000000000000000
>> [ 646.591836][ T5165] t1 : 0000000000000001 t2 : 0000000000000008 s0
>> : ffffffe008013ee0
>> [ 646.592509][ T5165] s1 : 00000000032f80d0 a0 : 0000000000000004 a1
>> : 0000000000000003
>> [ 646.593188][ T5165] a2 : 1ffffffc0cfac500 a3 : ffffffe0000c80cc a4
>> : 8d229faaffda9500
>> [ 646.593878][ T5165] a5 : 0000000000000000 a6 : 0000000000f00000 a7
>> : ffffffe000082eba
>> [ 646.594552][ T5165] s2 : 0000000000040000 s3 : ffffffe00c82c440 s4
>> : ffffffe00e61ffe0
>> [ 646.595253][ T5165] s5 : 0000000000004000 s6 : ffffffe067d57e00 s7
>> : ffffffe067d57850
>> [ 646.595938][ T5165] s8 : ffffffe067d56e18 s9 : ffffffe067d57e00
>> s10: ffffffe00c82c878
>> [ 646.596627][ T5165] s11: 000000967ba7a1cc t3 : 8d229faaffda9500 t4
>> : ffffffc4011bc79b
>> [ 646.597319][ T5165] t5 : ffffffc4011bc79d t6 : ffffffe008de3ce8
>> [ 646.597909][ T5165] status: 0000000000000120 badaddr:
>> 00000000032f80d0 cause: 000000000000000f
>> [ 646.598682][ T5165] Call Trace:
>> [ 646.599294][ T5165] [<ffffffe00008c8b0>] schedule_tail+0x72/0xb2
>> [ 646.600115][ T5165] [<ffffffe000005570>] ret_from_exception+0x0/0x14
>> [ 646.601333][ T5165] Dumping ftrace buffer:
>> [ 646.602322][ T5165] (ftrace buffer empty)
>> [ 646.663691][ T5165] ---[ end trace e7b7847ce74cdfca ]---
>
> Is it reasonable that schedule_tail is called from ret_from_exception?
> Maybe the issue is in ret_from_exception? I see it does something with
> registers.
I'd not noticed this with an earlier kernel (5.10 and the user-fault
check patches) but this may be an qemu issue?
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-11 10:41 ` Ben Dooks
0 siblings, 0 replies; 51+ messages in thread
From: Ben Dooks @ 2021-03-11 10:41 UTC (permalink / raw)
To: Dmitry Vyukov, Alex Ghiti
Cc: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv,
Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
On 11/03/2021 06:52, Dmitry Vyukov wrote:
> On Thu, Mar 11, 2021 at 7:50 AM Dmitry Vyukov <dvyukov@google.com> wrote:
>>
>> On Thu, Mar 11, 2021 at 7:40 AM Alex Ghiti <alex@ghiti.fr> wrote:
>>>
>>> Hi Ben,
>>>
>>> Le 3/10/21 à 5:24 PM, Ben Dooks a écrit :
>>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
>>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
>>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> syzbot found the following issue on:
>>>>>>
>>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
>>>>>> arch_dup_tas..
>>>>>> git tree:
>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>>>>>> console output: https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>>>>>> kernel config:
>>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>>>>>> dashboard link:
>>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>>>>>> userspace arch: riscv64
>>>>>>
>>>>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>>>>
>>>>>> IMPORTANT: if you fix the issue, please add the following tag to the
>>>>>> commit:
>>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>>>>>
>>>>> +riscv maintainers
>>>>>
>>>>> This is riscv64-specific.
>>>>> I've seen similar crashes in put_user in other places. It looks like
>>>>> put_user crashes in the user address is not mapped/protected (?).
>>>>
>>>> The unmapped case should have been handled.
>>>>
>>>> I think this issue is that the check for user-mode access added. From
>>>> what I read the code may be wrong in
>>>>
>>>> + if (!user_mode(regs) && addr < TASK_SIZE &&
>>>> + unlikely(!(regs->status & SR_SUM)))
>>>> + die_kernel_fault("access to user memory without uaccess routines",
>>>> + addr, regs);
>>>>
>>>> I think the SR_SUM check might be wrong, as I read the standard the
>>>> SR_SUM should be set to disable user-space access. So the check
>>>> should be unlikely(regs->status & SR_SUM) to say access without
>>>> having disabled the protection.
>>>
>>> The check that is done seems correct to me: "The SUM (permit Supervisor
>>> User Memory access) bit modifies the privilege with which S-mode loads
>>> and stores access virtual memory. *When SUM=0, S-mode memory accesses
>>> to pages that are accessible by U-mode (U=1 in Figure 4.15) will fault*.
>>> When SUM=1, these accesses are permitted.SUM has no effect when
>>> page-based virtual memory is not in effect".
>>>
>>> I will try to reproduce the problem locally.
>>
>> Weird. It crashes with this all the time:
>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>>
>> Even on trivial programs that almost don't do anything.
>> Maybe it's qemu bug? Do registers look sane in the dump? That SR_SUM, etc.
>>
>>
>> 00:13:27 executing program 1:
>> openat$drirender128(0xffffffffffffff9c,
>> &(0x7f0000000040)='/dev/dri/renderD128\x00', 0x0, 0x0)
>>
>> [ 812.318182][ T4833] Unable to handle kernel access to user memory
>> without uaccess routines at virtual address 00000000250b60d0
>> [ 812.322304][ T4833] Oops [#1]
>> [ 812.323196][ T4833] Modules linked in:
>> [ 812.324110][ T4833] CPU: 1 PID: 4833 Comm: syz-executor.1 Not
>> tainted 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0
>> [ 812.325862][ T4833] Hardware name: riscv-virtio,qemu (DT)
>> [ 812.327561][ T4833] epc : schedule_tail+0x72/0xb2
>> [ 812.328640][ T4833] ra : schedule_tail+0x70/0xb2
>> [ 812.330088][ T4833] epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp
>> : ffffffe0238bbec0
>> [ 812.331312][ T4833] gp : ffffffe005d25378 tp : ffffffe00a275b00 t0
>> : 0000000000000000
>> [ 812.333014][ T4833] t1 : 0000000000000001 t2 : 00000000000f4240 s0
>> : ffffffe0238bbee0
>> [ 812.334137][ T4833] s1 : 00000000250b60d0 a0 : 0000000000000036 a1
>> : 0000000000000003
>> [ 812.336063][ T4833] a2 : 1ffffffc0cfa8b00 a3 : ffffffe0000c80cc a4
>> : 7f467e72c6adf800
>> [ 812.337398][ T4833] a5 : 0000000000000000 a6 : 0000000000f00000 a7
>> : ffffffe0000f8c84
>> [ 812.339287][ T4833] s2 : 0000000000040000 s3 : ffffffe0077a96c0 s4
>> : ffffffe020e67fe0
>> [ 812.340658][ T4833] s5 : 0000000000004020 s6 : ffffffe0077a9b58 s7
>> : ffffffe067d74850
>> [ 812.342492][ T4833] s8 : ffffffe067d73e18 s9 : 0000000000000000
>> s10: ffffffe00bd72280
>> [ 812.343668][ T4833] s11: 000000bd067bf638 t3 : 7f467e72c6adf800 t4
>> : ffffffc403ee7fb2
>> [ 812.345510][ T4833] t5 : ffffffc403ee7fba t6 : 0000000000040000
>> [ 812.347004][ T4833] status: 0000000000000120 badaddr:
>> 00000000250b60d0 cause: 000000000000000f
>> [ 812.348091][ T4833] Call Trace:
>> [ 812.349291][ T4833] [<ffffffe00008c8b0>] schedule_tail+0x72/0xb2
>> [ 812.350796][ T4833] [<ffffffe000005570>] ret_from_exception+0x0/0x14
>> [ 812.352799][ T4833] Dumping ftrace buffer:
>> [ 812.354328][ T4833] (ftrace buffer empty)
>> [ 812.428145][ T4833] ---[ end trace 94b077e4d677ee73 ]---
>>
>>
>> 00:10:42 executing program 1:
>> bpf$ENABLE_STATS(0x20, 0x0, 0x0)
>> bpf$ENABLE_STATS(0x20, 0x0, 0x0)
>>
>> [ 646.536862][ T5163] loop0: detected capacity change from 0 to 1
>> [ 646.566730][ T5165] Unable to handle kernel access to user memory
>> without uaccess routines at virtual address 00000000032f80d0
>> [ 646.586024][ T5165] Oops [#1]
>> [ 646.586640][ T5165] Modules linked in:
>> [ 646.587350][ T5165] CPU: 1 PID: 5165 Comm: syz-executor.1 Not
>> tainted 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0
>> [ 646.588209][ T5165] Hardware name: riscv-virtio,qemu (DT)
>> [ 646.589019][ T5165] epc : schedule_tail+0x72/0xb2
>> [ 646.589811][ T5165] ra : schedule_tail+0x70/0xb2
>> [ 646.590435][ T5165] epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp
>> : ffffffe008013ec0
>> [ 646.591142][ T5165] gp : ffffffe005d25378 tp : ffffffe007634440 t0
>> : 0000000000000000
>> [ 646.591836][ T5165] t1 : 0000000000000001 t2 : 0000000000000008 s0
>> : ffffffe008013ee0
>> [ 646.592509][ T5165] s1 : 00000000032f80d0 a0 : 0000000000000004 a1
>> : 0000000000000003
>> [ 646.593188][ T5165] a2 : 1ffffffc0cfac500 a3 : ffffffe0000c80cc a4
>> : 8d229faaffda9500
>> [ 646.593878][ T5165] a5 : 0000000000000000 a6 : 0000000000f00000 a7
>> : ffffffe000082eba
>> [ 646.594552][ T5165] s2 : 0000000000040000 s3 : ffffffe00c82c440 s4
>> : ffffffe00e61ffe0
>> [ 646.595253][ T5165] s5 : 0000000000004000 s6 : ffffffe067d57e00 s7
>> : ffffffe067d57850
>> [ 646.595938][ T5165] s8 : ffffffe067d56e18 s9 : ffffffe067d57e00
>> s10: ffffffe00c82c878
>> [ 646.596627][ T5165] s11: 000000967ba7a1cc t3 : 8d229faaffda9500 t4
>> : ffffffc4011bc79b
>> [ 646.597319][ T5165] t5 : ffffffc4011bc79d t6 : ffffffe008de3ce8
>> [ 646.597909][ T5165] status: 0000000000000120 badaddr:
>> 00000000032f80d0 cause: 000000000000000f
>> [ 646.598682][ T5165] Call Trace:
>> [ 646.599294][ T5165] [<ffffffe00008c8b0>] schedule_tail+0x72/0xb2
>> [ 646.600115][ T5165] [<ffffffe000005570>] ret_from_exception+0x0/0x14
>> [ 646.601333][ T5165] Dumping ftrace buffer:
>> [ 646.602322][ T5165] (ftrace buffer empty)
>> [ 646.663691][ T5165] ---[ end trace e7b7847ce74cdfca ]---
>
> Is it reasonable that schedule_tail is called from ret_from_exception?
> Maybe the issue is in ret_from_exception? I see it does something with
> registers.
I'd not noticed this with an earlier kernel (5.10 and the user-fault
check patches) but this may be an qemu issue?
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-10 17:16 ` Dmitry Vyukov
@ 2021-03-12 13:49 ` Ben Dooks
-1 siblings, 0 replies; 51+ messages in thread
From: Ben Dooks @ 2021-03-12 13:49 UTC (permalink / raw)
To: Dmitry Vyukov, syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou,
linux-riscv
Cc: Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
On 10/03/2021 17:16, Dmitry Vyukov wrote:
> On Wed, Mar 10, 2021 at 5:46 PM syzbot
> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>
>> Hello,
>>
>> syzbot found the following issue on:
>>
>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for arch_dup_tas..
>> git tree: git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>> console output: https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>> kernel config: https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>> dashboard link: https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>> userspace arch: riscv64
>>
>> Unfortunately, I don't have any reproducer for this issue yet.
>>
>> IMPORTANT: if you fix the issue, please add the following tag to the commit:
>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>
> +riscv maintainers
>
> This is riscv64-specific.
> I've seen similar crashes in put_user in other places. It looks like
> put_user crashes in the user address is not mapped/protected (?).
I've been having a look, and this seems to be down to access of the
tsk->set_child_tid variable. I assume the fuzzing here is to pass a
bad address to clone?
From looking at the code, the put_user() code should have set the
relevant SR_SUM bit (the value for this, which is 1<<18 is in the
s2 register in the crash report) and from looking at the compiler
output from my gcc-10, the code looks to be dong the relevant csrs
and then csrc around the put_user
So currently I do not understand how the above could have happened
over than something re-tried the code seqeunce and ended up retrying
the faulting instruction without the SR_SUM bit set.
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-12 13:49 ` Ben Dooks
0 siblings, 0 replies; 51+ messages in thread
From: Ben Dooks @ 2021-03-12 13:49 UTC (permalink / raw)
To: Dmitry Vyukov, syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou,
linux-riscv
Cc: Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
On 10/03/2021 17:16, Dmitry Vyukov wrote:
> On Wed, Mar 10, 2021 at 5:46 PM syzbot
> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>
>> Hello,
>>
>> syzbot found the following issue on:
>>
>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for arch_dup_tas..
>> git tree: git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>> console output: https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>> kernel config: https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>> dashboard link: https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>> userspace arch: riscv64
>>
>> Unfortunately, I don't have any reproducer for this issue yet.
>>
>> IMPORTANT: if you fix the issue, please add the following tag to the commit:
>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>
> +riscv maintainers
>
> This is riscv64-specific.
> I've seen similar crashes in put_user in other places. It looks like
> put_user crashes in the user address is not mapped/protected (?).
I've been having a look, and this seems to be down to access of the
tsk->set_child_tid variable. I assume the fuzzing here is to pass a
bad address to clone?
From looking at the code, the put_user() code should have set the
relevant SR_SUM bit (the value for this, which is 1<<18 is in the
s2 register in the crash report) and from looking at the compiler
output from my gcc-10, the code looks to be dong the relevant csrs
and then csrc around the put_user
So currently I do not understand how the above could have happened
over than something re-tried the code seqeunce and ended up retrying
the faulting instruction without the SR_SUM bit set.
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-12 13:49 ` Ben Dooks
@ 2021-03-12 15:12 ` Dmitry Vyukov
-1 siblings, 0 replies; 51+ messages in thread
From: Dmitry Vyukov @ 2021-03-12 15:12 UTC (permalink / raw)
To: Ben Dooks
Cc: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv,
Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>
> On 10/03/2021 17:16, Dmitry Vyukov wrote:
> > On Wed, Mar 10, 2021 at 5:46 PM syzbot
> > <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
> >>
> >> Hello,
> >>
> >> syzbot found the following issue on:
> >>
> >> HEAD commit: 0d7588ab riscv: process: Fix no prototype for arch_dup_tas..
> >> git tree: git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
> >> console output: https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
> >> kernel config: https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
> >> dashboard link: https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
> >> userspace arch: riscv64
> >>
> >> Unfortunately, I don't have any reproducer for this issue yet.
> >>
> >> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> >> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
> >
> > +riscv maintainers
> >
> > This is riscv64-specific.
> > I've seen similar crashes in put_user in other places. It looks like
> > put_user crashes in the user address is not mapped/protected (?).
>
> I've been having a look, and this seems to be down to access of the
> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
> bad address to clone?
>
> From looking at the code, the put_user() code should have set the
> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
> s2 register in the crash report) and from looking at the compiler
> output from my gcc-10, the code looks to be dong the relevant csrs
> and then csrc around the put_user
>
> So currently I do not understand how the above could have happened
> over than something re-tried the code seqeunce and ended up retrying
> the faulting instruction without the SR_SUM bit set.
I would maybe blame qemu for randomly resetting SR_SUM, but it's
strange that 99% of these crashes are in schedule_tail. If it would be
qemu, then they would be more evenly distributed...
Another observation: looking at a dozen of crash logs, in none of
these cases fuzzer was actually trying to fuzz clone with some insane
arguments. So it looks like completely normal clone's (e..g coming
from pthread_create) result in this crash.
I also wonder why there is ret_from_exception, is it normal? I see
handle_exception disables SR_SUM:
https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-12 15:12 ` Dmitry Vyukov
0 siblings, 0 replies; 51+ messages in thread
From: Dmitry Vyukov @ 2021-03-12 15:12 UTC (permalink / raw)
To: Ben Dooks
Cc: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv,
Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>
> On 10/03/2021 17:16, Dmitry Vyukov wrote:
> > On Wed, Mar 10, 2021 at 5:46 PM syzbot
> > <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
> >>
> >> Hello,
> >>
> >> syzbot found the following issue on:
> >>
> >> HEAD commit: 0d7588ab riscv: process: Fix no prototype for arch_dup_tas..
> >> git tree: git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
> >> console output: https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
> >> kernel config: https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
> >> dashboard link: https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
> >> userspace arch: riscv64
> >>
> >> Unfortunately, I don't have any reproducer for this issue yet.
> >>
> >> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> >> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
> >
> > +riscv maintainers
> >
> > This is riscv64-specific.
> > I've seen similar crashes in put_user in other places. It looks like
> > put_user crashes in the user address is not mapped/protected (?).
>
> I've been having a look, and this seems to be down to access of the
> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
> bad address to clone?
>
> From looking at the code, the put_user() code should have set the
> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
> s2 register in the crash report) and from looking at the compiler
> output from my gcc-10, the code looks to be dong the relevant csrs
> and then csrc around the put_user
>
> So currently I do not understand how the above could have happened
> over than something re-tried the code seqeunce and ended up retrying
> the faulting instruction without the SR_SUM bit set.
I would maybe blame qemu for randomly resetting SR_SUM, but it's
strange that 99% of these crashes are in schedule_tail. If it would be
qemu, then they would be more evenly distributed...
Another observation: looking at a dozen of crash logs, in none of
these cases fuzzer was actually trying to fuzz clone with some insane
arguments. So it looks like completely normal clone's (e..g coming
from pthread_create) result in this crash.
I also wonder why there is ret_from_exception, is it normal? I see
handle_exception disables SR_SUM:
https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-12 15:12 ` Dmitry Vyukov
@ 2021-03-12 16:25 ` Alex Ghiti
-1 siblings, 0 replies; 51+ messages in thread
From: Alex Ghiti @ 2021-03-12 16:25 UTC (permalink / raw)
To: Dmitry Vyukov, Ben Dooks
Cc: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv,
Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
Le 3/12/21 à 10:12 AM, Dmitry Vyukov a écrit :
> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>>
>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>>>
>>>> Hello,
>>>>
>>>> syzbot found the following issue on:
>>>>
>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for arch_dup_tas..
>>>> git tree: git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>>>> console output: https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>>>> kernel config: https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>>>> dashboard link: https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>>>> userspace arch: riscv64
>>>>
>>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>>
>>>> IMPORTANT: if you fix the issue, please add the following tag to the commit:
>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>>>
>>> +riscv maintainers
>>>
>>> This is riscv64-specific.
>>> I've seen similar crashes in put_user in other places. It looks like
>>> put_user crashes in the user address is not mapped/protected (?).
>>
>> I've been having a look, and this seems to be down to access of the
>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
>> bad address to clone?
>>
>> From looking at the code, the put_user() code should have set the
>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
>> s2 register in the crash report) and from looking at the compiler
>> output from my gcc-10, the code looks to be dong the relevant csrs
>> and then csrc around the put_user
>>
>> So currently I do not understand how the above could have happened
>> over than something re-tried the code seqeunce and ended up retrying
>> the faulting instruction without the SR_SUM bit set.
>
> I would maybe blame qemu for randomly resetting SR_SUM, but it's
> strange that 99% of these crashes are in schedule_tail. If it would be
> qemu, then they would be more evenly distributed...
>
> Another observation: looking at a dozen of crash logs, in none of
> these cases fuzzer was actually trying to fuzz clone with some insane
> arguments. So it looks like completely normal clone's (e..g coming
> from pthread_create) result in this crash.
>
> I also wonder why there is ret_from_exception, is it normal? I see
> handle_exception disables SR_SUM:
csrrc does the right thing: it cleans SR_SUM bit in status but saves the
previous value that will get correctly restored.
("The CSRRC (Atomic Read and Clear Bits in CSR) instruction reads the
value of the CSR, zero-extends the value to XLEN bits, and writes it to
integer registerrd. The initial value in integerregisterrs1is treated
as a bit mask that specifies bit positions to be cleared in the CSR. Any
bitthat is high inrs1will cause the corresponding bit to be cleared in
the CSR, if that CSR bit iswritable. Other bits in the CSR are
unaffected.")
> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
Still no luck for the moment, can't reproduce it locally, my test is
maybe not that good (I created threads all day long in order to trigger
the put_user of schedule_tail).
Given that the path you mention works most of the time, and that the
status register in the stack trace shows the SUM bit is not set whereas
it is set in put_user, I'm leaning toward some race condition (maybe an
interrupt that arrives at the "wrong" time) or a qemu issue as you
mentioned.
To eliminate qemu issues, do you have access to some HW ? Or to
different qemu versions ?
>
> _______________________________________________
> linux-riscv mailing list
> linux-riscv@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-riscv
>
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-12 16:25 ` Alex Ghiti
0 siblings, 0 replies; 51+ messages in thread
From: Alex Ghiti @ 2021-03-12 16:25 UTC (permalink / raw)
To: Dmitry Vyukov, Ben Dooks
Cc: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv,
Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
Le 3/12/21 à 10:12 AM, Dmitry Vyukov a écrit :
> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>>
>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>>>
>>>> Hello,
>>>>
>>>> syzbot found the following issue on:
>>>>
>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for arch_dup_tas..
>>>> git tree: git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>>>> console output: https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>>>> kernel config: https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>>>> dashboard link: https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>>>> userspace arch: riscv64
>>>>
>>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>>
>>>> IMPORTANT: if you fix the issue, please add the following tag to the commit:
>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>>>
>>> +riscv maintainers
>>>
>>> This is riscv64-specific.
>>> I've seen similar crashes in put_user in other places. It looks like
>>> put_user crashes in the user address is not mapped/protected (?).
>>
>> I've been having a look, and this seems to be down to access of the
>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
>> bad address to clone?
>>
>> From looking at the code, the put_user() code should have set the
>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
>> s2 register in the crash report) and from looking at the compiler
>> output from my gcc-10, the code looks to be dong the relevant csrs
>> and then csrc around the put_user
>>
>> So currently I do not understand how the above could have happened
>> over than something re-tried the code seqeunce and ended up retrying
>> the faulting instruction without the SR_SUM bit set.
>
> I would maybe blame qemu for randomly resetting SR_SUM, but it's
> strange that 99% of these crashes are in schedule_tail. If it would be
> qemu, then they would be more evenly distributed...
>
> Another observation: looking at a dozen of crash logs, in none of
> these cases fuzzer was actually trying to fuzz clone with some insane
> arguments. So it looks like completely normal clone's (e..g coming
> from pthread_create) result in this crash.
>
> I also wonder why there is ret_from_exception, is it normal? I see
> handle_exception disables SR_SUM:
csrrc does the right thing: it cleans SR_SUM bit in status but saves the
previous value that will get correctly restored.
("The CSRRC (Atomic Read and Clear Bits in CSR) instruction reads the
value of the CSR, zero-extends the value to XLEN bits, and writes it to
integer registerrd. The initial value in integerregisterrs1is treated
as a bit mask that specifies bit positions to be cleared in the CSR. Any
bitthat is high inrs1will cause the corresponding bit to be cleared in
the CSR, if that CSR bit iswritable. Other bits in the CSR are
unaffected.")
> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
Still no luck for the moment, can't reproduce it locally, my test is
maybe not that good (I created threads all day long in order to trigger
the put_user of schedule_tail).
Given that the path you mention works most of the time, and that the
status register in the stack trace shows the SUM bit is not set whereas
it is set in put_user, I'm leaning toward some race condition (maybe an
interrupt that arrives at the "wrong" time) or a qemu issue as you
mentioned.
To eliminate qemu issues, do you have access to some HW ? Or to
different qemu versions ?
>
> _______________________________________________
> linux-riscv mailing list
> linux-riscv@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-riscv
>
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-12 15:12 ` Dmitry Vyukov
@ 2021-03-12 16:30 ` Ben Dooks
-1 siblings, 0 replies; 51+ messages in thread
From: Ben Dooks @ 2021-03-12 16:30 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv,
Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
On 12/03/2021 15:12, Dmitry Vyukov wrote:
> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>>
>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>>>
>>>> Hello,
>>>>
>>>> syzbot found the following issue on:
>>>>
>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for arch_dup_tas..
>>>> git tree: git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>>>> console output: https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>>>> kernel config: https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>>>> dashboard link: https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>>>> userspace arch: riscv64
>>>>
>>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>>
>>>> IMPORTANT: if you fix the issue, please add the following tag to the commit:
>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>>>
>>> +riscv maintainers
>>>
>>> This is riscv64-specific.
>>> I've seen similar crashes in put_user in other places. It looks like
>>> put_user crashes in the user address is not mapped/protected (?).
>>
>> I've been having a look, and this seems to be down to access of the
>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
>> bad address to clone?
>>
>> From looking at the code, the put_user() code should have set the
>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
>> s2 register in the crash report) and from looking at the compiler
>> output from my gcc-10, the code looks to be dong the relevant csrs
>> and then csrc around the put_user
>>
>> So currently I do not understand how the above could have happened
>> over than something re-tried the code seqeunce and ended up retrying
>> the faulting instruction without the SR_SUM bit set.
>
> I would maybe blame qemu for randomly resetting SR_SUM, but it's
> strange that 99% of these crashes are in schedule_tail. If it would be
> qemu, then they would be more evenly distributed...
>
> Another observation: looking at a dozen of crash logs, in none of
> these cases fuzzer was actually trying to fuzz clone with some insane
> arguments. So it looks like completely normal clone's (e..g coming
> from pthread_create) result in this crash.
>
> I also wonder why there is ret_from_exception, is it normal? I see
> handle_exception disables SR_SUM:
> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
So I think if SR_SUM is set, then it faults the access to user memory
which the _user() routines clear to allow them access.
I'm thinking there is at least one issue here:
- the test in fault is the wrong way around for die kernel
- the handler only catches this if the page has yet to be mapped.
So I think the test should be:
if (!user_mode(regs) && addr < TASK_SIZE &&
unlikely(regs->status & SR_SUM)
This then should continue on and allow the rest of the handler to
complete mapping the page if it is not there.
I have been trying to create a very simple clone test, but so far it
has yet to actually trigger anything.
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-12 16:30 ` Ben Dooks
0 siblings, 0 replies; 51+ messages in thread
From: Ben Dooks @ 2021-03-12 16:30 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv,
Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
On 12/03/2021 15:12, Dmitry Vyukov wrote:
> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>>
>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>>>
>>>> Hello,
>>>>
>>>> syzbot found the following issue on:
>>>>
>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for arch_dup_tas..
>>>> git tree: git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>>>> console output: https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>>>> kernel config: https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>>>> dashboard link: https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>>>> userspace arch: riscv64
>>>>
>>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>>
>>>> IMPORTANT: if you fix the issue, please add the following tag to the commit:
>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>>>
>>> +riscv maintainers
>>>
>>> This is riscv64-specific.
>>> I've seen similar crashes in put_user in other places. It looks like
>>> put_user crashes in the user address is not mapped/protected (?).
>>
>> I've been having a look, and this seems to be down to access of the
>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
>> bad address to clone?
>>
>> From looking at the code, the put_user() code should have set the
>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
>> s2 register in the crash report) and from looking at the compiler
>> output from my gcc-10, the code looks to be dong the relevant csrs
>> and then csrc around the put_user
>>
>> So currently I do not understand how the above could have happened
>> over than something re-tried the code seqeunce and ended up retrying
>> the faulting instruction without the SR_SUM bit set.
>
> I would maybe blame qemu for randomly resetting SR_SUM, but it's
> strange that 99% of these crashes are in schedule_tail. If it would be
> qemu, then they would be more evenly distributed...
>
> Another observation: looking at a dozen of crash logs, in none of
> these cases fuzzer was actually trying to fuzz clone with some insane
> arguments. So it looks like completely normal clone's (e..g coming
> from pthread_create) result in this crash.
>
> I also wonder why there is ret_from_exception, is it normal? I see
> handle_exception disables SR_SUM:
> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
So I think if SR_SUM is set, then it faults the access to user memory
which the _user() routines clear to allow them access.
I'm thinking there is at least one issue here:
- the test in fault is the wrong way around for die kernel
- the handler only catches this if the page has yet to be mapped.
So I think the test should be:
if (!user_mode(regs) && addr < TASK_SIZE &&
unlikely(regs->status & SR_SUM)
This then should continue on and allow the rest of the handler to
complete mapping the page if it is not there.
I have been trying to create a very simple clone test, but so far it
has yet to actually trigger anything.
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-12 16:30 ` Ben Dooks
@ 2021-03-12 16:34 ` Ben Dooks
-1 siblings, 0 replies; 51+ messages in thread
From: Ben Dooks @ 2021-03-12 16:34 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv,
Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
On 12/03/2021 16:30, Ben Dooks wrote:
> On 12/03/2021 15:12, Dmitry Vyukov wrote:
>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
>> wrote:
>>>
>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> syzbot found the following issue on:
>>>>>
>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
>>>>> arch_dup_tas..
>>>>> git tree:
>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>>>>> console output:
>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>>>>> kernel config:
>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>>>>> dashboard link:
>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>>>>> userspace arch: riscv64
>>>>>
>>>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>>>
>>>>> IMPORTANT: if you fix the issue, please add the following tag to
>>>>> the commit:
>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>>>>
>>>> +riscv maintainers
>>>>
>>>> This is riscv64-specific.
>>>> I've seen similar crashes in put_user in other places. It looks like
>>>> put_user crashes in the user address is not mapped/protected (?).
>>>
>>> I've been having a look, and this seems to be down to access of the
>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
>>> bad address to clone?
>>>
>>> From looking at the code, the put_user() code should have set the
>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
>>> s2 register in the crash report) and from looking at the compiler
>>> output from my gcc-10, the code looks to be dong the relevant csrs
>>> and then csrc around the put_user
>>>
>>> So currently I do not understand how the above could have happened
>>> over than something re-tried the code seqeunce and ended up retrying
>>> the faulting instruction without the SR_SUM bit set.
>>
>> I would maybe blame qemu for randomly resetting SR_SUM, but it's
>> strange that 99% of these crashes are in schedule_tail. If it would be
>> qemu, then they would be more evenly distributed...
>>
>> Another observation: looking at a dozen of crash logs, in none of
>> these cases fuzzer was actually trying to fuzz clone with some insane
>> arguments. So it looks like completely normal clone's (e..g coming
>> from pthread_create) result in this crash.
>>
>> I also wonder why there is ret_from_exception, is it normal? I see
>> handle_exception disables SR_SUM:
>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
>>
>
> So I think if SR_SUM is set, then it faults the access to user memory
> which the _user() routines clear to allow them access.
>
> I'm thinking there is at least one issue here:
>
> - the test in fault is the wrong way around for die kernel
> - the handler only catches this if the page has yet to be mapped.
>
> So I think the test should be:
>
> if (!user_mode(regs) && addr < TASK_SIZE &&
> unlikely(regs->status & SR_SUM)
>
> This then should continue on and allow the rest of the handler to
> complete mapping the page if it is not there.
>
> I have been trying to create a very simple clone test, but so far it
> has yet to actually trigger anything.
I should have added there doesn't seem to be a good way to use mmap()
to allocate memory but not insert a vm-mapping post the mmap().
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-12 16:34 ` Ben Dooks
0 siblings, 0 replies; 51+ messages in thread
From: Ben Dooks @ 2021-03-12 16:34 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv,
Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
On 12/03/2021 16:30, Ben Dooks wrote:
> On 12/03/2021 15:12, Dmitry Vyukov wrote:
>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
>> wrote:
>>>
>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> syzbot found the following issue on:
>>>>>
>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
>>>>> arch_dup_tas..
>>>>> git tree:
>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>>>>> console output:
>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>>>>> kernel config:
>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>>>>> dashboard link:
>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>>>>> userspace arch: riscv64
>>>>>
>>>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>>>
>>>>> IMPORTANT: if you fix the issue, please add the following tag to
>>>>> the commit:
>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>>>>
>>>> +riscv maintainers
>>>>
>>>> This is riscv64-specific.
>>>> I've seen similar crashes in put_user in other places. It looks like
>>>> put_user crashes in the user address is not mapped/protected (?).
>>>
>>> I've been having a look, and this seems to be down to access of the
>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
>>> bad address to clone?
>>>
>>> From looking at the code, the put_user() code should have set the
>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
>>> s2 register in the crash report) and from looking at the compiler
>>> output from my gcc-10, the code looks to be dong the relevant csrs
>>> and then csrc around the put_user
>>>
>>> So currently I do not understand how the above could have happened
>>> over than something re-tried the code seqeunce and ended up retrying
>>> the faulting instruction without the SR_SUM bit set.
>>
>> I would maybe blame qemu for randomly resetting SR_SUM, but it's
>> strange that 99% of these crashes are in schedule_tail. If it would be
>> qemu, then they would be more evenly distributed...
>>
>> Another observation: looking at a dozen of crash logs, in none of
>> these cases fuzzer was actually trying to fuzz clone with some insane
>> arguments. So it looks like completely normal clone's (e..g coming
>> from pthread_create) result in this crash.
>>
>> I also wonder why there is ret_from_exception, is it normal? I see
>> handle_exception disables SR_SUM:
>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
>>
>
> So I think if SR_SUM is set, then it faults the access to user memory
> which the _user() routines clear to allow them access.
>
> I'm thinking there is at least one issue here:
>
> - the test in fault is the wrong way around for die kernel
> - the handler only catches this if the page has yet to be mapped.
>
> So I think the test should be:
>
> if (!user_mode(regs) && addr < TASK_SIZE &&
> unlikely(regs->status & SR_SUM)
>
> This then should continue on and allow the rest of the handler to
> complete mapping the page if it is not there.
>
> I have been trying to create a very simple clone test, but so far it
> has yet to actually trigger anything.
I should have added there doesn't seem to be a good way to use mmap()
to allocate memory but not insert a vm-mapping post the mmap().
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-12 16:34 ` Ben Dooks
@ 2021-03-12 16:36 ` Ben Dooks
-1 siblings, 0 replies; 51+ messages in thread
From: Ben Dooks @ 2021-03-12 16:36 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv,
Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
On 12/03/2021 16:34, Ben Dooks wrote:
> On 12/03/2021 16:30, Ben Dooks wrote:
>> On 12/03/2021 15:12, Dmitry Vyukov wrote:
>>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
>>> wrote:
>>>>
>>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
>>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
>>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> syzbot found the following issue on:
>>>>>>
>>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
>>>>>> arch_dup_tas..
>>>>>> git tree:
>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>>>>>> console output:
>>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>>>>>> kernel config:
>>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>>>>>> dashboard link:
>>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>>>>>> userspace arch: riscv64
>>>>>>
>>>>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>>>>
>>>>>> IMPORTANT: if you fix the issue, please add the following tag to
>>>>>> the commit:
>>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>>>>>
>>>>> +riscv maintainers
>>>>>
>>>>> This is riscv64-specific.
>>>>> I've seen similar crashes in put_user in other places. It looks like
>>>>> put_user crashes in the user address is not mapped/protected (?).
>>>>
>>>> I've been having a look, and this seems to be down to access of the
>>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
>>>> bad address to clone?
>>>>
>>>> From looking at the code, the put_user() code should have set the
>>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
>>>> s2 register in the crash report) and from looking at the compiler
>>>> output from my gcc-10, the code looks to be dong the relevant csrs
>>>> and then csrc around the put_user
>>>>
>>>> So currently I do not understand how the above could have happened
>>>> over than something re-tried the code seqeunce and ended up retrying
>>>> the faulting instruction without the SR_SUM bit set.
>>>
>>> I would maybe blame qemu for randomly resetting SR_SUM, but it's
>>> strange that 99% of these crashes are in schedule_tail. If it would be
>>> qemu, then they would be more evenly distributed...
>>>
>>> Another observation: looking at a dozen of crash logs, in none of
>>> these cases fuzzer was actually trying to fuzz clone with some insane
>>> arguments. So it looks like completely normal clone's (e..g coming
>>> from pthread_create) result in this crash.
>>>
>>> I also wonder why there is ret_from_exception, is it normal? I see
>>> handle_exception disables SR_SUM:
>>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
>>>
>>
>> So I think if SR_SUM is set, then it faults the access to user memory
>> which the _user() routines clear to allow them access.
>>
>> I'm thinking there is at least one issue here:
>>
>> - the test in fault is the wrong way around for die kernel
>> - the handler only catches this if the page has yet to be mapped.
>>
>> So I think the test should be:
>>
>> if (!user_mode(regs) && addr < TASK_SIZE &&
>> unlikely(regs->status & SR_SUM)
>>
>> This then should continue on and allow the rest of the handler to
>> complete mapping the page if it is not there.
>>
>> I have been trying to create a very simple clone test, but so far it
>> has yet to actually trigger anything.
>
> I should have added there doesn't seem to be a good way to use mmap()
> to allocate memory but not insert a vm-mapping post the mmap().
>
>
How difficult is it to try building a branch with the above test
modified?
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-12 16:36 ` Ben Dooks
0 siblings, 0 replies; 51+ messages in thread
From: Ben Dooks @ 2021-03-12 16:36 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv,
Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
On 12/03/2021 16:34, Ben Dooks wrote:
> On 12/03/2021 16:30, Ben Dooks wrote:
>> On 12/03/2021 15:12, Dmitry Vyukov wrote:
>>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
>>> wrote:
>>>>
>>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
>>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
>>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> syzbot found the following issue on:
>>>>>>
>>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
>>>>>> arch_dup_tas..
>>>>>> git tree:
>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>>>>>> console output:
>>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>>>>>> kernel config:
>>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>>>>>> dashboard link:
>>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>>>>>> userspace arch: riscv64
>>>>>>
>>>>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>>>>
>>>>>> IMPORTANT: if you fix the issue, please add the following tag to
>>>>>> the commit:
>>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>>>>>
>>>>> +riscv maintainers
>>>>>
>>>>> This is riscv64-specific.
>>>>> I've seen similar crashes in put_user in other places. It looks like
>>>>> put_user crashes in the user address is not mapped/protected (?).
>>>>
>>>> I've been having a look, and this seems to be down to access of the
>>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
>>>> bad address to clone?
>>>>
>>>> From looking at the code, the put_user() code should have set the
>>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
>>>> s2 register in the crash report) and from looking at the compiler
>>>> output from my gcc-10, the code looks to be dong the relevant csrs
>>>> and then csrc around the put_user
>>>>
>>>> So currently I do not understand how the above could have happened
>>>> over than something re-tried the code seqeunce and ended up retrying
>>>> the faulting instruction without the SR_SUM bit set.
>>>
>>> I would maybe blame qemu for randomly resetting SR_SUM, but it's
>>> strange that 99% of these crashes are in schedule_tail. If it would be
>>> qemu, then they would be more evenly distributed...
>>>
>>> Another observation: looking at a dozen of crash logs, in none of
>>> these cases fuzzer was actually trying to fuzz clone with some insane
>>> arguments. So it looks like completely normal clone's (e..g coming
>>> from pthread_create) result in this crash.
>>>
>>> I also wonder why there is ret_from_exception, is it normal? I see
>>> handle_exception disables SR_SUM:
>>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
>>>
>>
>> So I think if SR_SUM is set, then it faults the access to user memory
>> which the _user() routines clear to allow them access.
>>
>> I'm thinking there is at least one issue here:
>>
>> - the test in fault is the wrong way around for die kernel
>> - the handler only catches this if the page has yet to be mapped.
>>
>> So I think the test should be:
>>
>> if (!user_mode(regs) && addr < TASK_SIZE &&
>> unlikely(regs->status & SR_SUM)
>>
>> This then should continue on and allow the rest of the handler to
>> complete mapping the page if it is not there.
>>
>> I have been trying to create a very simple clone test, but so far it
>> has yet to actually trigger anything.
>
> I should have added there doesn't seem to be a good way to use mmap()
> to allocate memory but not insert a vm-mapping post the mmap().
>
>
How difficult is it to try building a branch with the above test
modified?
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-12 16:36 ` Ben Dooks
@ 2021-03-12 17:34 ` Dmitry Vyukov
-1 siblings, 0 replies; 51+ messages in thread
From: Dmitry Vyukov @ 2021-03-12 17:34 UTC (permalink / raw)
To: Ben Dooks
Cc: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv,
Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
On Fri, Mar 12, 2021 at 5:36 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>
> On 12/03/2021 16:34, Ben Dooks wrote:
> > On 12/03/2021 16:30, Ben Dooks wrote:
> >> On 12/03/2021 15:12, Dmitry Vyukov wrote:
> >>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
> >>> wrote:
> >>>>
> >>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
> >>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
> >>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
> >>>>>>
> >>>>>> Hello,
> >>>>>>
> >>>>>> syzbot found the following issue on:
> >>>>>>
> >>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
> >>>>>> arch_dup_tas..
> >>>>>> git tree:
> >>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
> >>>>>> console output:
> >>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
> >>>>>> kernel config:
> >>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
> >>>>>> dashboard link:
> >>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
> >>>>>> userspace arch: riscv64
> >>>>>>
> >>>>>> Unfortunately, I don't have any reproducer for this issue yet.
> >>>>>>
> >>>>>> IMPORTANT: if you fix the issue, please add the following tag to
> >>>>>> the commit:
> >>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
> >>>>>
> >>>>> +riscv maintainers
> >>>>>
> >>>>> This is riscv64-specific.
> >>>>> I've seen similar crashes in put_user in other places. It looks like
> >>>>> put_user crashes in the user address is not mapped/protected (?).
> >>>>
> >>>> I've been having a look, and this seems to be down to access of the
> >>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
> >>>> bad address to clone?
> >>>>
> >>>> From looking at the code, the put_user() code should have set the
> >>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
> >>>> s2 register in the crash report) and from looking at the compiler
> >>>> output from my gcc-10, the code looks to be dong the relevant csrs
> >>>> and then csrc around the put_user
> >>>>
> >>>> So currently I do not understand how the above could have happened
> >>>> over than something re-tried the code seqeunce and ended up retrying
> >>>> the faulting instruction without the SR_SUM bit set.
> >>>
> >>> I would maybe blame qemu for randomly resetting SR_SUM, but it's
> >>> strange that 99% of these crashes are in schedule_tail. If it would be
> >>> qemu, then they would be more evenly distributed...
> >>>
> >>> Another observation: looking at a dozen of crash logs, in none of
> >>> these cases fuzzer was actually trying to fuzz clone with some insane
> >>> arguments. So it looks like completely normal clone's (e..g coming
> >>> from pthread_create) result in this crash.
> >>>
> >>> I also wonder why there is ret_from_exception, is it normal? I see
> >>> handle_exception disables SR_SUM:
> >>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
> >>>
> >>
> >> So I think if SR_SUM is set, then it faults the access to user memory
> >> which the _user() routines clear to allow them access.
> >>
> >> I'm thinking there is at least one issue here:
> >>
> >> - the test in fault is the wrong way around for die kernel
> >> - the handler only catches this if the page has yet to be mapped.
> >>
> >> So I think the test should be:
> >>
> >> if (!user_mode(regs) && addr < TASK_SIZE &&
> >> unlikely(regs->status & SR_SUM)
> >>
> >> This then should continue on and allow the rest of the handler to
> >> complete mapping the page if it is not there.
> >>
> >> I have been trying to create a very simple clone test, but so far it
> >> has yet to actually trigger anything.
> >
> > I should have added there doesn't seem to be a good way to use mmap()
> > to allocate memory but not insert a vm-mapping post the mmap().
> >
> How difficult is it to try building a branch with the above test
> modified?
I don't have access to hardware, I don't have other qemu versions ready to use.
But I can teach you how to run syzkaller locally :)
I am not sure anybody run it on real riscv hardware at all. When
Tobias ported syzkaller, Tobias also used qemu I think.
I am now building with an inverted check to test locally.
I don't fully understand but this code, but does handle_exception
reset SR_SUM around do_page_fault? If so, then looking at SR_SUM in
do_page_fault won't work with positive nor negative check.
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-12 17:34 ` Dmitry Vyukov
0 siblings, 0 replies; 51+ messages in thread
From: Dmitry Vyukov @ 2021-03-12 17:34 UTC (permalink / raw)
To: Ben Dooks
Cc: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv,
Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
On Fri, Mar 12, 2021 at 5:36 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>
> On 12/03/2021 16:34, Ben Dooks wrote:
> > On 12/03/2021 16:30, Ben Dooks wrote:
> >> On 12/03/2021 15:12, Dmitry Vyukov wrote:
> >>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
> >>> wrote:
> >>>>
> >>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
> >>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
> >>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
> >>>>>>
> >>>>>> Hello,
> >>>>>>
> >>>>>> syzbot found the following issue on:
> >>>>>>
> >>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
> >>>>>> arch_dup_tas..
> >>>>>> git tree:
> >>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
> >>>>>> console output:
> >>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
> >>>>>> kernel config:
> >>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
> >>>>>> dashboard link:
> >>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
> >>>>>> userspace arch: riscv64
> >>>>>>
> >>>>>> Unfortunately, I don't have any reproducer for this issue yet.
> >>>>>>
> >>>>>> IMPORTANT: if you fix the issue, please add the following tag to
> >>>>>> the commit:
> >>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
> >>>>>
> >>>>> +riscv maintainers
> >>>>>
> >>>>> This is riscv64-specific.
> >>>>> I've seen similar crashes in put_user in other places. It looks like
> >>>>> put_user crashes in the user address is not mapped/protected (?).
> >>>>
> >>>> I've been having a look, and this seems to be down to access of the
> >>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
> >>>> bad address to clone?
> >>>>
> >>>> From looking at the code, the put_user() code should have set the
> >>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
> >>>> s2 register in the crash report) and from looking at the compiler
> >>>> output from my gcc-10, the code looks to be dong the relevant csrs
> >>>> and then csrc around the put_user
> >>>>
> >>>> So currently I do not understand how the above could have happened
> >>>> over than something re-tried the code seqeunce and ended up retrying
> >>>> the faulting instruction without the SR_SUM bit set.
> >>>
> >>> I would maybe blame qemu for randomly resetting SR_SUM, but it's
> >>> strange that 99% of these crashes are in schedule_tail. If it would be
> >>> qemu, then they would be more evenly distributed...
> >>>
> >>> Another observation: looking at a dozen of crash logs, in none of
> >>> these cases fuzzer was actually trying to fuzz clone with some insane
> >>> arguments. So it looks like completely normal clone's (e..g coming
> >>> from pthread_create) result in this crash.
> >>>
> >>> I also wonder why there is ret_from_exception, is it normal? I see
> >>> handle_exception disables SR_SUM:
> >>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
> >>>
> >>
> >> So I think if SR_SUM is set, then it faults the access to user memory
> >> which the _user() routines clear to allow them access.
> >>
> >> I'm thinking there is at least one issue here:
> >>
> >> - the test in fault is the wrong way around for die kernel
> >> - the handler only catches this if the page has yet to be mapped.
> >>
> >> So I think the test should be:
> >>
> >> if (!user_mode(regs) && addr < TASK_SIZE &&
> >> unlikely(regs->status & SR_SUM)
> >>
> >> This then should continue on and allow the rest of the handler to
> >> complete mapping the page if it is not there.
> >>
> >> I have been trying to create a very simple clone test, but so far it
> >> has yet to actually trigger anything.
> >
> > I should have added there doesn't seem to be a good way to use mmap()
> > to allocate memory but not insert a vm-mapping post the mmap().
> >
> How difficult is it to try building a branch with the above test
> modified?
I don't have access to hardware, I don't have other qemu versions ready to use.
But I can teach you how to run syzkaller locally :)
I am not sure anybody run it on real riscv hardware at all. When
Tobias ported syzkaller, Tobias also used qemu I think.
I am now building with an inverted check to test locally.
I don't fully understand but this code, but does handle_exception
reset SR_SUM around do_page_fault? If so, then looking at SR_SUM in
do_page_fault won't work with positive nor negative check.
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-12 17:34 ` Dmitry Vyukov
@ 2021-03-12 17:38 ` Dmitry Vyukov
-1 siblings, 0 replies; 51+ messages in thread
From: Dmitry Vyukov @ 2021-03-12 17:38 UTC (permalink / raw)
To: Ben Dooks
Cc: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv,
Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
On Fri, Mar 12, 2021 at 6:34 PM Dmitry Vyukov <dvyukov@google.com> wrote:
>
> On Fri, Mar 12, 2021 at 5:36 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
> >
> > On 12/03/2021 16:34, Ben Dooks wrote:
> > > On 12/03/2021 16:30, Ben Dooks wrote:
> > >> On 12/03/2021 15:12, Dmitry Vyukov wrote:
> > >>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
> > >>> wrote:
> > >>>>
> > >>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
> > >>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
> > >>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
> > >>>>>>
> > >>>>>> Hello,
> > >>>>>>
> > >>>>>> syzbot found the following issue on:
> > >>>>>>
> > >>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
> > >>>>>> arch_dup_tas..
> > >>>>>> git tree:
> > >>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
> > >>>>>> console output:
> > >>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
> > >>>>>> kernel config:
> > >>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
> > >>>>>> dashboard link:
> > >>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
> > >>>>>> userspace arch: riscv64
> > >>>>>>
> > >>>>>> Unfortunately, I don't have any reproducer for this issue yet.
> > >>>>>>
> > >>>>>> IMPORTANT: if you fix the issue, please add the following tag to
> > >>>>>> the commit:
> > >>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
> > >>>>>
> > >>>>> +riscv maintainers
> > >>>>>
> > >>>>> This is riscv64-specific.
> > >>>>> I've seen similar crashes in put_user in other places. It looks like
> > >>>>> put_user crashes in the user address is not mapped/protected (?).
> > >>>>
> > >>>> I've been having a look, and this seems to be down to access of the
> > >>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
> > >>>> bad address to clone?
> > >>>>
> > >>>> From looking at the code, the put_user() code should have set the
> > >>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
> > >>>> s2 register in the crash report) and from looking at the compiler
> > >>>> output from my gcc-10, the code looks to be dong the relevant csrs
> > >>>> and then csrc around the put_user
> > >>>>
> > >>>> So currently I do not understand how the above could have happened
> > >>>> over than something re-tried the code seqeunce and ended up retrying
> > >>>> the faulting instruction without the SR_SUM bit set.
> > >>>
> > >>> I would maybe blame qemu for randomly resetting SR_SUM, but it's
> > >>> strange that 99% of these crashes are in schedule_tail. If it would be
> > >>> qemu, then they would be more evenly distributed...
> > >>>
> > >>> Another observation: looking at a dozen of crash logs, in none of
> > >>> these cases fuzzer was actually trying to fuzz clone with some insane
> > >>> arguments. So it looks like completely normal clone's (e..g coming
> > >>> from pthread_create) result in this crash.
> > >>>
> > >>> I also wonder why there is ret_from_exception, is it normal? I see
> > >>> handle_exception disables SR_SUM:
> > >>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
> > >>>
> > >>
> > >> So I think if SR_SUM is set, then it faults the access to user memory
> > >> which the _user() routines clear to allow them access.
> > >>
> > >> I'm thinking there is at least one issue here:
> > >>
> > >> - the test in fault is the wrong way around for die kernel
> > >> - the handler only catches this if the page has yet to be mapped.
> > >>
> > >> So I think the test should be:
> > >>
> > >> if (!user_mode(regs) && addr < TASK_SIZE &&
> > >> unlikely(regs->status & SR_SUM)
> > >>
> > >> This then should continue on and allow the rest of the handler to
> > >> complete mapping the page if it is not there.
> > >>
> > >> I have been trying to create a very simple clone test, but so far it
> > >> has yet to actually trigger anything.
> > >
> > > I should have added there doesn't seem to be a good way to use mmap()
> > > to allocate memory but not insert a vm-mapping post the mmap().
> > >
> > How difficult is it to try building a branch with the above test
> > modified?
>
> I don't have access to hardware, I don't have other qemu versions ready to use.
> But I can teach you how to run syzkaller locally :)
> I am not sure anybody run it on real riscv hardware at all. When
> Tobias ported syzkaller, Tobias also used qemu I think.
>
> I am now building with an inverted check to test locally.
>
> I don't fully understand but this code, but does handle_exception
> reset SR_SUM around do_page_fault? If so, then looking at SR_SUM in
> do_page_fault won't work with positive nor negative check.
The inverted check crashes during boot:
--- a/arch/riscv/mm/fault.c
+++ b/arch/riscv/mm/fault.c
@@ -249,7 +249,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
flags |= FAULT_FLAG_USER;
if (!user_mode(regs) && addr < TASK_SIZE &&
- unlikely(!(regs->status & SR_SUM)))
+ unlikely(regs->status & SR_SUM))
die_kernel_fault("access to user memory without
uaccess routines",
addr, regs);
[ 77.349329][ T1] Run /sbin/init as init process
[ 77.868371][ T1] Unable to handle kernel access to user memory
without uaccess routines at virtual address 00000000000e8e39
[ 77.870355][ T1] Oops [#1]
[ 77.870766][ T1] Modules linked in:
[ 77.871326][ T1] CPU: 0 PID: 1 Comm: init Not tainted
5.12.0-rc2-00010-g0d7588ab9ef9-dirty #42
[ 77.872057][ T1] Hardware name: riscv-virtio,qemu (DT)
[ 77.872620][ T1] epc : __clear_user+0x36/0x4e
[ 77.873285][ T1] ra : padzero+0x9c/0xb0
[ 77.873849][ T1] epc : ffffffe000bb7136 ra : ffffffe0004f42a0 sp
: ffffffe006f8fbc0
[ 77.874438][ T1] gp : ffffffe005d25718 tp : ffffffe006f98000 t0
: 00000000000e8e40
[ 77.875031][ T1] t1 : 00000000000e9000 t2 : 000000000001c49c s0
: ffffffe006f8fbf0
[ 77.875618][ T1] s1 : 00000000000001c7 a0 : 00000000000e8e39 a1
: 00000000000001c7
[ 77.876204][ T1] a2 : 0000000000000002 a3 : 00000000000e9000 a4
: ffffffe006f99000
[ 77.876787][ T1] a5 : 0000000000000000 a6 : 0000000000f00000 a7
: ffffffe00031c088
[ 77.877367][ T1] s2 : 00000000000e8e39 s3 : 0000000000001000 s4
: 0000003ffffffe39
[ 77.877952][ T1] s5 : 00000000000e8e39 s6 : 00000000000e9570 s7
: 00000000000e8e39
[ 77.878535][ T1] s8 : 0000000000000001 s9 : 00000000000e8e39
s10: ffffffe00c65f608
[ 77.879126][ T1] s11: ffffffe00816e8d8 t3 : ea3af0fa372b8300 t4
: 0000000000000003
[ 77.879711][ T1] t5 : ffffffc401dc45d8 t6 : 0000000000040000
[ 77.880209][ T1] status: 0000000000040120 badaddr:
00000000000e8e39 cause: 000000000000000f
[ 77.880846][ T1] Call Trace:
[ 77.881213][ T1] [<ffffffe000bb7136>] __clear_user+0x36/0x4e
[ 77.881912][ T1] [<ffffffe0004f523e>] load_elf_binary+0xf8a/0x2400
[ 77.882562][ T1] [<ffffffe0003e1802>] bprm_execve+0x5b0/0x1080
[ 77.883145][ T1] [<ffffffe0003e38bc>] kernel_execve+0x204/0x288
[ 77.883727][ T1] [<ffffffe003b70e94>] run_init_process+0x1fe/0x212
[ 77.884337][ T1] [<ffffffe003b70ec6>] try_to_run_init_process+0x1e/0x66
[ 77.884956][ T1] [<ffffffe003bc0864>] kernel_init+0x14a/0x200
[ 77.885541][ T1] [<ffffffe000005570>] ret_from_exception+0x0/0x14
[ 77.886955][ T1] ---[ end trace 1e934d07b8a4bed8 ]---
[ 77.887705][ T1] Kernel panic - not syncing: Fatal exception
[ 77.888333][ T1] SMP: stopping secondary CPUs
[ 77.889357][ T1] Rebooting in 86400 seconds..
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-12 17:38 ` Dmitry Vyukov
0 siblings, 0 replies; 51+ messages in thread
From: Dmitry Vyukov @ 2021-03-12 17:38 UTC (permalink / raw)
To: Ben Dooks
Cc: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv,
Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
On Fri, Mar 12, 2021 at 6:34 PM Dmitry Vyukov <dvyukov@google.com> wrote:
>
> On Fri, Mar 12, 2021 at 5:36 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
> >
> > On 12/03/2021 16:34, Ben Dooks wrote:
> > > On 12/03/2021 16:30, Ben Dooks wrote:
> > >> On 12/03/2021 15:12, Dmitry Vyukov wrote:
> > >>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
> > >>> wrote:
> > >>>>
> > >>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
> > >>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
> > >>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
> > >>>>>>
> > >>>>>> Hello,
> > >>>>>>
> > >>>>>> syzbot found the following issue on:
> > >>>>>>
> > >>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
> > >>>>>> arch_dup_tas..
> > >>>>>> git tree:
> > >>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
> > >>>>>> console output:
> > >>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
> > >>>>>> kernel config:
> > >>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
> > >>>>>> dashboard link:
> > >>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
> > >>>>>> userspace arch: riscv64
> > >>>>>>
> > >>>>>> Unfortunately, I don't have any reproducer for this issue yet.
> > >>>>>>
> > >>>>>> IMPORTANT: if you fix the issue, please add the following tag to
> > >>>>>> the commit:
> > >>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
> > >>>>>
> > >>>>> +riscv maintainers
> > >>>>>
> > >>>>> This is riscv64-specific.
> > >>>>> I've seen similar crashes in put_user in other places. It looks like
> > >>>>> put_user crashes in the user address is not mapped/protected (?).
> > >>>>
> > >>>> I've been having a look, and this seems to be down to access of the
> > >>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
> > >>>> bad address to clone?
> > >>>>
> > >>>> From looking at the code, the put_user() code should have set the
> > >>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
> > >>>> s2 register in the crash report) and from looking at the compiler
> > >>>> output from my gcc-10, the code looks to be dong the relevant csrs
> > >>>> and then csrc around the put_user
> > >>>>
> > >>>> So currently I do not understand how the above could have happened
> > >>>> over than something re-tried the code seqeunce and ended up retrying
> > >>>> the faulting instruction without the SR_SUM bit set.
> > >>>
> > >>> I would maybe blame qemu for randomly resetting SR_SUM, but it's
> > >>> strange that 99% of these crashes are in schedule_tail. If it would be
> > >>> qemu, then they would be more evenly distributed...
> > >>>
> > >>> Another observation: looking at a dozen of crash logs, in none of
> > >>> these cases fuzzer was actually trying to fuzz clone with some insane
> > >>> arguments. So it looks like completely normal clone's (e..g coming
> > >>> from pthread_create) result in this crash.
> > >>>
> > >>> I also wonder why there is ret_from_exception, is it normal? I see
> > >>> handle_exception disables SR_SUM:
> > >>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
> > >>>
> > >>
> > >> So I think if SR_SUM is set, then it faults the access to user memory
> > >> which the _user() routines clear to allow them access.
> > >>
> > >> I'm thinking there is at least one issue here:
> > >>
> > >> - the test in fault is the wrong way around for die kernel
> > >> - the handler only catches this if the page has yet to be mapped.
> > >>
> > >> So I think the test should be:
> > >>
> > >> if (!user_mode(regs) && addr < TASK_SIZE &&
> > >> unlikely(regs->status & SR_SUM)
> > >>
> > >> This then should continue on and allow the rest of the handler to
> > >> complete mapping the page if it is not there.
> > >>
> > >> I have been trying to create a very simple clone test, but so far it
> > >> has yet to actually trigger anything.
> > >
> > > I should have added there doesn't seem to be a good way to use mmap()
> > > to allocate memory but not insert a vm-mapping post the mmap().
> > >
> > How difficult is it to try building a branch with the above test
> > modified?
>
> I don't have access to hardware, I don't have other qemu versions ready to use.
> But I can teach you how to run syzkaller locally :)
> I am not sure anybody run it on real riscv hardware at all. When
> Tobias ported syzkaller, Tobias also used qemu I think.
>
> I am now building with an inverted check to test locally.
>
> I don't fully understand but this code, but does handle_exception
> reset SR_SUM around do_page_fault? If so, then looking at SR_SUM in
> do_page_fault won't work with positive nor negative check.
The inverted check crashes during boot:
--- a/arch/riscv/mm/fault.c
+++ b/arch/riscv/mm/fault.c
@@ -249,7 +249,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
flags |= FAULT_FLAG_USER;
if (!user_mode(regs) && addr < TASK_SIZE &&
- unlikely(!(regs->status & SR_SUM)))
+ unlikely(regs->status & SR_SUM))
die_kernel_fault("access to user memory without
uaccess routines",
addr, regs);
[ 77.349329][ T1] Run /sbin/init as init process
[ 77.868371][ T1] Unable to handle kernel access to user memory
without uaccess routines at virtual address 00000000000e8e39
[ 77.870355][ T1] Oops [#1]
[ 77.870766][ T1] Modules linked in:
[ 77.871326][ T1] CPU: 0 PID: 1 Comm: init Not tainted
5.12.0-rc2-00010-g0d7588ab9ef9-dirty #42
[ 77.872057][ T1] Hardware name: riscv-virtio,qemu (DT)
[ 77.872620][ T1] epc : __clear_user+0x36/0x4e
[ 77.873285][ T1] ra : padzero+0x9c/0xb0
[ 77.873849][ T1] epc : ffffffe000bb7136 ra : ffffffe0004f42a0 sp
: ffffffe006f8fbc0
[ 77.874438][ T1] gp : ffffffe005d25718 tp : ffffffe006f98000 t0
: 00000000000e8e40
[ 77.875031][ T1] t1 : 00000000000e9000 t2 : 000000000001c49c s0
: ffffffe006f8fbf0
[ 77.875618][ T1] s1 : 00000000000001c7 a0 : 00000000000e8e39 a1
: 00000000000001c7
[ 77.876204][ T1] a2 : 0000000000000002 a3 : 00000000000e9000 a4
: ffffffe006f99000
[ 77.876787][ T1] a5 : 0000000000000000 a6 : 0000000000f00000 a7
: ffffffe00031c088
[ 77.877367][ T1] s2 : 00000000000e8e39 s3 : 0000000000001000 s4
: 0000003ffffffe39
[ 77.877952][ T1] s5 : 00000000000e8e39 s6 : 00000000000e9570 s7
: 00000000000e8e39
[ 77.878535][ T1] s8 : 0000000000000001 s9 : 00000000000e8e39
s10: ffffffe00c65f608
[ 77.879126][ T1] s11: ffffffe00816e8d8 t3 : ea3af0fa372b8300 t4
: 0000000000000003
[ 77.879711][ T1] t5 : ffffffc401dc45d8 t6 : 0000000000040000
[ 77.880209][ T1] status: 0000000000040120 badaddr:
00000000000e8e39 cause: 000000000000000f
[ 77.880846][ T1] Call Trace:
[ 77.881213][ T1] [<ffffffe000bb7136>] __clear_user+0x36/0x4e
[ 77.881912][ T1] [<ffffffe0004f523e>] load_elf_binary+0xf8a/0x2400
[ 77.882562][ T1] [<ffffffe0003e1802>] bprm_execve+0x5b0/0x1080
[ 77.883145][ T1] [<ffffffe0003e38bc>] kernel_execve+0x204/0x288
[ 77.883727][ T1] [<ffffffe003b70e94>] run_init_process+0x1fe/0x212
[ 77.884337][ T1] [<ffffffe003b70ec6>] try_to_run_init_process+0x1e/0x66
[ 77.884956][ T1] [<ffffffe003bc0864>] kernel_init+0x14a/0x200
[ 77.885541][ T1] [<ffffffe000005570>] ret_from_exception+0x0/0x14
[ 77.886955][ T1] ---[ end trace 1e934d07b8a4bed8 ]---
[ 77.887705][ T1] Kernel panic - not syncing: Fatal exception
[ 77.888333][ T1] SMP: stopping secondary CPUs
[ 77.889357][ T1] Rebooting in 86400 seconds..
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-12 16:25 ` Alex Ghiti
@ 2021-03-12 20:12 ` Ben Dooks
-1 siblings, 0 replies; 51+ messages in thread
From: Ben Dooks @ 2021-03-12 20:12 UTC (permalink / raw)
To: Alex Ghiti, Dmitry Vyukov
Cc: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv,
Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
On 12/03/2021 16:25, Alex Ghiti wrote:
>
>
> Le 3/12/21 à 10:12 AM, Dmitry Vyukov a écrit :
>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
>> wrote:
>>>
>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> syzbot found the following issue on:
>>>>>
>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
>>>>> arch_dup_tas..
>>>>> git tree:
>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>>>>> console output:
>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>>>>> kernel config:
>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>>>>> dashboard link:
>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>>>>> userspace arch: riscv64
>>>>>
>>>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>>>
>>>>> IMPORTANT: if you fix the issue, please add the following tag to
>>>>> the commit:
>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>>>>
>>>> +riscv maintainers
>>>>
>>>> This is riscv64-specific.
>>>> I've seen similar crashes in put_user in other places. It looks like
>>>> put_user crashes in the user address is not mapped/protected (?).
>>>
>>> I've been having a look, and this seems to be down to access of the
>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
>>> bad address to clone?
>>>
>>> From looking at the code, the put_user() code should have set the
>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
>>> s2 register in the crash report) and from looking at the compiler
>>> output from my gcc-10, the code looks to be dong the relevant csrs
>>> and then csrc around the put_user
>>>
>>> So currently I do not understand how the above could have happened
>>> over than something re-tried the code seqeunce and ended up retrying
>>> the faulting instruction without the SR_SUM bit set.
>>
>> I would maybe blame qemu for randomly resetting SR_SUM, but it's
>> strange that 99% of these crashes are in schedule_tail. If it would be
>> qemu, then they would be more evenly distributed...
>>
>> Another observation: looking at a dozen of crash logs, in none of
>> these cases fuzzer was actually trying to fuzz clone with some insane
>> arguments. So it looks like completely normal clone's (e..g coming
>> from pthread_create) result in this crash.
>>
>> I also wonder why there is ret_from_exception, is it normal? I see
>> handle_exception disables SR_SUM:
>
> csrrc does the right thing: it cleans SR_SUM bit in status but saves the
> previous value that will get correctly restored.
>
> ("The CSRRC (Atomic Read and Clear Bits in CSR) instruction reads the
> value of the CSR, zero-extends the value to XLEN bits, and writes it to
> integer registerrd. The initial value in integerregisterrs1is treated
> as a bit mask that specifies bit positions to be cleared in the CSR. Any
> bitthat is high inrs1will cause the corresponding bit to be cleared in
> the CSR, if that CSR bit iswritable. Other bits in the CSR are
> unaffected.")
I think there may also be an understanding issue on what the SR_SUM
bit does. I thought if it is set, M->U accesses would fault, which is
why it gets set early on. But from reading the uaccess code it looks
like the uaccess code sets it on entry and then clears on exit.
I am very confused. Is there a master reference for rv64?
https://people.eecs.berkeley.edu/~krste/papers/riscv-privileged-v1.9.pdf
seems to state PUM is the SR_SUM bit, and that (if set) disabled
Quote:
The PUM (Protect User Memory) bit modifies the privilege with which
S-mode loads, stores, and instruction fetches access virtual memory.
When PUM=0, translation and protection behave as normal. When PUM=1,
S-mode memory accesses to pages that are accessible by U-mode (U=1 in
Figure 4.19) will fault. PUM has no effect when executing in U-mode
>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
>>
>
> Still no luck for the moment, can't reproduce it locally, my test is
> maybe not that good (I created threads all day long in order to trigger
> the put_user of schedule_tail).
It may of course depend on memory and other stuff. I did try to see if
it was possible to clone() with the child_tid address being a valid but
not mapped page...
> Given that the path you mention works most of the time, and that the
> status register in the stack trace shows the SUM bit is not set whereas
> it is set in put_user, I'm leaning toward some race condition (maybe an
> interrupt that arrives at the "wrong" time) or a qemu issue as you
> mentioned.
I suppose this is possible. From what I read it should get to the
point of being there with the SUM flag cleared, so either something
went wrong in trying to fix the instruction up or there's some other
error we're missing.
> To eliminate qemu issues, do you have access to some HW ? Or to
> different qemu versions ?
I do have access to a Microchip Polarfire board. I just need the
instructions on how to setup the test-code to make it work on the
hardware.
The config supplied takes /ages/ to boot on qemu even on my ryzen9.
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-12 20:12 ` Ben Dooks
0 siblings, 0 replies; 51+ messages in thread
From: Ben Dooks @ 2021-03-12 20:12 UTC (permalink / raw)
To: Alex Ghiti, Dmitry Vyukov
Cc: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv,
Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot
On 12/03/2021 16:25, Alex Ghiti wrote:
>
>
> Le 3/12/21 à 10:12 AM, Dmitry Vyukov a écrit :
>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
>> wrote:
>>>
>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> syzbot found the following issue on:
>>>>>
>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
>>>>> arch_dup_tas..
>>>>> git tree:
>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>>>>> console output:
>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>>>>> kernel config:
>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>>>>> dashboard link:
>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>>>>> userspace arch: riscv64
>>>>>
>>>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>>>
>>>>> IMPORTANT: if you fix the issue, please add the following tag to
>>>>> the commit:
>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>>>>
>>>> +riscv maintainers
>>>>
>>>> This is riscv64-specific.
>>>> I've seen similar crashes in put_user in other places. It looks like
>>>> put_user crashes in the user address is not mapped/protected (?).
>>>
>>> I've been having a look, and this seems to be down to access of the
>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
>>> bad address to clone?
>>>
>>> From looking at the code, the put_user() code should have set the
>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
>>> s2 register in the crash report) and from looking at the compiler
>>> output from my gcc-10, the code looks to be dong the relevant csrs
>>> and then csrc around the put_user
>>>
>>> So currently I do not understand how the above could have happened
>>> over than something re-tried the code seqeunce and ended up retrying
>>> the faulting instruction without the SR_SUM bit set.
>>
>> I would maybe blame qemu for randomly resetting SR_SUM, but it's
>> strange that 99% of these crashes are in schedule_tail. If it would be
>> qemu, then they would be more evenly distributed...
>>
>> Another observation: looking at a dozen of crash logs, in none of
>> these cases fuzzer was actually trying to fuzz clone with some insane
>> arguments. So it looks like completely normal clone's (e..g coming
>> from pthread_create) result in this crash.
>>
>> I also wonder why there is ret_from_exception, is it normal? I see
>> handle_exception disables SR_SUM:
>
> csrrc does the right thing: it cleans SR_SUM bit in status but saves the
> previous value that will get correctly restored.
>
> ("The CSRRC (Atomic Read and Clear Bits in CSR) instruction reads the
> value of the CSR, zero-extends the value to XLEN bits, and writes it to
> integer registerrd. The initial value in integerregisterrs1is treated
> as a bit mask that specifies bit positions to be cleared in the CSR. Any
> bitthat is high inrs1will cause the corresponding bit to be cleared in
> the CSR, if that CSR bit iswritable. Other bits in the CSR are
> unaffected.")
I think there may also be an understanding issue on what the SR_SUM
bit does. I thought if it is set, M->U accesses would fault, which is
why it gets set early on. But from reading the uaccess code it looks
like the uaccess code sets it on entry and then clears on exit.
I am very confused. Is there a master reference for rv64?
https://people.eecs.berkeley.edu/~krste/papers/riscv-privileged-v1.9.pdf
seems to state PUM is the SR_SUM bit, and that (if set) disabled
Quote:
The PUM (Protect User Memory) bit modifies the privilege with which
S-mode loads, stores, and instruction fetches access virtual memory.
When PUM=0, translation and protection behave as normal. When PUM=1,
S-mode memory accesses to pages that are accessible by U-mode (U=1 in
Figure 4.19) will fault. PUM has no effect when executing in U-mode
>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
>>
>
> Still no luck for the moment, can't reproduce it locally, my test is
> maybe not that good (I created threads all day long in order to trigger
> the put_user of schedule_tail).
It may of course depend on memory and other stuff. I did try to see if
it was possible to clone() with the child_tid address being a valid but
not mapped page...
> Given that the path you mention works most of the time, and that the
> status register in the stack trace shows the SUM bit is not set whereas
> it is set in put_user, I'm leaning toward some race condition (maybe an
> interrupt that arrives at the "wrong" time) or a qemu issue as you
> mentioned.
I suppose this is possible. From what I read it should get to the
point of being there with the SUM flag cleared, so either something
went wrong in trying to fix the instruction up or there's some other
error we're missing.
> To eliminate qemu issues, do you have access to some HW ? Or to
> different qemu versions ?
I do have access to a Microchip Polarfire board. I just need the
instructions on how to setup the test-code to make it work on the
hardware.
The config supplied takes /ages/ to boot on qemu even on my ryzen9.
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-12 20:12 ` Ben Dooks
@ 2021-03-13 7:20 ` Dmitry Vyukov
-1 siblings, 0 replies; 51+ messages in thread
From: Dmitry Vyukov @ 2021-03-13 7:20 UTC (permalink / raw)
To: Ben Dooks
Cc: Alex Ghiti, syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou,
linux-riscv, Daniel Bristot de Oliveira, Benjamin Segall,
dietmar.eggemann, Juri Lelli, LKML, Mel Gorman, Ingo Molnar,
Peter Zijlstra, Steven Rostedt, syzkaller-bugs, Vincent Guittot
On Fri, Mar 12, 2021 at 9:12 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>
> On 12/03/2021 16:25, Alex Ghiti wrote:
> >
> >
> > Le 3/12/21 à 10:12 AM, Dmitry Vyukov a écrit :
> >> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
> >> wrote:
> >>>
> >>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
> >>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
> >>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
> >>>>>
> >>>>> Hello,
> >>>>>
> >>>>> syzbot found the following issue on:
> >>>>>
> >>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
> >>>>> arch_dup_tas..
> >>>>> git tree:
> >>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
> >>>>> console output:
> >>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
> >>>>> kernel config:
> >>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
> >>>>> dashboard link:
> >>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
> >>>>> userspace arch: riscv64
> >>>>>
> >>>>> Unfortunately, I don't have any reproducer for this issue yet.
> >>>>>
> >>>>> IMPORTANT: if you fix the issue, please add the following tag to
> >>>>> the commit:
> >>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
> >>>>
> >>>> +riscv maintainers
> >>>>
> >>>> This is riscv64-specific.
> >>>> I've seen similar crashes in put_user in other places. It looks like
> >>>> put_user crashes in the user address is not mapped/protected (?).
> >>>
> >>> I've been having a look, and this seems to be down to access of the
> >>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
> >>> bad address to clone?
> >>>
> >>> From looking at the code, the put_user() code should have set the
> >>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
> >>> s2 register in the crash report) and from looking at the compiler
> >>> output from my gcc-10, the code looks to be dong the relevant csrs
> >>> and then csrc around the put_user
> >>>
> >>> So currently I do not understand how the above could have happened
> >>> over than something re-tried the code seqeunce and ended up retrying
> >>> the faulting instruction without the SR_SUM bit set.
> >>
> >> I would maybe blame qemu for randomly resetting SR_SUM, but it's
> >> strange that 99% of these crashes are in schedule_tail. If it would be
> >> qemu, then they would be more evenly distributed...
> >>
> >> Another observation: looking at a dozen of crash logs, in none of
> >> these cases fuzzer was actually trying to fuzz clone with some insane
> >> arguments. So it looks like completely normal clone's (e..g coming
> >> from pthread_create) result in this crash.
> >>
> >> I also wonder why there is ret_from_exception, is it normal? I see
> >> handle_exception disables SR_SUM:
> >
> > csrrc does the right thing: it cleans SR_SUM bit in status but saves the
> > previous value that will get correctly restored.
> >
> > ("The CSRRC (Atomic Read and Clear Bits in CSR) instruction reads the
> > value of the CSR, zero-extends the value to XLEN bits, and writes it to
> > integer registerrd. The initial value in integerregisterrs1is treated
> > as a bit mask that specifies bit positions to be cleared in the CSR. Any
> > bitthat is high inrs1will cause the corresponding bit to be cleared in
> > the CSR, if that CSR bit iswritable. Other bits in the CSR are
> > unaffected.")
>
> I think there may also be an understanding issue on what the SR_SUM
> bit does. I thought if it is set, M->U accesses would fault, which is
> why it gets set early on. But from reading the uaccess code it looks
> like the uaccess code sets it on entry and then clears on exit.
>
> I am very confused. Is there a master reference for rv64?
>
> https://people.eecs.berkeley.edu/~krste/papers/riscv-privileged-v1.9.pdf
> seems to state PUM is the SR_SUM bit, and that (if set) disabled
>
> Quote:
> The PUM (Protect User Memory) bit modifies the privilege with which
> S-mode loads, stores, and instruction fetches access virtual memory.
> When PUM=0, translation and protection behave as normal. When PUM=1,
> S-mode memory accesses to pages that are accessible by U-mode (U=1 in
> Figure 4.19) will fault. PUM has no effect when executing in U-mode
>
>
> >> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
> >>
> >
> > Still no luck for the moment, can't reproduce it locally, my test is
> > maybe not that good (I created threads all day long in order to trigger
> > the put_user of schedule_tail).
>
> It may of course depend on memory and other stuff. I did try to see if
> it was possible to clone() with the child_tid address being a valid but
> not mapped page...
>
> > Given that the path you mention works most of the time, and that the
> > status register in the stack trace shows the SUM bit is not set whereas
> > it is set in put_user, I'm leaning toward some race condition (maybe an
> > interrupt that arrives at the "wrong" time) or a qemu issue as you
> > mentioned.
>
> I suppose this is possible. From what I read it should get to the
> point of being there with the SUM flag cleared, so either something
> went wrong in trying to fix the instruction up or there's some other
> error we're missing.
>
> > To eliminate qemu issues, do you have access to some HW ? Or to
> > different qemu versions ?
>
> I do have access to a Microchip Polarfire board. I just need the
> instructions on how to setup the test-code to make it work on the
> hardware.
For full syzkaller support, it would need to know how to reboot these
boards and get access to the console.
syzkaller has a stop-gap VM backend which just uses ssh to a physical
machine and expects the kernel to reboot on its own after any crashes.
But I actually managed to reproduce it in an even simpler setup.
Assuming you have Go 1.15 and riscv64 cross-compiler gcc installed
$ go get -u -d github.com/google/syzkaller/...
$ cd $GOPATH/src/github.com/google/syzkaller
$ make stress executor TARGETARCH=riscv64
$ scp bin/linux_riscv64/syz-execprog bin/linux_riscv64/syz-executor
your_machine:/
Then run ./syz-stress on the machine.
On the first run it crashed it with some other bug, on the second run
I got the crash in schedule_tail.
With qemu tcg I also added -slowdown=10 flag to syz-stress to scale
all timeouts, if native execution is faster, then you don't need it.
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-13 7:20 ` Dmitry Vyukov
0 siblings, 0 replies; 51+ messages in thread
From: Dmitry Vyukov @ 2021-03-13 7:20 UTC (permalink / raw)
To: Ben Dooks
Cc: Alex Ghiti, syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou,
linux-riscv, Daniel Bristot de Oliveira, Benjamin Segall,
dietmar.eggemann, Juri Lelli, LKML, Mel Gorman, Ingo Molnar,
Peter Zijlstra, Steven Rostedt, syzkaller-bugs, Vincent Guittot
On Fri, Mar 12, 2021 at 9:12 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>
> On 12/03/2021 16:25, Alex Ghiti wrote:
> >
> >
> > Le 3/12/21 à 10:12 AM, Dmitry Vyukov a écrit :
> >> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
> >> wrote:
> >>>
> >>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
> >>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
> >>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
> >>>>>
> >>>>> Hello,
> >>>>>
> >>>>> syzbot found the following issue on:
> >>>>>
> >>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
> >>>>> arch_dup_tas..
> >>>>> git tree:
> >>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
> >>>>> console output:
> >>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
> >>>>> kernel config:
> >>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
> >>>>> dashboard link:
> >>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
> >>>>> userspace arch: riscv64
> >>>>>
> >>>>> Unfortunately, I don't have any reproducer for this issue yet.
> >>>>>
> >>>>> IMPORTANT: if you fix the issue, please add the following tag to
> >>>>> the commit:
> >>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
> >>>>
> >>>> +riscv maintainers
> >>>>
> >>>> This is riscv64-specific.
> >>>> I've seen similar crashes in put_user in other places. It looks like
> >>>> put_user crashes in the user address is not mapped/protected (?).
> >>>
> >>> I've been having a look, and this seems to be down to access of the
> >>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
> >>> bad address to clone?
> >>>
> >>> From looking at the code, the put_user() code should have set the
> >>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
> >>> s2 register in the crash report) and from looking at the compiler
> >>> output from my gcc-10, the code looks to be dong the relevant csrs
> >>> and then csrc around the put_user
> >>>
> >>> So currently I do not understand how the above could have happened
> >>> over than something re-tried the code seqeunce and ended up retrying
> >>> the faulting instruction without the SR_SUM bit set.
> >>
> >> I would maybe blame qemu for randomly resetting SR_SUM, but it's
> >> strange that 99% of these crashes are in schedule_tail. If it would be
> >> qemu, then they would be more evenly distributed...
> >>
> >> Another observation: looking at a dozen of crash logs, in none of
> >> these cases fuzzer was actually trying to fuzz clone with some insane
> >> arguments. So it looks like completely normal clone's (e..g coming
> >> from pthread_create) result in this crash.
> >>
> >> I also wonder why there is ret_from_exception, is it normal? I see
> >> handle_exception disables SR_SUM:
> >
> > csrrc does the right thing: it cleans SR_SUM bit in status but saves the
> > previous value that will get correctly restored.
> >
> > ("The CSRRC (Atomic Read and Clear Bits in CSR) instruction reads the
> > value of the CSR, zero-extends the value to XLEN bits, and writes it to
> > integer registerrd. The initial value in integerregisterrs1is treated
> > as a bit mask that specifies bit positions to be cleared in the CSR. Any
> > bitthat is high inrs1will cause the corresponding bit to be cleared in
> > the CSR, if that CSR bit iswritable. Other bits in the CSR are
> > unaffected.")
>
> I think there may also be an understanding issue on what the SR_SUM
> bit does. I thought if it is set, M->U accesses would fault, which is
> why it gets set early on. But from reading the uaccess code it looks
> like the uaccess code sets it on entry and then clears on exit.
>
> I am very confused. Is there a master reference for rv64?
>
> https://people.eecs.berkeley.edu/~krste/papers/riscv-privileged-v1.9.pdf
> seems to state PUM is the SR_SUM bit, and that (if set) disabled
>
> Quote:
> The PUM (Protect User Memory) bit modifies the privilege with which
> S-mode loads, stores, and instruction fetches access virtual memory.
> When PUM=0, translation and protection behave as normal. When PUM=1,
> S-mode memory accesses to pages that are accessible by U-mode (U=1 in
> Figure 4.19) will fault. PUM has no effect when executing in U-mode
>
>
> >> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
> >>
> >
> > Still no luck for the moment, can't reproduce it locally, my test is
> > maybe not that good (I created threads all day long in order to trigger
> > the put_user of schedule_tail).
>
> It may of course depend on memory and other stuff. I did try to see if
> it was possible to clone() with the child_tid address being a valid but
> not mapped page...
>
> > Given that the path you mention works most of the time, and that the
> > status register in the stack trace shows the SUM bit is not set whereas
> > it is set in put_user, I'm leaning toward some race condition (maybe an
> > interrupt that arrives at the "wrong" time) or a qemu issue as you
> > mentioned.
>
> I suppose this is possible. From what I read it should get to the
> point of being there with the SUM flag cleared, so either something
> went wrong in trying to fix the instruction up or there's some other
> error we're missing.
>
> > To eliminate qemu issues, do you have access to some HW ? Or to
> > different qemu versions ?
>
> I do have access to a Microchip Polarfire board. I just need the
> instructions on how to setup the test-code to make it work on the
> hardware.
For full syzkaller support, it would need to know how to reboot these
boards and get access to the console.
syzkaller has a stop-gap VM backend which just uses ssh to a physical
machine and expects the kernel to reboot on its own after any crashes.
But I actually managed to reproduce it in an even simpler setup.
Assuming you have Go 1.15 and riscv64 cross-compiler gcc installed
$ go get -u -d github.com/google/syzkaller/...
$ cd $GOPATH/src/github.com/google/syzkaller
$ make stress executor TARGETARCH=riscv64
$ scp bin/linux_riscv64/syz-execprog bin/linux_riscv64/syz-executor
your_machine:/
Then run ./syz-stress on the machine.
On the first run it crashed it with some other bug, on the second run
I got the crash in schedule_tail.
With qemu tcg I also added -slowdown=10 flag to syz-stress to scale
all timeouts, if native execution is faster, then you don't need it.
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-13 7:20 ` Dmitry Vyukov
@ 2021-03-15 16:55 ` Ben Dooks
-1 siblings, 0 replies; 51+ messages in thread
From: Ben Dooks @ 2021-03-15 16:55 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: Alex Ghiti, syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou,
linux-riscv, Daniel Bristot de Oliveira, Benjamin Segall,
dietmar.eggemann, Juri Lelli, LKML, Mel Gorman, Ingo Molnar,
Peter Zijlstra, Steven Rostedt, syzkaller-bugs, Vincent Guittot
On 13/03/2021 07:20, Dmitry Vyukov wrote:
> On Fri, Mar 12, 2021 at 9:12 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>>> Still no luck for the moment, can't reproduce it locally, my test is
>>> maybe not that good (I created threads all day long in order to trigger
>>> the put_user of schedule_tail).
>>
>> It may of course depend on memory and other stuff. I did try to see if
>> it was possible to clone() with the child_tid address being a valid but
>> not mapped page...
>>
>>> Given that the path you mention works most of the time, and that the
>>> status register in the stack trace shows the SUM bit is not set whereas
>>> it is set in put_user, I'm leaning toward some race condition (maybe an
>>> interrupt that arrives at the "wrong" time) or a qemu issue as you
>>> mentioned.
>>
>> I suppose this is possible. From what I read it should get to the
>> point of being there with the SUM flag cleared, so either something
>> went wrong in trying to fix the instruction up or there's some other
>> error we're missing.
>>
>>> To eliminate qemu issues, do you have access to some HW ? Or to
>>> different qemu versions ?
>>
>> I do have access to a Microchip Polarfire board. I just need the
>> instructions on how to setup the test-code to make it work on the
>> hardware.
>
> For full syzkaller support, it would need to know how to reboot these
> boards and get access to the console.
> syzkaller has a stop-gap VM backend which just uses ssh to a physical
> machine and expects the kernel to reboot on its own after any crashes.
>
> But I actually managed to reproduce it in an even simpler setup.
> Assuming you have Go 1.15 and riscv64 cross-compiler gcc installed
>
> $ go get -u -d github.com/google/syzkaller/...
> $ cd $GOPATH/src/github.com/google/syzkaller
> $ make stress executor TARGETARCH=riscv64
> $ scp bin/linux_riscv64/syz-execprog bin/linux_riscv64/syz-executor
> your_machine:/
>
> Then run ./syz-stress on the machine.
> On the first run it crashed it with some other bug, on the second run
> I got the crash in schedule_tail.
> With qemu tcg I also added -slowdown=10 flag to syz-stress to scale
> all timeouts, if native execution is faster, then you don't need it.
I have built the tools and got it to start.
It would be helpful for the dashboard to give the qemu version and
how it was launched (memory, cpus etc)
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-15 16:55 ` Ben Dooks
0 siblings, 0 replies; 51+ messages in thread
From: Ben Dooks @ 2021-03-15 16:55 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: Alex Ghiti, syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou,
linux-riscv, Daniel Bristot de Oliveira, Benjamin Segall,
dietmar.eggemann, Juri Lelli, LKML, Mel Gorman, Ingo Molnar,
Peter Zijlstra, Steven Rostedt, syzkaller-bugs, Vincent Guittot
On 13/03/2021 07:20, Dmitry Vyukov wrote:
> On Fri, Mar 12, 2021 at 9:12 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>>> Still no luck for the moment, can't reproduce it locally, my test is
>>> maybe not that good (I created threads all day long in order to trigger
>>> the put_user of schedule_tail).
>>
>> It may of course depend on memory and other stuff. I did try to see if
>> it was possible to clone() with the child_tid address being a valid but
>> not mapped page...
>>
>>> Given that the path you mention works most of the time, and that the
>>> status register in the stack trace shows the SUM bit is not set whereas
>>> it is set in put_user, I'm leaning toward some race condition (maybe an
>>> interrupt that arrives at the "wrong" time) or a qemu issue as you
>>> mentioned.
>>
>> I suppose this is possible. From what I read it should get to the
>> point of being there with the SUM flag cleared, so either something
>> went wrong in trying to fix the instruction up or there's some other
>> error we're missing.
>>
>>> To eliminate qemu issues, do you have access to some HW ? Or to
>>> different qemu versions ?
>>
>> I do have access to a Microchip Polarfire board. I just need the
>> instructions on how to setup the test-code to make it work on the
>> hardware.
>
> For full syzkaller support, it would need to know how to reboot these
> boards and get access to the console.
> syzkaller has a stop-gap VM backend which just uses ssh to a physical
> machine and expects the kernel to reboot on its own after any crashes.
>
> But I actually managed to reproduce it in an even simpler setup.
> Assuming you have Go 1.15 and riscv64 cross-compiler gcc installed
>
> $ go get -u -d github.com/google/syzkaller/...
> $ cd $GOPATH/src/github.com/google/syzkaller
> $ make stress executor TARGETARCH=riscv64
> $ scp bin/linux_riscv64/syz-execprog bin/linux_riscv64/syz-executor
> your_machine:/
>
> Then run ./syz-stress on the machine.
> On the first run it crashed it with some other bug, on the second run
> I got the crash in schedule_tail.
> With qemu tcg I also added -slowdown=10 flag to syz-stress to scale
> all timeouts, if native execution is faster, then you don't need it.
I have built the tools and got it to start.
It would be helpful for the dashboard to give the qemu version and
how it was launched (memory, cpus etc)
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-13 7:20 ` Dmitry Vyukov
@ 2021-03-15 21:38 ` Ben Dooks
-1 siblings, 0 replies; 51+ messages in thread
From: Ben Dooks @ 2021-03-15 21:38 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: Alex Ghiti, syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou,
linux-riscv, Daniel Bristot de Oliveira, Benjamin Segall,
dietmar.eggemann, Juri Lelli, LKML, Mel Gorman, Ingo Molnar,
Peter Zijlstra, Steven Rostedt, syzkaller-bugs, Vincent Guittot
On 13/03/2021 07:20, Dmitry Vyukov wrote:
> On Fri, Mar 12, 2021 at 9:12 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>>
>> On 12/03/2021 16:25, Alex Ghiti wrote:
>>>
>>>
>>> Le 3/12/21 à 10:12 AM, Dmitry Vyukov a écrit :
>>>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
>>>> wrote:
>>>>>
>>>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
>>>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
>>>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> syzbot found the following issue on:
>>>>>>>
>>>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
>>>>>>> arch_dup_tas..
>>>>>>> git tree:
>>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>>>>>>> console output:
>>>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>>>>>>> kernel config:
>>>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>>>>>>> dashboard link:
>>>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>>>>>>> userspace arch: riscv64
>>>>>>>
>>>>>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>>>>>
>>>>>>> IMPORTANT: if you fix the issue, please add the following tag to
>>>>>>> the commit:
>>>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>>>>>>
>>>>>> +riscv maintainers
>>>>>>
>>>>>> This is riscv64-specific.
>>>>>> I've seen similar crashes in put_user in other places. It looks like
>>>>>> put_user crashes in the user address is not mapped/protected (?).
>>>>>
>>>>> I've been having a look, and this seems to be down to access of the
>>>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
>>>>> bad address to clone?
>>>>>
>>>>> From looking at the code, the put_user() code should have set the
>>>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
>>>>> s2 register in the crash report) and from looking at the compiler
>>>>> output from my gcc-10, the code looks to be dong the relevant csrs
>>>>> and then csrc around the put_user
>>>>>
>>>>> So currently I do not understand how the above could have happened
>>>>> over than something re-tried the code seqeunce and ended up retrying
>>>>> the faulting instruction without the SR_SUM bit set.
>>>>
>>>> I would maybe blame qemu for randomly resetting SR_SUM, but it's
>>>> strange that 99% of these crashes are in schedule_tail. If it would be
>>>> qemu, then they would be more evenly distributed...
>>>>
>>>> Another observation: looking at a dozen of crash logs, in none of
>>>> these cases fuzzer was actually trying to fuzz clone with some insane
>>>> arguments. So it looks like completely normal clone's (e..g coming
>>>> from pthread_create) result in this crash.
>>>>
>>>> I also wonder why there is ret_from_exception, is it normal? I see
>>>> handle_exception disables SR_SUM:
>>>
>>> csrrc does the right thing: it cleans SR_SUM bit in status but saves the
>>> previous value that will get correctly restored.
>>>
>>> ("The CSRRC (Atomic Read and Clear Bits in CSR) instruction reads the
>>> value of the CSR, zero-extends the value to XLEN bits, and writes it to
>>> integer registerrd. The initial value in integerregisterrs1is treated
>>> as a bit mask that specifies bit positions to be cleared in the CSR. Any
>>> bitthat is high inrs1will cause the corresponding bit to be cleared in
>>> the CSR, if that CSR bit iswritable. Other bits in the CSR are
>>> unaffected.")
>>
>> I think there may also be an understanding issue on what the SR_SUM
>> bit does. I thought if it is set, M->U accesses would fault, which is
>> why it gets set early on. But from reading the uaccess code it looks
>> like the uaccess code sets it on entry and then clears on exit.
>>
>> I am very confused. Is there a master reference for rv64?
>>
>> https://people.eecs.berkeley.edu/~krste/papers/riscv-privileged-v1.9.pdf
>> seems to state PUM is the SR_SUM bit, and that (if set) disabled
>>
>> Quote:
>> The PUM (Protect User Memory) bit modifies the privilege with which
>> S-mode loads, stores, and instruction fetches access virtual memory.
>> When PUM=0, translation and protection behave as normal. When PUM=1,
>> S-mode memory accesses to pages that are accessible by U-mode (U=1 in
>> Figure 4.19) will fault. PUM has no effect when executing in U-mode
>>
>>
>>>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
>>>>
>>>
>>> Still no luck for the moment, can't reproduce it locally, my test is
>>> maybe not that good (I created threads all day long in order to trigger
>>> the put_user of schedule_tail).
>>
>> It may of course depend on memory and other stuff. I did try to see if
>> it was possible to clone() with the child_tid address being a valid but
>> not mapped page...
>>
>>> Given that the path you mention works most of the time, and that the
>>> status register in the stack trace shows the SUM bit is not set whereas
>>> it is set in put_user, I'm leaning toward some race condition (maybe an
>>> interrupt that arrives at the "wrong" time) or a qemu issue as you
>>> mentioned.
>>
>> I suppose this is possible. From what I read it should get to the
>> point of being there with the SUM flag cleared, so either something
>> went wrong in trying to fix the instruction up or there's some other
>> error we're missing.
>>
>>> To eliminate qemu issues, do you have access to some HW ? Or to
>>> different qemu versions ?
>>
>> I do have access to a Microchip Polarfire board. I just need the
>> instructions on how to setup the test-code to make it work on the
>> hardware.
>
> For full syzkaller support, it would need to know how to reboot these
> boards and get access to the console.
> syzkaller has a stop-gap VM backend which just uses ssh to a physical
> machine and expects the kernel to reboot on its own after any crashes.
>
> But I actually managed to reproduce it in an even simpler setup.
> Assuming you have Go 1.15 and riscv64 cross-compiler gcc installed
>
> $ go get -u -d github.com/google/syzkaller/...
> $ cd $GOPATH/src/github.com/google/syzkaller
> $ make stress executor TARGETARCH=riscv64
> $ scp bin/linux_riscv64/syz-execprog bin/linux_riscv64/syz-executor
> your_machine:/
>
> Then run ./syz-stress on the machine.
> On the first run it crashed it with some other bug, on the second run
> I got the crash in schedule_tail.
> With qemu tcg I also added -slowdown=10 flag to syz-stress to scale
> all timeouts, if native execution is faster, then you don't need it.
Ok, not sure what's going on. I get a lot of errors similar to:
>
> 2021/03/15 21:35:20 transitively unsupported: ioctl$SNAPSHOT_CREATE_IMAGE: no syscalls can create resource fd_snapshot, enable some syscalls that can create it [openat$snapshot]
Followed by:
> 2021/03/15 21:35:48 executed 0 programs
> 2021/03/15 21:35:48 failed to create execution environment: failed to mmap shm file: invalid argument
The qemu is 5.2.0 and root is Debian/unstable riscv64 (same as chroot
used to build the syz tools)
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-15 21:38 ` Ben Dooks
0 siblings, 0 replies; 51+ messages in thread
From: Ben Dooks @ 2021-03-15 21:38 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: Alex Ghiti, syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou,
linux-riscv, Daniel Bristot de Oliveira, Benjamin Segall,
dietmar.eggemann, Juri Lelli, LKML, Mel Gorman, Ingo Molnar,
Peter Zijlstra, Steven Rostedt, syzkaller-bugs, Vincent Guittot
On 13/03/2021 07:20, Dmitry Vyukov wrote:
> On Fri, Mar 12, 2021 at 9:12 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>>
>> On 12/03/2021 16:25, Alex Ghiti wrote:
>>>
>>>
>>> Le 3/12/21 à 10:12 AM, Dmitry Vyukov a écrit :
>>>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
>>>> wrote:
>>>>>
>>>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
>>>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
>>>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> syzbot found the following issue on:
>>>>>>>
>>>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
>>>>>>> arch_dup_tas..
>>>>>>> git tree:
>>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>>>>>>> console output:
>>>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>>>>>>> kernel config:
>>>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>>>>>>> dashboard link:
>>>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>>>>>>> userspace arch: riscv64
>>>>>>>
>>>>>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>>>>>
>>>>>>> IMPORTANT: if you fix the issue, please add the following tag to
>>>>>>> the commit:
>>>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>>>>>>
>>>>>> +riscv maintainers
>>>>>>
>>>>>> This is riscv64-specific.
>>>>>> I've seen similar crashes in put_user in other places. It looks like
>>>>>> put_user crashes in the user address is not mapped/protected (?).
>>>>>
>>>>> I've been having a look, and this seems to be down to access of the
>>>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
>>>>> bad address to clone?
>>>>>
>>>>> From looking at the code, the put_user() code should have set the
>>>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
>>>>> s2 register in the crash report) and from looking at the compiler
>>>>> output from my gcc-10, the code looks to be dong the relevant csrs
>>>>> and then csrc around the put_user
>>>>>
>>>>> So currently I do not understand how the above could have happened
>>>>> over than something re-tried the code seqeunce and ended up retrying
>>>>> the faulting instruction without the SR_SUM bit set.
>>>>
>>>> I would maybe blame qemu for randomly resetting SR_SUM, but it's
>>>> strange that 99% of these crashes are in schedule_tail. If it would be
>>>> qemu, then they would be more evenly distributed...
>>>>
>>>> Another observation: looking at a dozen of crash logs, in none of
>>>> these cases fuzzer was actually trying to fuzz clone with some insane
>>>> arguments. So it looks like completely normal clone's (e..g coming
>>>> from pthread_create) result in this crash.
>>>>
>>>> I also wonder why there is ret_from_exception, is it normal? I see
>>>> handle_exception disables SR_SUM:
>>>
>>> csrrc does the right thing: it cleans SR_SUM bit in status but saves the
>>> previous value that will get correctly restored.
>>>
>>> ("The CSRRC (Atomic Read and Clear Bits in CSR) instruction reads the
>>> value of the CSR, zero-extends the value to XLEN bits, and writes it to
>>> integer registerrd. The initial value in integerregisterrs1is treated
>>> as a bit mask that specifies bit positions to be cleared in the CSR. Any
>>> bitthat is high inrs1will cause the corresponding bit to be cleared in
>>> the CSR, if that CSR bit iswritable. Other bits in the CSR are
>>> unaffected.")
>>
>> I think there may also be an understanding issue on what the SR_SUM
>> bit does. I thought if it is set, M->U accesses would fault, which is
>> why it gets set early on. But from reading the uaccess code it looks
>> like the uaccess code sets it on entry and then clears on exit.
>>
>> I am very confused. Is there a master reference for rv64?
>>
>> https://people.eecs.berkeley.edu/~krste/papers/riscv-privileged-v1.9.pdf
>> seems to state PUM is the SR_SUM bit, and that (if set) disabled
>>
>> Quote:
>> The PUM (Protect User Memory) bit modifies the privilege with which
>> S-mode loads, stores, and instruction fetches access virtual memory.
>> When PUM=0, translation and protection behave as normal. When PUM=1,
>> S-mode memory accesses to pages that are accessible by U-mode (U=1 in
>> Figure 4.19) will fault. PUM has no effect when executing in U-mode
>>
>>
>>>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
>>>>
>>>
>>> Still no luck for the moment, can't reproduce it locally, my test is
>>> maybe not that good (I created threads all day long in order to trigger
>>> the put_user of schedule_tail).
>>
>> It may of course depend on memory and other stuff. I did try to see if
>> it was possible to clone() with the child_tid address being a valid but
>> not mapped page...
>>
>>> Given that the path you mention works most of the time, and that the
>>> status register in the stack trace shows the SUM bit is not set whereas
>>> it is set in put_user, I'm leaning toward some race condition (maybe an
>>> interrupt that arrives at the "wrong" time) or a qemu issue as you
>>> mentioned.
>>
>> I suppose this is possible. From what I read it should get to the
>> point of being there with the SUM flag cleared, so either something
>> went wrong in trying to fix the instruction up or there's some other
>> error we're missing.
>>
>>> To eliminate qemu issues, do you have access to some HW ? Or to
>>> different qemu versions ?
>>
>> I do have access to a Microchip Polarfire board. I just need the
>> instructions on how to setup the test-code to make it work on the
>> hardware.
>
> For full syzkaller support, it would need to know how to reboot these
> boards and get access to the console.
> syzkaller has a stop-gap VM backend which just uses ssh to a physical
> machine and expects the kernel to reboot on its own after any crashes.
>
> But I actually managed to reproduce it in an even simpler setup.
> Assuming you have Go 1.15 and riscv64 cross-compiler gcc installed
>
> $ go get -u -d github.com/google/syzkaller/...
> $ cd $GOPATH/src/github.com/google/syzkaller
> $ make stress executor TARGETARCH=riscv64
> $ scp bin/linux_riscv64/syz-execprog bin/linux_riscv64/syz-executor
> your_machine:/
>
> Then run ./syz-stress on the machine.
> On the first run it crashed it with some other bug, on the second run
> I got the crash in schedule_tail.
> With qemu tcg I also added -slowdown=10 flag to syz-stress to scale
> all timeouts, if native execution is faster, then you don't need it.
Ok, not sure what's going on. I get a lot of errors similar to:
>
> 2021/03/15 21:35:20 transitively unsupported: ioctl$SNAPSHOT_CREATE_IMAGE: no syscalls can create resource fd_snapshot, enable some syscalls that can create it [openat$snapshot]
Followed by:
> 2021/03/15 21:35:48 executed 0 programs
> 2021/03/15 21:35:48 failed to create execution environment: failed to mmap shm file: invalid argument
The qemu is 5.2.0 and root is Debian/unstable riscv64 (same as chroot
used to build the syz tools)
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-15 21:38 ` Ben Dooks
@ 2021-03-16 8:52 ` Dmitry Vyukov
-1 siblings, 0 replies; 51+ messages in thread
From: Dmitry Vyukov @ 2021-03-16 8:52 UTC (permalink / raw)
To: Ben Dooks
Cc: Alex Ghiti, syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou,
linux-riscv, Daniel Bristot de Oliveira, Benjamin Segall,
dietmar.eggemann, Juri Lelli, LKML, Mel Gorman, Ingo Molnar,
Peter Zijlstra, Steven Rostedt, syzkaller-bugs, Vincent Guittot
On Mon, Mar 15, 2021 at 10:38 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>
> On 13/03/2021 07:20, Dmitry Vyukov wrote:
> > On Fri, Mar 12, 2021 at 9:12 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
> >>
> >> On 12/03/2021 16:25, Alex Ghiti wrote:
> >>>
> >>>
> >>> Le 3/12/21 à 10:12 AM, Dmitry Vyukov a écrit :
> >>>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
> >>>> wrote:
> >>>>>
> >>>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
> >>>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
> >>>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
> >>>>>>>
> >>>>>>> Hello,
> >>>>>>>
> >>>>>>> syzbot found the following issue on:
> >>>>>>>
> >>>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
> >>>>>>> arch_dup_tas..
> >>>>>>> git tree:
> >>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
> >>>>>>> console output:
> >>>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
> >>>>>>> kernel config:
> >>>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
> >>>>>>> dashboard link:
> >>>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
> >>>>>>> userspace arch: riscv64
> >>>>>>>
> >>>>>>> Unfortunately, I don't have any reproducer for this issue yet.
> >>>>>>>
> >>>>>>> IMPORTANT: if you fix the issue, please add the following tag to
> >>>>>>> the commit:
> >>>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
> >>>>>>
> >>>>>> +riscv maintainers
> >>>>>>
> >>>>>> This is riscv64-specific.
> >>>>>> I've seen similar crashes in put_user in other places. It looks like
> >>>>>> put_user crashes in the user address is not mapped/protected (?).
> >>>>>
> >>>>> I've been having a look, and this seems to be down to access of the
> >>>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
> >>>>> bad address to clone?
> >>>>>
> >>>>> From looking at the code, the put_user() code should have set the
> >>>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
> >>>>> s2 register in the crash report) and from looking at the compiler
> >>>>> output from my gcc-10, the code looks to be dong the relevant csrs
> >>>>> and then csrc around the put_user
> >>>>>
> >>>>> So currently I do not understand how the above could have happened
> >>>>> over than something re-tried the code seqeunce and ended up retrying
> >>>>> the faulting instruction without the SR_SUM bit set.
> >>>>
> >>>> I would maybe blame qemu for randomly resetting SR_SUM, but it's
> >>>> strange that 99% of these crashes are in schedule_tail. If it would be
> >>>> qemu, then they would be more evenly distributed...
> >>>>
> >>>> Another observation: looking at a dozen of crash logs, in none of
> >>>> these cases fuzzer was actually trying to fuzz clone with some insane
> >>>> arguments. So it looks like completely normal clone's (e..g coming
> >>>> from pthread_create) result in this crash.
> >>>>
> >>>> I also wonder why there is ret_from_exception, is it normal? I see
> >>>> handle_exception disables SR_SUM:
> >>>
> >>> csrrc does the right thing: it cleans SR_SUM bit in status but saves the
> >>> previous value that will get correctly restored.
> >>>
> >>> ("The CSRRC (Atomic Read and Clear Bits in CSR) instruction reads the
> >>> value of the CSR, zero-extends the value to XLEN bits, and writes it to
> >>> integer registerrd. The initial value in integerregisterrs1is treated
> >>> as a bit mask that specifies bit positions to be cleared in the CSR. Any
> >>> bitthat is high inrs1will cause the corresponding bit to be cleared in
> >>> the CSR, if that CSR bit iswritable. Other bits in the CSR are
> >>> unaffected.")
> >>
> >> I think there may also be an understanding issue on what the SR_SUM
> >> bit does. I thought if it is set, M->U accesses would fault, which is
> >> why it gets set early on. But from reading the uaccess code it looks
> >> like the uaccess code sets it on entry and then clears on exit.
> >>
> >> I am very confused. Is there a master reference for rv64?
> >>
> >> https://people.eecs.berkeley.edu/~krste/papers/riscv-privileged-v1.9.pdf
> >> seems to state PUM is the SR_SUM bit, and that (if set) disabled
> >>
> >> Quote:
> >> The PUM (Protect User Memory) bit modifies the privilege with which
> >> S-mode loads, stores, and instruction fetches access virtual memory.
> >> When PUM=0, translation and protection behave as normal. When PUM=1,
> >> S-mode memory accesses to pages that are accessible by U-mode (U=1 in
> >> Figure 4.19) will fault. PUM has no effect when executing in U-mode
> >>
> >>
> >>>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
> >>>>
> >>>
> >>> Still no luck for the moment, can't reproduce it locally, my test is
> >>> maybe not that good (I created threads all day long in order to trigger
> >>> the put_user of schedule_tail).
> >>
> >> It may of course depend on memory and other stuff. I did try to see if
> >> it was possible to clone() with the child_tid address being a valid but
> >> not mapped page...
> >>
> >>> Given that the path you mention works most of the time, and that the
> >>> status register in the stack trace shows the SUM bit is not set whereas
> >>> it is set in put_user, I'm leaning toward some race condition (maybe an
> >>> interrupt that arrives at the "wrong" time) or a qemu issue as you
> >>> mentioned.
> >>
> >> I suppose this is possible. From what I read it should get to the
> >> point of being there with the SUM flag cleared, so either something
> >> went wrong in trying to fix the instruction up or there's some other
> >> error we're missing.
> >>
> >>> To eliminate qemu issues, do you have access to some HW ? Or to
> >>> different qemu versions ?
> >>
> >> I do have access to a Microchip Polarfire board. I just need the
> >> instructions on how to setup the test-code to make it work on the
> >> hardware.
> >
> > For full syzkaller support, it would need to know how to reboot these
> > boards and get access to the console.
> > syzkaller has a stop-gap VM backend which just uses ssh to a physical
> > machine and expects the kernel to reboot on its own after any crashes.
> >
> > But I actually managed to reproduce it in an even simpler setup.
> > Assuming you have Go 1.15 and riscv64 cross-compiler gcc installed
> >
> > $ go get -u -d github.com/google/syzkaller/...
> > $ cd $GOPATH/src/github.com/google/syzkaller
> > $ make stress executor TARGETARCH=riscv64
> > $ scp bin/linux_riscv64/syz-execprog bin/linux_riscv64/syz-executor
> > your_machine:/
> >
> > Then run ./syz-stress on the machine.
> > On the first run it crashed it with some other bug, on the second run
> > I got the crash in schedule_tail.
> > With qemu tcg I also added -slowdown=10 flag to syz-stress to scale
> > all timeouts, if native execution is faster, then you don't need it.
>
> Ok, not sure what's going on. I get a lot of errors similar to:
> >
> > 2021/03/15 21:35:20 transitively unsupported: ioctl$SNAPSHOT_CREATE_IMAGE: no syscalls can create resource fd_snapshot, enable some syscalls that can create it [openat$snapshot]
This is not an error, just a notification that some syscalls are not
enabled in the kernel and won't be fuzzed.
> Followed by:
>
> > 2021/03/15 21:35:48 executed 0 programs
> > 2021/03/15 21:35:48 failed to create execution environment: failed to mmap shm file: invalid argument
>
> The qemu is 5.2.0 and root is Debian/unstable riscv64 (same as chroot
> used to build the syz tools)
This is an error. But I see it the first time ever.
It comes from here:
https://github.com/google/syzkaller/blob/fdb2bb2c23ee709880407f56307e2800ad27e9ae/pkg/osutil/osutil_unix.go#L119-L121
There should be pretty simple logic inside of syscall.Mmap. Perhaps
you are using some older Go toolchain with incomplete riscv support?
I think I've used 1.14 and 1.15. But there is already 1.16. You can
always download a toolchain here:
https://golang.org/dl/
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-16 8:52 ` Dmitry Vyukov
0 siblings, 0 replies; 51+ messages in thread
From: Dmitry Vyukov @ 2021-03-16 8:52 UTC (permalink / raw)
To: Ben Dooks
Cc: Alex Ghiti, syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou,
linux-riscv, Daniel Bristot de Oliveira, Benjamin Segall,
dietmar.eggemann, Juri Lelli, LKML, Mel Gorman, Ingo Molnar,
Peter Zijlstra, Steven Rostedt, syzkaller-bugs, Vincent Guittot
On Mon, Mar 15, 2021 at 10:38 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>
> On 13/03/2021 07:20, Dmitry Vyukov wrote:
> > On Fri, Mar 12, 2021 at 9:12 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
> >>
> >> On 12/03/2021 16:25, Alex Ghiti wrote:
> >>>
> >>>
> >>> Le 3/12/21 à 10:12 AM, Dmitry Vyukov a écrit :
> >>>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
> >>>> wrote:
> >>>>>
> >>>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
> >>>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
> >>>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
> >>>>>>>
> >>>>>>> Hello,
> >>>>>>>
> >>>>>>> syzbot found the following issue on:
> >>>>>>>
> >>>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
> >>>>>>> arch_dup_tas..
> >>>>>>> git tree:
> >>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
> >>>>>>> console output:
> >>>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
> >>>>>>> kernel config:
> >>>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
> >>>>>>> dashboard link:
> >>>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
> >>>>>>> userspace arch: riscv64
> >>>>>>>
> >>>>>>> Unfortunately, I don't have any reproducer for this issue yet.
> >>>>>>>
> >>>>>>> IMPORTANT: if you fix the issue, please add the following tag to
> >>>>>>> the commit:
> >>>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
> >>>>>>
> >>>>>> +riscv maintainers
> >>>>>>
> >>>>>> This is riscv64-specific.
> >>>>>> I've seen similar crashes in put_user in other places. It looks like
> >>>>>> put_user crashes in the user address is not mapped/protected (?).
> >>>>>
> >>>>> I've been having a look, and this seems to be down to access of the
> >>>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
> >>>>> bad address to clone?
> >>>>>
> >>>>> From looking at the code, the put_user() code should have set the
> >>>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
> >>>>> s2 register in the crash report) and from looking at the compiler
> >>>>> output from my gcc-10, the code looks to be dong the relevant csrs
> >>>>> and then csrc around the put_user
> >>>>>
> >>>>> So currently I do not understand how the above could have happened
> >>>>> over than something re-tried the code seqeunce and ended up retrying
> >>>>> the faulting instruction without the SR_SUM bit set.
> >>>>
> >>>> I would maybe blame qemu for randomly resetting SR_SUM, but it's
> >>>> strange that 99% of these crashes are in schedule_tail. If it would be
> >>>> qemu, then they would be more evenly distributed...
> >>>>
> >>>> Another observation: looking at a dozen of crash logs, in none of
> >>>> these cases fuzzer was actually trying to fuzz clone with some insane
> >>>> arguments. So it looks like completely normal clone's (e..g coming
> >>>> from pthread_create) result in this crash.
> >>>>
> >>>> I also wonder why there is ret_from_exception, is it normal? I see
> >>>> handle_exception disables SR_SUM:
> >>>
> >>> csrrc does the right thing: it cleans SR_SUM bit in status but saves the
> >>> previous value that will get correctly restored.
> >>>
> >>> ("The CSRRC (Atomic Read and Clear Bits in CSR) instruction reads the
> >>> value of the CSR, zero-extends the value to XLEN bits, and writes it to
> >>> integer registerrd. The initial value in integerregisterrs1is treated
> >>> as a bit mask that specifies bit positions to be cleared in the CSR. Any
> >>> bitthat is high inrs1will cause the corresponding bit to be cleared in
> >>> the CSR, if that CSR bit iswritable. Other bits in the CSR are
> >>> unaffected.")
> >>
> >> I think there may also be an understanding issue on what the SR_SUM
> >> bit does. I thought if it is set, M->U accesses would fault, which is
> >> why it gets set early on. But from reading the uaccess code it looks
> >> like the uaccess code sets it on entry and then clears on exit.
> >>
> >> I am very confused. Is there a master reference for rv64?
> >>
> >> https://people.eecs.berkeley.edu/~krste/papers/riscv-privileged-v1.9.pdf
> >> seems to state PUM is the SR_SUM bit, and that (if set) disabled
> >>
> >> Quote:
> >> The PUM (Protect User Memory) bit modifies the privilege with which
> >> S-mode loads, stores, and instruction fetches access virtual memory.
> >> When PUM=0, translation and protection behave as normal. When PUM=1,
> >> S-mode memory accesses to pages that are accessible by U-mode (U=1 in
> >> Figure 4.19) will fault. PUM has no effect when executing in U-mode
> >>
> >>
> >>>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
> >>>>
> >>>
> >>> Still no luck for the moment, can't reproduce it locally, my test is
> >>> maybe not that good (I created threads all day long in order to trigger
> >>> the put_user of schedule_tail).
> >>
> >> It may of course depend on memory and other stuff. I did try to see if
> >> it was possible to clone() with the child_tid address being a valid but
> >> not mapped page...
> >>
> >>> Given that the path you mention works most of the time, and that the
> >>> status register in the stack trace shows the SUM bit is not set whereas
> >>> it is set in put_user, I'm leaning toward some race condition (maybe an
> >>> interrupt that arrives at the "wrong" time) or a qemu issue as you
> >>> mentioned.
> >>
> >> I suppose this is possible. From what I read it should get to the
> >> point of being there with the SUM flag cleared, so either something
> >> went wrong in trying to fix the instruction up or there's some other
> >> error we're missing.
> >>
> >>> To eliminate qemu issues, do you have access to some HW ? Or to
> >>> different qemu versions ?
> >>
> >> I do have access to a Microchip Polarfire board. I just need the
> >> instructions on how to setup the test-code to make it work on the
> >> hardware.
> >
> > For full syzkaller support, it would need to know how to reboot these
> > boards and get access to the console.
> > syzkaller has a stop-gap VM backend which just uses ssh to a physical
> > machine and expects the kernel to reboot on its own after any crashes.
> >
> > But I actually managed to reproduce it in an even simpler setup.
> > Assuming you have Go 1.15 and riscv64 cross-compiler gcc installed
> >
> > $ go get -u -d github.com/google/syzkaller/...
> > $ cd $GOPATH/src/github.com/google/syzkaller
> > $ make stress executor TARGETARCH=riscv64
> > $ scp bin/linux_riscv64/syz-execprog bin/linux_riscv64/syz-executor
> > your_machine:/
> >
> > Then run ./syz-stress on the machine.
> > On the first run it crashed it with some other bug, on the second run
> > I got the crash in schedule_tail.
> > With qemu tcg I also added -slowdown=10 flag to syz-stress to scale
> > all timeouts, if native execution is faster, then you don't need it.
>
> Ok, not sure what's going on. I get a lot of errors similar to:
> >
> > 2021/03/15 21:35:20 transitively unsupported: ioctl$SNAPSHOT_CREATE_IMAGE: no syscalls can create resource fd_snapshot, enable some syscalls that can create it [openat$snapshot]
This is not an error, just a notification that some syscalls are not
enabled in the kernel and won't be fuzzed.
> Followed by:
>
> > 2021/03/15 21:35:48 executed 0 programs
> > 2021/03/15 21:35:48 failed to create execution environment: failed to mmap shm file: invalid argument
>
> The qemu is 5.2.0 and root is Debian/unstable riscv64 (same as chroot
> used to build the syz tools)
This is an error. But I see it the first time ever.
It comes from here:
https://github.com/google/syzkaller/blob/fdb2bb2c23ee709880407f56307e2800ad27e9ae/pkg/osutil/osutil_unix.go#L119-L121
There should be pretty simple logic inside of syscall.Mmap. Perhaps
you are using some older Go toolchain with incomplete riscv support?
I think I've used 1.14 and 1.15. But there is already 1.16. You can
always download a toolchain here:
https://golang.org/dl/
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-16 8:52 ` Dmitry Vyukov
@ 2021-03-16 11:35 ` Ben Dooks
-1 siblings, 0 replies; 51+ messages in thread
From: Ben Dooks @ 2021-03-16 11:35 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: Alex Ghiti, syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou,
linux-riscv, Daniel Bristot de Oliveira, Benjamin Segall,
dietmar.eggemann, Juri Lelli, LKML, Mel Gorman, Ingo Molnar,
Peter Zijlstra, Steven Rostedt, syzkaller-bugs, Vincent Guittot
On 16/03/2021 08:52, Dmitry Vyukov wrote:
> On Mon, Mar 15, 2021 at 10:38 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>>
>> On 13/03/2021 07:20, Dmitry Vyukov wrote:
>>> On Fri, Mar 12, 2021 at 9:12 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>>>>
>>>> On 12/03/2021 16:25, Alex Ghiti wrote:
>>>>>
>>>>>
>>>>> Le 3/12/21 à 10:12 AM, Dmitry Vyukov a écrit :
>>>>>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
>>>>>> wrote:
>>>>>>>
>>>>>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
>>>>>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
>>>>>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> syzbot found the following issue on:
>>>>>>>>>
>>>>>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
>>>>>>>>> arch_dup_tas..
>>>>>>>>> git tree:
>>>>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>>>>>>>>> console output:
>>>>>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>>>>>>>>> kernel config:
>>>>>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>>>>>>>>> dashboard link:
>>>>>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>>>>>>>>> userspace arch: riscv64
>>>>>>>>>
>>>>>>>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>>>>>>>
>>>>>>>>> IMPORTANT: if you fix the issue, please add the following tag to
>>>>>>>>> the commit:
>>>>>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>>>>>>>>
>>>>>>>> +riscv maintainers
>>>>>>>>
>>>>>>>> This is riscv64-specific.
>>>>>>>> I've seen similar crashes in put_user in other places. It looks like
>>>>>>>> put_user crashes in the user address is not mapped/protected (?).
>>>>>>>
>>>>>>> I've been having a look, and this seems to be down to access of the
>>>>>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
>>>>>>> bad address to clone?
>>>>>>>
>>>>>>> From looking at the code, the put_user() code should have set the
>>>>>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
>>>>>>> s2 register in the crash report) and from looking at the compiler
>>>>>>> output from my gcc-10, the code looks to be dong the relevant csrs
>>>>>>> and then csrc around the put_user
>>>>>>>
>>>>>>> So currently I do not understand how the above could have happened
>>>>>>> over than something re-tried the code seqeunce and ended up retrying
>>>>>>> the faulting instruction without the SR_SUM bit set.
>>>>>>
>>>>>> I would maybe blame qemu for randomly resetting SR_SUM, but it's
>>>>>> strange that 99% of these crashes are in schedule_tail. If it would be
>>>>>> qemu, then they would be more evenly distributed...
>>>>>>
>>>>>> Another observation: looking at a dozen of crash logs, in none of
>>>>>> these cases fuzzer was actually trying to fuzz clone with some insane
>>>>>> arguments. So it looks like completely normal clone's (e..g coming
>>>>>> from pthread_create) result in this crash.
>>>>>>
>>>>>> I also wonder why there is ret_from_exception, is it normal? I see
>>>>>> handle_exception disables SR_SUM:
>>>>>
>>>>> csrrc does the right thing: it cleans SR_SUM bit in status but saves the
>>>>> previous value that will get correctly restored.
>>>>>
>>>>> ("The CSRRC (Atomic Read and Clear Bits in CSR) instruction reads the
>>>>> value of the CSR, zero-extends the value to XLEN bits, and writes it to
>>>>> integer registerrd. The initial value in integerregisterrs1is treated
>>>>> as a bit mask that specifies bit positions to be cleared in the CSR. Any
>>>>> bitthat is high inrs1will cause the corresponding bit to be cleared in
>>>>> the CSR, if that CSR bit iswritable. Other bits in the CSR are
>>>>> unaffected.")
>>>>
>>>> I think there may also be an understanding issue on what the SR_SUM
>>>> bit does. I thought if it is set, M->U accesses would fault, which is
>>>> why it gets set early on. But from reading the uaccess code it looks
>>>> like the uaccess code sets it on entry and then clears on exit.
>>>>
>>>> I am very confused. Is there a master reference for rv64?
>>>>
>>>> https://people.eecs.berkeley.edu/~krste/papers/riscv-privileged-v1.9.pdf
>>>> seems to state PUM is the SR_SUM bit, and that (if set) disabled
>>>>
>>>> Quote:
>>>> The PUM (Protect User Memory) bit modifies the privilege with which
>>>> S-mode loads, stores, and instruction fetches access virtual memory.
>>>> When PUM=0, translation and protection behave as normal. When PUM=1,
>>>> S-mode memory accesses to pages that are accessible by U-mode (U=1 in
>>>> Figure 4.19) will fault. PUM has no effect when executing in U-mode
>>>>
>>>>
>>>>>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
>>>>>>
>>>>>
>>>>> Still no luck for the moment, can't reproduce it locally, my test is
>>>>> maybe not that good (I created threads all day long in order to trigger
>>>>> the put_user of schedule_tail).
>>>>
>>>> It may of course depend on memory and other stuff. I did try to see if
>>>> it was possible to clone() with the child_tid address being a valid but
>>>> not mapped page...
>>>>
>>>>> Given that the path you mention works most of the time, and that the
>>>>> status register in the stack trace shows the SUM bit is not set whereas
>>>>> it is set in put_user, I'm leaning toward some race condition (maybe an
>>>>> interrupt that arrives at the "wrong" time) or a qemu issue as you
>>>>> mentioned.
>>>>
>>>> I suppose this is possible. From what I read it should get to the
>>>> point of being there with the SUM flag cleared, so either something
>>>> went wrong in trying to fix the instruction up or there's some other
>>>> error we're missing.
>>>>
>>>>> To eliminate qemu issues, do you have access to some HW ? Or to
>>>>> different qemu versions ?
>>>>
>>>> I do have access to a Microchip Polarfire board. I just need the
>>>> instructions on how to setup the test-code to make it work on the
>>>> hardware.
>>>
>>> For full syzkaller support, it would need to know how to reboot these
>>> boards and get access to the console.
>>> syzkaller has a stop-gap VM backend which just uses ssh to a physical
>>> machine and expects the kernel to reboot on its own after any crashes.
>>>
>>> But I actually managed to reproduce it in an even simpler setup.
>>> Assuming you have Go 1.15 and riscv64 cross-compiler gcc installed
>>>
>>> $ go get -u -d github.com/google/syzkaller/...
>>> $ cd $GOPATH/src/github.com/google/syzkaller
>>> $ make stress executor TARGETARCH=riscv64
>>> $ scp bin/linux_riscv64/syz-execprog bin/linux_riscv64/syz-executor
>>> your_machine:/
>>>
>>> Then run ./syz-stress on the machine.
>>> On the first run it crashed it with some other bug, on the second run
>>> I got the crash in schedule_tail.
>>> With qemu tcg I also added -slowdown=10 flag to syz-stress to scale
>>> all timeouts, if native execution is faster, then you don't need it.
>>
>> Ok, not sure what's going on. I get a lot of errors similar to:
>>>
>>> 2021/03/15 21:35:20 transitively unsupported: ioctl$SNAPSHOT_CREATE_IMAGE: no syscalls can create resource fd_snapshot, enable some syscalls that can create it [openat$snapshot]
>
> This is not an error, just a notification that some syscalls are not
> enabled in the kernel and won't be fuzzed.
>
>> Followed by:
>>
>>> 2021/03/15 21:35:48 executed 0 programs
>>> 2021/03/15 21:35:48 failed to create execution environment: failed to mmap shm file: invalid argument
>>
>> The qemu is 5.2.0 and root is Debian/unstable riscv64 (same as chroot
>> used to build the syz tools)
>
> This is an error. But I see it the first time ever.
> It comes from here:
> https://github.com/google/syzkaller/blob/fdb2bb2c23ee709880407f56307e2800ad27e9ae/pkg/osutil/osutil_unix.go#L119-L121
> There should be pretty simple logic inside of syscall.Mmap. Perhaps
> you are using some older Go toolchain with incomplete riscv support?
> I think I've used 1.14 and 1.15. But there is already 1.16. You can
> always download a toolchain here:
> https://golang.org/dl/
Hmm it would have been useful to print out what file it failed to map.
I've got go 1.15 from the debian/unstable riscv64 chroot.
I'll have a look at this in a bit to see if it throws the same issue on
a real system.
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-16 11:35 ` Ben Dooks
0 siblings, 0 replies; 51+ messages in thread
From: Ben Dooks @ 2021-03-16 11:35 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: Alex Ghiti, syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou,
linux-riscv, Daniel Bristot de Oliveira, Benjamin Segall,
dietmar.eggemann, Juri Lelli, LKML, Mel Gorman, Ingo Molnar,
Peter Zijlstra, Steven Rostedt, syzkaller-bugs, Vincent Guittot
On 16/03/2021 08:52, Dmitry Vyukov wrote:
> On Mon, Mar 15, 2021 at 10:38 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>>
>> On 13/03/2021 07:20, Dmitry Vyukov wrote:
>>> On Fri, Mar 12, 2021 at 9:12 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>>>>
>>>> On 12/03/2021 16:25, Alex Ghiti wrote:
>>>>>
>>>>>
>>>>> Le 3/12/21 à 10:12 AM, Dmitry Vyukov a écrit :
>>>>>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
>>>>>> wrote:
>>>>>>>
>>>>>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
>>>>>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
>>>>>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> syzbot found the following issue on:
>>>>>>>>>
>>>>>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
>>>>>>>>> arch_dup_tas..
>>>>>>>>> git tree:
>>>>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>>>>>>>>> console output:
>>>>>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>>>>>>>>> kernel config:
>>>>>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>>>>>>>>> dashboard link:
>>>>>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>>>>>>>>> userspace arch: riscv64
>>>>>>>>>
>>>>>>>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>>>>>>>
>>>>>>>>> IMPORTANT: if you fix the issue, please add the following tag to
>>>>>>>>> the commit:
>>>>>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>>>>>>>>
>>>>>>>> +riscv maintainers
>>>>>>>>
>>>>>>>> This is riscv64-specific.
>>>>>>>> I've seen similar crashes in put_user in other places. It looks like
>>>>>>>> put_user crashes in the user address is not mapped/protected (?).
>>>>>>>
>>>>>>> I've been having a look, and this seems to be down to access of the
>>>>>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
>>>>>>> bad address to clone?
>>>>>>>
>>>>>>> From looking at the code, the put_user() code should have set the
>>>>>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
>>>>>>> s2 register in the crash report) and from looking at the compiler
>>>>>>> output from my gcc-10, the code looks to be dong the relevant csrs
>>>>>>> and then csrc around the put_user
>>>>>>>
>>>>>>> So currently I do not understand how the above could have happened
>>>>>>> over than something re-tried the code seqeunce and ended up retrying
>>>>>>> the faulting instruction without the SR_SUM bit set.
>>>>>>
>>>>>> I would maybe blame qemu for randomly resetting SR_SUM, but it's
>>>>>> strange that 99% of these crashes are in schedule_tail. If it would be
>>>>>> qemu, then they would be more evenly distributed...
>>>>>>
>>>>>> Another observation: looking at a dozen of crash logs, in none of
>>>>>> these cases fuzzer was actually trying to fuzz clone with some insane
>>>>>> arguments. So it looks like completely normal clone's (e..g coming
>>>>>> from pthread_create) result in this crash.
>>>>>>
>>>>>> I also wonder why there is ret_from_exception, is it normal? I see
>>>>>> handle_exception disables SR_SUM:
>>>>>
>>>>> csrrc does the right thing: it cleans SR_SUM bit in status but saves the
>>>>> previous value that will get correctly restored.
>>>>>
>>>>> ("The CSRRC (Atomic Read and Clear Bits in CSR) instruction reads the
>>>>> value of the CSR, zero-extends the value to XLEN bits, and writes it to
>>>>> integer registerrd. The initial value in integerregisterrs1is treated
>>>>> as a bit mask that specifies bit positions to be cleared in the CSR. Any
>>>>> bitthat is high inrs1will cause the corresponding bit to be cleared in
>>>>> the CSR, if that CSR bit iswritable. Other bits in the CSR are
>>>>> unaffected.")
>>>>
>>>> I think there may also be an understanding issue on what the SR_SUM
>>>> bit does. I thought if it is set, M->U accesses would fault, which is
>>>> why it gets set early on. But from reading the uaccess code it looks
>>>> like the uaccess code sets it on entry and then clears on exit.
>>>>
>>>> I am very confused. Is there a master reference for rv64?
>>>>
>>>> https://people.eecs.berkeley.edu/~krste/papers/riscv-privileged-v1.9.pdf
>>>> seems to state PUM is the SR_SUM bit, and that (if set) disabled
>>>>
>>>> Quote:
>>>> The PUM (Protect User Memory) bit modifies the privilege with which
>>>> S-mode loads, stores, and instruction fetches access virtual memory.
>>>> When PUM=0, translation and protection behave as normal. When PUM=1,
>>>> S-mode memory accesses to pages that are accessible by U-mode (U=1 in
>>>> Figure 4.19) will fault. PUM has no effect when executing in U-mode
>>>>
>>>>
>>>>>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
>>>>>>
>>>>>
>>>>> Still no luck for the moment, can't reproduce it locally, my test is
>>>>> maybe not that good (I created threads all day long in order to trigger
>>>>> the put_user of schedule_tail).
>>>>
>>>> It may of course depend on memory and other stuff. I did try to see if
>>>> it was possible to clone() with the child_tid address being a valid but
>>>> not mapped page...
>>>>
>>>>> Given that the path you mention works most of the time, and that the
>>>>> status register in the stack trace shows the SUM bit is not set whereas
>>>>> it is set in put_user, I'm leaning toward some race condition (maybe an
>>>>> interrupt that arrives at the "wrong" time) or a qemu issue as you
>>>>> mentioned.
>>>>
>>>> I suppose this is possible. From what I read it should get to the
>>>> point of being there with the SUM flag cleared, so either something
>>>> went wrong in trying to fix the instruction up or there's some other
>>>> error we're missing.
>>>>
>>>>> To eliminate qemu issues, do you have access to some HW ? Or to
>>>>> different qemu versions ?
>>>>
>>>> I do have access to a Microchip Polarfire board. I just need the
>>>> instructions on how to setup the test-code to make it work on the
>>>> hardware.
>>>
>>> For full syzkaller support, it would need to know how to reboot these
>>> boards and get access to the console.
>>> syzkaller has a stop-gap VM backend which just uses ssh to a physical
>>> machine and expects the kernel to reboot on its own after any crashes.
>>>
>>> But I actually managed to reproduce it in an even simpler setup.
>>> Assuming you have Go 1.15 and riscv64 cross-compiler gcc installed
>>>
>>> $ go get -u -d github.com/google/syzkaller/...
>>> $ cd $GOPATH/src/github.com/google/syzkaller
>>> $ make stress executor TARGETARCH=riscv64
>>> $ scp bin/linux_riscv64/syz-execprog bin/linux_riscv64/syz-executor
>>> your_machine:/
>>>
>>> Then run ./syz-stress on the machine.
>>> On the first run it crashed it with some other bug, on the second run
>>> I got the crash in schedule_tail.
>>> With qemu tcg I also added -slowdown=10 flag to syz-stress to scale
>>> all timeouts, if native execution is faster, then you don't need it.
>>
>> Ok, not sure what's going on. I get a lot of errors similar to:
>>>
>>> 2021/03/15 21:35:20 transitively unsupported: ioctl$SNAPSHOT_CREATE_IMAGE: no syscalls can create resource fd_snapshot, enable some syscalls that can create it [openat$snapshot]
>
> This is not an error, just a notification that some syscalls are not
> enabled in the kernel and won't be fuzzed.
>
>> Followed by:
>>
>>> 2021/03/15 21:35:48 executed 0 programs
>>> 2021/03/15 21:35:48 failed to create execution environment: failed to mmap shm file: invalid argument
>>
>> The qemu is 5.2.0 and root is Debian/unstable riscv64 (same as chroot
>> used to build the syz tools)
>
> This is an error. But I see it the first time ever.
> It comes from here:
> https://github.com/google/syzkaller/blob/fdb2bb2c23ee709880407f56307e2800ad27e9ae/pkg/osutil/osutil_unix.go#L119-L121
> There should be pretty simple logic inside of syscall.Mmap. Perhaps
> you are using some older Go toolchain with incomplete riscv support?
> I think I've used 1.14 and 1.15. But there is already 1.16. You can
> always download a toolchain here:
> https://golang.org/dl/
Hmm it would have been useful to print out what file it failed to map.
I've got go 1.15 from the debian/unstable riscv64 chroot.
I'll have a look at this in a bit to see if it throws the same issue on
a real system.
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-16 11:35 ` Ben Dooks
@ 2021-03-16 11:44 ` Dmitry Vyukov
-1 siblings, 0 replies; 51+ messages in thread
From: Dmitry Vyukov @ 2021-03-16 11:44 UTC (permalink / raw)
To: Ben Dooks
Cc: Alex Ghiti, syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou,
linux-riscv, Daniel Bristot de Oliveira, Benjamin Segall,
dietmar.eggemann, Juri Lelli, LKML, Mel Gorman, Ingo Molnar,
Peter Zijlstra, Steven Rostedt, syzkaller-bugs, Vincent Guittot
On Tue, Mar 16, 2021 at 12:35 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
> >>>> On 12/03/2021 16:25, Alex Ghiti wrote:
> >>>>>
> >>>>>
> >>>>> Le 3/12/21 à 10:12 AM, Dmitry Vyukov a écrit :
> >>>>>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
> >>>>>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
> >>>>>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>> Hello,
> >>>>>>>>>
> >>>>>>>>> syzbot found the following issue on:
> >>>>>>>>>
> >>>>>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
> >>>>>>>>> arch_dup_tas..
> >>>>>>>>> git tree:
> >>>>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
> >>>>>>>>> console output:
> >>>>>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
> >>>>>>>>> kernel config:
> >>>>>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
> >>>>>>>>> dashboard link:
> >>>>>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
> >>>>>>>>> userspace arch: riscv64
> >>>>>>>>>
> >>>>>>>>> Unfortunately, I don't have any reproducer for this issue yet.
> >>>>>>>>>
> >>>>>>>>> IMPORTANT: if you fix the issue, please add the following tag to
> >>>>>>>>> the commit:
> >>>>>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
> >>>>>>>>
> >>>>>>>> +riscv maintainers
> >>>>>>>>
> >>>>>>>> This is riscv64-specific.
> >>>>>>>> I've seen similar crashes in put_user in other places. It looks like
> >>>>>>>> put_user crashes in the user address is not mapped/protected (?).
> >>>>>>>
> >>>>>>> I've been having a look, and this seems to be down to access of the
> >>>>>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
> >>>>>>> bad address to clone?
> >>>>>>>
> >>>>>>> From looking at the code, the put_user() code should have set the
> >>>>>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
> >>>>>>> s2 register in the crash report) and from looking at the compiler
> >>>>>>> output from my gcc-10, the code looks to be dong the relevant csrs
> >>>>>>> and then csrc around the put_user
> >>>>>>>
> >>>>>>> So currently I do not understand how the above could have happened
> >>>>>>> over than something re-tried the code seqeunce and ended up retrying
> >>>>>>> the faulting instruction without the SR_SUM bit set.
> >>>>>>
> >>>>>> I would maybe blame qemu for randomly resetting SR_SUM, but it's
> >>>>>> strange that 99% of these crashes are in schedule_tail. If it would be
> >>>>>> qemu, then they would be more evenly distributed...
> >>>>>>
> >>>>>> Another observation: looking at a dozen of crash logs, in none of
> >>>>>> these cases fuzzer was actually trying to fuzz clone with some insane
> >>>>>> arguments. So it looks like completely normal clone's (e..g coming
> >>>>>> from pthread_create) result in this crash.
> >>>>>>
> >>>>>> I also wonder why there is ret_from_exception, is it normal? I see
> >>>>>> handle_exception disables SR_SUM:
> >>>>>
> >>>>> csrrc does the right thing: it cleans SR_SUM bit in status but saves the
> >>>>> previous value that will get correctly restored.
> >>>>>
> >>>>> ("The CSRRC (Atomic Read and Clear Bits in CSR) instruction reads the
> >>>>> value of the CSR, zero-extends the value to XLEN bits, and writes it to
> >>>>> integer registerrd. The initial value in integerregisterrs1is treated
> >>>>> as a bit mask that specifies bit positions to be cleared in the CSR. Any
> >>>>> bitthat is high inrs1will cause the corresponding bit to be cleared in
> >>>>> the CSR, if that CSR bit iswritable. Other bits in the CSR are
> >>>>> unaffected.")
> >>>>
> >>>> I think there may also be an understanding issue on what the SR_SUM
> >>>> bit does. I thought if it is set, M->U accesses would fault, which is
> >>>> why it gets set early on. But from reading the uaccess code it looks
> >>>> like the uaccess code sets it on entry and then clears on exit.
> >>>>
> >>>> I am very confused. Is there a master reference for rv64?
> >>>>
> >>>> https://people.eecs.berkeley.edu/~krste/papers/riscv-privileged-v1.9.pdf
> >>>> seems to state PUM is the SR_SUM bit, and that (if set) disabled
> >>>>
> >>>> Quote:
> >>>> The PUM (Protect User Memory) bit modifies the privilege with which
> >>>> S-mode loads, stores, and instruction fetches access virtual memory.
> >>>> When PUM=0, translation and protection behave as normal. When PUM=1,
> >>>> S-mode memory accesses to pages that are accessible by U-mode (U=1 in
> >>>> Figure 4.19) will fault. PUM has no effect when executing in U-mode
> >>>>
> >>>>
> >>>>>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
> >>>>>>
> >>>>>
> >>>>> Still no luck for the moment, can't reproduce it locally, my test is
> >>>>> maybe not that good (I created threads all day long in order to trigger
> >>>>> the put_user of schedule_tail).
> >>>>
> >>>> It may of course depend on memory and other stuff. I did try to see if
> >>>> it was possible to clone() with the child_tid address being a valid but
> >>>> not mapped page...
> >>>>
> >>>>> Given that the path you mention works most of the time, and that the
> >>>>> status register in the stack trace shows the SUM bit is not set whereas
> >>>>> it is set in put_user, I'm leaning toward some race condition (maybe an
> >>>>> interrupt that arrives at the "wrong" time) or a qemu issue as you
> >>>>> mentioned.
> >>>>
> >>>> I suppose this is possible. From what I read it should get to the
> >>>> point of being there with the SUM flag cleared, so either something
> >>>> went wrong in trying to fix the instruction up or there's some other
> >>>> error we're missing.
> >>>>
> >>>>> To eliminate qemu issues, do you have access to some HW ? Or to
> >>>>> different qemu versions ?
> >>>>
> >>>> I do have access to a Microchip Polarfire board. I just need the
> >>>> instructions on how to setup the test-code to make it work on the
> >>>> hardware.
> >>>
> >>> For full syzkaller support, it would need to know how to reboot these
> >>> boards and get access to the console.
> >>> syzkaller has a stop-gap VM backend which just uses ssh to a physical
> >>> machine and expects the kernel to reboot on its own after any crashes.
> >>>
> >>> But I actually managed to reproduce it in an even simpler setup.
> >>> Assuming you have Go 1.15 and riscv64 cross-compiler gcc installed
> >>>
> >>> $ go get -u -d github.com/google/syzkaller/...
> >>> $ cd $GOPATH/src/github.com/google/syzkaller
> >>> $ make stress executor TARGETARCH=riscv64
> >>> $ scp bin/linux_riscv64/syz-execprog bin/linux_riscv64/syz-executor
> >>> your_machine:/
> >>>
> >>> Then run ./syz-stress on the machine.
> >>> On the first run it crashed it with some other bug, on the second run
> >>> I got the crash in schedule_tail.
> >>> With qemu tcg I also added -slowdown=10 flag to syz-stress to scale
> >>> all timeouts, if native execution is faster, then you don't need it.
> >>
> >> Ok, not sure what's going on. I get a lot of errors similar to:
> >>>
> >>> 2021/03/15 21:35:20 transitively unsupported: ioctl$SNAPSHOT_CREATE_IMAGE: no syscalls can create resource fd_snapshot, enable some syscalls that can create it [openat$snapshot]
> >
> > This is not an error, just a notification that some syscalls are not
> > enabled in the kernel and won't be fuzzed.
> >
> >> Followed by:
> >>
> >>> 2021/03/15 21:35:48 executed 0 programs
> >>> 2021/03/15 21:35:48 failed to create execution environment: failed to mmap shm file: invalid argument
> >>
> >> The qemu is 5.2.0 and root is Debian/unstable riscv64 (same as chroot
> >> used to build the syz tools)
> >
> > This is an error. But I see it the first time ever.
> > It comes from here:
> > https://github.com/google/syzkaller/blob/fdb2bb2c23ee709880407f56307e2800ad27e9ae/pkg/osutil/osutil_unix.go#L119-L121
> > There should be pretty simple logic inside of syscall.Mmap. Perhaps
> > you are using some older Go toolchain with incomplete riscv support?
> > I think I've used 1.14 and 1.15. But there is already 1.16. You can
> > always download a toolchain here:
> > https://golang.org/dl/
>
> Hmm it would have been useful to print out what file it failed to map.
What do you want to do with the file name? It's not one of
pre-existing files, so the name won't tell the user much. It's just a
temp file, it won't exist afterwards and it's easy to create an
equivalent file.
It was created in that function with:
f, err = ioutil.TempFile("./", "syzkaller-shm")
if err != nil {
err = fmt.Errorf("failed to create temp file: %v", err)
return
}
if err = f.Truncate(int64(size)); err != nil {
err = fmt.Errorf("failed to truncate shm file: %v", err)
f.Close()
os.Remove(f.Name())
return
}
f.Close()
fname := f.Name()
f, err = os.OpenFile(f.Name(), os.O_RDWR, DefaultFilePerm)
if err != nil {
err = fmt.Errorf("failed to open shm file: %v", err)
os.Remove(fname)
return
}
> I've got go 1.15 from the debian/unstable riscv64 chroot.
> I'll have a look at this in a bit to see if it throws the same issue on
> a real system.
>
>
> --
> Ben Dooks http://www.codethink.co.uk/
> Senior Engineer Codethink - Providing Genius
>
> https://www.codethink.co.uk/privacy.html
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/8ebea51d-b03c-e6de-fa1c-d47091c54e45%40codethink.co.uk.
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-16 11:44 ` Dmitry Vyukov
0 siblings, 0 replies; 51+ messages in thread
From: Dmitry Vyukov @ 2021-03-16 11:44 UTC (permalink / raw)
To: Ben Dooks
Cc: Alex Ghiti, syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou,
linux-riscv, Daniel Bristot de Oliveira, Benjamin Segall,
dietmar.eggemann, Juri Lelli, LKML, Mel Gorman, Ingo Molnar,
Peter Zijlstra, Steven Rostedt, syzkaller-bugs, Vincent Guittot
On Tue, Mar 16, 2021 at 12:35 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
> >>>> On 12/03/2021 16:25, Alex Ghiti wrote:
> >>>>>
> >>>>>
> >>>>> Le 3/12/21 à 10:12 AM, Dmitry Vyukov a écrit :
> >>>>>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
> >>>>>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
> >>>>>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>> Hello,
> >>>>>>>>>
> >>>>>>>>> syzbot found the following issue on:
> >>>>>>>>>
> >>>>>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
> >>>>>>>>> arch_dup_tas..
> >>>>>>>>> git tree:
> >>>>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
> >>>>>>>>> console output:
> >>>>>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
> >>>>>>>>> kernel config:
> >>>>>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
> >>>>>>>>> dashboard link:
> >>>>>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
> >>>>>>>>> userspace arch: riscv64
> >>>>>>>>>
> >>>>>>>>> Unfortunately, I don't have any reproducer for this issue yet.
> >>>>>>>>>
> >>>>>>>>> IMPORTANT: if you fix the issue, please add the following tag to
> >>>>>>>>> the commit:
> >>>>>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
> >>>>>>>>
> >>>>>>>> +riscv maintainers
> >>>>>>>>
> >>>>>>>> This is riscv64-specific.
> >>>>>>>> I've seen similar crashes in put_user in other places. It looks like
> >>>>>>>> put_user crashes in the user address is not mapped/protected (?).
> >>>>>>>
> >>>>>>> I've been having a look, and this seems to be down to access of the
> >>>>>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
> >>>>>>> bad address to clone?
> >>>>>>>
> >>>>>>> From looking at the code, the put_user() code should have set the
> >>>>>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
> >>>>>>> s2 register in the crash report) and from looking at the compiler
> >>>>>>> output from my gcc-10, the code looks to be dong the relevant csrs
> >>>>>>> and then csrc around the put_user
> >>>>>>>
> >>>>>>> So currently I do not understand how the above could have happened
> >>>>>>> over than something re-tried the code seqeunce and ended up retrying
> >>>>>>> the faulting instruction without the SR_SUM bit set.
> >>>>>>
> >>>>>> I would maybe blame qemu for randomly resetting SR_SUM, but it's
> >>>>>> strange that 99% of these crashes are in schedule_tail. If it would be
> >>>>>> qemu, then they would be more evenly distributed...
> >>>>>>
> >>>>>> Another observation: looking at a dozen of crash logs, in none of
> >>>>>> these cases fuzzer was actually trying to fuzz clone with some insane
> >>>>>> arguments. So it looks like completely normal clone's (e..g coming
> >>>>>> from pthread_create) result in this crash.
> >>>>>>
> >>>>>> I also wonder why there is ret_from_exception, is it normal? I see
> >>>>>> handle_exception disables SR_SUM:
> >>>>>
> >>>>> csrrc does the right thing: it cleans SR_SUM bit in status but saves the
> >>>>> previous value that will get correctly restored.
> >>>>>
> >>>>> ("The CSRRC (Atomic Read and Clear Bits in CSR) instruction reads the
> >>>>> value of the CSR, zero-extends the value to XLEN bits, and writes it to
> >>>>> integer registerrd. The initial value in integerregisterrs1is treated
> >>>>> as a bit mask that specifies bit positions to be cleared in the CSR. Any
> >>>>> bitthat is high inrs1will cause the corresponding bit to be cleared in
> >>>>> the CSR, if that CSR bit iswritable. Other bits in the CSR are
> >>>>> unaffected.")
> >>>>
> >>>> I think there may also be an understanding issue on what the SR_SUM
> >>>> bit does. I thought if it is set, M->U accesses would fault, which is
> >>>> why it gets set early on. But from reading the uaccess code it looks
> >>>> like the uaccess code sets it on entry and then clears on exit.
> >>>>
> >>>> I am very confused. Is there a master reference for rv64?
> >>>>
> >>>> https://people.eecs.berkeley.edu/~krste/papers/riscv-privileged-v1.9.pdf
> >>>> seems to state PUM is the SR_SUM bit, and that (if set) disabled
> >>>>
> >>>> Quote:
> >>>> The PUM (Protect User Memory) bit modifies the privilege with which
> >>>> S-mode loads, stores, and instruction fetches access virtual memory.
> >>>> When PUM=0, translation and protection behave as normal. When PUM=1,
> >>>> S-mode memory accesses to pages that are accessible by U-mode (U=1 in
> >>>> Figure 4.19) will fault. PUM has no effect when executing in U-mode
> >>>>
> >>>>
> >>>>>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
> >>>>>>
> >>>>>
> >>>>> Still no luck for the moment, can't reproduce it locally, my test is
> >>>>> maybe not that good (I created threads all day long in order to trigger
> >>>>> the put_user of schedule_tail).
> >>>>
> >>>> It may of course depend on memory and other stuff. I did try to see if
> >>>> it was possible to clone() with the child_tid address being a valid but
> >>>> not mapped page...
> >>>>
> >>>>> Given that the path you mention works most of the time, and that the
> >>>>> status register in the stack trace shows the SUM bit is not set whereas
> >>>>> it is set in put_user, I'm leaning toward some race condition (maybe an
> >>>>> interrupt that arrives at the "wrong" time) or a qemu issue as you
> >>>>> mentioned.
> >>>>
> >>>> I suppose this is possible. From what I read it should get to the
> >>>> point of being there with the SUM flag cleared, so either something
> >>>> went wrong in trying to fix the instruction up or there's some other
> >>>> error we're missing.
> >>>>
> >>>>> To eliminate qemu issues, do you have access to some HW ? Or to
> >>>>> different qemu versions ?
> >>>>
> >>>> I do have access to a Microchip Polarfire board. I just need the
> >>>> instructions on how to setup the test-code to make it work on the
> >>>> hardware.
> >>>
> >>> For full syzkaller support, it would need to know how to reboot these
> >>> boards and get access to the console.
> >>> syzkaller has a stop-gap VM backend which just uses ssh to a physical
> >>> machine and expects the kernel to reboot on its own after any crashes.
> >>>
> >>> But I actually managed to reproduce it in an even simpler setup.
> >>> Assuming you have Go 1.15 and riscv64 cross-compiler gcc installed
> >>>
> >>> $ go get -u -d github.com/google/syzkaller/...
> >>> $ cd $GOPATH/src/github.com/google/syzkaller
> >>> $ make stress executor TARGETARCH=riscv64
> >>> $ scp bin/linux_riscv64/syz-execprog bin/linux_riscv64/syz-executor
> >>> your_machine:/
> >>>
> >>> Then run ./syz-stress on the machine.
> >>> On the first run it crashed it with some other bug, on the second run
> >>> I got the crash in schedule_tail.
> >>> With qemu tcg I also added -slowdown=10 flag to syz-stress to scale
> >>> all timeouts, if native execution is faster, then you don't need it.
> >>
> >> Ok, not sure what's going on. I get a lot of errors similar to:
> >>>
> >>> 2021/03/15 21:35:20 transitively unsupported: ioctl$SNAPSHOT_CREATE_IMAGE: no syscalls can create resource fd_snapshot, enable some syscalls that can create it [openat$snapshot]
> >
> > This is not an error, just a notification that some syscalls are not
> > enabled in the kernel and won't be fuzzed.
> >
> >> Followed by:
> >>
> >>> 2021/03/15 21:35:48 executed 0 programs
> >>> 2021/03/15 21:35:48 failed to create execution environment: failed to mmap shm file: invalid argument
> >>
> >> The qemu is 5.2.0 and root is Debian/unstable riscv64 (same as chroot
> >> used to build the syz tools)
> >
> > This is an error. But I see it the first time ever.
> > It comes from here:
> > https://github.com/google/syzkaller/blob/fdb2bb2c23ee709880407f56307e2800ad27e9ae/pkg/osutil/osutil_unix.go#L119-L121
> > There should be pretty simple logic inside of syscall.Mmap. Perhaps
> > you are using some older Go toolchain with incomplete riscv support?
> > I think I've used 1.14 and 1.15. But there is already 1.16. You can
> > always download a toolchain here:
> > https://golang.org/dl/
>
> Hmm it would have been useful to print out what file it failed to map.
What do you want to do with the file name? It's not one of
pre-existing files, so the name won't tell the user much. It's just a
temp file, it won't exist afterwards and it's easy to create an
equivalent file.
It was created in that function with:
f, err = ioutil.TempFile("./", "syzkaller-shm")
if err != nil {
err = fmt.Errorf("failed to create temp file: %v", err)
return
}
if err = f.Truncate(int64(size)); err != nil {
err = fmt.Errorf("failed to truncate shm file: %v", err)
f.Close()
os.Remove(f.Name())
return
}
f.Close()
fname := f.Name()
f, err = os.OpenFile(f.Name(), os.O_RDWR, DefaultFilePerm)
if err != nil {
err = fmt.Errorf("failed to open shm file: %v", err)
os.Remove(fname)
return
}
> I've got go 1.15 from the debian/unstable riscv64 chroot.
> I'll have a look at this in a bit to see if it throws the same issue on
> a real system.
>
>
> --
> Ben Dooks http://www.codethink.co.uk/
> Senior Engineer Codethink - Providing Genius
>
> https://www.codethink.co.uk/privacy.html
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/8ebea51d-b03c-e6de-fa1c-d47091c54e45%40codethink.co.uk.
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-12 17:38 ` Dmitry Vyukov
@ 2021-03-18 9:41 ` Ben Dooks
-1 siblings, 0 replies; 51+ messages in thread
From: Ben Dooks @ 2021-03-18 9:41 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv,
Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot, Terry Hu,
Javier Jardón
On 12/03/2021 17:38, Dmitry Vyukov wrote:
> On Fri, Mar 12, 2021 at 6:34 PM Dmitry Vyukov <dvyukov@google.com> wrote:
>>
>> On Fri, Mar 12, 2021 at 5:36 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>>>
>>> On 12/03/2021 16:34, Ben Dooks wrote:
>>>> On 12/03/2021 16:30, Ben Dooks wrote:
>>>>> On 12/03/2021 15:12, Dmitry Vyukov wrote:
>>>>>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
>>>>>> wrote:
>>>>>>>
>>>>>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
>>>>>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
>>>>>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> syzbot found the following issue on:
>>>>>>>>>
>>>>>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
>>>>>>>>> arch_dup_tas..
>>>>>>>>> git tree:
>>>>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>>>>>>>>> console output:
>>>>>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>>>>>>>>> kernel config:
>>>>>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>>>>>>>>> dashboard link:
>>>>>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>>>>>>>>> userspace arch: riscv64
>>>>>>>>>
>>>>>>>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>>>>>>>
>>>>>>>>> IMPORTANT: if you fix the issue, please add the following tag to
>>>>>>>>> the commit:
>>>>>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>>>>>>>>
>>>>>>>> +riscv maintainers
>>>>>>>>
>>>>>>>> This is riscv64-specific.
>>>>>>>> I've seen similar crashes in put_user in other places. It looks like
>>>>>>>> put_user crashes in the user address is not mapped/protected (?).
>>>>>>>
>>>>>>> I've been having a look, and this seems to be down to access of the
>>>>>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
>>>>>>> bad address to clone?
>>>>>>>
>>>>>>> From looking at the code, the put_user() code should have set the
>>>>>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
>>>>>>> s2 register in the crash report) and from looking at the compiler
>>>>>>> output from my gcc-10, the code looks to be dong the relevant csrs
>>>>>>> and then csrc around the put_user
>>>>>>>
>>>>>>> So currently I do not understand how the above could have happened
>>>>>>> over than something re-tried the code seqeunce and ended up retrying
>>>>>>> the faulting instruction without the SR_SUM bit set.
>>>>>>
>>>>>> I would maybe blame qemu for randomly resetting SR_SUM, but it's
>>>>>> strange that 99% of these crashes are in schedule_tail. If it would be
>>>>>> qemu, then they would be more evenly distributed...
>>>>>>
>>>>>> Another observation: looking at a dozen of crash logs, in none of
>>>>>> these cases fuzzer was actually trying to fuzz clone with some insane
>>>>>> arguments. So it looks like completely normal clone's (e..g coming
>>>>>> from pthread_create) result in this crash.
>>>>>>
>>>>>> I also wonder why there is ret_from_exception, is it normal? I see
>>>>>> handle_exception disables SR_SUM:
>>>>>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
>>>>>>
>>>>>
>>>>> So I think if SR_SUM is set, then it faults the access to user memory
>>>>> which the _user() routines clear to allow them access.
>>>>>
>>>>> I'm thinking there is at least one issue here:
>>>>>
>>>>> - the test in fault is the wrong way around for die kernel
>>>>> - the handler only catches this if the page has yet to be mapped.
>>>>>
>>>>> So I think the test should be:
>>>>>
>>>>> if (!user_mode(regs) && addr < TASK_SIZE &&
>>>>> unlikely(regs->status & SR_SUM)
>>>>>
>>>>> This then should continue on and allow the rest of the handler to
>>>>> complete mapping the page if it is not there.
>>>>>
>>>>> I have been trying to create a very simple clone test, but so far it
>>>>> has yet to actually trigger anything.
>>>>
>>>> I should have added there doesn't seem to be a good way to use mmap()
>>>> to allocate memory but not insert a vm-mapping post the mmap().
>>>>
>>> How difficult is it to try building a branch with the above test
>>> modified?
>>
>> I don't have access to hardware, I don't have other qemu versions ready to use.
>> But I can teach you how to run syzkaller locally :)
>> I am not sure anybody run it on real riscv hardware at all. When
>> Tobias ported syzkaller, Tobias also used qemu I think.
>>
>> I am now building with an inverted check to test locally.
>>
>> I don't fully understand but this code, but does handle_exception
>> reset SR_SUM around do_page_fault? If so, then looking at SR_SUM in
>> do_page_fault won't work with positive nor negative check.
>
>
> The inverted check crashes during boot:
>
> --- a/arch/riscv/mm/fault.c
> +++ b/arch/riscv/mm/fault.c
> @@ -249,7 +249,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> flags |= FAULT_FLAG_USER;
>
> if (!user_mode(regs) && addr < TASK_SIZE &&
> - unlikely(!(regs->status & SR_SUM)))
> + unlikely(regs->status & SR_SUM))
> die_kernel_fault("access to user memory without
> uaccess routines",
> addr, regs);
>
>
> [ 77.349329][ T1] Run /sbin/init as init process
> [ 77.868371][ T1] Unable to handle kernel access to user memory
> without uaccess routines at virtual address 00000000000e8e39
> [ 77.870355][ T1] Oops [#1]
> [ 77.870766][ T1] Modules linked in:
> [ 77.871326][ T1] CPU: 0 PID: 1 Comm: init Not tainted
> 5.12.0-rc2-00010-g0d7588ab9ef9-dirty #42
> [ 77.872057][ T1] Hardware name: riscv-virtio,qemu (DT)
> [ 77.872620][ T1] epc : __clear_user+0x36/0x4e
> [ 77.873285][ T1] ra : padzero+0x9c/0xb0
> [ 77.873849][ T1] epc : ffffffe000bb7136 ra : ffffffe0004f42a0 sp
> : ffffffe006f8fbc0
> [ 77.874438][ T1] gp : ffffffe005d25718 tp : ffffffe006f98000 t0
> : 00000000000e8e40
> [ 77.875031][ T1] t1 : 00000000000e9000 t2 : 000000000001c49c s0
> : ffffffe006f8fbf0
> [ 77.875618][ T1] s1 : 00000000000001c7 a0 : 00000000000e8e39 a1
> : 00000000000001c7
> [ 77.876204][ T1] a2 : 0000000000000002 a3 : 00000000000e9000 a4
> : ffffffe006f99000
> [ 77.876787][ T1] a5 : 0000000000000000 a6 : 0000000000f00000 a7
> : ffffffe00031c088
> [ 77.877367][ T1] s2 : 00000000000e8e39 s3 : 0000000000001000 s4
> : 0000003ffffffe39
> [ 77.877952][ T1] s5 : 00000000000e8e39 s6 : 00000000000e9570 s7
> : 00000000000e8e39
> [ 77.878535][ T1] s8 : 0000000000000001 s9 : 00000000000e8e39
> s10: ffffffe00c65f608
> [ 77.879126][ T1] s11: ffffffe00816e8d8 t3 : ea3af0fa372b8300 t4
> : 0000000000000003
> [ 77.879711][ T1] t5 : ffffffc401dc45d8 t6 : 0000000000040000
> [ 77.880209][ T1] status: 0000000000040120 badaddr:
> 00000000000e8e39 cause: 000000000000000f
> [ 77.880846][ T1] Call Trace:
> [ 77.881213][ T1] [<ffffffe000bb7136>] __clear_user+0x36/0x4e
> [ 77.881912][ T1] [<ffffffe0004f523e>] load_elf_binary+0xf8a/0x2400
> [ 77.882562][ T1] [<ffffffe0003e1802>] bprm_execve+0x5b0/0x1080
> [ 77.883145][ T1] [<ffffffe0003e38bc>] kernel_execve+0x204/0x288
> [ 77.883727][ T1] [<ffffffe003b70e94>] run_init_process+0x1fe/0x212
> [ 77.884337][ T1] [<ffffffe003b70ec6>] try_to_run_init_process+0x1e/0x66
> [ 77.884956][ T1] [<ffffffe003bc0864>] kernel_init+0x14a/0x200
> [ 77.885541][ T1] [<ffffffe000005570>] ret_from_exception+0x0/0x14
> [ 77.886955][ T1] ---[ end trace 1e934d07b8a4bed8 ]---
> [ 77.887705][ T1] Kernel panic - not syncing: Fatal exception
> [ 77.888333][ T1] SMP: stopping secondary CPUs
> [ 77.889357][ T1] Rebooting in 86400 seconds..
I have reproduced this on qemu, not managed to get the real hardwre
working with this branch yet.
I have a working hypothesis now, having added debug to check the
sstatus.SR_SUM flag and reviewed the assembly, I think this is
what is happening:
C code of "put_user(func(), address)" is generating code to do:
1: __enable_user_access();
2: cpu_reg = func();
3: assembly for *address = cpu_reg;
4: __disable_user_access();
I think the call to func() with all the sanitisers enabled allow
the func() to possibly schedule out. The __swtich_to() code does
not restore the original status registers which means that if
there is IO during the sleep SR_SUM may end up being cleared and
never re-set. We get back to 3 and fault as 2 cleared the result of 1.
It is very possible no-one has seen this before as generally the
functions involved in feeding put_user() are fairly small and thus
this system is both under load and has some reason to schedule then
this bug has probably been rare to unseen.
I think the correct solution is to store the SR_SUM bit status in
the thread_struct and make __switch_to() save/restore this when
changing between tasks/threads. Trying to re-order the code to
force swapping of 1 and 2 may reduce the bug's window.
Further thinking of the order of 1 and 2 is that we should probably
fix that order so that func() is not run with the user-space access
protection disabled.
I'll try and make some sort of of small test case to avoid having
to run syz-stress to provoke this.
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-18 9:41 ` Ben Dooks
0 siblings, 0 replies; 51+ messages in thread
From: Ben Dooks @ 2021-03-18 9:41 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv,
Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot, Terry Hu,
Javier Jardón
On 12/03/2021 17:38, Dmitry Vyukov wrote:
> On Fri, Mar 12, 2021 at 6:34 PM Dmitry Vyukov <dvyukov@google.com> wrote:
>>
>> On Fri, Mar 12, 2021 at 5:36 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>>>
>>> On 12/03/2021 16:34, Ben Dooks wrote:
>>>> On 12/03/2021 16:30, Ben Dooks wrote:
>>>>> On 12/03/2021 15:12, Dmitry Vyukov wrote:
>>>>>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
>>>>>> wrote:
>>>>>>>
>>>>>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
>>>>>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
>>>>>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> syzbot found the following issue on:
>>>>>>>>>
>>>>>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
>>>>>>>>> arch_dup_tas..
>>>>>>>>> git tree:
>>>>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>>>>>>>>> console output:
>>>>>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>>>>>>>>> kernel config:
>>>>>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>>>>>>>>> dashboard link:
>>>>>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>>>>>>>>> userspace arch: riscv64
>>>>>>>>>
>>>>>>>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>>>>>>>
>>>>>>>>> IMPORTANT: if you fix the issue, please add the following tag to
>>>>>>>>> the commit:
>>>>>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>>>>>>>>
>>>>>>>> +riscv maintainers
>>>>>>>>
>>>>>>>> This is riscv64-specific.
>>>>>>>> I've seen similar crashes in put_user in other places. It looks like
>>>>>>>> put_user crashes in the user address is not mapped/protected (?).
>>>>>>>
>>>>>>> I've been having a look, and this seems to be down to access of the
>>>>>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
>>>>>>> bad address to clone?
>>>>>>>
>>>>>>> From looking at the code, the put_user() code should have set the
>>>>>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
>>>>>>> s2 register in the crash report) and from looking at the compiler
>>>>>>> output from my gcc-10, the code looks to be dong the relevant csrs
>>>>>>> and then csrc around the put_user
>>>>>>>
>>>>>>> So currently I do not understand how the above could have happened
>>>>>>> over than something re-tried the code seqeunce and ended up retrying
>>>>>>> the faulting instruction without the SR_SUM bit set.
>>>>>>
>>>>>> I would maybe blame qemu for randomly resetting SR_SUM, but it's
>>>>>> strange that 99% of these crashes are in schedule_tail. If it would be
>>>>>> qemu, then they would be more evenly distributed...
>>>>>>
>>>>>> Another observation: looking at a dozen of crash logs, in none of
>>>>>> these cases fuzzer was actually trying to fuzz clone with some insane
>>>>>> arguments. So it looks like completely normal clone's (e..g coming
>>>>>> from pthread_create) result in this crash.
>>>>>>
>>>>>> I also wonder why there is ret_from_exception, is it normal? I see
>>>>>> handle_exception disables SR_SUM:
>>>>>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
>>>>>>
>>>>>
>>>>> So I think if SR_SUM is set, then it faults the access to user memory
>>>>> which the _user() routines clear to allow them access.
>>>>>
>>>>> I'm thinking there is at least one issue here:
>>>>>
>>>>> - the test in fault is the wrong way around for die kernel
>>>>> - the handler only catches this if the page has yet to be mapped.
>>>>>
>>>>> So I think the test should be:
>>>>>
>>>>> if (!user_mode(regs) && addr < TASK_SIZE &&
>>>>> unlikely(regs->status & SR_SUM)
>>>>>
>>>>> This then should continue on and allow the rest of the handler to
>>>>> complete mapping the page if it is not there.
>>>>>
>>>>> I have been trying to create a very simple clone test, but so far it
>>>>> has yet to actually trigger anything.
>>>>
>>>> I should have added there doesn't seem to be a good way to use mmap()
>>>> to allocate memory but not insert a vm-mapping post the mmap().
>>>>
>>> How difficult is it to try building a branch with the above test
>>> modified?
>>
>> I don't have access to hardware, I don't have other qemu versions ready to use.
>> But I can teach you how to run syzkaller locally :)
>> I am not sure anybody run it on real riscv hardware at all. When
>> Tobias ported syzkaller, Tobias also used qemu I think.
>>
>> I am now building with an inverted check to test locally.
>>
>> I don't fully understand but this code, but does handle_exception
>> reset SR_SUM around do_page_fault? If so, then looking at SR_SUM in
>> do_page_fault won't work with positive nor negative check.
>
>
> The inverted check crashes during boot:
>
> --- a/arch/riscv/mm/fault.c
> +++ b/arch/riscv/mm/fault.c
> @@ -249,7 +249,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> flags |= FAULT_FLAG_USER;
>
> if (!user_mode(regs) && addr < TASK_SIZE &&
> - unlikely(!(regs->status & SR_SUM)))
> + unlikely(regs->status & SR_SUM))
> die_kernel_fault("access to user memory without
> uaccess routines",
> addr, regs);
>
>
> [ 77.349329][ T1] Run /sbin/init as init process
> [ 77.868371][ T1] Unable to handle kernel access to user memory
> without uaccess routines at virtual address 00000000000e8e39
> [ 77.870355][ T1] Oops [#1]
> [ 77.870766][ T1] Modules linked in:
> [ 77.871326][ T1] CPU: 0 PID: 1 Comm: init Not tainted
> 5.12.0-rc2-00010-g0d7588ab9ef9-dirty #42
> [ 77.872057][ T1] Hardware name: riscv-virtio,qemu (DT)
> [ 77.872620][ T1] epc : __clear_user+0x36/0x4e
> [ 77.873285][ T1] ra : padzero+0x9c/0xb0
> [ 77.873849][ T1] epc : ffffffe000bb7136 ra : ffffffe0004f42a0 sp
> : ffffffe006f8fbc0
> [ 77.874438][ T1] gp : ffffffe005d25718 tp : ffffffe006f98000 t0
> : 00000000000e8e40
> [ 77.875031][ T1] t1 : 00000000000e9000 t2 : 000000000001c49c s0
> : ffffffe006f8fbf0
> [ 77.875618][ T1] s1 : 00000000000001c7 a0 : 00000000000e8e39 a1
> : 00000000000001c7
> [ 77.876204][ T1] a2 : 0000000000000002 a3 : 00000000000e9000 a4
> : ffffffe006f99000
> [ 77.876787][ T1] a5 : 0000000000000000 a6 : 0000000000f00000 a7
> : ffffffe00031c088
> [ 77.877367][ T1] s2 : 00000000000e8e39 s3 : 0000000000001000 s4
> : 0000003ffffffe39
> [ 77.877952][ T1] s5 : 00000000000e8e39 s6 : 00000000000e9570 s7
> : 00000000000e8e39
> [ 77.878535][ T1] s8 : 0000000000000001 s9 : 00000000000e8e39
> s10: ffffffe00c65f608
> [ 77.879126][ T1] s11: ffffffe00816e8d8 t3 : ea3af0fa372b8300 t4
> : 0000000000000003
> [ 77.879711][ T1] t5 : ffffffc401dc45d8 t6 : 0000000000040000
> [ 77.880209][ T1] status: 0000000000040120 badaddr:
> 00000000000e8e39 cause: 000000000000000f
> [ 77.880846][ T1] Call Trace:
> [ 77.881213][ T1] [<ffffffe000bb7136>] __clear_user+0x36/0x4e
> [ 77.881912][ T1] [<ffffffe0004f523e>] load_elf_binary+0xf8a/0x2400
> [ 77.882562][ T1] [<ffffffe0003e1802>] bprm_execve+0x5b0/0x1080
> [ 77.883145][ T1] [<ffffffe0003e38bc>] kernel_execve+0x204/0x288
> [ 77.883727][ T1] [<ffffffe003b70e94>] run_init_process+0x1fe/0x212
> [ 77.884337][ T1] [<ffffffe003b70ec6>] try_to_run_init_process+0x1e/0x66
> [ 77.884956][ T1] [<ffffffe003bc0864>] kernel_init+0x14a/0x200
> [ 77.885541][ T1] [<ffffffe000005570>] ret_from_exception+0x0/0x14
> [ 77.886955][ T1] ---[ end trace 1e934d07b8a4bed8 ]---
> [ 77.887705][ T1] Kernel panic - not syncing: Fatal exception
> [ 77.888333][ T1] SMP: stopping secondary CPUs
> [ 77.889357][ T1] Rebooting in 86400 seconds..
I have reproduced this on qemu, not managed to get the real hardwre
working with this branch yet.
I have a working hypothesis now, having added debug to check the
sstatus.SR_SUM flag and reviewed the assembly, I think this is
what is happening:
C code of "put_user(func(), address)" is generating code to do:
1: __enable_user_access();
2: cpu_reg = func();
3: assembly for *address = cpu_reg;
4: __disable_user_access();
I think the call to func() with all the sanitisers enabled allow
the func() to possibly schedule out. The __swtich_to() code does
not restore the original status registers which means that if
there is IO during the sleep SR_SUM may end up being cleared and
never re-set. We get back to 3 and fault as 2 cleared the result of 1.
It is very possible no-one has seen this before as generally the
functions involved in feeding put_user() are fairly small and thus
this system is both under load and has some reason to schedule then
this bug has probably been rare to unseen.
I think the correct solution is to store the SR_SUM bit status in
the thread_struct and make __switch_to() save/restore this when
changing between tasks/threads. Trying to re-order the code to
force swapping of 1 and 2 may reduce the bug's window.
Further thinking of the order of 1 and 2 is that we should probably
fix that order so that func() is not run with the user-space access
protection disabled.
I'll try and make some sort of of small test case to avoid having
to run syz-stress to provoke this.
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-18 9:41 ` Ben Dooks
@ 2021-03-18 10:05 ` Dmitry Vyukov
-1 siblings, 0 replies; 51+ messages in thread
From: Dmitry Vyukov @ 2021-03-18 10:05 UTC (permalink / raw)
To: Ben Dooks
Cc: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv,
Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot, Terry Hu,
Javier Jardón
On Thu, Mar 18, 2021 at 10:41 AM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>
> On 12/03/2021 17:38, Dmitry Vyukov wrote:
> > On Fri, Mar 12, 2021 at 6:34 PM Dmitry Vyukov <dvyukov@google.com> wrote:
> >>
> >> On Fri, Mar 12, 2021 at 5:36 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
> >>>
> >>> On 12/03/2021 16:34, Ben Dooks wrote:
> >>>> On 12/03/2021 16:30, Ben Dooks wrote:
> >>>>> On 12/03/2021 15:12, Dmitry Vyukov wrote:
> >>>>>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
> >>>>>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
> >>>>>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>> Hello,
> >>>>>>>>>
> >>>>>>>>> syzbot found the following issue on:
> >>>>>>>>>
> >>>>>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
> >>>>>>>>> arch_dup_tas..
> >>>>>>>>> git tree:
> >>>>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
> >>>>>>>>> console output:
> >>>>>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
> >>>>>>>>> kernel config:
> >>>>>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
> >>>>>>>>> dashboard link:
> >>>>>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
> >>>>>>>>> userspace arch: riscv64
> >>>>>>>>>
> >>>>>>>>> Unfortunately, I don't have any reproducer for this issue yet.
> >>>>>>>>>
> >>>>>>>>> IMPORTANT: if you fix the issue, please add the following tag to
> >>>>>>>>> the commit:
> >>>>>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
> >>>>>>>>
> >>>>>>>> +riscv maintainers
> >>>>>>>>
> >>>>>>>> This is riscv64-specific.
> >>>>>>>> I've seen similar crashes in put_user in other places. It looks like
> >>>>>>>> put_user crashes in the user address is not mapped/protected (?).
> >>>>>>>
> >>>>>>> I've been having a look, and this seems to be down to access of the
> >>>>>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
> >>>>>>> bad address to clone?
> >>>>>>>
> >>>>>>> From looking at the code, the put_user() code should have set the
> >>>>>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
> >>>>>>> s2 register in the crash report) and from looking at the compiler
> >>>>>>> output from my gcc-10, the code looks to be dong the relevant csrs
> >>>>>>> and then csrc around the put_user
> >>>>>>>
> >>>>>>> So currently I do not understand how the above could have happened
> >>>>>>> over than something re-tried the code seqeunce and ended up retrying
> >>>>>>> the faulting instruction without the SR_SUM bit set.
> >>>>>>
> >>>>>> I would maybe blame qemu for randomly resetting SR_SUM, but it's
> >>>>>> strange that 99% of these crashes are in schedule_tail. If it would be
> >>>>>> qemu, then they would be more evenly distributed...
> >>>>>>
> >>>>>> Another observation: looking at a dozen of crash logs, in none of
> >>>>>> these cases fuzzer was actually trying to fuzz clone with some insane
> >>>>>> arguments. So it looks like completely normal clone's (e..g coming
> >>>>>> from pthread_create) result in this crash.
> >>>>>>
> >>>>>> I also wonder why there is ret_from_exception, is it normal? I see
> >>>>>> handle_exception disables SR_SUM:
> >>>>>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
> >>>>>>
> >>>>>
> >>>>> So I think if SR_SUM is set, then it faults the access to user memory
> >>>>> which the _user() routines clear to allow them access.
> >>>>>
> >>>>> I'm thinking there is at least one issue here:
> >>>>>
> >>>>> - the test in fault is the wrong way around for die kernel
> >>>>> - the handler only catches this if the page has yet to be mapped.
> >>>>>
> >>>>> So I think the test should be:
> >>>>>
> >>>>> if (!user_mode(regs) && addr < TASK_SIZE &&
> >>>>> unlikely(regs->status & SR_SUM)
> >>>>>
> >>>>> This then should continue on and allow the rest of the handler to
> >>>>> complete mapping the page if it is not there.
> >>>>>
> >>>>> I have been trying to create a very simple clone test, but so far it
> >>>>> has yet to actually trigger anything.
> >>>>
> >>>> I should have added there doesn't seem to be a good way to use mmap()
> >>>> to allocate memory but not insert a vm-mapping post the mmap().
> >>>>
> >>> How difficult is it to try building a branch with the above test
> >>> modified?
> >>
> >> I don't have access to hardware, I don't have other qemu versions ready to use.
> >> But I can teach you how to run syzkaller locally :)
> >> I am not sure anybody run it on real riscv hardware at all. When
> >> Tobias ported syzkaller, Tobias also used qemu I think.
> >>
> >> I am now building with an inverted check to test locally.
> >>
> >> I don't fully understand but this code, but does handle_exception
> >> reset SR_SUM around do_page_fault? If so, then looking at SR_SUM in
> >> do_page_fault won't work with positive nor negative check.
> >
> >
> > The inverted check crashes during boot:
> >
> > --- a/arch/riscv/mm/fault.c
> > +++ b/arch/riscv/mm/fault.c
> > @@ -249,7 +249,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> > flags |= FAULT_FLAG_USER;
> >
> > if (!user_mode(regs) && addr < TASK_SIZE &&
> > - unlikely(!(regs->status & SR_SUM)))
> > + unlikely(regs->status & SR_SUM))
> > die_kernel_fault("access to user memory without
> > uaccess routines",
> > addr, regs);
> >
> >
> > [ 77.349329][ T1] Run /sbin/init as init process
> > [ 77.868371][ T1] Unable to handle kernel access to user memory
> > without uaccess routines at virtual address 00000000000e8e39
> > [ 77.870355][ T1] Oops [#1]
> > [ 77.870766][ T1] Modules linked in:
> > [ 77.871326][ T1] CPU: 0 PID: 1 Comm: init Not tainted
> > 5.12.0-rc2-00010-g0d7588ab9ef9-dirty #42
> > [ 77.872057][ T1] Hardware name: riscv-virtio,qemu (DT)
> > [ 77.872620][ T1] epc : __clear_user+0x36/0x4e
> > [ 77.873285][ T1] ra : padzero+0x9c/0xb0
> > [ 77.873849][ T1] epc : ffffffe000bb7136 ra : ffffffe0004f42a0 sp
> > : ffffffe006f8fbc0
> > [ 77.874438][ T1] gp : ffffffe005d25718 tp : ffffffe006f98000 t0
> > : 00000000000e8e40
> > [ 77.875031][ T1] t1 : 00000000000e9000 t2 : 000000000001c49c s0
> > : ffffffe006f8fbf0
> > [ 77.875618][ T1] s1 : 00000000000001c7 a0 : 00000000000e8e39 a1
> > : 00000000000001c7
> > [ 77.876204][ T1] a2 : 0000000000000002 a3 : 00000000000e9000 a4
> > : ffffffe006f99000
> > [ 77.876787][ T1] a5 : 0000000000000000 a6 : 0000000000f00000 a7
> > : ffffffe00031c088
> > [ 77.877367][ T1] s2 : 00000000000e8e39 s3 : 0000000000001000 s4
> > : 0000003ffffffe39
> > [ 77.877952][ T1] s5 : 00000000000e8e39 s6 : 00000000000e9570 s7
> > : 00000000000e8e39
> > [ 77.878535][ T1] s8 : 0000000000000001 s9 : 00000000000e8e39
> > s10: ffffffe00c65f608
> > [ 77.879126][ T1] s11: ffffffe00816e8d8 t3 : ea3af0fa372b8300 t4
> > : 0000000000000003
> > [ 77.879711][ T1] t5 : ffffffc401dc45d8 t6 : 0000000000040000
> > [ 77.880209][ T1] status: 0000000000040120 badaddr:
> > 00000000000e8e39 cause: 000000000000000f
> > [ 77.880846][ T1] Call Trace:
> > [ 77.881213][ T1] [<ffffffe000bb7136>] __clear_user+0x36/0x4e
> > [ 77.881912][ T1] [<ffffffe0004f523e>] load_elf_binary+0xf8a/0x2400
> > [ 77.882562][ T1] [<ffffffe0003e1802>] bprm_execve+0x5b0/0x1080
> > [ 77.883145][ T1] [<ffffffe0003e38bc>] kernel_execve+0x204/0x288
> > [ 77.883727][ T1] [<ffffffe003b70e94>] run_init_process+0x1fe/0x212
> > [ 77.884337][ T1] [<ffffffe003b70ec6>] try_to_run_init_process+0x1e/0x66
> > [ 77.884956][ T1] [<ffffffe003bc0864>] kernel_init+0x14a/0x200
> > [ 77.885541][ T1] [<ffffffe000005570>] ret_from_exception+0x0/0x14
> > [ 77.886955][ T1] ---[ end trace 1e934d07b8a4bed8 ]---
> > [ 77.887705][ T1] Kernel panic - not syncing: Fatal exception
> > [ 77.888333][ T1] SMP: stopping secondary CPUs
> > [ 77.889357][ T1] Rebooting in 86400 seconds..
>
> I have reproduced this on qemu, not managed to get the real hardwre
> working with this branch yet.
>
> I have a working hypothesis now, having added debug to check the
> sstatus.SR_SUM flag and reviewed the assembly, I think this is
> what is happening:
>
> C code of "put_user(func(), address)" is generating code to do:
>
> 1: __enable_user_access();
> 2: cpu_reg = func();
> 3: assembly for *address = cpu_reg;
> 4: __disable_user_access();
>
> I think the call to func() with all the sanitisers enabled allow
> the func() to possibly schedule out. The __swtich_to() code does
> not restore the original status registers which means that if
> there is IO during the sleep SR_SUM may end up being cleared and
> never re-set. We get back to 3 and fault as 2 cleared the result of 1.
>
> It is very possible no-one has seen this before as generally the
> functions involved in feeding put_user() are fairly small and thus
> this system is both under load and has some reason to schedule then
> this bug has probably been rare to unseen.
>
> I think the correct solution is to store the SR_SUM bit status in
> the thread_struct and make __switch_to() save/restore this when
> changing between tasks/threads. Trying to re-order the code to
> force swapping of 1 and 2 may reduce the bug's window.
>
> Further thinking of the order of 1 and 2 is that we should probably
> fix that order so that func() is not run with the user-space access
> protection disabled.
>
> I'll try and make some sort of of small test case to avoid having
> to run syz-stress to provoke this.
Ouch!
Can't preempt kernel schedule at almost any instruction where
preemption is not disabled explicitly? But if it's disabled, then the
instrumented code won't schedule as well, right? I suspect this may be
quite a bad issue for preempt kernels.
Shouldn't __put_user materialize the expression in a local var using
__typeof__ magic before __enable_user_access? I suspect it may
potentially lead to quite bad security implications.
It can also make sense to add checks to schedule to check that it's
not called from unexpected contexts.
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-18 10:05 ` Dmitry Vyukov
0 siblings, 0 replies; 51+ messages in thread
From: Dmitry Vyukov @ 2021-03-18 10:05 UTC (permalink / raw)
To: Ben Dooks
Cc: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv,
Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot, Terry Hu,
Javier Jardón
On Thu, Mar 18, 2021 at 10:41 AM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>
> On 12/03/2021 17:38, Dmitry Vyukov wrote:
> > On Fri, Mar 12, 2021 at 6:34 PM Dmitry Vyukov <dvyukov@google.com> wrote:
> >>
> >> On Fri, Mar 12, 2021 at 5:36 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
> >>>
> >>> On 12/03/2021 16:34, Ben Dooks wrote:
> >>>> On 12/03/2021 16:30, Ben Dooks wrote:
> >>>>> On 12/03/2021 15:12, Dmitry Vyukov wrote:
> >>>>>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
> >>>>>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
> >>>>>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>> Hello,
> >>>>>>>>>
> >>>>>>>>> syzbot found the following issue on:
> >>>>>>>>>
> >>>>>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
> >>>>>>>>> arch_dup_tas..
> >>>>>>>>> git tree:
> >>>>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
> >>>>>>>>> console output:
> >>>>>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
> >>>>>>>>> kernel config:
> >>>>>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
> >>>>>>>>> dashboard link:
> >>>>>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
> >>>>>>>>> userspace arch: riscv64
> >>>>>>>>>
> >>>>>>>>> Unfortunately, I don't have any reproducer for this issue yet.
> >>>>>>>>>
> >>>>>>>>> IMPORTANT: if you fix the issue, please add the following tag to
> >>>>>>>>> the commit:
> >>>>>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
> >>>>>>>>
> >>>>>>>> +riscv maintainers
> >>>>>>>>
> >>>>>>>> This is riscv64-specific.
> >>>>>>>> I've seen similar crashes in put_user in other places. It looks like
> >>>>>>>> put_user crashes in the user address is not mapped/protected (?).
> >>>>>>>
> >>>>>>> I've been having a look, and this seems to be down to access of the
> >>>>>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
> >>>>>>> bad address to clone?
> >>>>>>>
> >>>>>>> From looking at the code, the put_user() code should have set the
> >>>>>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
> >>>>>>> s2 register in the crash report) and from looking at the compiler
> >>>>>>> output from my gcc-10, the code looks to be dong the relevant csrs
> >>>>>>> and then csrc around the put_user
> >>>>>>>
> >>>>>>> So currently I do not understand how the above could have happened
> >>>>>>> over than something re-tried the code seqeunce and ended up retrying
> >>>>>>> the faulting instruction without the SR_SUM bit set.
> >>>>>>
> >>>>>> I would maybe blame qemu for randomly resetting SR_SUM, but it's
> >>>>>> strange that 99% of these crashes are in schedule_tail. If it would be
> >>>>>> qemu, then they would be more evenly distributed...
> >>>>>>
> >>>>>> Another observation: looking at a dozen of crash logs, in none of
> >>>>>> these cases fuzzer was actually trying to fuzz clone with some insane
> >>>>>> arguments. So it looks like completely normal clone's (e..g coming
> >>>>>> from pthread_create) result in this crash.
> >>>>>>
> >>>>>> I also wonder why there is ret_from_exception, is it normal? I see
> >>>>>> handle_exception disables SR_SUM:
> >>>>>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
> >>>>>>
> >>>>>
> >>>>> So I think if SR_SUM is set, then it faults the access to user memory
> >>>>> which the _user() routines clear to allow them access.
> >>>>>
> >>>>> I'm thinking there is at least one issue here:
> >>>>>
> >>>>> - the test in fault is the wrong way around for die kernel
> >>>>> - the handler only catches this if the page has yet to be mapped.
> >>>>>
> >>>>> So I think the test should be:
> >>>>>
> >>>>> if (!user_mode(regs) && addr < TASK_SIZE &&
> >>>>> unlikely(regs->status & SR_SUM)
> >>>>>
> >>>>> This then should continue on and allow the rest of the handler to
> >>>>> complete mapping the page if it is not there.
> >>>>>
> >>>>> I have been trying to create a very simple clone test, but so far it
> >>>>> has yet to actually trigger anything.
> >>>>
> >>>> I should have added there doesn't seem to be a good way to use mmap()
> >>>> to allocate memory but not insert a vm-mapping post the mmap().
> >>>>
> >>> How difficult is it to try building a branch with the above test
> >>> modified?
> >>
> >> I don't have access to hardware, I don't have other qemu versions ready to use.
> >> But I can teach you how to run syzkaller locally :)
> >> I am not sure anybody run it on real riscv hardware at all. When
> >> Tobias ported syzkaller, Tobias also used qemu I think.
> >>
> >> I am now building with an inverted check to test locally.
> >>
> >> I don't fully understand but this code, but does handle_exception
> >> reset SR_SUM around do_page_fault? If so, then looking at SR_SUM in
> >> do_page_fault won't work with positive nor negative check.
> >
> >
> > The inverted check crashes during boot:
> >
> > --- a/arch/riscv/mm/fault.c
> > +++ b/arch/riscv/mm/fault.c
> > @@ -249,7 +249,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> > flags |= FAULT_FLAG_USER;
> >
> > if (!user_mode(regs) && addr < TASK_SIZE &&
> > - unlikely(!(regs->status & SR_SUM)))
> > + unlikely(regs->status & SR_SUM))
> > die_kernel_fault("access to user memory without
> > uaccess routines",
> > addr, regs);
> >
> >
> > [ 77.349329][ T1] Run /sbin/init as init process
> > [ 77.868371][ T1] Unable to handle kernel access to user memory
> > without uaccess routines at virtual address 00000000000e8e39
> > [ 77.870355][ T1] Oops [#1]
> > [ 77.870766][ T1] Modules linked in:
> > [ 77.871326][ T1] CPU: 0 PID: 1 Comm: init Not tainted
> > 5.12.0-rc2-00010-g0d7588ab9ef9-dirty #42
> > [ 77.872057][ T1] Hardware name: riscv-virtio,qemu (DT)
> > [ 77.872620][ T1] epc : __clear_user+0x36/0x4e
> > [ 77.873285][ T1] ra : padzero+0x9c/0xb0
> > [ 77.873849][ T1] epc : ffffffe000bb7136 ra : ffffffe0004f42a0 sp
> > : ffffffe006f8fbc0
> > [ 77.874438][ T1] gp : ffffffe005d25718 tp : ffffffe006f98000 t0
> > : 00000000000e8e40
> > [ 77.875031][ T1] t1 : 00000000000e9000 t2 : 000000000001c49c s0
> > : ffffffe006f8fbf0
> > [ 77.875618][ T1] s1 : 00000000000001c7 a0 : 00000000000e8e39 a1
> > : 00000000000001c7
> > [ 77.876204][ T1] a2 : 0000000000000002 a3 : 00000000000e9000 a4
> > : ffffffe006f99000
> > [ 77.876787][ T1] a5 : 0000000000000000 a6 : 0000000000f00000 a7
> > : ffffffe00031c088
> > [ 77.877367][ T1] s2 : 00000000000e8e39 s3 : 0000000000001000 s4
> > : 0000003ffffffe39
> > [ 77.877952][ T1] s5 : 00000000000e8e39 s6 : 00000000000e9570 s7
> > : 00000000000e8e39
> > [ 77.878535][ T1] s8 : 0000000000000001 s9 : 00000000000e8e39
> > s10: ffffffe00c65f608
> > [ 77.879126][ T1] s11: ffffffe00816e8d8 t3 : ea3af0fa372b8300 t4
> > : 0000000000000003
> > [ 77.879711][ T1] t5 : ffffffc401dc45d8 t6 : 0000000000040000
> > [ 77.880209][ T1] status: 0000000000040120 badaddr:
> > 00000000000e8e39 cause: 000000000000000f
> > [ 77.880846][ T1] Call Trace:
> > [ 77.881213][ T1] [<ffffffe000bb7136>] __clear_user+0x36/0x4e
> > [ 77.881912][ T1] [<ffffffe0004f523e>] load_elf_binary+0xf8a/0x2400
> > [ 77.882562][ T1] [<ffffffe0003e1802>] bprm_execve+0x5b0/0x1080
> > [ 77.883145][ T1] [<ffffffe0003e38bc>] kernel_execve+0x204/0x288
> > [ 77.883727][ T1] [<ffffffe003b70e94>] run_init_process+0x1fe/0x212
> > [ 77.884337][ T1] [<ffffffe003b70ec6>] try_to_run_init_process+0x1e/0x66
> > [ 77.884956][ T1] [<ffffffe003bc0864>] kernel_init+0x14a/0x200
> > [ 77.885541][ T1] [<ffffffe000005570>] ret_from_exception+0x0/0x14
> > [ 77.886955][ T1] ---[ end trace 1e934d07b8a4bed8 ]---
> > [ 77.887705][ T1] Kernel panic - not syncing: Fatal exception
> > [ 77.888333][ T1] SMP: stopping secondary CPUs
> > [ 77.889357][ T1] Rebooting in 86400 seconds..
>
> I have reproduced this on qemu, not managed to get the real hardwre
> working with this branch yet.
>
> I have a working hypothesis now, having added debug to check the
> sstatus.SR_SUM flag and reviewed the assembly, I think this is
> what is happening:
>
> C code of "put_user(func(), address)" is generating code to do:
>
> 1: __enable_user_access();
> 2: cpu_reg = func();
> 3: assembly for *address = cpu_reg;
> 4: __disable_user_access();
>
> I think the call to func() with all the sanitisers enabled allow
> the func() to possibly schedule out. The __swtich_to() code does
> not restore the original status registers which means that if
> there is IO during the sleep SR_SUM may end up being cleared and
> never re-set. We get back to 3 and fault as 2 cleared the result of 1.
>
> It is very possible no-one has seen this before as generally the
> functions involved in feeding put_user() are fairly small and thus
> this system is both under load and has some reason to schedule then
> this bug has probably been rare to unseen.
>
> I think the correct solution is to store the SR_SUM bit status in
> the thread_struct and make __switch_to() save/restore this when
> changing between tasks/threads. Trying to re-order the code to
> force swapping of 1 and 2 may reduce the bug's window.
>
> Further thinking of the order of 1 and 2 is that we should probably
> fix that order so that func() is not run with the user-space access
> protection disabled.
>
> I'll try and make some sort of of small test case to avoid having
> to run syz-stress to provoke this.
Ouch!
Can't preempt kernel schedule at almost any instruction where
preemption is not disabled explicitly? But if it's disabled, then the
instrumented code won't schedule as well, right? I suspect this may be
quite a bad issue for preempt kernels.
Shouldn't __put_user materialize the expression in a local var using
__typeof__ magic before __enable_user_access? I suspect it may
potentially lead to quite bad security implications.
It can also make sense to add checks to schedule to check that it's
not called from unexpected contexts.
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-18 10:05 ` Dmitry Vyukov
@ 2021-03-18 12:52 ` Ben Dooks
-1 siblings, 0 replies; 51+ messages in thread
From: Ben Dooks @ 2021-03-18 12:52 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv,
Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot, Terry Hu,
Javier Jardón
On 18/03/2021 10:05, Dmitry Vyukov wrote:
> On Thu, Mar 18, 2021 at 10:41 AM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>>
>> On 12/03/2021 17:38, Dmitry Vyukov wrote:
>>> On Fri, Mar 12, 2021 at 6:34 PM Dmitry Vyukov <dvyukov@google.com> wrote:
>>>>
>>>> On Fri, Mar 12, 2021 at 5:36 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>>>>>
>>>>> On 12/03/2021 16:34, Ben Dooks wrote:
>>>>>> On 12/03/2021 16:30, Ben Dooks wrote:
>>>>>>> On 12/03/2021 15:12, Dmitry Vyukov wrote:
>>>>>>>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
>>>>>>>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
>>>>>>>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> syzbot found the following issue on:
>>>>>>>>>>>
>>>>>>>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
>>>>>>>>>>> arch_dup_tas..
>>>>>>>>>>> git tree:
>>>>>>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>>>>>>>>>>> console output:
>>>>>>>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>>>>>>>>>>> kernel config:
>>>>>>>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>>>>>>>>>>> dashboard link:
>>>>>>>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>>>>>>>>>>> userspace arch: riscv64
>>>>>>>>>>>
>>>>>>>>>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>>>>>>>>>
>>>>>>>>>>> IMPORTANT: if you fix the issue, please add the following tag to
>>>>>>>>>>> the commit:
>>>>>>>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>>>>>>>>>>
>>>>>>>>>> +riscv maintainers
>>>>>>>>>>
>>>>>>>>>> This is riscv64-specific.
>>>>>>>>>> I've seen similar crashes in put_user in other places. It looks like
>>>>>>>>>> put_user crashes in the user address is not mapped/protected (?).
>>>>>>>>>
>>>>>>>>> I've been having a look, and this seems to be down to access of the
>>>>>>>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
>>>>>>>>> bad address to clone?
>>>>>>>>>
>>>>>>>>> From looking at the code, the put_user() code should have set the
>>>>>>>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
>>>>>>>>> s2 register in the crash report) and from looking at the compiler
>>>>>>>>> output from my gcc-10, the code looks to be dong the relevant csrs
>>>>>>>>> and then csrc around the put_user
>>>>>>>>>
>>>>>>>>> So currently I do not understand how the above could have happened
>>>>>>>>> over than something re-tried the code seqeunce and ended up retrying
>>>>>>>>> the faulting instruction without the SR_SUM bit set.
>>>>>>>>
>>>>>>>> I would maybe blame qemu for randomly resetting SR_SUM, but it's
>>>>>>>> strange that 99% of these crashes are in schedule_tail. If it would be
>>>>>>>> qemu, then they would be more evenly distributed...
>>>>>>>>
>>>>>>>> Another observation: looking at a dozen of crash logs, in none of
>>>>>>>> these cases fuzzer was actually trying to fuzz clone with some insane
>>>>>>>> arguments. So it looks like completely normal clone's (e..g coming
>>>>>>>> from pthread_create) result in this crash.
>>>>>>>>
>>>>>>>> I also wonder why there is ret_from_exception, is it normal? I see
>>>>>>>> handle_exception disables SR_SUM:
>>>>>>>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
>>>>>>>>
>>>>>>>
>>>>>>> So I think if SR_SUM is set, then it faults the access to user memory
>>>>>>> which the _user() routines clear to allow them access.
>>>>>>>
>>>>>>> I'm thinking there is at least one issue here:
>>>>>>>
>>>>>>> - the test in fault is the wrong way around for die kernel
>>>>>>> - the handler only catches this if the page has yet to be mapped.
>>>>>>>
>>>>>>> So I think the test should be:
>>>>>>>
>>>>>>> if (!user_mode(regs) && addr < TASK_SIZE &&
>>>>>>> unlikely(regs->status & SR_SUM)
>>>>>>>
>>>>>>> This then should continue on and allow the rest of the handler to
>>>>>>> complete mapping the page if it is not there.
>>>>>>>
>>>>>>> I have been trying to create a very simple clone test, but so far it
>>>>>>> has yet to actually trigger anything.
>>>>>>
>>>>>> I should have added there doesn't seem to be a good way to use mmap()
>>>>>> to allocate memory but not insert a vm-mapping post the mmap().
>>>>>>
>>>>> How difficult is it to try building a branch with the above test
>>>>> modified?
>>>>
>>>> I don't have access to hardware, I don't have other qemu versions ready to use.
>>>> But I can teach you how to run syzkaller locally :)
>>>> I am not sure anybody run it on real riscv hardware at all. When
>>>> Tobias ported syzkaller, Tobias also used qemu I think.
>>>>
>>>> I am now building with an inverted check to test locally.
>>>>
>>>> I don't fully understand but this code, but does handle_exception
>>>> reset SR_SUM around do_page_fault? If so, then looking at SR_SUM in
>>>> do_page_fault won't work with positive nor negative check.
>>>
>>>
>>> The inverted check crashes during boot:
>>>
>>> --- a/arch/riscv/mm/fault.c
>>> +++ b/arch/riscv/mm/fault.c
>>> @@ -249,7 +249,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
>>> flags |= FAULT_FLAG_USER;
>>>
>>> if (!user_mode(regs) && addr < TASK_SIZE &&
>>> - unlikely(!(regs->status & SR_SUM)))
>>> + unlikely(regs->status & SR_SUM))
>>> die_kernel_fault("access to user memory without
>>> uaccess routines",
>>> addr, regs);
>>>
>>>
>>> [ 77.349329][ T1] Run /sbin/init as init process
>>> [ 77.868371][ T1] Unable to handle kernel access to user memory
>>> without uaccess routines at virtual address 00000000000e8e39
>>> [ 77.870355][ T1] Oops [#1]
>>> [ 77.870766][ T1] Modules linked in:
>>> [ 77.871326][ T1] CPU: 0 PID: 1 Comm: init Not tainted
>>> 5.12.0-rc2-00010-g0d7588ab9ef9-dirty #42
>>> [ 77.872057][ T1] Hardware name: riscv-virtio,qemu (DT)
>>> [ 77.872620][ T1] epc : __clear_user+0x36/0x4e
>>> [ 77.873285][ T1] ra : padzero+0x9c/0xb0
>>> [ 77.873849][ T1] epc : ffffffe000bb7136 ra : ffffffe0004f42a0 sp
>>> : ffffffe006f8fbc0
>>> [ 77.874438][ T1] gp : ffffffe005d25718 tp : ffffffe006f98000 t0
>>> : 00000000000e8e40
>>> [ 77.875031][ T1] t1 : 00000000000e9000 t2 : 000000000001c49c s0
>>> : ffffffe006f8fbf0
>>> [ 77.875618][ T1] s1 : 00000000000001c7 a0 : 00000000000e8e39 a1
>>> : 00000000000001c7
>>> [ 77.876204][ T1] a2 : 0000000000000002 a3 : 00000000000e9000 a4
>>> : ffffffe006f99000
>>> [ 77.876787][ T1] a5 : 0000000000000000 a6 : 0000000000f00000 a7
>>> : ffffffe00031c088
>>> [ 77.877367][ T1] s2 : 00000000000e8e39 s3 : 0000000000001000 s4
>>> : 0000003ffffffe39
>>> [ 77.877952][ T1] s5 : 00000000000e8e39 s6 : 00000000000e9570 s7
>>> : 00000000000e8e39
>>> [ 77.878535][ T1] s8 : 0000000000000001 s9 : 00000000000e8e39
>>> s10: ffffffe00c65f608
>>> [ 77.879126][ T1] s11: ffffffe00816e8d8 t3 : ea3af0fa372b8300 t4
>>> : 0000000000000003
>>> [ 77.879711][ T1] t5 : ffffffc401dc45d8 t6 : 0000000000040000
>>> [ 77.880209][ T1] status: 0000000000040120 badaddr:
>>> 00000000000e8e39 cause: 000000000000000f
>>> [ 77.880846][ T1] Call Trace:
>>> [ 77.881213][ T1] [<ffffffe000bb7136>] __clear_user+0x36/0x4e
>>> [ 77.881912][ T1] [<ffffffe0004f523e>] load_elf_binary+0xf8a/0x2400
>>> [ 77.882562][ T1] [<ffffffe0003e1802>] bprm_execve+0x5b0/0x1080
>>> [ 77.883145][ T1] [<ffffffe0003e38bc>] kernel_execve+0x204/0x288
>>> [ 77.883727][ T1] [<ffffffe003b70e94>] run_init_process+0x1fe/0x212
>>> [ 77.884337][ T1] [<ffffffe003b70ec6>] try_to_run_init_process+0x1e/0x66
>>> [ 77.884956][ T1] [<ffffffe003bc0864>] kernel_init+0x14a/0x200
>>> [ 77.885541][ T1] [<ffffffe000005570>] ret_from_exception+0x0/0x14
>>> [ 77.886955][ T1] ---[ end trace 1e934d07b8a4bed8 ]---
>>> [ 77.887705][ T1] Kernel panic - not syncing: Fatal exception
>>> [ 77.888333][ T1] SMP: stopping secondary CPUs
>>> [ 77.889357][ T1] Rebooting in 86400 seconds..
>>
>> I have reproduced this on qemu, not managed to get the real hardwre
>> working with this branch yet.
>>
>> I have a working hypothesis now, having added debug to check the
>> sstatus.SR_SUM flag and reviewed the assembly, I think this is
>> what is happening:
>>
>> C code of "put_user(func(), address)" is generating code to do:
>>
>> 1: __enable_user_access();
>> 2: cpu_reg = func();
>> 3: assembly for *address = cpu_reg;
>> 4: __disable_user_access();
>>
>> I think the call to func() with all the sanitisers enabled allow
>> the func() to possibly schedule out. The __swtich_to() code does
>> not restore the original status registers which means that if
>> there is IO during the sleep SR_SUM may end up being cleared and
>> never re-set. We get back to 3 and fault as 2 cleared the result of 1.
>>
>> It is very possible no-one has seen this before as generally the
>> functions involved in feeding put_user() are fairly small and thus
>> this system is both under load and has some reason to schedule then
>> this bug has probably been rare to unseen.
>>
>> I think the correct solution is to store the SR_SUM bit status in
>> the thread_struct and make __switch_to() save/restore this when
>> changing between tasks/threads. Trying to re-order the code to
>> force swapping of 1 and 2 may reduce the bug's window.
>>
>> Further thinking of the order of 1 and 2 is that we should probably
>> fix that order so that func() is not run with the user-space access
>> protection disabled.
>>
>> I'll try and make some sort of of small test case to avoid having
>> to run syz-stress to provoke this.
>
> Ouch!
> Can't preempt kernel schedule at almost any instruction where
> preemption is not disabled explicitly? But if it's disabled, then the
> instrumented code won't schedule as well, right? I suspect this may be
> quite a bad issue for preempt kernels.
>
> Shouldn't __put_user materialize the expression in a local var using
> __typeof__ magic before __enable_user_access? I suspect it may
> potentially lead to quite bad security implications.
>
> It can also make sense to add checks to schedule to check that it's
> not called from unexpected contexts.
I wrote a kernel thread that does:
#define rd_sstatus() ({ unsigned long result; asm(" csrr %0, sstatus" :
"=r"(result) :: "memory"); result; })
static int test_thread1(void *data)
{
unsigned int cpu = (unsigned int)data;
unsigned long status;
pr_info("%s: thread starting on cpu %d\n", __func__, cpu);
while (!kthread_should_stop()) {
status = rd_sstatus();
if (status & SR_SUM)
printk_ratelimited("%s: found sstaus=0x%08lx\n",
__func__, status);
msleep(1);
}
pr_info("%s: thread exiting\n", __func__);
return 0;
}
And under the syz-stress I have the following
[ 1192.124018] test_thread1: found sstaus=0x00040022
this thread does not do any IO operations yet during a stress run
it got entered with SR_SUM set (the 0x00040000) in the sstatus
field.
I think this is proof that #1 this is /rare/ and #2 we need to
make __switch_user save at-least the SR_SUM field.
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-18 12:52 ` Ben Dooks
0 siblings, 0 replies; 51+ messages in thread
From: Ben Dooks @ 2021-03-18 12:52 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou, linux-riscv,
Daniel Bristot de Oliveira, Benjamin Segall, dietmar.eggemann,
Juri Lelli, LKML, Mel Gorman, Ingo Molnar, Peter Zijlstra,
Steven Rostedt, syzkaller-bugs, Vincent Guittot, Terry Hu,
Javier Jardón
On 18/03/2021 10:05, Dmitry Vyukov wrote:
> On Thu, Mar 18, 2021 at 10:41 AM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>>
>> On 12/03/2021 17:38, Dmitry Vyukov wrote:
>>> On Fri, Mar 12, 2021 at 6:34 PM Dmitry Vyukov <dvyukov@google.com> wrote:
>>>>
>>>> On Fri, Mar 12, 2021 at 5:36 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>>>>>
>>>>> On 12/03/2021 16:34, Ben Dooks wrote:
>>>>>> On 12/03/2021 16:30, Ben Dooks wrote:
>>>>>>> On 12/03/2021 15:12, Dmitry Vyukov wrote:
>>>>>>>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks <ben.dooks@codethink.co.uk>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> On 10/03/2021 17:16, Dmitry Vyukov wrote:
>>>>>>>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot
>>>>>>>>>> <syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> syzbot found the following issue on:
>>>>>>>>>>>
>>>>>>>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for
>>>>>>>>>>> arch_dup_tas..
>>>>>>>>>>> git tree:
>>>>>>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes
>>>>>>>>>>> console output:
>>>>>>>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000
>>>>>>>>>>> kernel config:
>>>>>>>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136
>>>>>>>>>>> dashboard link:
>>>>>>>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69
>>>>>>>>>>> userspace arch: riscv64
>>>>>>>>>>>
>>>>>>>>>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>>>>>>>>>
>>>>>>>>>>> IMPORTANT: if you fix the issue, please add the following tag to
>>>>>>>>>>> the commit:
>>>>>>>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com
>>>>>>>>>>
>>>>>>>>>> +riscv maintainers
>>>>>>>>>>
>>>>>>>>>> This is riscv64-specific.
>>>>>>>>>> I've seen similar crashes in put_user in other places. It looks like
>>>>>>>>>> put_user crashes in the user address is not mapped/protected (?).
>>>>>>>>>
>>>>>>>>> I've been having a look, and this seems to be down to access of the
>>>>>>>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a
>>>>>>>>> bad address to clone?
>>>>>>>>>
>>>>>>>>> From looking at the code, the put_user() code should have set the
>>>>>>>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the
>>>>>>>>> s2 register in the crash report) and from looking at the compiler
>>>>>>>>> output from my gcc-10, the code looks to be dong the relevant csrs
>>>>>>>>> and then csrc around the put_user
>>>>>>>>>
>>>>>>>>> So currently I do not understand how the above could have happened
>>>>>>>>> over than something re-tried the code seqeunce and ended up retrying
>>>>>>>>> the faulting instruction without the SR_SUM bit set.
>>>>>>>>
>>>>>>>> I would maybe blame qemu for randomly resetting SR_SUM, but it's
>>>>>>>> strange that 99% of these crashes are in schedule_tail. If it would be
>>>>>>>> qemu, then they would be more evenly distributed...
>>>>>>>>
>>>>>>>> Another observation: looking at a dozen of crash logs, in none of
>>>>>>>> these cases fuzzer was actually trying to fuzz clone with some insane
>>>>>>>> arguments. So it looks like completely normal clone's (e..g coming
>>>>>>>> from pthread_create) result in this crash.
>>>>>>>>
>>>>>>>> I also wonder why there is ret_from_exception, is it normal? I see
>>>>>>>> handle_exception disables SR_SUM:
>>>>>>>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73
>>>>>>>>
>>>>>>>
>>>>>>> So I think if SR_SUM is set, then it faults the access to user memory
>>>>>>> which the _user() routines clear to allow them access.
>>>>>>>
>>>>>>> I'm thinking there is at least one issue here:
>>>>>>>
>>>>>>> - the test in fault is the wrong way around for die kernel
>>>>>>> - the handler only catches this if the page has yet to be mapped.
>>>>>>>
>>>>>>> So I think the test should be:
>>>>>>>
>>>>>>> if (!user_mode(regs) && addr < TASK_SIZE &&
>>>>>>> unlikely(regs->status & SR_SUM)
>>>>>>>
>>>>>>> This then should continue on and allow the rest of the handler to
>>>>>>> complete mapping the page if it is not there.
>>>>>>>
>>>>>>> I have been trying to create a very simple clone test, but so far it
>>>>>>> has yet to actually trigger anything.
>>>>>>
>>>>>> I should have added there doesn't seem to be a good way to use mmap()
>>>>>> to allocate memory but not insert a vm-mapping post the mmap().
>>>>>>
>>>>> How difficult is it to try building a branch with the above test
>>>>> modified?
>>>>
>>>> I don't have access to hardware, I don't have other qemu versions ready to use.
>>>> But I can teach you how to run syzkaller locally :)
>>>> I am not sure anybody run it on real riscv hardware at all. When
>>>> Tobias ported syzkaller, Tobias also used qemu I think.
>>>>
>>>> I am now building with an inverted check to test locally.
>>>>
>>>> I don't fully understand but this code, but does handle_exception
>>>> reset SR_SUM around do_page_fault? If so, then looking at SR_SUM in
>>>> do_page_fault won't work with positive nor negative check.
>>>
>>>
>>> The inverted check crashes during boot:
>>>
>>> --- a/arch/riscv/mm/fault.c
>>> +++ b/arch/riscv/mm/fault.c
>>> @@ -249,7 +249,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
>>> flags |= FAULT_FLAG_USER;
>>>
>>> if (!user_mode(regs) && addr < TASK_SIZE &&
>>> - unlikely(!(regs->status & SR_SUM)))
>>> + unlikely(regs->status & SR_SUM))
>>> die_kernel_fault("access to user memory without
>>> uaccess routines",
>>> addr, regs);
>>>
>>>
>>> [ 77.349329][ T1] Run /sbin/init as init process
>>> [ 77.868371][ T1] Unable to handle kernel access to user memory
>>> without uaccess routines at virtual address 00000000000e8e39
>>> [ 77.870355][ T1] Oops [#1]
>>> [ 77.870766][ T1] Modules linked in:
>>> [ 77.871326][ T1] CPU: 0 PID: 1 Comm: init Not tainted
>>> 5.12.0-rc2-00010-g0d7588ab9ef9-dirty #42
>>> [ 77.872057][ T1] Hardware name: riscv-virtio,qemu (DT)
>>> [ 77.872620][ T1] epc : __clear_user+0x36/0x4e
>>> [ 77.873285][ T1] ra : padzero+0x9c/0xb0
>>> [ 77.873849][ T1] epc : ffffffe000bb7136 ra : ffffffe0004f42a0 sp
>>> : ffffffe006f8fbc0
>>> [ 77.874438][ T1] gp : ffffffe005d25718 tp : ffffffe006f98000 t0
>>> : 00000000000e8e40
>>> [ 77.875031][ T1] t1 : 00000000000e9000 t2 : 000000000001c49c s0
>>> : ffffffe006f8fbf0
>>> [ 77.875618][ T1] s1 : 00000000000001c7 a0 : 00000000000e8e39 a1
>>> : 00000000000001c7
>>> [ 77.876204][ T1] a2 : 0000000000000002 a3 : 00000000000e9000 a4
>>> : ffffffe006f99000
>>> [ 77.876787][ T1] a5 : 0000000000000000 a6 : 0000000000f00000 a7
>>> : ffffffe00031c088
>>> [ 77.877367][ T1] s2 : 00000000000e8e39 s3 : 0000000000001000 s4
>>> : 0000003ffffffe39
>>> [ 77.877952][ T1] s5 : 00000000000e8e39 s6 : 00000000000e9570 s7
>>> : 00000000000e8e39
>>> [ 77.878535][ T1] s8 : 0000000000000001 s9 : 00000000000e8e39
>>> s10: ffffffe00c65f608
>>> [ 77.879126][ T1] s11: ffffffe00816e8d8 t3 : ea3af0fa372b8300 t4
>>> : 0000000000000003
>>> [ 77.879711][ T1] t5 : ffffffc401dc45d8 t6 : 0000000000040000
>>> [ 77.880209][ T1] status: 0000000000040120 badaddr:
>>> 00000000000e8e39 cause: 000000000000000f
>>> [ 77.880846][ T1] Call Trace:
>>> [ 77.881213][ T1] [<ffffffe000bb7136>] __clear_user+0x36/0x4e
>>> [ 77.881912][ T1] [<ffffffe0004f523e>] load_elf_binary+0xf8a/0x2400
>>> [ 77.882562][ T1] [<ffffffe0003e1802>] bprm_execve+0x5b0/0x1080
>>> [ 77.883145][ T1] [<ffffffe0003e38bc>] kernel_execve+0x204/0x288
>>> [ 77.883727][ T1] [<ffffffe003b70e94>] run_init_process+0x1fe/0x212
>>> [ 77.884337][ T1] [<ffffffe003b70ec6>] try_to_run_init_process+0x1e/0x66
>>> [ 77.884956][ T1] [<ffffffe003bc0864>] kernel_init+0x14a/0x200
>>> [ 77.885541][ T1] [<ffffffe000005570>] ret_from_exception+0x0/0x14
>>> [ 77.886955][ T1] ---[ end trace 1e934d07b8a4bed8 ]---
>>> [ 77.887705][ T1] Kernel panic - not syncing: Fatal exception
>>> [ 77.888333][ T1] SMP: stopping secondary CPUs
>>> [ 77.889357][ T1] Rebooting in 86400 seconds..
>>
>> I have reproduced this on qemu, not managed to get the real hardwre
>> working with this branch yet.
>>
>> I have a working hypothesis now, having added debug to check the
>> sstatus.SR_SUM flag and reviewed the assembly, I think this is
>> what is happening:
>>
>> C code of "put_user(func(), address)" is generating code to do:
>>
>> 1: __enable_user_access();
>> 2: cpu_reg = func();
>> 3: assembly for *address = cpu_reg;
>> 4: __disable_user_access();
>>
>> I think the call to func() with all the sanitisers enabled allow
>> the func() to possibly schedule out. The __swtich_to() code does
>> not restore the original status registers which means that if
>> there is IO during the sleep SR_SUM may end up being cleared and
>> never re-set. We get back to 3 and fault as 2 cleared the result of 1.
>>
>> It is very possible no-one has seen this before as generally the
>> functions involved in feeding put_user() are fairly small and thus
>> this system is both under load and has some reason to schedule then
>> this bug has probably been rare to unseen.
>>
>> I think the correct solution is to store the SR_SUM bit status in
>> the thread_struct and make __switch_to() save/restore this when
>> changing between tasks/threads. Trying to re-order the code to
>> force swapping of 1 and 2 may reduce the bug's window.
>>
>> Further thinking of the order of 1 and 2 is that we should probably
>> fix that order so that func() is not run with the user-space access
>> protection disabled.
>>
>> I'll try and make some sort of of small test case to avoid having
>> to run syz-stress to provoke this.
>
> Ouch!
> Can't preempt kernel schedule at almost any instruction where
> preemption is not disabled explicitly? But if it's disabled, then the
> instrumented code won't schedule as well, right? I suspect this may be
> quite a bad issue for preempt kernels.
>
> Shouldn't __put_user materialize the expression in a local var using
> __typeof__ magic before __enable_user_access? I suspect it may
> potentially lead to quite bad security implications.
>
> It can also make sense to add checks to schedule to check that it's
> not called from unexpected contexts.
I wrote a kernel thread that does:
#define rd_sstatus() ({ unsigned long result; asm(" csrr %0, sstatus" :
"=r"(result) :: "memory"); result; })
static int test_thread1(void *data)
{
unsigned int cpu = (unsigned int)data;
unsigned long status;
pr_info("%s: thread starting on cpu %d\n", __func__, cpu);
while (!kthread_should_stop()) {
status = rd_sstatus();
if (status & SR_SUM)
printk_ratelimited("%s: found sstaus=0x%08lx\n",
__func__, status);
msleep(1);
}
pr_info("%s: thread exiting\n", __func__);
return 0;
}
And under the syz-stress I have the following
[ 1192.124018] test_thread1: found sstaus=0x00040022
this thread does not do any IO operations yet during a stress run
it got entered with SR_SUM set (the 0x00040000) in the sstatus
field.
I think this is proof that #1 this is /rare/ and #2 we need to
make __switch_user save at-least the SR_SUM field.
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
2021-03-15 16:55 ` Ben Dooks
@ 2021-03-18 14:34 ` Dmitry Vyukov
-1 siblings, 0 replies; 51+ messages in thread
From: Dmitry Vyukov @ 2021-03-18 14:34 UTC (permalink / raw)
To: Ben Dooks
Cc: Alex Ghiti, syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou,
linux-riscv, Daniel Bristot de Oliveira, Benjamin Segall,
dietmar.eggemann, Juri Lelli, LKML, Mel Gorman, Ingo Molnar,
Peter Zijlstra, Steven Rostedt, syzkaller-bugs, Vincent Guittot
On Mon, Mar 15, 2021 at 5:55 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
> > On Fri, Mar 12, 2021 at 9:12 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>
> >>> Still no luck for the moment, can't reproduce it locally, my test is
> >>> maybe not that good (I created threads all day long in order to trigger
> >>> the put_user of schedule_tail).
> >>
> >> It may of course depend on memory and other stuff. I did try to see if
> >> it was possible to clone() with the child_tid address being a valid but
> >> not mapped page...
> >>
> >>> Given that the path you mention works most of the time, and that the
> >>> status register in the stack trace shows the SUM bit is not set whereas
> >>> it is set in put_user, I'm leaning toward some race condition (maybe an
> >>> interrupt that arrives at the "wrong" time) or a qemu issue as you
> >>> mentioned.
> >>
> >> I suppose this is possible. From what I read it should get to the
> >> point of being there with the SUM flag cleared, so either something
> >> went wrong in trying to fix the instruction up or there's some other
> >> error we're missing.
> >>
> >>> To eliminate qemu issues, do you have access to some HW ? Or to
> >>> different qemu versions ?
> >>
> >> I do have access to a Microchip Polarfire board. I just need the
> >> instructions on how to setup the test-code to make it work on the
> >> hardware.
> >
> > For full syzkaller support, it would need to know how to reboot these
> > boards and get access to the console.
> > syzkaller has a stop-gap VM backend which just uses ssh to a physical
> > machine and expects the kernel to reboot on its own after any crashes.
> >
> > But I actually managed to reproduce it in an even simpler setup.
> > Assuming you have Go 1.15 and riscv64 cross-compiler gcc installed
> >
> > $ go get -u -d github.com/google/syzkaller/...
> > $ cd $GOPATH/src/github.com/google/syzkaller
> > $ make stress executor TARGETARCH=riscv64
> > $ scp bin/linux_riscv64/syz-execprog bin/linux_riscv64/syz-executor
> > your_machine:/
> >
> > Then run ./syz-stress on the machine.
> > On the first run it crashed it with some other bug, on the second run
> > I got the crash in schedule_tail.
> > With qemu tcg I also added -slowdown=10 flag to syz-stress to scale
> > all timeouts, if native execution is faster, then you don't need it.
>
> I have built the tools and got it to start.
>
> It would be helpful for the dashboard to give the qemu version and
> how it was launched (memory, cpus etc)
Hi Ben,
syzbot will show info about qemu version/args in "VM info" column then
this commit is deployed (should happen by tomorrow);
https://github.com/google/syzkaller/commit/4a3131941837f1fab321bcdfcac13ac4fb480684
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail
@ 2021-03-18 14:34 ` Dmitry Vyukov
0 siblings, 0 replies; 51+ messages in thread
From: Dmitry Vyukov @ 2021-03-18 14:34 UTC (permalink / raw)
To: Ben Dooks
Cc: Alex Ghiti, syzbot, Paul Walmsley, Palmer Dabbelt, Albert Ou,
linux-riscv, Daniel Bristot de Oliveira, Benjamin Segall,
dietmar.eggemann, Juri Lelli, LKML, Mel Gorman, Ingo Molnar,
Peter Zijlstra, Steven Rostedt, syzkaller-bugs, Vincent Guittot
On Mon, Mar 15, 2021 at 5:55 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
> > On Fri, Mar 12, 2021 at 9:12 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>
> >>> Still no luck for the moment, can't reproduce it locally, my test is
> >>> maybe not that good (I created threads all day long in order to trigger
> >>> the put_user of schedule_tail).
> >>
> >> It may of course depend on memory and other stuff. I did try to see if
> >> it was possible to clone() with the child_tid address being a valid but
> >> not mapped page...
> >>
> >>> Given that the path you mention works most of the time, and that the
> >>> status register in the stack trace shows the SUM bit is not set whereas
> >>> it is set in put_user, I'm leaning toward some race condition (maybe an
> >>> interrupt that arrives at the "wrong" time) or a qemu issue as you
> >>> mentioned.
> >>
> >> I suppose this is possible. From what I read it should get to the
> >> point of being there with the SUM flag cleared, so either something
> >> went wrong in trying to fix the instruction up or there's some other
> >> error we're missing.
> >>
> >>> To eliminate qemu issues, do you have access to some HW ? Or to
> >>> different qemu versions ?
> >>
> >> I do have access to a Microchip Polarfire board. I just need the
> >> instructions on how to setup the test-code to make it work on the
> >> hardware.
> >
> > For full syzkaller support, it would need to know how to reboot these
> > boards and get access to the console.
> > syzkaller has a stop-gap VM backend which just uses ssh to a physical
> > machine and expects the kernel to reboot on its own after any crashes.
> >
> > But I actually managed to reproduce it in an even simpler setup.
> > Assuming you have Go 1.15 and riscv64 cross-compiler gcc installed
> >
> > $ go get -u -d github.com/google/syzkaller/...
> > $ cd $GOPATH/src/github.com/google/syzkaller
> > $ make stress executor TARGETARCH=riscv64
> > $ scp bin/linux_riscv64/syz-execprog bin/linux_riscv64/syz-executor
> > your_machine:/
> >
> > Then run ./syz-stress on the machine.
> > On the first run it crashed it with some other bug, on the second run
> > I got the crash in schedule_tail.
> > With qemu tcg I also added -slowdown=10 flag to syz-stress to scale
> > all timeouts, if native execution is faster, then you don't need it.
>
> I have built the tools and got it to start.
>
> It would be helpful for the dashboard to give the qemu version and
> how it was launched (memory, cpus etc)
Hi Ben,
syzbot will show info about qemu version/args in "VM info" column then
this commit is deployed (should happen by tomorrow);
https://github.com/google/syzkaller/commit/4a3131941837f1fab321bcdfcac13ac4fb480684
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 51+ messages in thread
end of thread, other threads:[~2021-03-18 14:35 UTC | newest]
Thread overview: 51+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-10 16:46 [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail syzbot
2021-03-10 17:16 ` Dmitry Vyukov
2021-03-10 17:16 ` Dmitry Vyukov
2021-03-10 22:24 ` Ben Dooks
2021-03-10 22:24 ` Ben Dooks
2021-03-11 6:39 ` Alex Ghiti
2021-03-11 6:39 ` Alex Ghiti
2021-03-11 6:50 ` Dmitry Vyukov
2021-03-11 6:50 ` Dmitry Vyukov
2021-03-11 6:52 ` Dmitry Vyukov
2021-03-11 6:52 ` Dmitry Vyukov
2021-03-11 10:41 ` Ben Dooks
2021-03-11 10:41 ` Ben Dooks
2021-03-12 13:49 ` Ben Dooks
2021-03-12 13:49 ` Ben Dooks
2021-03-12 15:12 ` Dmitry Vyukov
2021-03-12 15:12 ` Dmitry Vyukov
2021-03-12 16:25 ` Alex Ghiti
2021-03-12 16:25 ` Alex Ghiti
2021-03-12 20:12 ` Ben Dooks
2021-03-12 20:12 ` Ben Dooks
2021-03-13 7:20 ` Dmitry Vyukov
2021-03-13 7:20 ` Dmitry Vyukov
2021-03-15 16:55 ` Ben Dooks
2021-03-15 16:55 ` Ben Dooks
2021-03-18 14:34 ` Dmitry Vyukov
2021-03-18 14:34 ` Dmitry Vyukov
2021-03-15 21:38 ` Ben Dooks
2021-03-15 21:38 ` Ben Dooks
2021-03-16 8:52 ` Dmitry Vyukov
2021-03-16 8:52 ` Dmitry Vyukov
2021-03-16 11:35 ` Ben Dooks
2021-03-16 11:35 ` Ben Dooks
2021-03-16 11:44 ` Dmitry Vyukov
2021-03-16 11:44 ` Dmitry Vyukov
2021-03-12 16:30 ` Ben Dooks
2021-03-12 16:30 ` Ben Dooks
2021-03-12 16:34 ` Ben Dooks
2021-03-12 16:34 ` Ben Dooks
2021-03-12 16:36 ` Ben Dooks
2021-03-12 16:36 ` Ben Dooks
2021-03-12 17:34 ` Dmitry Vyukov
2021-03-12 17:34 ` Dmitry Vyukov
2021-03-12 17:38 ` Dmitry Vyukov
2021-03-12 17:38 ` Dmitry Vyukov
2021-03-18 9:41 ` Ben Dooks
2021-03-18 9:41 ` Ben Dooks
2021-03-18 10:05 ` Dmitry Vyukov
2021-03-18 10:05 ` Dmitry Vyukov
2021-03-18 12:52 ` Ben Dooks
2021-03-18 12:52 ` Ben Dooks
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.