* [syzbot] possible deadlock in pipe_write @ 2022-03-21 2:34 syzbot 2022-03-21 14:41 ` Jann Horn 2022-03-21 16:03 ` David Howells 0 siblings, 2 replies; 6+ messages in thread From: syzbot @ 2022-03-21 2:34 UTC (permalink / raw) To: linux-fsdevel, linux-kernel, syzkaller-bugs, viro Hello, syzbot found the following issue on: HEAD commit: 56e337f2cf13 Revert "gpio: Revert regression in sysfs-gpio.. git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=13f00f7e700000 kernel config: https://syzkaller.appspot.com/x/.config?x=d35f9bc6884af6c9 dashboard link: https://syzkaller.appspot.com/bug?extid=011e4ea1da6692cf881c compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=133235c5700000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=1248ca89700000 Bisection is inconclusive: the issue happens on the oldest tested release. bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=12f235c5700000 final oops: https://syzkaller.appspot.com/x/report.txt?x=11f235c5700000 console output: https://syzkaller.appspot.com/x/log.txt?x=16f235c5700000 IMPORTANT: if you fix the issue, please add the following tag to the commit: Reported-by: syzbot+011e4ea1da6692cf881c@syzkaller.appspotmail.com ============================================ WARNING: possible recursive locking detected 5.17.0-rc8-syzkaller-00003-g56e337f2cf13 #0 Not tainted -------------------------------------------- syz-executor190/3593 is trying to acquire lock: ffff888078020868 (&pipe->mutex/1){+.+.}-{3:3}, at: __pipe_lock fs/pipe.c:103 [inline] ffff888078020868 (&pipe->mutex/1){+.+.}-{3:3}, at: pipe_write+0x132/0x1c00 fs/pipe.c:431 but task is already holding lock: ffff888078020468 (&pipe->mutex/1){+.+.}-{3:3}, at: pipe_lock_nested fs/pipe.c:82 [inline] ffff888078020468 (&pipe->mutex/1){+.+.}-{3:3}, at: pipe_lock fs/pipe.c:90 [inline] ffff888078020468 (&pipe->mutex/1){+.+.}-{3:3}, at: pipe_wait_readable+0x39b/0x420 fs/pipe.c:1049 other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock(&pipe->mutex/1); lock(&pipe->mutex/1); *** DEADLOCK *** May be due to missing lock nesting notation 1 lock held by syz-executor190/3593: #0: ffff888078020468 (&pipe->mutex/1){+.+.}-{3:3}, at: pipe_lock_nested fs/pipe.c:82 [inline] #0: ffff888078020468 (&pipe->mutex/1){+.+.}-{3:3}, at: pipe_lock fs/pipe.c:90 [inline] #0: ffff888078020468 (&pipe->mutex/1){+.+.}-{3:3}, at: pipe_wait_readable+0x39b/0x420 fs/pipe.c:1049 stack backtrace: CPU: 1 PID: 3593 Comm: syz-executor190 Not tainted 5.17.0-rc8-syzkaller-00003-g56e337f2cf13 #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: <TASK> __dump_stack lib/dump_stack.c:88 [inline] dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106 print_deadlock_bug kernel/locking/lockdep.c:2956 [inline] check_deadlock kernel/locking/lockdep.c:2999 [inline] validate_chain kernel/locking/lockdep.c:3788 [inline] __lock_acquire.cold+0x213/0x3a9 kernel/locking/lockdep.c:5027 lock_acquire kernel/locking/lockdep.c:5639 [inline] lock_acquire+0x1ab/0x510 kernel/locking/lockdep.c:5604 __mutex_lock_common kernel/locking/mutex.c:600 [inline] __mutex_lock+0x12f/0x12f0 kernel/locking/mutex.c:733 __pipe_lock fs/pipe.c:103 [inline] pipe_write+0x132/0x1c00 fs/pipe.c:431 call_write_iter include/linux/fs.h:2074 [inline] do_iter_readv_writev+0x47a/0x750 fs/read_write.c:725 do_iter_write+0x188/0x710 fs/read_write.c:851 vfs_iter_write+0x70/0xa0 fs/read_write.c:892 iter_file_splice_write+0x723/0xc70 fs/splice.c:689 do_splice_from fs/splice.c:767 [inline] do_splice+0xb7e/0x1960 fs/splice.c:1079 __do_splice+0x134/0x250 fs/splice.c:1144 __do_sys_splice fs/splice.c:1350 [inline] __se_sys_splice fs/splice.c:1332 [inline] __x64_sys_splice+0x198/0x250 fs/splice.c:1332 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x7fb9ac14bca9 Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 81 14 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007fb9ac0fe308 EFLAGS: 00000246 ORIG_RAX: 0000000000000113 RAX: fffffff --- This report is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at syzkaller@googlegroups.com. syzbot will keep track of this issue. See: https://goo.gl/tpsmEJ#status for how to communicate with syzbot. For information about bisection process see: https://goo.gl/tpsmEJ#bisection syzbot can test patches for this issue, for details see: https://goo.gl/tpsmEJ#testing-patches ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [syzbot] possible deadlock in pipe_write 2022-03-21 2:34 [syzbot] possible deadlock in pipe_write syzbot @ 2022-03-21 14:41 ` Jann Horn 2022-03-21 16:03 ` David Howells 1 sibling, 0 replies; 6+ messages in thread From: Jann Horn @ 2022-03-21 14:41 UTC (permalink / raw) To: syzbot, David Howells; +Cc: linux-fsdevel, linux-kernel, syzkaller-bugs, viro This also looks like a watch_queue bug. On Mon, Mar 21, 2022 at 3:34 AM syzbot <syzbot+011e4ea1da6692cf881c@syzkaller.appspotmail.com> wrote: > > Hello, > > syzbot found the following issue on: > > HEAD commit: 56e337f2cf13 Revert "gpio: Revert regression in sysfs-gpio.. > git tree: upstream > console output: https://syzkaller.appspot.com/x/log.txt?x=13f00f7e700000 > kernel config: https://syzkaller.appspot.com/x/.config?x=d35f9bc6884af6c9 > dashboard link: https://syzkaller.appspot.com/bug?extid=011e4ea1da6692cf881c > compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=133235c5700000 The syz reproducer is: #{"threaded":true,"procs":1,"slowdown":1,"sandbox":"","close_fds":false} pipe(&(0x7f0000000240)={<r0=>0xffffffffffffffff, <r1=>0xffffffffffffffff}) pipe2(&(0x7f00000001c0)={0xffffffffffffffff, <r2=>0xffffffffffffffff}, 0x80) splice(r0, 0x0, r2, 0x0, 0x1ff, 0x0) vmsplice(r1, &(0x7f00000006c0)=[{&(0x7f0000000080)="b5", 0x1}], 0x1, 0x0) That 0x80 is O_NOTIFICATION_PIPE (==O_EXCL). It looks like the bug is that when you try to splice between a normal pipe and a notification pipe, get_pipe_info(..., true) fails, so splice() falls back to treating the notification pipe like a normal pipe - so we end up in iter_file_splice_write(), which first locks the input pipe, then calls vfs_iter_write(), which locks the output pipe. I think this probably (?) can't actually lead to deadlocks, since you'd need another way to nest locking a normal pipe into locking a watch_queue pipe, but the lockdep annotations don't make that clear. > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=1248ca89700000 > > Bisection is inconclusive: the issue happens on the oldest tested release. > > bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=12f235c5700000 > final oops: https://syzkaller.appspot.com/x/report.txt?x=11f235c5700000 > console output: https://syzkaller.appspot.com/x/log.txt?x=16f235c5700000 > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > Reported-by: syzbot+011e4ea1da6692cf881c@syzkaller.appspotmail.com > > ============================================ > WARNING: possible recursive locking detected > 5.17.0-rc8-syzkaller-00003-g56e337f2cf13 #0 Not tainted > -------------------------------------------- > syz-executor190/3593 is trying to acquire lock: > ffff888078020868 (&pipe->mutex/1){+.+.}-{3:3}, at: __pipe_lock fs/pipe.c:103 [inline] > ffff888078020868 (&pipe->mutex/1){+.+.}-{3:3}, at: pipe_write+0x132/0x1c00 fs/pipe.c:431 > > but task is already holding lock: > ffff888078020468 (&pipe->mutex/1){+.+.}-{3:3}, at: pipe_lock_nested fs/pipe.c:82 [inline] > ffff888078020468 (&pipe->mutex/1){+.+.}-{3:3}, at: pipe_lock fs/pipe.c:90 [inline] > ffff888078020468 (&pipe->mutex/1){+.+.}-{3:3}, at: pipe_wait_readable+0x39b/0x420 fs/pipe.c:1049 > > other info that might help us debug this: > Possible unsafe locking scenario: > > CPU0 > ---- > lock(&pipe->mutex/1); > lock(&pipe->mutex/1); > > *** DEADLOCK *** > > May be due to missing lock nesting notation > > 1 lock held by syz-executor190/3593: > #0: ffff888078020468 (&pipe->mutex/1){+.+.}-{3:3}, at: pipe_lock_nested fs/pipe.c:82 [inline] > #0: ffff888078020468 (&pipe->mutex/1){+.+.}-{3:3}, at: pipe_lock fs/pipe.c:90 [inline] > #0: ffff888078020468 (&pipe->mutex/1){+.+.}-{3:3}, at: pipe_wait_readable+0x39b/0x420 fs/pipe.c:1049 > > stack backtrace: > CPU: 1 PID: 3593 Comm: syz-executor190 Not tainted 5.17.0-rc8-syzkaller-00003-g56e337f2cf13 #0 > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 > Call Trace: > <TASK> > __dump_stack lib/dump_stack.c:88 [inline] > dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106 > print_deadlock_bug kernel/locking/lockdep.c:2956 [inline] > check_deadlock kernel/locking/lockdep.c:2999 [inline] > validate_chain kernel/locking/lockdep.c:3788 [inline] > __lock_acquire.cold+0x213/0x3a9 kernel/locking/lockdep.c:5027 > lock_acquire kernel/locking/lockdep.c:5639 [inline] > lock_acquire+0x1ab/0x510 kernel/locking/lockdep.c:5604 > __mutex_lock_common kernel/locking/mutex.c:600 [inline] > __mutex_lock+0x12f/0x12f0 kernel/locking/mutex.c:733 > __pipe_lock fs/pipe.c:103 [inline] > pipe_write+0x132/0x1c00 fs/pipe.c:431 > call_write_iter include/linux/fs.h:2074 [inline] > do_iter_readv_writev+0x47a/0x750 fs/read_write.c:725 > do_iter_write+0x188/0x710 fs/read_write.c:851 > vfs_iter_write+0x70/0xa0 fs/read_write.c:892 > iter_file_splice_write+0x723/0xc70 fs/splice.c:689 > do_splice_from fs/splice.c:767 [inline] > do_splice+0xb7e/0x1960 fs/splice.c:1079 > __do_splice+0x134/0x250 fs/splice.c:1144 > __do_sys_splice fs/splice.c:1350 [inline] > __se_sys_splice fs/splice.c:1332 [inline] > __x64_sys_splice+0x198/0x250 fs/splice.c:1332 > do_syscall_x64 arch/x86/entry/common.c:50 [inline] > do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80 > entry_SYSCALL_64_after_hwframe+0x44/0xae > RIP: 0033:0x7fb9ac14bca9 > Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 81 14 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48 > RSP: 002b:00007fb9ac0fe308 EFLAGS: 00000246 ORIG_RAX: 0000000000000113 > RAX: fffffff > > > --- > This report is generated by a bot. It may contain errors. > See https://goo.gl/tpsmEJ for more information about syzbot. > syzbot engineers can be reached at syzkaller@googlegroups.com. > > syzbot will keep track of this issue. See: > https://goo.gl/tpsmEJ#status for how to communicate with syzbot. > For information about bisection process see: https://goo.gl/tpsmEJ#bisection > syzbot can test patches for this issue, for details see: > https://goo.gl/tpsmEJ#testing-patches > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [syzbot] possible deadlock in pipe_write 2022-03-21 2:34 [syzbot] possible deadlock in pipe_write syzbot 2022-03-21 14:41 ` Jann Horn @ 2022-03-21 16:03 ` David Howells 2022-03-21 16:17 ` Jann Horn 1 sibling, 1 reply; 6+ messages in thread From: David Howells @ 2022-03-21 16:03 UTC (permalink / raw) To: Jann Horn Cc: dhowells, syzbot, linux-fsdevel, linux-kernel, syzkaller-bugs, viro Jann Horn <jannh@google.com> wrote: > The syz reproducer is: > > #{"threaded":true,"procs":1,"slowdown":1,"sandbox":"","close_fds":false} > pipe(&(0x7f0000000240)={<r0=>0xffffffffffffffff, <r1=>0xffffffffffffffff}) > pipe2(&(0x7f00000001c0)={0xffffffffffffffff, <r2=>0xffffffffffffffff}, 0x80) > splice(r0, 0x0, r2, 0x0, 0x1ff, 0x0) > vmsplice(r1, &(0x7f00000006c0)=[{&(0x7f0000000080)="b5", 0x1}], 0x1, 0x0) > > That 0x80 is O_NOTIFICATION_PIPE (==O_EXCL). > > It looks like the bug is that when you try to splice between a normal > pipe and a notification pipe, get_pipe_info(..., true) fails, so > splice() falls back to treating the notification pipe like a normal > pipe - so we end up in iter_file_splice_write(), which first locks the > input pipe, then calls vfs_iter_write(), which locks the output pipe. > > I think this probably (?) can't actually lead to deadlocks, since > you'd need another way to nest locking a normal pipe into locking a > watch_queue pipe, but the lockdep annotations don't make that clear. Is this then a bug/feature in iter_file_splice_write() rather than in the watch queue code, per se? David ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [syzbot] possible deadlock in pipe_write 2022-03-21 16:03 ` David Howells @ 2022-03-21 16:17 ` Jann Horn 2023-11-24 14:38 ` Jann Horn 0 siblings, 1 reply; 6+ messages in thread From: Jann Horn @ 2022-03-21 16:17 UTC (permalink / raw) To: David Howells; +Cc: syzbot, linux-fsdevel, linux-kernel, syzkaller-bugs, viro On Mon, Mar 21, 2022 at 5:03 PM David Howells <dhowells@redhat.com> wrote: > Jann Horn <jannh@google.com> wrote: > > > The syz reproducer is: > > > > #{"threaded":true,"procs":1,"slowdown":1,"sandbox":"","close_fds":false} > > pipe(&(0x7f0000000240)={<r0=>0xffffffffffffffff, <r1=>0xffffffffffffffff}) > > pipe2(&(0x7f00000001c0)={0xffffffffffffffff, <r2=>0xffffffffffffffff}, 0x80) > > splice(r0, 0x0, r2, 0x0, 0x1ff, 0x0) > > vmsplice(r1, &(0x7f00000006c0)=[{&(0x7f0000000080)="b5", 0x1}], 0x1, 0x0) > > > > That 0x80 is O_NOTIFICATION_PIPE (==O_EXCL). > > > > It looks like the bug is that when you try to splice between a normal > > pipe and a notification pipe, get_pipe_info(..., true) fails, so > > splice() falls back to treating the notification pipe like a normal > > pipe - so we end up in iter_file_splice_write(), which first locks the > > input pipe, then calls vfs_iter_write(), which locks the output pipe. > > > > I think this probably (?) can't actually lead to deadlocks, since > > you'd need another way to nest locking a normal pipe into locking a > > watch_queue pipe, but the lockdep annotations don't make that clear. > > Is this then a bug/feature in iter_file_splice_write() rather than in the > watch queue code, per se? I think at least when you call splice() on two normal pipes from userspace, it'll never go through this codepath for real pipes, because pipe-to-pipe splicing is special-cased? And sendfile() bails out in that case because pipes don't have a .splice_read() handler. And with notification pipes, we don't take that special path in splice(), and so we hit the lockdep warning. But I don't know whether that makes it the fault of notification pipes... Maybe it would be enough to just move the "if (pipe->watch_queue)" check in pipe_write() up above the __pipe_lock(pipe)? ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [syzbot] possible deadlock in pipe_write 2022-03-21 16:17 ` Jann Horn @ 2023-11-24 14:38 ` Jann Horn 2023-11-24 15:03 ` [syzbot] [fs?] " syzbot 0 siblings, 1 reply; 6+ messages in thread From: Jann Horn @ 2023-11-24 14:38 UTC (permalink / raw) To: David Howells; +Cc: syzbot, linux-fsdevel, linux-kernel, syzkaller-bugs, viro On Mon, Mar 21, 2022 at 5:17 PM Jann Horn <jannh@google.com> wrote: > On Mon, Mar 21, 2022 at 5:03 PM David Howells <dhowells@redhat.com> wrote: > > Jann Horn <jannh@google.com> wrote: > > > > > The syz reproducer is: > > > > > > #{"threaded":true,"procs":1,"slowdown":1,"sandbox":"","close_fds":false} > > > pipe(&(0x7f0000000240)={<r0=>0xffffffffffffffff, <r1=>0xffffffffffffffff}) > > > pipe2(&(0x7f00000001c0)={0xffffffffffffffff, <r2=>0xffffffffffffffff}, 0x80) > > > splice(r0, 0x0, r2, 0x0, 0x1ff, 0x0) > > > vmsplice(r1, &(0x7f00000006c0)=[{&(0x7f0000000080)="b5", 0x1}], 0x1, 0x0) > > > > > > That 0x80 is O_NOTIFICATION_PIPE (==O_EXCL). > > > > > > It looks like the bug is that when you try to splice between a normal > > > pipe and a notification pipe, get_pipe_info(..., true) fails, so > > > splice() falls back to treating the notification pipe like a normal > > > pipe - so we end up in iter_file_splice_write(), which first locks the > > > input pipe, then calls vfs_iter_write(), which locks the output pipe. > > > > > > I think this probably (?) can't actually lead to deadlocks, since > > > you'd need another way to nest locking a normal pipe into locking a > > > watch_queue pipe, but the lockdep annotations don't make that clear. > > > > Is this then a bug/feature in iter_file_splice_write() rather than in the > > watch queue code, per se? > > I think at least when you call splice() on two normal pipes from > userspace, it'll never go through this codepath for real pipes, > because pipe-to-pipe splicing is special-cased? And sendfile() bails > out in that case because pipes don't have a .splice_read() handler. > > And with notification pipes, we don't take that special path in > splice(), and so we hit the lockdep warning. But I don't know whether > that makes it the fault of notification pipes... > > Maybe it would be enough to just move the "if (pipe->watch_queue)" > check in pipe_write() up above the __pipe_lock(pipe)? [coming back to this thread 1.5 years later...] I've turned that idea into a fix, let's have syzbot try it out before I submit the fix patch: #syz test: https://github.com/thejh/linux.git 56c486e68166 ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [syzbot] [fs?] possible deadlock in pipe_write 2023-11-24 14:38 ` Jann Horn @ 2023-11-24 15:03 ` syzbot 0 siblings, 0 replies; 6+ messages in thread From: syzbot @ 2023-11-24 15:03 UTC (permalink / raw) To: dhowells, jannh, linux-fsdevel, linux-kernel, syzkaller-bugs, viro Hello, syzbot has tested the proposed patch and the reproducer did not trigger any issue: Reported-and-tested-by: syzbot+011e4ea1da6692cf881c@syzkaller.appspotmail.com Tested on: commit: 56c486e6 fs/pipe: Fix lockdep false-positive in watchq.. git tree: https://github.com/thejh/linux.git console output: https://syzkaller.appspot.com/x/log.txt?x=128b880ce80000 kernel config: https://syzkaller.appspot.com/x/.config?x=1e6a76f6c7029ca2 dashboard link: https://syzkaller.appspot.com/bug?extid=011e4ea1da6692cf881c compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 Note: no patches were applied. Note: testing is done by a robot and is best-effort only. ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-11-24 15:03 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-03-21 2:34 [syzbot] possible deadlock in pipe_write syzbot 2022-03-21 14:41 ` Jann Horn 2022-03-21 16:03 ` David Howells 2022-03-21 16:17 ` Jann Horn 2023-11-24 14:38 ` Jann Horn 2023-11-24 15:03 ` [syzbot] [fs?] " syzbot
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.