All of lore.kernel.org
 help / color / mirror / Atom feed
* [syzbot] INFO: rcu detected stall in sys_lsetxattr
@ 2022-04-19 16:16 syzbot
  2022-04-20 12:27 ` Christian Brauner
  2022-04-20 13:19 ` [PATCH] fs: unset MNT_WRITE_HOLD on failure Christian Brauner
  0 siblings, 2 replies; 6+ messages in thread
From: syzbot @ 2022-04-19 16:16 UTC (permalink / raw)
  To: fweisbec, linux-fsdevel, linux-kernel, mingo, syzkaller-bugs, tglx, viro

Hello,

syzbot found the following issue on:

HEAD commit:    40354149f4d7 Add linux-next specific files for 20220414
git tree:       linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=16ae0bd0f00000
kernel config:  https://syzkaller.appspot.com/x/.config?x=a44d62051576f6f5
dashboard link: https://syzkaller.appspot.com/bug?extid=306090cfa3294f0bbfb3
compiler:       gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=164417ccf00000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=104d63d0f00000

The issue was bisected to:

commit e257039f0fc7da36ac3a522ef9a5cb4ae7852e67
Author: Al Viro <viro@zeniv.linux.org.uk>
Date:   Tue Mar 1 04:04:20 2022 +0000

    mount_setattr(): clean the control flow and calling conventions

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=14622210f00000
final oops:     https://syzkaller.appspot.com/x/report.txt?x=16622210f00000
console output: https://syzkaller.appspot.com/x/log.txt?x=12622210f00000

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+306090cfa3294f0bbfb3@syzkaller.appspotmail.com
Fixes: e257039f0fc7 ("mount_setattr(): clean the control flow and calling conventions")

rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 	1-....: (10499 ticks this GP) idle=23b/1/0x4000000000000000 softirq=5447/5447 fqs=5210 
	(t=10500 jiffies g=4401 q=63 ncpus=2)
NMI backtrace for cpu 1
CPU: 1 PID: 3614 Comm: syz-executor153 Not tainted 5.18.0-rc2-next-20220414-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
 nmi_cpu_backtrace.cold+0x47/0x144 lib/nmi_backtrace.c:111
 nmi_trigger_cpumask_backtrace+0x1e6/0x230 lib/nmi_backtrace.c:62
 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
 rcu_dump_cpu_stacks+0x262/0x3f0 kernel/rcu/tree_stall.h:369
 print_cpu_stall kernel/rcu/tree_stall.h:665 [inline]
 check_cpu_stall kernel/rcu/tree_stall.h:749 [inline]
 rcu_pending kernel/rcu/tree.c:4010 [inline]
 rcu_sched_clock_irq.cold+0x144/0x8fc kernel/rcu/tree.c:2675
 update_process_times+0x16d/0x200 kernel/time/timer.c:1811
 tick_sched_handle+0x9b/0x180 kernel/time/tick-sched.c:243
 tick_sched_timer+0xee/0x120 kernel/time/tick-sched.c:1473
 __run_hrtimer kernel/time/hrtimer.c:1685 [inline]
 __hrtimer_run_queues+0x1c0/0xe50 kernel/time/hrtimer.c:1749
 hrtimer_interrupt+0x31c/0x790 kernel/time/hrtimer.c:1811
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1086 [inline]
 __sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1103
 sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1097
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:649
RIP: 0010:__mnt_want_write+0xdd/0x2e0 fs/namespace.c:348
Code: 00 02 00 00 89 de e8 22 64 9c ff 85 db 74 42 48 b8 00 00 00 00 00 fc ff df 4d 89 ec 49 c1 ec 03 49 01 c4 e8 e5 61 9c ff f3 90 <41> 0f b6 04 24 84 c0 74 08 3c 03 0f 8e 99 01 00 00 8b 5d 10 31 ff
RSP: 0018:ffffc90003acfdf0 EFLAGS: 00000293
RAX: 0000000000000000 RBX: 0000000000000200 RCX: 0000000000000000
RDX: ffff8880209957c0 RSI: ffffffff81dda00b RDI: 0000000000000003
RBP: ffff888140174c20 R08: 0000000000000000 R09: 0000000000000000
R10: ffffffff81dda030 R11: 1ffffffff1f09899 R12: ffffed102802e986
R13: ffff888140174c30 R14: ffff888140174c50 R15: 0000000000000000
 mnt_want_write+0x13d/0x3e0 fs/namespace.c:394
 path_setxattr+0xb2/0x1c0 fs/xattr.c:627
 __do_sys_lsetxattr fs/xattr.c:652 [inline]
 __se_sys_lsetxattr fs/xattr.c:648 [inline]
 __x64_sys_lsetxattr+0xbd/0x150 fs/xattr.c:648
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7f7262608cc9
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 11 15 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f72625992f8 EFLAGS: 00000246 ORIG_RAX: 00000000000000bd
RAX: ffffffffffffffda RBX: 00007f72626904b0 RCX: 00007f7262608cc9
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000020000600
RBP: 00007f726265e074 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0030656c69662f2e
R13: 695f70756f72672c R14: 695f726573752c30 R15: 00007f72626904b8
 </TASK>
----------------
Code disassembly (best guess):
   0:	00 02                	add    %al,(%rdx)
   2:	00 00                	add    %al,(%rax)
   4:	89 de                	mov    %ebx,%esi
   6:	e8 22 64 9c ff       	callq  0xff9c642d
   b:	85 db                	test   %ebx,%ebx
   d:	74 42                	je     0x51
   f:	48 b8 00 00 00 00 00 	movabs $0xdffffc0000000000,%rax
  16:	fc ff df
  19:	4d 89 ec             	mov    %r13,%r12
  1c:	49 c1 ec 03          	shr    $0x3,%r12
  20:	49 01 c4             	add    %rax,%r12
  23:	e8 e5 61 9c ff       	callq  0xff9c620d
  28:	f3 90                	pause
* 2a:	41 0f b6 04 24       	movzbl (%r12),%eax <-- trapping instruction
  2f:	84 c0                	test   %al,%al
  31:	74 08                	je     0x3b
  33:	3c 03                	cmp    $0x3,%al
  35:	0f 8e 99 01 00 00    	jle    0x1d4
  3b:	8b 5d 10             	mov    0x10(%rbp),%ebx
  3e:	31 ff                	xor    %edi,%edi


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
For information about bisection process see: https://goo.gl/tpsmEJ#bisection
syzbot can test patches for this issue, for details see:
https://goo.gl/tpsmEJ#testing-patches

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [syzbot] INFO: rcu detected stall in sys_lsetxattr
  2022-04-19 16:16 [syzbot] INFO: rcu detected stall in sys_lsetxattr syzbot
@ 2022-04-20 12:27 ` Christian Brauner
  2022-04-20 13:01   ` syzbot
  2022-04-20 13:19 ` [PATCH] fs: unset MNT_WRITE_HOLD on failure Christian Brauner
  1 sibling, 1 reply; 6+ messages in thread
From: Christian Brauner @ 2022-04-20 12:27 UTC (permalink / raw)
  To: syzbot
  Cc: fweisbec, linux-fsdevel, linux-kernel, mingo, syzkaller-bugs, tglx, viro

On Tue, Apr 19, 2022 at 09:16:20AM -0700, syzbot wrote:
> Hello,
> 
> syzbot found the following issue on:
> 
> HEAD commit:    40354149f4d7 Add linux-next specific files for 20220414
> git tree:       linux-next
> console output: https://syzkaller.appspot.com/x/log.txt?x=16ae0bd0f00000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=a44d62051576f6f5
> dashboard link: https://syzkaller.appspot.com/bug?extid=306090cfa3294f0bbfb3
> compiler:       gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=164417ccf00000
> C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=104d63d0f00000
> 
> The issue was bisected to:
> 
> commit e257039f0fc7da36ac3a522ef9a5cb4ae7852e67
> Author: Al Viro <viro@zeniv.linux.org.uk>
> Date:   Tue Mar 1 04:04:20 2022 +0000
> 
>     mount_setattr(): clean the control flow and calling conventions
> 
> bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=14622210f00000
> final oops:     https://syzkaller.appspot.com/x/report.txt?x=16622210f00000
> console output: https://syzkaller.appspot.com/x/log.txt?x=12622210f00000
> 
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+306090cfa3294f0bbfb3@syzkaller.appspotmail.com
> Fixes: e257039f0fc7 ("mount_setattr(): clean the control flow and calling conventions")

#syz test: git://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux.git fs.mount_setattr.cleanup

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [syzbot] INFO: rcu detected stall in sys_lsetxattr
  2022-04-20 12:27 ` Christian Brauner
@ 2022-04-20 13:01   ` syzbot
  0 siblings, 0 replies; 6+ messages in thread
From: syzbot @ 2022-04-20 13:01 UTC (permalink / raw)
  To: brauner, fweisbec, linux-fsdevel, linux-kernel, mingo,
	syzkaller-bugs, tglx, viro

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by: syzbot+306090cfa3294f0bbfb3@syzkaller.appspotmail.com

Tested on:

commit:         bbc1e8c5 fs: unset MNT_WRITE_HOLD on failure
git tree:       git://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux.git fs.mount_setattr.cleanup
kernel config:  https://syzkaller.appspot.com/x/.config?x=c01066a8395ef6d7
dashboard link: https://syzkaller.appspot.com/bug?extid=306090cfa3294f0bbfb3
compiler:       gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2

Note: no patches were applied.
Note: testing is done by a robot and is best-effort only.

^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH] fs: unset MNT_WRITE_HOLD on failure
  2022-04-19 16:16 [syzbot] INFO: rcu detected stall in sys_lsetxattr syzbot
  2022-04-20 12:27 ` Christian Brauner
@ 2022-04-20 13:19 ` Christian Brauner
  2022-04-21 13:47   ` Christian Brauner
  2022-04-21 14:43   ` Christoph Hellwig
  1 sibling, 2 replies; 6+ messages in thread
From: Christian Brauner @ 2022-04-20 13:19 UTC (permalink / raw)
  To: linux-fsdevel
  Cc: Christian Brauner, Christoph Hellwig, Al Viro, Hillf Danton,
	fweisbec, mingo, syzkaller-bugs, tglx,
	syzbot+10a16d1c43580983f6a2, syzbot+306090cfa3294f0bbfb3

After mnt_hold_writers() has been called we will always have set MNT_WRITE_HOLD
and consequently we always need to pair mnt_hold_writers() with
mnt_unhold_writers(). After the recent cleanup in [1] where Al switched from a
do-while to a for loop the cleanup currently fails to unset MNT_WRITE_HOLD for
the first mount that was changed. Fix this and make sure that the first mount
will be cleaned up and add some comments to make it more obvious.

Reported-by: syzbot+10a16d1c43580983f6a2@syzkaller.appspotmail.com
Reported-by: syzbot+306090cfa3294f0bbfb3@syzkaller.appspotmail.com
Fixes: e257039f0fc7 ("mount_setattr(): clean the control flow and calling conventions") [1]
Link: https://lore.kernel.org/lkml/0000000000007cc21d05dd0432b8@google.com
Link: https://lore.kernel.org/lkml/00000000000080e10e05dd043247@google.com
Cc: Hillf Danton <hdanton@sina.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Christian Brauner (Microsoft) <brauner@kernel.org>
---
This should fix the syzbot issue. This is only relevant for making a
mount or mount tree read-only:
1. successul recursive read-only mount tree change:
   Cleanup loop isn't executed.
2. failed recursive read-only mount tree change:
   m will point to the mount we failed to handle. The cleanup loop will
   run until p == m and then terminate.
3. successful single read-only mount change:
   Cleanup loop won't be executed.
4. failed single read-only mount change:
   m will point to mnt and the cleanup loop will terminate if p == m.
I don't think there's any other weird corner cases since we now that
MNT_WRITE_HOLD can only have been set by us as it requires
lock_mount_hash() which we hold. So unconditionally unsetting it is
fine. But please make sure to take a close look at the changed loop.
---
 fs/namespace.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/fs/namespace.c b/fs/namespace.c
index a0a36bfa3aa0..afe2b64b14f1 100644
--- a/fs/namespace.c
+++ b/fs/namespace.c
@@ -4058,10 +4058,22 @@ static int mount_setattr_prepare(struct mount_kattr *kattr, struct mount *mnt)
 	if (err) {
 		struct mount *p;
 
-		for (p = mnt; p != m; p = next_mnt(p, mnt)) {
+		/*
+		 * If we had to call mnt_hold_writers() MNT_WRITE_HOLD will
+		 * be set in @mnt_flags. The loop unsets MNT_WRITE_HOLD for all
+		 * mounts and needs to take care to include the first mount.
+		 */
+		for (p = mnt; p; p = next_mnt(p, mnt)) {
 			/* If we had to hold writers unblock them. */
 			if (p->mnt.mnt_flags & MNT_WRITE_HOLD)
 				mnt_unhold_writers(p);
+
+			/*
+			 * We're done once the first mount we changed got
+			 * MNT_WRITE_HOLD unset.
+			 */
+			if (p == m)
+				break;
 		}
 	}
 	return err;

base-commit: b2d229d4ddb17db541098b83524d901257e93845
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 6+ messages in thread

* Re: [PATCH] fs: unset MNT_WRITE_HOLD on failure
  2022-04-20 13:19 ` [PATCH] fs: unset MNT_WRITE_HOLD on failure Christian Brauner
@ 2022-04-21 13:47   ` Christian Brauner
  2022-04-21 14:43   ` Christoph Hellwig
  1 sibling, 0 replies; 6+ messages in thread
From: Christian Brauner @ 2022-04-21 13:47 UTC (permalink / raw)
  To: linux-fsdevel
  Cc: Christoph Hellwig, Al Viro, Hillf Danton, fweisbec, mingo,
	syzkaller-bugs, tglx, syzbot+10a16d1c43580983f6a2,
	syzbot+306090cfa3294f0bbfb3

On Wed, Apr 20, 2022 at 03:19:25PM +0200, Christian Brauner wrote:
> After mnt_hold_writers() has been called we will always have set MNT_WRITE_HOLD
> and consequently we always need to pair mnt_hold_writers() with
> mnt_unhold_writers(). After the recent cleanup in [1] where Al switched from a
> do-while to a for loop the cleanup currently fails to unset MNT_WRITE_HOLD for
> the first mount that was changed. Fix this and make sure that the first mount
> will be cleaned up and add some comments to make it more obvious.
> 
> Reported-by: syzbot+10a16d1c43580983f6a2@syzkaller.appspotmail.com
> Reported-by: syzbot+306090cfa3294f0bbfb3@syzkaller.appspotmail.com
> Fixes: e257039f0fc7 ("mount_setattr(): clean the control flow and calling conventions") [1]
> Link: https://lore.kernel.org/lkml/0000000000007cc21d05dd0432b8@google.com
> Link: https://lore.kernel.org/lkml/00000000000080e10e05dd043247@google.com
> Cc: Hillf Danton <hdanton@sina.com>
> Cc: Christoph Hellwig <hch@lst.de>
> Cc: Al Viro <viro@zeniv.linux.org.uk>
> Signed-off-by: Christian Brauner (Microsoft) <brauner@kernel.org>
> ---
> This should fix the syzbot issue. This is only relevant for making a
> mount or mount tree read-only:
> 1. successul recursive read-only mount tree change:
>    Cleanup loop isn't executed.
> 2. failed recursive read-only mount tree change:
>    m will point to the mount we failed to handle. The cleanup loop will
>    run until p == m and then terminate.
> 3. successful single read-only mount change:
>    Cleanup loop won't be executed.
> 4. failed single read-only mount change:
>    m will point to mnt and the cleanup loop will terminate if p == m.
> I don't think there's any other weird corner cases since we now that
> MNT_WRITE_HOLD can only have been set by us as it requires
> lock_mount_hash() which we hold. So unconditionally unsetting it is
> fine. But please make sure to take a close look at the changed loop.
> ---

Unless I hear objections I'll route this upstream before -rc4 to get
this fixed because it's pretty trivial to trigger this.

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] fs: unset MNT_WRITE_HOLD on failure
  2022-04-20 13:19 ` [PATCH] fs: unset MNT_WRITE_HOLD on failure Christian Brauner
  2022-04-21 13:47   ` Christian Brauner
@ 2022-04-21 14:43   ` Christoph Hellwig
  1 sibling, 0 replies; 6+ messages in thread
From: Christoph Hellwig @ 2022-04-21 14:43 UTC (permalink / raw)
  To: Christian Brauner
  Cc: linux-fsdevel, Christoph Hellwig, Al Viro, Hillf Danton,
	fweisbec, mingo, syzkaller-bugs, tglx,
	syzbot+10a16d1c43580983f6a2, syzbot+306090cfa3294f0bbfb3

Looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2022-04-21 14:44 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-04-19 16:16 [syzbot] INFO: rcu detected stall in sys_lsetxattr syzbot
2022-04-20 12:27 ` Christian Brauner
2022-04-20 13:01   ` syzbot
2022-04-20 13:19 ` [PATCH] fs: unset MNT_WRITE_HOLD on failure Christian Brauner
2022-04-21 13:47   ` Christian Brauner
2022-04-21 14:43   ` Christoph Hellwig

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.