linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* WARNING in ext4_invalidatepage
@ 2018-10-08 16:18 syzbot
  2018-10-08 16:29 ` Dmitry Vyukov
  0 siblings, 1 reply; 9+ messages in thread
From: syzbot @ 2018-10-08 16:18 UTC (permalink / raw)
  To: adilger.kernel, linux-ext4, linux-kernel, syzkaller-bugs, tytso

Hello,

syzbot found the following crash on:

HEAD commit:    c1d84a1b42ef Merge git://git.kernel.org/pub/scm/linux/kern..
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=161588d6400000
kernel config:  https://syzkaller.appspot.com/x/.config?x=c0af03fe452b65fb
dashboard link: https://syzkaller.appspot.com/bug?extid=85da7ac734f7ba432ee4
compiler:       gcc (GCC) 8.0.1 20180413 (experimental)

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+85da7ac734f7ba432ee4@syzkaller.appspotmail.com

WARNING: CPU: 1 PID: 17203 at fs/ext4/inode.c:3353  
ext4_invalidatepage+0x1c7/0x5b0 fs/ext4/inode.c:3353
Kernel panic - not syncing: panic_on_warn set ...

CPU: 1 PID: 17203 Comm: syz-executor5 Not tainted 4.19.0-rc6+ #48
kobject: 'loop3' (000000002393f9b0): kobject_uevent_env
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS  
Google 01/01/2011
Call Trace:
  __dump_stack lib/dump_stack.c:77 [inline]
  dump_stack+0x1c4/0x2b4 lib/dump_stack.c:113
kobject: 'loop3' (000000002393f9b0): fill_kobj_path: path  
= '/devices/virtual/block/loop3'
  panic+0x238/0x4e7 kernel/panic.c:184
  __warn.cold.8+0x163/0x1ba kernel/panic.c:536
  report_bug+0x254/0x2d0 lib/bug.c:186
  fixup_bug arch/x86/kernel/traps.c:178 [inline]
  do_error_trap+0x1fc/0x4d0 arch/x86/kernel/traps.c:296
  do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:316
  invalid_op+0x14/0x20 arch/x86/entry/entry_64.S:993
RIP: 0010:ext4_invalidatepage+0x1c7/0x5b0 fs/ext4/inode.c:3353
Code: 80 3c 02 00 0f 85 a7 03 00 00 4d 8b 6d 00 31 ff 49 c1 ed 11 41 83 e5  
01 4c 89 ee e8 43 ea 67 ff 4d 85 ed 74 07 e8 09 e9 67 ff <0f> 0b e8 02 e9  
67 ff 8b b5 34 ff ff ff 44 89 fa 4c 89 e7 e8 91 3a
RSP: 0018:ffff88018590ec78 EFLAGS: 00010212
RAX: 0000000000040000 RBX: 1ffff10030b21d91 RCX: ffffc9000dae4000
RDX: 0000000000000ece RSI: ffffffff8216ec87 RDI: 0000000000000007
RBP: ffff88018590ed50 R08: ffff880183018300 R09: fffff94000ceffc6
R10: fffff94000ceffc6 R11: ffffea000677fe33 R12: ffffea000677fe00
R13: 0000000000000001 R14: ffffea000677fe08 R15: 0000000000001000
  do_invalidatepage mm/truncate.c:165 [inline]
  truncate_cleanup_page+0x5ac/0xa90 mm/truncate.c:187
  truncate_inode_page+0x107/0x1a0 mm/truncate.c:229
  truncate_inode_pages_range+0x1382/0x2d50 mm/truncate.c:451
  truncate_inode_pages+0x24/0x30 mm/truncate.c:478
  swap_inode_boot_loader fs/ext4/ioctl.c:123 [inline]
  ext4_ioctl+0x1e3b/0x4210 fs/ext4/ioctl.c:865
  vfs_ioctl fs/ioctl.c:46 [inline]
  file_ioctl fs/ioctl.c:501 [inline]
  do_vfs_ioctl+0x1de/0x1720 fs/ioctl.c:685
  ksys_ioctl+0xa9/0xd0 fs/ioctl.c:702
  __do_sys_ioctl fs/ioctl.c:709 [inline]
  __se_sys_ioctl fs/ioctl.c:707 [inline]
  __x64_sys_ioctl+0x73/0xb0 fs/ioctl.c:707
  do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290
  entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x457579
Code: 1d b4 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7  
48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff  
ff 0f 83 eb b3 fb ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007fa151655c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000457579
RDX: 0000000020000000 RSI: 0000000000006611 RDI: 0000000000000006
RBP: 000000000072bfa0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007fa1516566d4
R13: 00000000004bf60e R14: 00000000004cf4a8 R15: 00000000ffffffff
Kernel Offset: disabled
Rebooting in 86400 seconds..


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with  
syzbot.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: WARNING in ext4_invalidatepage
  2018-10-08 16:18 WARNING in ext4_invalidatepage syzbot
@ 2018-10-08 16:29 ` Dmitry Vyukov
  2018-10-09  1:34   ` Theodore Y. Ts'o
  0 siblings, 1 reply; 9+ messages in thread
From: Dmitry Vyukov @ 2018-10-08 16:29 UTC (permalink / raw)
  To: syzbot
  Cc: Andreas Dilger, linux-ext4, LKML, syzkaller-bugs, Theodore Ts'o

On Mon, Oct 8, 2018 at 6:18 PM, syzbot
<syzbot+85da7ac734f7ba432ee4@syzkaller.appspotmail.com> wrote:
> Hello,
>
> syzbot found the following crash on:
>
> HEAD commit:    c1d84a1b42ef Merge git://git.kernel.org/pub/scm/linux/kern..
> git tree:       upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=161588d6400000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=c0af03fe452b65fb
> dashboard link: https://syzkaller.appspot.com/bug?extid=85da7ac734f7ba432ee4
> compiler:       gcc (GCC) 8.0.1 20180413 (experimental)
>
> Unfortunately, I don't have any reproducer for this crash yet.
>
> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> Reported-by: syzbot+85da7ac734f7ba432ee4@syzkaller.appspotmail.com


The program that triggered it did the following:

05:23:28 executing program 5:
r0 = creat(&(0x7f00000001c0)='./file0\x00', 0x0)
socketpair$unix(0x1, 0x1, 0x0, &(0x7f0000000380)={0xffffffffffffffff,
<r1=>0xffffffffffffffff})
write$RDMA_USER_CM_CMD_CREATE_ID(r0, &(0x7f0000000240)={0x0, 0x18,
0xfa00, {0x0, &(0x7f0000000200)}}, 0x20)
ioctl$PERF_EVENT_IOC_ENABLE(r1, 0x8912, 0x400200)
ioctl$EXT4_IOC_SETFLAGS(r0, 0x4008660f, &(0x7f0000000000)=0x4000)

Seems like some bad interaction of unix sockets and ext4.


> WARNING: CPU: 1 PID: 17203 at fs/ext4/inode.c:3353
> ext4_invalidatepage+0x1c7/0x5b0 fs/ext4/inode.c:3353
> Kernel panic - not syncing: panic_on_warn set ...
>
> CPU: 1 PID: 17203 Comm: syz-executor5 Not tainted 4.19.0-rc6+ #48
> kobject: 'loop3' (000000002393f9b0): kobject_uevent_env
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
> Google 01/01/2011
> Call Trace:
>  __dump_stack lib/dump_stack.c:77 [inline]
>  dump_stack+0x1c4/0x2b4 lib/dump_stack.c:113
> kobject: 'loop3' (000000002393f9b0): fill_kobj_path: path =
> '/devices/virtual/block/loop3'
>  panic+0x238/0x4e7 kernel/panic.c:184
>  __warn.cold.8+0x163/0x1ba kernel/panic.c:536
>  report_bug+0x254/0x2d0 lib/bug.c:186
>  fixup_bug arch/x86/kernel/traps.c:178 [inline]
>  do_error_trap+0x1fc/0x4d0 arch/x86/kernel/traps.c:296
>  do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:316
>  invalid_op+0x14/0x20 arch/x86/entry/entry_64.S:993
> RIP: 0010:ext4_invalidatepage+0x1c7/0x5b0 fs/ext4/inode.c:3353
> Code: 80 3c 02 00 0f 85 a7 03 00 00 4d 8b 6d 00 31 ff 49 c1 ed 11 41 83 e5
> 01 4c 89 ee e8 43 ea 67 ff 4d 85 ed 74 07 e8 09 e9 67 ff <0f> 0b e8 02 e9 67
> ff 8b b5 34 ff ff ff 44 89 fa 4c 89 e7 e8 91 3a
> RSP: 0018:ffff88018590ec78 EFLAGS: 00010212
> RAX: 0000000000040000 RBX: 1ffff10030b21d91 RCX: ffffc9000dae4000
> RDX: 0000000000000ece RSI: ffffffff8216ec87 RDI: 0000000000000007
> RBP: ffff88018590ed50 R08: ffff880183018300 R09: fffff94000ceffc6
> R10: fffff94000ceffc6 R11: ffffea000677fe33 R12: ffffea000677fe00
> R13: 0000000000000001 R14: ffffea000677fe08 R15: 0000000000001000
>  do_invalidatepage mm/truncate.c:165 [inline]
>  truncate_cleanup_page+0x5ac/0xa90 mm/truncate.c:187
>  truncate_inode_page+0x107/0x1a0 mm/truncate.c:229
>  truncate_inode_pages_range+0x1382/0x2d50 mm/truncate.c:451
>  truncate_inode_pages+0x24/0x30 mm/truncate.c:478
>  swap_inode_boot_loader fs/ext4/ioctl.c:123 [inline]
>  ext4_ioctl+0x1e3b/0x4210 fs/ext4/ioctl.c:865
>  vfs_ioctl fs/ioctl.c:46 [inline]
>  file_ioctl fs/ioctl.c:501 [inline]
>  do_vfs_ioctl+0x1de/0x1720 fs/ioctl.c:685
>  ksys_ioctl+0xa9/0xd0 fs/ioctl.c:702
>  __do_sys_ioctl fs/ioctl.c:709 [inline]
>  __se_sys_ioctl fs/ioctl.c:707 [inline]
>  __x64_sys_ioctl+0x73/0xb0 fs/ioctl.c:707
>  do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290
>  entry_SYSCALL_64_after_hwframe+0x49/0xbe
> RIP: 0033:0x457579
> Code: 1d b4 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7
> 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff
> 0f 83 eb b3 fb ff c3 66 2e 0f 1f 84 00 00 00 00
> RSP: 002b:00007fa151655c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
> RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000457579
> RDX: 0000000020000000 RSI: 0000000000006611 RDI: 0000000000000006
> RBP: 000000000072bfa0 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000246 R12: 00007fa1516566d4
> R13: 00000000004bf60e R14: 00000000004cf4a8 R15: 00000000ffffffff
> Kernel Offset: disabled
> Rebooting in 86400 seconds..
>
>
> ---
> This bug is generated by a bot. It may contain errors.
> See https://goo.gl/tpsmEJ for more information about syzbot.
> syzbot engineers can be reached at syzkaller@googlegroups.com.
>
> syzbot will keep track of this bug report. See:
> https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with
> syzbot.
>
> --
> You received this message because you are subscribed to the Google Groups
> "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to syzkaller-bugs+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/syzkaller-bugs/0000000000002753c60577b9f707%40google.com.
> For more options, visit https://groups.google.com/d/optout.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: WARNING in ext4_invalidatepage
  2018-10-08 16:29 ` Dmitry Vyukov
@ 2018-10-09  1:34   ` Theodore Y. Ts'o
  2018-10-15 13:22     ` Dmitry Vyukov
  0 siblings, 1 reply; 9+ messages in thread
From: Theodore Y. Ts'o @ 2018-10-09  1:34 UTC (permalink / raw)
  To: Dmitry Vyukov; +Cc: syzbot, Andreas Dilger, linux-ext4, LKML, syzkaller-bugs

On Mon, Oct 08, 2018 at 06:29:54PM +0200, Dmitry Vyukov wrote:
> 
> The program that triggered it did the following:
> 
> 05:23:28 executing program 5:
> r0 = creat(&(0x7f00000001c0)='./file0\x00', 0x0)
> socketpair$unix(0x1, 0x1, 0x0, &(0x7f0000000380)={0xffffffffffffffff,
> <r1=>0xffffffffffffffff})
> write$RDMA_USER_CM_CMD_CREATE_ID(r0, &(0x7f0000000240)={0x0, 0x18,
> 0xfa00, {0x0, &(0x7f0000000200)}}, 0x20)

This looks like it's doing an ioctl-like thing which is now restricted
to root --- it looks like people can do arbitrary stupid things with
it?

https://www.openwall.com/lists/oss-security/2016/05/09/11

> ioctl$PERF_EVENT_IOC_ENABLE(r1, 0x8912, 0x400200)
> ioctl$EXT4_IOC_SETFLAGS(r0, 0x4008660f, &(0x7f0000000000)=0x4000)

Um, this doesn't seem to correspond to the stack trace:

> >  do_invalidatepage mm/truncate.c:165 [inline]
> >  truncate_cleanup_page+0x5ac/0xa90 mm/truncate.c:187
> >  truncate_inode_page+0x107/0x1a0 mm/truncate.c:229
> >  truncate_inode_pages_range+0x1382/0x2d50 mm/truncate.c:451
> >  truncate_inode_pages+0x24/0x30 mm/truncate.c:478
> >  swap_inode_boot_loader fs/ext4/ioctl.c:123 [inline]
> >  ext4_ioctl+0x1e3b/0x4210 fs/ext4/ioctl.c:865
> >  vfs_ioctl fs/ioctl.c:46 [inline]
> >  file_ioctl fs/ioctl.c:501 [inline]
> >  do_vfs_ioctl+0x1de/0x1720 fs/ioctl.c:685
> >  ksys_ioctl+0xa9/0xd0 fs/ioctl.c:702
> >  __do_sys_ioctl fs/ioctl.c:709 [inline]
> >  __se_sys_ioctl fs/ioctl.c:707 [inline]
> >  __x64_sys_ioctl+0x73/0xb0 fs/ioctl.c:707
> >  do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290
> >  entry_SYSCALL_64_after_hwframe+0x49/0xbe

This looks like the program that triggered the crash was running
EXT4_IOC_SWAP_BOOT.  It may have been racing against something that's
was using RDMA, but that's not clear at all, since the write command
appears to be some weird ioctl-like interface that requires root.

It's too bad that syzbot wasn't able to come up with a clean
reproducer.  I've been doing some work to robustify the
EXT4_IOC_SWAP_ROOT.  If we had a reproducer I'd see if this patch
would address it.

	http://patchwork.ozlabs.org/patch/978083/

      	      	       	    	    	    - Ted


> > WARNING: CPU: 1 PID: 17203 at fs/ext4/inode.c:3353
> > ext4_invalidatepage+0x1c7/0x5b0 fs/ext4/inode.c:3353
> > Kernel panic - not syncing: panic_on_warn set ...
> >
> > CPU: 1 PID: 17203 Comm: syz-executor5 Not tainted 4.19.0-rc6+ #48
> > kobject: 'loop3' (000000002393f9b0): kobject_uevent_env
> > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
> > Google 01/01/2011
> > Call Trace:
> >  __dump_stack lib/dump_stack.c:77 [inline]
> >  dump_stack+0x1c4/0x2b4 lib/dump_stack.c:113
> > kobject: 'loop3' (000000002393f9b0): fill_kobj_path: path =
> > '/devices/virtual/block/loop3'
> >  panic+0x238/0x4e7 kernel/panic.c:184
> >  __warn.cold.8+0x163/0x1ba kernel/panic.c:536
> >  report_bug+0x254/0x2d0 lib/bug.c:186
> >  fixup_bug arch/x86/kernel/traps.c:178 [inline]
> >  do_error_trap+0x1fc/0x4d0 arch/x86/kernel/traps.c:296
> >  do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:316
> >  invalid_op+0x14/0x20 arch/x86/entry/entry_64.S:993
> > RIP: 0010:ext4_invalidatepage+0x1c7/0x5b0 fs/ext4/inode.c:3353
> > Code: 80 3c 02 00 0f 85 a7 03 00 00 4d 8b 6d 00 31 ff 49 c1 ed 11 41 83 e5
> > 01 4c 89 ee e8 43 ea 67 ff 4d 85 ed 74 07 e8 09 e9 67 ff <0f> 0b e8 02 e9 67
> > ff 8b b5 34 ff ff ff 44 89 fa 4c 89 e7 e8 91 3a
> > RSP: 0018:ffff88018590ec78 EFLAGS: 00010212
> > RAX: 0000000000040000 RBX: 1ffff10030b21d91 RCX: ffffc9000dae4000
> > RDX: 0000000000000ece RSI: ffffffff8216ec87 RDI: 0000000000000007
> > RBP: ffff88018590ed50 R08: ffff880183018300 R09: fffff94000ceffc6
> > R10: fffff94000ceffc6 R11: ffffea000677fe33 R12: ffffea000677fe00
> > R13: 0000000000000001 R14: ffffea000677fe08 R15: 0000000000001000
> >  do_invalidatepage mm/truncate.c:165 [inline]
> >  truncate_cleanup_page+0x5ac/0xa90 mm/truncate.c:187
> >  truncate_inode_page+0x107/0x1a0 mm/truncate.c:229
> >  truncate_inode_pages_range+0x1382/0x2d50 mm/truncate.c:451
> >  truncate_inode_pages+0x24/0x30 mm/truncate.c:478
> >  swap_inode_boot_loader fs/ext4/ioctl.c:123 [inline]
> >  ext4_ioctl+0x1e3b/0x4210 fs/ext4/ioctl.c:865
> >  vfs_ioctl fs/ioctl.c:46 [inline]
> >  file_ioctl fs/ioctl.c:501 [inline]
> >  do_vfs_ioctl+0x1de/0x1720 fs/ioctl.c:685
> >  ksys_ioctl+0xa9/0xd0 fs/ioctl.c:702
> >  __do_sys_ioctl fs/ioctl.c:709 [inline]
> >  __se_sys_ioctl fs/ioctl.c:707 [inline]
> >  __x64_sys_ioctl+0x73/0xb0 fs/ioctl.c:707
> >  do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290
> >  entry_SYSCALL_64_after_hwframe+0x49/0xbe
> > RIP: 0033:0x457579
> > Code: 1d b4 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7
> > 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff
> > 0f 83 eb b3 fb ff c3 66 2e 0f 1f 84 00 00 00 00
> > RSP: 002b:00007fa151655c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
> > RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000457579
> > RDX: 0000000020000000 RSI: 0000000000006611 RDI: 0000000000000006
> > RBP: 000000000072bfa0 R08: 0000000000000000 R09: 0000000000000000
> > R10: 0000000000000000 R11: 0000000000000246 R12: 00007fa1516566d4
> > R13: 00000000004bf60e R14: 00000000004cf4a8 R15: 00000000ffffffff
> > Kernel Offset: disabled
> > Rebooting in 86400 seconds..
> >
> >
> > ---
> > This bug is generated by a bot. It may contain errors.
> > See https://goo.gl/tpsmEJ for more information about syzbot.
> > syzbot engineers can be reached at syzkaller@googlegroups.com.
> >
> > syzbot will keep track of this bug report. See:
> > https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with
> > syzbot.
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "syzkaller-bugs" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to syzkaller-bugs+unsubscribe@googlegroups.com.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/syzkaller-bugs/0000000000002753c60577b9f707%40google.com.
> > For more options, visit https://groups.google.com/d/optout.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: WARNING in ext4_invalidatepage
  2018-10-09  1:34   ` Theodore Y. Ts'o
@ 2018-10-15 13:22     ` Dmitry Vyukov
  2018-10-15 18:08       ` Theodore Y. Ts'o
  0 siblings, 1 reply; 9+ messages in thread
From: Dmitry Vyukov @ 2018-10-15 13:22 UTC (permalink / raw)
  To: Theodore Y. Ts'o, Dmitry Vyukov, syzbot, Andreas Dilger,
	linux-ext4, LKML, syzkaller-bugs

On Tue, Oct 9, 2018 at 3:34 AM, Theodore Y. Ts'o <tytso@mit.edu> wrote:
> On Mon, Oct 08, 2018 at 06:29:54PM +0200, Dmitry Vyukov wrote:
>>
>> The program that triggered it did the following:
>>
>> 05:23:28 executing program 5:
>> r0 = creat(&(0x7f00000001c0)='./file0\x00', 0x0)
>> socketpair$unix(0x1, 0x1, 0x0, &(0x7f0000000380)={0xffffffffffffffff,
>> <r1=>0xffffffffffffffff})
>> write$RDMA_USER_CM_CMD_CREATE_ID(r0, &(0x7f0000000240)={0x0, 0x18,
>> 0xfa00, {0x0, &(0x7f0000000200)}}, 0x20)
>
> This looks like it's doing an ioctl-like thing which is now restricted
> to root --- it looks like people can do arbitrary stupid things with
> it?
>
> https://www.openwall.com/lists/oss-security/2016/05/09/11
>
>> ioctl$PERF_EVENT_IOC_ENABLE(r1, 0x8912, 0x400200)
>> ioctl$EXT4_IOC_SETFLAGS(r0, 0x4008660f, &(0x7f0000000000)=0x4000)
>
> Um, this doesn't seem to correspond to the stack trace:
>
>> >  do_invalidatepage mm/truncate.c:165 [inline]
>> >  truncate_cleanup_page+0x5ac/0xa90 mm/truncate.c:187
>> >  truncate_inode_page+0x107/0x1a0 mm/truncate.c:229
>> >  truncate_inode_pages_range+0x1382/0x2d50 mm/truncate.c:451
>> >  truncate_inode_pages+0x24/0x30 mm/truncate.c:478
>> >  swap_inode_boot_loader fs/ext4/ioctl.c:123 [inline]
>> >  ext4_ioctl+0x1e3b/0x4210 fs/ext4/ioctl.c:865
>> >  vfs_ioctl fs/ioctl.c:46 [inline]
>> >  file_ioctl fs/ioctl.c:501 [inline]
>> >  do_vfs_ioctl+0x1de/0x1720 fs/ioctl.c:685
>> >  ksys_ioctl+0xa9/0xd0 fs/ioctl.c:702
>> >  __do_sys_ioctl fs/ioctl.c:709 [inline]
>> >  __se_sys_ioctl fs/ioctl.c:707 [inline]
>> >  __x64_sys_ioctl+0x73/0xb0 fs/ioctl.c:707
>> >  do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290
>> >  entry_SYSCALL_64_after_hwframe+0x49/0xbe
>
> This looks like the program that triggered the crash was running
> EXT4_IOC_SWAP_BOOT.  It may have been racing against something that's
> was using RDMA, but that's not clear at all, since the write command
> appears to be some weird ioctl-like interface that requires root.
>
> It's too bad that syzbot wasn't able to come up with a clean
> reproducer.  I've been doing some work to robustify the
> EXT4_IOC_SWAP_ROOT.  If we had a reproducer I'd see if this patch
> would address it.
>
>         http://patchwork.ozlabs.org/patch/978083/



Hi Ted,

The RDMA thing seems to be red-herring, as it writes RDMA command into
a normal local file. These commands will only take effect if written
to /dev/infiniband/rdma_cm file.

Now that you mention EXT4_IOC_SWAP_BOOT, I think I looked at the wrong
program, there is a subsequent one that does ioctl(0x6611) where
0x6611 is in fact EXT4_IOC_SWAP_BOOT. So I think it's this one:

05:23:28 executing program 5:
r0 = creat(&(0x7f00000001c0)='./file0\x00', 0x0)
socketpair$unix(0x1, 0x1, 0x0, &(0x7f0000000380)={0xffffffffffffffff,
<r1=>0xffffffffffffffff})
write$RDMA_USER_CM_CMD_CREATE_ID(r0, &(0x7f0000000240)={0x0, 0x18,
0xfa00, {0x0, &(0x7f0000000200)}}, 0x20)
ioctl$PERF_EVENT_IOC_ENABLE(r1, 0x8912, 0x400200)
ioctl$EXT4_IOC_SETFLAGS(r0, 0x6611, &(0x7f0000000000)=0x4000)


I've tried to manually reply this program and the whole log too, but
it does not reproduce. This may be related to the fact that filesystem
accumulates too much global state, so probably first relevant part
happened long time ago, and then second relevant part happened later
and triggered the warning. But just re-doing the second part does not
reproduce the bug.
Unfortunately it's the reality of kernel testing. Maybe syzbot will
come up with repro later, or maybe we will see that it does not happen
anymore (fixed by your patch) and we will close it then.



>> > WARNING: CPU: 1 PID: 17203 at fs/ext4/inode.c:3353
>> > ext4_invalidatepage+0x1c7/0x5b0 fs/ext4/inode.c:3353
>> > Kernel panic - not syncing: panic_on_warn set ...
>> >
>> > CPU: 1 PID: 17203 Comm: syz-executor5 Not tainted 4.19.0-rc6+ #48
>> > kobject: 'loop3' (000000002393f9b0): kobject_uevent_env
>> > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
>> > Google 01/01/2011
>> > Call Trace:
>> >  __dump_stack lib/dump_stack.c:77 [inline]
>> >  dump_stack+0x1c4/0x2b4 lib/dump_stack.c:113
>> > kobject: 'loop3' (000000002393f9b0): fill_kobj_path: path =
>> > '/devices/virtual/block/loop3'
>> >  panic+0x238/0x4e7 kernel/panic.c:184
>> >  __warn.cold.8+0x163/0x1ba kernel/panic.c:536
>> >  report_bug+0x254/0x2d0 lib/bug.c:186
>> >  fixup_bug arch/x86/kernel/traps.c:178 [inline]
>> >  do_error_trap+0x1fc/0x4d0 arch/x86/kernel/traps.c:296
>> >  do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:316
>> >  invalid_op+0x14/0x20 arch/x86/entry/entry_64.S:993
>> > RIP: 0010:ext4_invalidatepage+0x1c7/0x5b0 fs/ext4/inode.c:3353
>> > Code: 80 3c 02 00 0f 85 a7 03 00 00 4d 8b 6d 00 31 ff 49 c1 ed 11 41 83 e5
>> > 01 4c 89 ee e8 43 ea 67 ff 4d 85 ed 74 07 e8 09 e9 67 ff <0f> 0b e8 02 e9 67
>> > ff 8b b5 34 ff ff ff 44 89 fa 4c 89 e7 e8 91 3a
>> > RSP: 0018:ffff88018590ec78 EFLAGS: 00010212
>> > RAX: 0000000000040000 RBX: 1ffff10030b21d91 RCX: ffffc9000dae4000
>> > RDX: 0000000000000ece RSI: ffffffff8216ec87 RDI: 0000000000000007
>> > RBP: ffff88018590ed50 R08: ffff880183018300 R09: fffff94000ceffc6
>> > R10: fffff94000ceffc6 R11: ffffea000677fe33 R12: ffffea000677fe00
>> > R13: 0000000000000001 R14: ffffea000677fe08 R15: 0000000000001000
>> >  do_invalidatepage mm/truncate.c:165 [inline]
>> >  truncate_cleanup_page+0x5ac/0xa90 mm/truncate.c:187
>> >  truncate_inode_page+0x107/0x1a0 mm/truncate.c:229
>> >  truncate_inode_pages_range+0x1382/0x2d50 mm/truncate.c:451
>> >  truncate_inode_pages+0x24/0x30 mm/truncate.c:478
>> >  swap_inode_boot_loader fs/ext4/ioctl.c:123 [inline]
>> >  ext4_ioctl+0x1e3b/0x4210 fs/ext4/ioctl.c:865
>> >  vfs_ioctl fs/ioctl.c:46 [inline]
>> >  file_ioctl fs/ioctl.c:501 [inline]
>> >  do_vfs_ioctl+0x1de/0x1720 fs/ioctl.c:685
>> >  ksys_ioctl+0xa9/0xd0 fs/ioctl.c:702
>> >  __do_sys_ioctl fs/ioctl.c:709 [inline]
>> >  __se_sys_ioctl fs/ioctl.c:707 [inline]
>> >  __x64_sys_ioctl+0x73/0xb0 fs/ioctl.c:707
>> >  do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290
>> >  entry_SYSCALL_64_after_hwframe+0x49/0xbe
>> > RIP: 0033:0x457579
>> > Code: 1d b4 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7
>> > 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff
>> > 0f 83 eb b3 fb ff c3 66 2e 0f 1f 84 00 00 00 00
>> > RSP: 002b:00007fa151655c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
>> > RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000457579
>> > RDX: 0000000020000000 RSI: 0000000000006611 RDI: 0000000000000006
>> > RBP: 000000000072bfa0 R08: 0000000000000000 R09: 0000000000000000
>> > R10: 0000000000000000 R11: 0000000000000246 R12: 00007fa1516566d4
>> > R13: 00000000004bf60e R14: 00000000004cf4a8 R15: 00000000ffffffff
>> > Kernel Offset: disabled
>> > Rebooting in 86400 seconds..
>> >
>> >
>> > ---
>> > This bug is generated by a bot. It may contain errors.
>> > See https://goo.gl/tpsmEJ for more information about syzbot.
>> > syzbot engineers can be reached at syzkaller@googlegroups.com.
>> >
>> > syzbot will keep track of this bug report. See:
>> > https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with
>> > syzbot.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: WARNING in ext4_invalidatepage
  2018-10-15 13:22     ` Dmitry Vyukov
@ 2018-10-15 18:08       ` Theodore Y. Ts'o
  2018-10-16 14:02         ` Dmitry Vyukov
  0 siblings, 1 reply; 9+ messages in thread
From: Theodore Y. Ts'o @ 2018-10-15 18:08 UTC (permalink / raw)
  To: Dmitry Vyukov; +Cc: syzbot, Andreas Dilger, linux-ext4, LKML, syzkaller-bugs

On Mon, Oct 15, 2018 at 03:22:42PM +0200, Dmitry Vyukov wrote:
> Now that you mention EXT4_IOC_SWAP_BOOT, I think I looked at the wrong
> program, there is a subsequent one that does ioctl(0x6611) where
> 0x6611 is in fact EXT4_IOC_SWAP_BOOT. So I think it's this one:
> 
> 05:23:28 executing program 5:
> r0 = creat(&(0x7f00000001c0)='./file0\x00', 0x0)
> socketpair$unix(0x1, 0x1, 0x0, &(0x7f0000000380)={0xffffffffffffffff,
> <r1=>0xffffffffffffffff})
> write$RDMA_USER_CM_CMD_CREATE_ID(r0, &(0x7f0000000240)={0x0, 0x18,
> 0xfa00, {0x0, &(0x7f0000000200)}}, 0x20)
> ioctl$PERF_EVENT_IOC_ENABLE(r1, 0x8912, 0x400200)
> ioctl$EXT4_IOC_SETFLAGS(r0, 0x6611, &(0x7f0000000000)=0x4000)

Ah, so is it a bug in Syzkaller that it is printing
ioctl$EXT4_IOC_SETFLAGS when 0x6611 is in fact EXT4_IOC_SWAP_BOOT,
right?

> I've tried to manually reply this program and the whole log too, but
> it does not reproduce. This may be related to the fact that filesystem
> accumulates too much global state, so probably first relevant part
> happened long time ago, and then second relevant part happened later
> and triggered the warning. But just re-doing the second part does not
> reproduce the bug.

It was probably some other process racing with EXT4_IOC_SWAP_BOOT.
The patch I referenced in my previous e-mail protects against
additional scenarios where someone might be trying to punch a whole
into a file that is being swapped into the bootloader ioctl.  This
particular ioctl isn't yet being used by anyone, so it had some other
issues as well, such as not interacting well with inline_data-enabled
file systems --- not that any bootloader would be small enough that it
would fit in an inline_data inode, but we're basically proofing the
code against a malicious (or buggy) root-privileged program... such as
syzbot.  :-)

						- Ted

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: WARNING in ext4_invalidatepage
  2018-10-15 18:08       ` Theodore Y. Ts'o
@ 2018-10-16 14:02         ` Dmitry Vyukov
  2018-10-16 15:53           ` Theodore Y. Ts'o
  0 siblings, 1 reply; 9+ messages in thread
From: Dmitry Vyukov @ 2018-10-16 14:02 UTC (permalink / raw)
  To: Theodore Y. Ts'o, Dmitry Vyukov, syzbot, Andreas Dilger,
	linux-ext4, LKML, syzkaller-bugs

On Mon, Oct 15, 2018 at 8:08 PM, Theodore Y. Ts'o <tytso@mit.edu> wrote:
> On Mon, Oct 15, 2018 at 03:22:42PM +0200, Dmitry Vyukov wrote:
>> Now that you mention EXT4_IOC_SWAP_BOOT, I think I looked at the wrong
>> program, there is a subsequent one that does ioctl(0x6611) where
>> 0x6611 is in fact EXT4_IOC_SWAP_BOOT. So I think it's this one:
>>
>> 05:23:28 executing program 5:
>> r0 = creat(&(0x7f00000001c0)='./file0\x00', 0x0)
>> socketpair$unix(0x1, 0x1, 0x0, &(0x7f0000000380)={0xffffffffffffffff,
>> <r1=>0xffffffffffffffff})
>> write$RDMA_USER_CM_CMD_CREATE_ID(r0, &(0x7f0000000240)={0x0, 0x18,
>> 0xfa00, {0x0, &(0x7f0000000200)}}, 0x20)
>> ioctl$PERF_EVENT_IOC_ENABLE(r1, 0x8912, 0x400200)
>> ioctl$EXT4_IOC_SETFLAGS(r0, 0x6611, &(0x7f0000000000)=0x4000)
>
> Ah, so is it a bug in Syzkaller that it is printing
> ioctl$EXT4_IOC_SETFLAGS when 0x6611 is in fact EXT4_IOC_SWAP_BOOT,
> right?

I am not sure how exactly this should be classified. To significant
degree these "$FOO" discriminations are informational and only really
affect the data format passed in. For ioctl/write it's not really
possible to know what exactly it will do if you just open a file
(which can be a link to into an overlay mount pointing to some special
file). Sometimes we also want to specifically spoof exact fd type (or
other resource type) for a syscall. Sometimes we discover other ioctl
command constants while running an ioctl, and then we just change the
command constant but leave the data as is.


>> I've tried to manually reply this program and the whole log too, but
>> it does not reproduce. This may be related to the fact that filesystem
>> accumulates too much global state, so probably first relevant part
>> happened long time ago, and then second relevant part happened later
>> and triggered the warning. But just re-doing the second part does not
>> reproduce the bug.
>
> It was probably some other process racing with EXT4_IOC_SWAP_BOOT.
> The patch I referenced in my previous e-mail protects against
> additional scenarios where someone might be trying to punch a whole
> into a file that is being swapped into the bootloader ioctl.  This
> particular ioctl isn't yet being used by anyone, so it had some other
> issues as well, such as not interacting well with inline_data-enabled
> file systems --- not that any bootloader would be small enough that it
> would fit in an inline_data inode, but we're basically proofing the
> code against a malicious (or buggy) root-privileged program... such as
> syzbot.  :-)

... or paving the way to opening all of this to non-root users. Why
not if not bugs? ;)


Back to this bug, I think we should wait to see if it happens more in
future and if syzkaller can come up with a repro. If it won't happen
more (perhaps fixed by your patch), then we will close it as obsolete.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: WARNING in ext4_invalidatepage
  2018-10-16 14:02         ` Dmitry Vyukov
@ 2018-10-16 15:53           ` Theodore Y. Ts'o
  2018-10-30 12:09             ` Dmitry Vyukov
  2018-10-30 12:16             ` Dmitry Vyukov
  0 siblings, 2 replies; 9+ messages in thread
From: Theodore Y. Ts'o @ 2018-10-16 15:53 UTC (permalink / raw)
  To: Dmitry Vyukov; +Cc: syzbot, Andreas Dilger, linux-ext4, LKML, syzkaller-bugs

On Tue, Oct 16, 2018 at 04:02:07PM +0200, Dmitry Vyukov wrote:
> I am not sure how exactly this should be classified. To significant
> degree these "$FOO" discriminations are informational and only really
> affect the data format passed in.

The problem is that putting something which is simply and plainly
*wrong* is going to be actively misleading.  You *have* to know what
ioctl you are trying to be trying to execute because some of them
require input data structures that you have to fill in, or that it
requires a file descriptor as opposed to an integer or a pointer to a
struct.  I assume you're not just picking ioctl numbers purely at
random, right?

So I assume you had a table that said, this ioctl, 0x6611 takes a file
descriptor, and has thus-and-so-a-name.  If that name is wrong, I'd
say it's pretty clearly a bug, right?

(I'll again gently point out that a tiny amount of work in making
syzkaller a tiny bit more developer toil would make it much more
effective, since all of this extra developer toil --- now I know I
can't even trust the $FOO discriminators to be right --- makes
syzkaller less scalable by pursuading developers that it's not
worthwhile to pay attention...)

> > The patch I referenced in my previous e-mail protects against
> > additional scenarios where someone might be trying to punch a whole
> > into a file that is being swapped into the bootloader ioctl.  This
> > particular ioctl isn't yet being used by anyone, so it had some other
> > issues as well, such as not interacting well with inline_data-enabled
> > file systems --- not that any bootloader would be small enough that it
> > would fit in an inline_data inode, but we're basically proofing the
> > code against a malicious (or buggy) root-privileged program... such as
> > syzbot.  :-)
> 
> ... or paving the way to opening all of this to non-root users. Why
> not if not bugs? ;)

The intent behind this particular ioctl is used to install a boot
loader.  It will *never* be opened to non-root users.  It doesn't even
make sense to make it available to pseudo-containers-root users.  :-)

							- Ted

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: WARNING in ext4_invalidatepage
  2018-10-16 15:53           ` Theodore Y. Ts'o
@ 2018-10-30 12:09             ` Dmitry Vyukov
  2018-10-30 12:16             ` Dmitry Vyukov
  1 sibling, 0 replies; 9+ messages in thread
From: Dmitry Vyukov @ 2018-10-30 12:09 UTC (permalink / raw)
  To: Theodore Y. Ts'o, Dmitry Vyukov, syzbot, Andreas Dilger,
	linux-ext4, LKML, syzkaller-bugs

On Tue, Oct 16, 2018 at 5:53 PM, Theodore Y. Ts'o <tytso@mit.edu> wrote:
> On Tue, Oct 16, 2018 at 04:02:07PM +0200, Dmitry Vyukov wrote:
>> I am not sure how exactly this should be classified. To significant
>> degree these "$FOO" discriminations are informational and only really
>> affect the data format passed in.
>
> The problem is that putting something which is simply and plainly
> *wrong* is going to be actively misleading.  You *have* to know what
> ioctl you are trying to be trying to execute because some of them
> require input data structures that you have to fill in, or that it
> requires a file descriptor as opposed to an integer or a pointer to a
> struct.  I assume you're not just picking ioctl numbers purely at
> random, right?
>
> So I assume you had a table that said, this ioctl, 0x6611 takes a file
> descriptor, and has thus-and-so-a-name.  If that name is wrong, I'd
> say it's pretty clearly a bug, right?

We have such table and it is correct. But it is legal and fine and
actually useful in context of testing to take an ioctl that takes a
pointer to one structure and give it pointer to another structure, or
plain int or just anything else. $FOO states what data layout is used
and actual cmd number and fd type what command is executed.

> (I'll again gently point out that a tiny amount of work in making
> syzkaller a tiny bit more developer toil would make it much more
> effective, since all of this extra developer toil --- now I know I
> can't even trust the $FOO discriminators to be right --- makes
> syzkaller less scalable by pursuading developers that it's not
> worthwhile to pay attention...)

Ted, I appreciate your suggestions. And the new table-structured email
format that you proposed is great.
But there are reasons for the current syzkaller program format. It's
not just there is a typo in some table that we need to fix. The $FOO
part encodes data layout and treatment of argument values (e.g. if
0xab actually takes a byte or a word, and what's it alignment). It's
not something that can be restored from anything else. So if we lose
this information, the program will useless and confusing in other
ways. For example, consider an ioctl accepts a struct with 2 byte
fields; but we actually executed it passing a struct with 2 int
fields. Now if you look at {0x1, 0x2} you will assume that these are
the byte values, but in reality it's 0x1 int which covered both 1-byte
arguments.
Also consider that ioctl cmd is actually meaningless in itself, and
it's generally not possible to statically know what exactly ioctl will
be executed. The crucial piece is fd type. I guess it boils down to
the halting problem to statically prove what fd type we will have in a
random program. Even worse, we can have a race on fd type, so it's
non-deterministic and even not detectable dynamically. Taking this
into account any attempts to preserve matching $FOO and cmd number are
pointless.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: WARNING in ext4_invalidatepage
  2018-10-16 15:53           ` Theodore Y. Ts'o
  2018-10-30 12:09             ` Dmitry Vyukov
@ 2018-10-30 12:16             ` Dmitry Vyukov
  1 sibling, 0 replies; 9+ messages in thread
From: Dmitry Vyukov @ 2018-10-30 12:16 UTC (permalink / raw)
  To: Theodore Y. Ts'o, Dmitry Vyukov, syzbot, Andreas Dilger,
	linux-ext4, LKML, syzkaller-bugs

On Tue, Oct 16, 2018 at 5:53 PM, Theodore Y. Ts'o <tytso@mit.edu> wrote:
>> > The patch I referenced in my previous e-mail protects against
>> > additional scenarios where someone might be trying to punch a whole
>> > into a file that is being swapped into the bootloader ioctl.  This
>> > particular ioctl isn't yet being used by anyone, so it had some other
>> > issues as well, such as not interacting well with inline_data-enabled
>> > file systems --- not that any bootloader would be small enough that it
>> > would fit in an inline_data inode, but we're basically proofing the
>> > code against a malicious (or buggy) root-privileged program... such as
>> > syzbot.  :-)
>>
>> ... or paving the way to opening all of this to non-root users. Why
>> not if not bugs? ;)
>
> The intent behind this particular ioctl is used to install a boot
> loader.  It will *never* be opened to non-root users.  It doesn't even
> make sense to make it available to pseudo-containers-root users.  :-)

But it does not have to be executed on the root fs, right? Or is it?
For example, if I need to build a bootable image (which I actually
need to do for syzkaller). In the end it boils down to just creating a
local, private file with some particular contents. Should be nobody's
business, right? But for some reason on Linux creation of a local file
requires root privileges multiple times along the way. As the result I
have run the whole service as root, degrading overall security. And I
can't say that I fully trust all code and data involved, but Linux
simply does not give me any choice.

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2018-10-30 12:16 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-10-08 16:18 WARNING in ext4_invalidatepage syzbot
2018-10-08 16:29 ` Dmitry Vyukov
2018-10-09  1:34   ` Theodore Y. Ts'o
2018-10-15 13:22     ` Dmitry Vyukov
2018-10-15 18:08       ` Theodore Y. Ts'o
2018-10-16 14:02         ` Dmitry Vyukov
2018-10-16 15:53           ` Theodore Y. Ts'o
2018-10-30 12:09             ` Dmitry Vyukov
2018-10-30 12:16             ` Dmitry Vyukov

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).