* WARNING in ext4_invalidatepage @ 2018-10-08 16:18 syzbot 2018-10-08 16:29 ` Dmitry Vyukov 0 siblings, 1 reply; 9+ messages in thread From: syzbot @ 2018-10-08 16:18 UTC (permalink / raw) To: adilger.kernel, linux-ext4, linux-kernel, syzkaller-bugs, tytso Hello, syzbot found the following crash on: HEAD commit: c1d84a1b42ef Merge git://git.kernel.org/pub/scm/linux/kern.. git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=161588d6400000 kernel config: https://syzkaller.appspot.com/x/.config?x=c0af03fe452b65fb dashboard link: https://syzkaller.appspot.com/bug?extid=85da7ac734f7ba432ee4 compiler: gcc (GCC) 8.0.1 20180413 (experimental) Unfortunately, I don't have any reproducer for this crash yet. IMPORTANT: if you fix the bug, please add the following tag to the commit: Reported-by: syzbot+85da7ac734f7ba432ee4@syzkaller.appspotmail.com WARNING: CPU: 1 PID: 17203 at fs/ext4/inode.c:3353 ext4_invalidatepage+0x1c7/0x5b0 fs/ext4/inode.c:3353 Kernel panic - not syncing: panic_on_warn set ... CPU: 1 PID: 17203 Comm: syz-executor5 Not tainted 4.19.0-rc6+ #48 kobject: 'loop3' (000000002393f9b0): kobject_uevent_env Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x1c4/0x2b4 lib/dump_stack.c:113 kobject: 'loop3' (000000002393f9b0): fill_kobj_path: path = '/devices/virtual/block/loop3' panic+0x238/0x4e7 kernel/panic.c:184 __warn.cold.8+0x163/0x1ba kernel/panic.c:536 report_bug+0x254/0x2d0 lib/bug.c:186 fixup_bug arch/x86/kernel/traps.c:178 [inline] do_error_trap+0x1fc/0x4d0 arch/x86/kernel/traps.c:296 do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:316 invalid_op+0x14/0x20 arch/x86/entry/entry_64.S:993 RIP: 0010:ext4_invalidatepage+0x1c7/0x5b0 fs/ext4/inode.c:3353 Code: 80 3c 02 00 0f 85 a7 03 00 00 4d 8b 6d 00 31 ff 49 c1 ed 11 41 83 e5 01 4c 89 ee e8 43 ea 67 ff 4d 85 ed 74 07 e8 09 e9 67 ff <0f> 0b e8 02 e9 67 ff 8b b5 34 ff ff ff 44 89 fa 4c 89 e7 e8 91 3a RSP: 0018:ffff88018590ec78 EFLAGS: 00010212 RAX: 0000000000040000 RBX: 1ffff10030b21d91 RCX: ffffc9000dae4000 RDX: 0000000000000ece RSI: ffffffff8216ec87 RDI: 0000000000000007 RBP: ffff88018590ed50 R08: ffff880183018300 R09: fffff94000ceffc6 R10: fffff94000ceffc6 R11: ffffea000677fe33 R12: ffffea000677fe00 R13: 0000000000000001 R14: ffffea000677fe08 R15: 0000000000001000 do_invalidatepage mm/truncate.c:165 [inline] truncate_cleanup_page+0x5ac/0xa90 mm/truncate.c:187 truncate_inode_page+0x107/0x1a0 mm/truncate.c:229 truncate_inode_pages_range+0x1382/0x2d50 mm/truncate.c:451 truncate_inode_pages+0x24/0x30 mm/truncate.c:478 swap_inode_boot_loader fs/ext4/ioctl.c:123 [inline] ext4_ioctl+0x1e3b/0x4210 fs/ext4/ioctl.c:865 vfs_ioctl fs/ioctl.c:46 [inline] file_ioctl fs/ioctl.c:501 [inline] do_vfs_ioctl+0x1de/0x1720 fs/ioctl.c:685 ksys_ioctl+0xa9/0xd0 fs/ioctl.c:702 __do_sys_ioctl fs/ioctl.c:709 [inline] __se_sys_ioctl fs/ioctl.c:707 [inline] __x64_sys_ioctl+0x73/0xb0 fs/ioctl.c:707 do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x457579 Code: 1d b4 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 eb b3 fb ff c3 66 2e 0f 1f 84 00 00 00 00 RSP: 002b:00007fa151655c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000457579 RDX: 0000000020000000 RSI: 0000000000006611 RDI: 0000000000000006 RBP: 000000000072bfa0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 00007fa1516566d4 R13: 00000000004bf60e R14: 00000000004cf4a8 R15: 00000000ffffffff Kernel Offset: disabled Rebooting in 86400 seconds.. --- This bug is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at syzkaller@googlegroups.com. syzbot will keep track of this bug report. See: https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with syzbot. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: WARNING in ext4_invalidatepage 2018-10-08 16:18 WARNING in ext4_invalidatepage syzbot @ 2018-10-08 16:29 ` Dmitry Vyukov 2018-10-09 1:34 ` Theodore Y. Ts'o 0 siblings, 1 reply; 9+ messages in thread From: Dmitry Vyukov @ 2018-10-08 16:29 UTC (permalink / raw) To: syzbot Cc: Andreas Dilger, linux-ext4, LKML, syzkaller-bugs, Theodore Ts'o On Mon, Oct 8, 2018 at 6:18 PM, syzbot <syzbot+85da7ac734f7ba432ee4@syzkaller.appspotmail.com> wrote: > Hello, > > syzbot found the following crash on: > > HEAD commit: c1d84a1b42ef Merge git://git.kernel.org/pub/scm/linux/kern.. > git tree: upstream > console output: https://syzkaller.appspot.com/x/log.txt?x=161588d6400000 > kernel config: https://syzkaller.appspot.com/x/.config?x=c0af03fe452b65fb > dashboard link: https://syzkaller.appspot.com/bug?extid=85da7ac734f7ba432ee4 > compiler: gcc (GCC) 8.0.1 20180413 (experimental) > > Unfortunately, I don't have any reproducer for this crash yet. > > IMPORTANT: if you fix the bug, please add the following tag to the commit: > Reported-by: syzbot+85da7ac734f7ba432ee4@syzkaller.appspotmail.com The program that triggered it did the following: 05:23:28 executing program 5: r0 = creat(&(0x7f00000001c0)='./file0\x00', 0x0) socketpair$unix(0x1, 0x1, 0x0, &(0x7f0000000380)={0xffffffffffffffff, <r1=>0xffffffffffffffff}) write$RDMA_USER_CM_CMD_CREATE_ID(r0, &(0x7f0000000240)={0x0, 0x18, 0xfa00, {0x0, &(0x7f0000000200)}}, 0x20) ioctl$PERF_EVENT_IOC_ENABLE(r1, 0x8912, 0x400200) ioctl$EXT4_IOC_SETFLAGS(r0, 0x4008660f, &(0x7f0000000000)=0x4000) Seems like some bad interaction of unix sockets and ext4. > WARNING: CPU: 1 PID: 17203 at fs/ext4/inode.c:3353 > ext4_invalidatepage+0x1c7/0x5b0 fs/ext4/inode.c:3353 > Kernel panic - not syncing: panic_on_warn set ... > > CPU: 1 PID: 17203 Comm: syz-executor5 Not tainted 4.19.0-rc6+ #48 > kobject: 'loop3' (000000002393f9b0): kobject_uevent_env > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS > Google 01/01/2011 > Call Trace: > __dump_stack lib/dump_stack.c:77 [inline] > dump_stack+0x1c4/0x2b4 lib/dump_stack.c:113 > kobject: 'loop3' (000000002393f9b0): fill_kobj_path: path = > '/devices/virtual/block/loop3' > panic+0x238/0x4e7 kernel/panic.c:184 > __warn.cold.8+0x163/0x1ba kernel/panic.c:536 > report_bug+0x254/0x2d0 lib/bug.c:186 > fixup_bug arch/x86/kernel/traps.c:178 [inline] > do_error_trap+0x1fc/0x4d0 arch/x86/kernel/traps.c:296 > do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:316 > invalid_op+0x14/0x20 arch/x86/entry/entry_64.S:993 > RIP: 0010:ext4_invalidatepage+0x1c7/0x5b0 fs/ext4/inode.c:3353 > Code: 80 3c 02 00 0f 85 a7 03 00 00 4d 8b 6d 00 31 ff 49 c1 ed 11 41 83 e5 > 01 4c 89 ee e8 43 ea 67 ff 4d 85 ed 74 07 e8 09 e9 67 ff <0f> 0b e8 02 e9 67 > ff 8b b5 34 ff ff ff 44 89 fa 4c 89 e7 e8 91 3a > RSP: 0018:ffff88018590ec78 EFLAGS: 00010212 > RAX: 0000000000040000 RBX: 1ffff10030b21d91 RCX: ffffc9000dae4000 > RDX: 0000000000000ece RSI: ffffffff8216ec87 RDI: 0000000000000007 > RBP: ffff88018590ed50 R08: ffff880183018300 R09: fffff94000ceffc6 > R10: fffff94000ceffc6 R11: ffffea000677fe33 R12: ffffea000677fe00 > R13: 0000000000000001 R14: ffffea000677fe08 R15: 0000000000001000 > do_invalidatepage mm/truncate.c:165 [inline] > truncate_cleanup_page+0x5ac/0xa90 mm/truncate.c:187 > truncate_inode_page+0x107/0x1a0 mm/truncate.c:229 > truncate_inode_pages_range+0x1382/0x2d50 mm/truncate.c:451 > truncate_inode_pages+0x24/0x30 mm/truncate.c:478 > swap_inode_boot_loader fs/ext4/ioctl.c:123 [inline] > ext4_ioctl+0x1e3b/0x4210 fs/ext4/ioctl.c:865 > vfs_ioctl fs/ioctl.c:46 [inline] > file_ioctl fs/ioctl.c:501 [inline] > do_vfs_ioctl+0x1de/0x1720 fs/ioctl.c:685 > ksys_ioctl+0xa9/0xd0 fs/ioctl.c:702 > __do_sys_ioctl fs/ioctl.c:709 [inline] > __se_sys_ioctl fs/ioctl.c:707 [inline] > __x64_sys_ioctl+0x73/0xb0 fs/ioctl.c:707 > do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 > entry_SYSCALL_64_after_hwframe+0x49/0xbe > RIP: 0033:0x457579 > Code: 1d b4 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 > 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff > 0f 83 eb b3 fb ff c3 66 2e 0f 1f 84 00 00 00 00 > RSP: 002b:00007fa151655c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 > RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000457579 > RDX: 0000000020000000 RSI: 0000000000006611 RDI: 0000000000000006 > RBP: 000000000072bfa0 R08: 0000000000000000 R09: 0000000000000000 > R10: 0000000000000000 R11: 0000000000000246 R12: 00007fa1516566d4 > R13: 00000000004bf60e R14: 00000000004cf4a8 R15: 00000000ffffffff > Kernel Offset: disabled > Rebooting in 86400 seconds.. > > > --- > This bug is generated by a bot. It may contain errors. > See https://goo.gl/tpsmEJ for more information about syzbot. > syzbot engineers can be reached at syzkaller@googlegroups.com. > > syzbot will keep track of this bug report. See: > https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with > syzbot. > > -- > You received this message because you are subscribed to the Google Groups > "syzkaller-bugs" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to syzkaller-bugs+unsubscribe@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/syzkaller-bugs/0000000000002753c60577b9f707%40google.com. > For more options, visit https://groups.google.com/d/optout. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: WARNING in ext4_invalidatepage 2018-10-08 16:29 ` Dmitry Vyukov @ 2018-10-09 1:34 ` Theodore Y. Ts'o 2018-10-15 13:22 ` Dmitry Vyukov 0 siblings, 1 reply; 9+ messages in thread From: Theodore Y. Ts'o @ 2018-10-09 1:34 UTC (permalink / raw) To: Dmitry Vyukov; +Cc: syzbot, Andreas Dilger, linux-ext4, LKML, syzkaller-bugs On Mon, Oct 08, 2018 at 06:29:54PM +0200, Dmitry Vyukov wrote: > > The program that triggered it did the following: > > 05:23:28 executing program 5: > r0 = creat(&(0x7f00000001c0)='./file0\x00', 0x0) > socketpair$unix(0x1, 0x1, 0x0, &(0x7f0000000380)={0xffffffffffffffff, > <r1=>0xffffffffffffffff}) > write$RDMA_USER_CM_CMD_CREATE_ID(r0, &(0x7f0000000240)={0x0, 0x18, > 0xfa00, {0x0, &(0x7f0000000200)}}, 0x20) This looks like it's doing an ioctl-like thing which is now restricted to root --- it looks like people can do arbitrary stupid things with it? https://www.openwall.com/lists/oss-security/2016/05/09/11 > ioctl$PERF_EVENT_IOC_ENABLE(r1, 0x8912, 0x400200) > ioctl$EXT4_IOC_SETFLAGS(r0, 0x4008660f, &(0x7f0000000000)=0x4000) Um, this doesn't seem to correspond to the stack trace: > > do_invalidatepage mm/truncate.c:165 [inline] > > truncate_cleanup_page+0x5ac/0xa90 mm/truncate.c:187 > > truncate_inode_page+0x107/0x1a0 mm/truncate.c:229 > > truncate_inode_pages_range+0x1382/0x2d50 mm/truncate.c:451 > > truncate_inode_pages+0x24/0x30 mm/truncate.c:478 > > swap_inode_boot_loader fs/ext4/ioctl.c:123 [inline] > > ext4_ioctl+0x1e3b/0x4210 fs/ext4/ioctl.c:865 > > vfs_ioctl fs/ioctl.c:46 [inline] > > file_ioctl fs/ioctl.c:501 [inline] > > do_vfs_ioctl+0x1de/0x1720 fs/ioctl.c:685 > > ksys_ioctl+0xa9/0xd0 fs/ioctl.c:702 > > __do_sys_ioctl fs/ioctl.c:709 [inline] > > __se_sys_ioctl fs/ioctl.c:707 [inline] > > __x64_sys_ioctl+0x73/0xb0 fs/ioctl.c:707 > > do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 > > entry_SYSCALL_64_after_hwframe+0x49/0xbe This looks like the program that triggered the crash was running EXT4_IOC_SWAP_BOOT. It may have been racing against something that's was using RDMA, but that's not clear at all, since the write command appears to be some weird ioctl-like interface that requires root. It's too bad that syzbot wasn't able to come up with a clean reproducer. I've been doing some work to robustify the EXT4_IOC_SWAP_ROOT. If we had a reproducer I'd see if this patch would address it. http://patchwork.ozlabs.org/patch/978083/ - Ted > > WARNING: CPU: 1 PID: 17203 at fs/ext4/inode.c:3353 > > ext4_invalidatepage+0x1c7/0x5b0 fs/ext4/inode.c:3353 > > Kernel panic - not syncing: panic_on_warn set ... > > > > CPU: 1 PID: 17203 Comm: syz-executor5 Not tainted 4.19.0-rc6+ #48 > > kobject: 'loop3' (000000002393f9b0): kobject_uevent_env > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS > > Google 01/01/2011 > > Call Trace: > > __dump_stack lib/dump_stack.c:77 [inline] > > dump_stack+0x1c4/0x2b4 lib/dump_stack.c:113 > > kobject: 'loop3' (000000002393f9b0): fill_kobj_path: path = > > '/devices/virtual/block/loop3' > > panic+0x238/0x4e7 kernel/panic.c:184 > > __warn.cold.8+0x163/0x1ba kernel/panic.c:536 > > report_bug+0x254/0x2d0 lib/bug.c:186 > > fixup_bug arch/x86/kernel/traps.c:178 [inline] > > do_error_trap+0x1fc/0x4d0 arch/x86/kernel/traps.c:296 > > do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:316 > > invalid_op+0x14/0x20 arch/x86/entry/entry_64.S:993 > > RIP: 0010:ext4_invalidatepage+0x1c7/0x5b0 fs/ext4/inode.c:3353 > > Code: 80 3c 02 00 0f 85 a7 03 00 00 4d 8b 6d 00 31 ff 49 c1 ed 11 41 83 e5 > > 01 4c 89 ee e8 43 ea 67 ff 4d 85 ed 74 07 e8 09 e9 67 ff <0f> 0b e8 02 e9 67 > > ff 8b b5 34 ff ff ff 44 89 fa 4c 89 e7 e8 91 3a > > RSP: 0018:ffff88018590ec78 EFLAGS: 00010212 > > RAX: 0000000000040000 RBX: 1ffff10030b21d91 RCX: ffffc9000dae4000 > > RDX: 0000000000000ece RSI: ffffffff8216ec87 RDI: 0000000000000007 > > RBP: ffff88018590ed50 R08: ffff880183018300 R09: fffff94000ceffc6 > > R10: fffff94000ceffc6 R11: ffffea000677fe33 R12: ffffea000677fe00 > > R13: 0000000000000001 R14: ffffea000677fe08 R15: 0000000000001000 > > do_invalidatepage mm/truncate.c:165 [inline] > > truncate_cleanup_page+0x5ac/0xa90 mm/truncate.c:187 > > truncate_inode_page+0x107/0x1a0 mm/truncate.c:229 > > truncate_inode_pages_range+0x1382/0x2d50 mm/truncate.c:451 > > truncate_inode_pages+0x24/0x30 mm/truncate.c:478 > > swap_inode_boot_loader fs/ext4/ioctl.c:123 [inline] > > ext4_ioctl+0x1e3b/0x4210 fs/ext4/ioctl.c:865 > > vfs_ioctl fs/ioctl.c:46 [inline] > > file_ioctl fs/ioctl.c:501 [inline] > > do_vfs_ioctl+0x1de/0x1720 fs/ioctl.c:685 > > ksys_ioctl+0xa9/0xd0 fs/ioctl.c:702 > > __do_sys_ioctl fs/ioctl.c:709 [inline] > > __se_sys_ioctl fs/ioctl.c:707 [inline] > > __x64_sys_ioctl+0x73/0xb0 fs/ioctl.c:707 > > do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 > > entry_SYSCALL_64_after_hwframe+0x49/0xbe > > RIP: 0033:0x457579 > > Code: 1d b4 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 > > 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff > > 0f 83 eb b3 fb ff c3 66 2e 0f 1f 84 00 00 00 00 > > RSP: 002b:00007fa151655c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 > > RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000457579 > > RDX: 0000000020000000 RSI: 0000000000006611 RDI: 0000000000000006 > > RBP: 000000000072bfa0 R08: 0000000000000000 R09: 0000000000000000 > > R10: 0000000000000000 R11: 0000000000000246 R12: 00007fa1516566d4 > > R13: 00000000004bf60e R14: 00000000004cf4a8 R15: 00000000ffffffff > > Kernel Offset: disabled > > Rebooting in 86400 seconds.. > > > > > > --- > > This bug is generated by a bot. It may contain errors. > > See https://goo.gl/tpsmEJ for more information about syzbot. > > syzbot engineers can be reached at syzkaller@googlegroups.com. > > > > syzbot will keep track of this bug report. See: > > https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with > > syzbot. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "syzkaller-bugs" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to syzkaller-bugs+unsubscribe@googlegroups.com. > > To view this discussion on the web visit > > https://groups.google.com/d/msgid/syzkaller-bugs/0000000000002753c60577b9f707%40google.com. > > For more options, visit https://groups.google.com/d/optout. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: WARNING in ext4_invalidatepage 2018-10-09 1:34 ` Theodore Y. Ts'o @ 2018-10-15 13:22 ` Dmitry Vyukov 2018-10-15 18:08 ` Theodore Y. Ts'o 0 siblings, 1 reply; 9+ messages in thread From: Dmitry Vyukov @ 2018-10-15 13:22 UTC (permalink / raw) To: Theodore Y. Ts'o, Dmitry Vyukov, syzbot, Andreas Dilger, linux-ext4, LKML, syzkaller-bugs On Tue, Oct 9, 2018 at 3:34 AM, Theodore Y. Ts'o <tytso@mit.edu> wrote: > On Mon, Oct 08, 2018 at 06:29:54PM +0200, Dmitry Vyukov wrote: >> >> The program that triggered it did the following: >> >> 05:23:28 executing program 5: >> r0 = creat(&(0x7f00000001c0)='./file0\x00', 0x0) >> socketpair$unix(0x1, 0x1, 0x0, &(0x7f0000000380)={0xffffffffffffffff, >> <r1=>0xffffffffffffffff}) >> write$RDMA_USER_CM_CMD_CREATE_ID(r0, &(0x7f0000000240)={0x0, 0x18, >> 0xfa00, {0x0, &(0x7f0000000200)}}, 0x20) > > This looks like it's doing an ioctl-like thing which is now restricted > to root --- it looks like people can do arbitrary stupid things with > it? > > https://www.openwall.com/lists/oss-security/2016/05/09/11 > >> ioctl$PERF_EVENT_IOC_ENABLE(r1, 0x8912, 0x400200) >> ioctl$EXT4_IOC_SETFLAGS(r0, 0x4008660f, &(0x7f0000000000)=0x4000) > > Um, this doesn't seem to correspond to the stack trace: > >> > do_invalidatepage mm/truncate.c:165 [inline] >> > truncate_cleanup_page+0x5ac/0xa90 mm/truncate.c:187 >> > truncate_inode_page+0x107/0x1a0 mm/truncate.c:229 >> > truncate_inode_pages_range+0x1382/0x2d50 mm/truncate.c:451 >> > truncate_inode_pages+0x24/0x30 mm/truncate.c:478 >> > swap_inode_boot_loader fs/ext4/ioctl.c:123 [inline] >> > ext4_ioctl+0x1e3b/0x4210 fs/ext4/ioctl.c:865 >> > vfs_ioctl fs/ioctl.c:46 [inline] >> > file_ioctl fs/ioctl.c:501 [inline] >> > do_vfs_ioctl+0x1de/0x1720 fs/ioctl.c:685 >> > ksys_ioctl+0xa9/0xd0 fs/ioctl.c:702 >> > __do_sys_ioctl fs/ioctl.c:709 [inline] >> > __se_sys_ioctl fs/ioctl.c:707 [inline] >> > __x64_sys_ioctl+0x73/0xb0 fs/ioctl.c:707 >> > do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 >> > entry_SYSCALL_64_after_hwframe+0x49/0xbe > > This looks like the program that triggered the crash was running > EXT4_IOC_SWAP_BOOT. It may have been racing against something that's > was using RDMA, but that's not clear at all, since the write command > appears to be some weird ioctl-like interface that requires root. > > It's too bad that syzbot wasn't able to come up with a clean > reproducer. I've been doing some work to robustify the > EXT4_IOC_SWAP_ROOT. If we had a reproducer I'd see if this patch > would address it. > > http://patchwork.ozlabs.org/patch/978083/ Hi Ted, The RDMA thing seems to be red-herring, as it writes RDMA command into a normal local file. These commands will only take effect if written to /dev/infiniband/rdma_cm file. Now that you mention EXT4_IOC_SWAP_BOOT, I think I looked at the wrong program, there is a subsequent one that does ioctl(0x6611) where 0x6611 is in fact EXT4_IOC_SWAP_BOOT. So I think it's this one: 05:23:28 executing program 5: r0 = creat(&(0x7f00000001c0)='./file0\x00', 0x0) socketpair$unix(0x1, 0x1, 0x0, &(0x7f0000000380)={0xffffffffffffffff, <r1=>0xffffffffffffffff}) write$RDMA_USER_CM_CMD_CREATE_ID(r0, &(0x7f0000000240)={0x0, 0x18, 0xfa00, {0x0, &(0x7f0000000200)}}, 0x20) ioctl$PERF_EVENT_IOC_ENABLE(r1, 0x8912, 0x400200) ioctl$EXT4_IOC_SETFLAGS(r0, 0x6611, &(0x7f0000000000)=0x4000) I've tried to manually reply this program and the whole log too, but it does not reproduce. This may be related to the fact that filesystem accumulates too much global state, so probably first relevant part happened long time ago, and then second relevant part happened later and triggered the warning. But just re-doing the second part does not reproduce the bug. Unfortunately it's the reality of kernel testing. Maybe syzbot will come up with repro later, or maybe we will see that it does not happen anymore (fixed by your patch) and we will close it then. >> > WARNING: CPU: 1 PID: 17203 at fs/ext4/inode.c:3353 >> > ext4_invalidatepage+0x1c7/0x5b0 fs/ext4/inode.c:3353 >> > Kernel panic - not syncing: panic_on_warn set ... >> > >> > CPU: 1 PID: 17203 Comm: syz-executor5 Not tainted 4.19.0-rc6+ #48 >> > kobject: 'loop3' (000000002393f9b0): kobject_uevent_env >> > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS >> > Google 01/01/2011 >> > Call Trace: >> > __dump_stack lib/dump_stack.c:77 [inline] >> > dump_stack+0x1c4/0x2b4 lib/dump_stack.c:113 >> > kobject: 'loop3' (000000002393f9b0): fill_kobj_path: path = >> > '/devices/virtual/block/loop3' >> > panic+0x238/0x4e7 kernel/panic.c:184 >> > __warn.cold.8+0x163/0x1ba kernel/panic.c:536 >> > report_bug+0x254/0x2d0 lib/bug.c:186 >> > fixup_bug arch/x86/kernel/traps.c:178 [inline] >> > do_error_trap+0x1fc/0x4d0 arch/x86/kernel/traps.c:296 >> > do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:316 >> > invalid_op+0x14/0x20 arch/x86/entry/entry_64.S:993 >> > RIP: 0010:ext4_invalidatepage+0x1c7/0x5b0 fs/ext4/inode.c:3353 >> > Code: 80 3c 02 00 0f 85 a7 03 00 00 4d 8b 6d 00 31 ff 49 c1 ed 11 41 83 e5 >> > 01 4c 89 ee e8 43 ea 67 ff 4d 85 ed 74 07 e8 09 e9 67 ff <0f> 0b e8 02 e9 67 >> > ff 8b b5 34 ff ff ff 44 89 fa 4c 89 e7 e8 91 3a >> > RSP: 0018:ffff88018590ec78 EFLAGS: 00010212 >> > RAX: 0000000000040000 RBX: 1ffff10030b21d91 RCX: ffffc9000dae4000 >> > RDX: 0000000000000ece RSI: ffffffff8216ec87 RDI: 0000000000000007 >> > RBP: ffff88018590ed50 R08: ffff880183018300 R09: fffff94000ceffc6 >> > R10: fffff94000ceffc6 R11: ffffea000677fe33 R12: ffffea000677fe00 >> > R13: 0000000000000001 R14: ffffea000677fe08 R15: 0000000000001000 >> > do_invalidatepage mm/truncate.c:165 [inline] >> > truncate_cleanup_page+0x5ac/0xa90 mm/truncate.c:187 >> > truncate_inode_page+0x107/0x1a0 mm/truncate.c:229 >> > truncate_inode_pages_range+0x1382/0x2d50 mm/truncate.c:451 >> > truncate_inode_pages+0x24/0x30 mm/truncate.c:478 >> > swap_inode_boot_loader fs/ext4/ioctl.c:123 [inline] >> > ext4_ioctl+0x1e3b/0x4210 fs/ext4/ioctl.c:865 >> > vfs_ioctl fs/ioctl.c:46 [inline] >> > file_ioctl fs/ioctl.c:501 [inline] >> > do_vfs_ioctl+0x1de/0x1720 fs/ioctl.c:685 >> > ksys_ioctl+0xa9/0xd0 fs/ioctl.c:702 >> > __do_sys_ioctl fs/ioctl.c:709 [inline] >> > __se_sys_ioctl fs/ioctl.c:707 [inline] >> > __x64_sys_ioctl+0x73/0xb0 fs/ioctl.c:707 >> > do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 >> > entry_SYSCALL_64_after_hwframe+0x49/0xbe >> > RIP: 0033:0x457579 >> > Code: 1d b4 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 >> > 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff >> > 0f 83 eb b3 fb ff c3 66 2e 0f 1f 84 00 00 00 00 >> > RSP: 002b:00007fa151655c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 >> > RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000457579 >> > RDX: 0000000020000000 RSI: 0000000000006611 RDI: 0000000000000006 >> > RBP: 000000000072bfa0 R08: 0000000000000000 R09: 0000000000000000 >> > R10: 0000000000000000 R11: 0000000000000246 R12: 00007fa1516566d4 >> > R13: 00000000004bf60e R14: 00000000004cf4a8 R15: 00000000ffffffff >> > Kernel Offset: disabled >> > Rebooting in 86400 seconds.. >> > >> > >> > --- >> > This bug is generated by a bot. It may contain errors. >> > See https://goo.gl/tpsmEJ for more information about syzbot. >> > syzbot engineers can be reached at syzkaller@googlegroups.com. >> > >> > syzbot will keep track of this bug report. See: >> > https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with >> > syzbot. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: WARNING in ext4_invalidatepage 2018-10-15 13:22 ` Dmitry Vyukov @ 2018-10-15 18:08 ` Theodore Y. Ts'o 2018-10-16 14:02 ` Dmitry Vyukov 0 siblings, 1 reply; 9+ messages in thread From: Theodore Y. Ts'o @ 2018-10-15 18:08 UTC (permalink / raw) To: Dmitry Vyukov; +Cc: syzbot, Andreas Dilger, linux-ext4, LKML, syzkaller-bugs On Mon, Oct 15, 2018 at 03:22:42PM +0200, Dmitry Vyukov wrote: > Now that you mention EXT4_IOC_SWAP_BOOT, I think I looked at the wrong > program, there is a subsequent one that does ioctl(0x6611) where > 0x6611 is in fact EXT4_IOC_SWAP_BOOT. So I think it's this one: > > 05:23:28 executing program 5: > r0 = creat(&(0x7f00000001c0)='./file0\x00', 0x0) > socketpair$unix(0x1, 0x1, 0x0, &(0x7f0000000380)={0xffffffffffffffff, > <r1=>0xffffffffffffffff}) > write$RDMA_USER_CM_CMD_CREATE_ID(r0, &(0x7f0000000240)={0x0, 0x18, > 0xfa00, {0x0, &(0x7f0000000200)}}, 0x20) > ioctl$PERF_EVENT_IOC_ENABLE(r1, 0x8912, 0x400200) > ioctl$EXT4_IOC_SETFLAGS(r0, 0x6611, &(0x7f0000000000)=0x4000) Ah, so is it a bug in Syzkaller that it is printing ioctl$EXT4_IOC_SETFLAGS when 0x6611 is in fact EXT4_IOC_SWAP_BOOT, right? > I've tried to manually reply this program and the whole log too, but > it does not reproduce. This may be related to the fact that filesystem > accumulates too much global state, so probably first relevant part > happened long time ago, and then second relevant part happened later > and triggered the warning. But just re-doing the second part does not > reproduce the bug. It was probably some other process racing with EXT4_IOC_SWAP_BOOT. The patch I referenced in my previous e-mail protects against additional scenarios where someone might be trying to punch a whole into a file that is being swapped into the bootloader ioctl. This particular ioctl isn't yet being used by anyone, so it had some other issues as well, such as not interacting well with inline_data-enabled file systems --- not that any bootloader would be small enough that it would fit in an inline_data inode, but we're basically proofing the code against a malicious (or buggy) root-privileged program... such as syzbot. :-) - Ted ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: WARNING in ext4_invalidatepage 2018-10-15 18:08 ` Theodore Y. Ts'o @ 2018-10-16 14:02 ` Dmitry Vyukov 2018-10-16 15:53 ` Theodore Y. Ts'o 0 siblings, 1 reply; 9+ messages in thread From: Dmitry Vyukov @ 2018-10-16 14:02 UTC (permalink / raw) To: Theodore Y. Ts'o, Dmitry Vyukov, syzbot, Andreas Dilger, linux-ext4, LKML, syzkaller-bugs On Mon, Oct 15, 2018 at 8:08 PM, Theodore Y. Ts'o <tytso@mit.edu> wrote: > On Mon, Oct 15, 2018 at 03:22:42PM +0200, Dmitry Vyukov wrote: >> Now that you mention EXT4_IOC_SWAP_BOOT, I think I looked at the wrong >> program, there is a subsequent one that does ioctl(0x6611) where >> 0x6611 is in fact EXT4_IOC_SWAP_BOOT. So I think it's this one: >> >> 05:23:28 executing program 5: >> r0 = creat(&(0x7f00000001c0)='./file0\x00', 0x0) >> socketpair$unix(0x1, 0x1, 0x0, &(0x7f0000000380)={0xffffffffffffffff, >> <r1=>0xffffffffffffffff}) >> write$RDMA_USER_CM_CMD_CREATE_ID(r0, &(0x7f0000000240)={0x0, 0x18, >> 0xfa00, {0x0, &(0x7f0000000200)}}, 0x20) >> ioctl$PERF_EVENT_IOC_ENABLE(r1, 0x8912, 0x400200) >> ioctl$EXT4_IOC_SETFLAGS(r0, 0x6611, &(0x7f0000000000)=0x4000) > > Ah, so is it a bug in Syzkaller that it is printing > ioctl$EXT4_IOC_SETFLAGS when 0x6611 is in fact EXT4_IOC_SWAP_BOOT, > right? I am not sure how exactly this should be classified. To significant degree these "$FOO" discriminations are informational and only really affect the data format passed in. For ioctl/write it's not really possible to know what exactly it will do if you just open a file (which can be a link to into an overlay mount pointing to some special file). Sometimes we also want to specifically spoof exact fd type (or other resource type) for a syscall. Sometimes we discover other ioctl command constants while running an ioctl, and then we just change the command constant but leave the data as is. >> I've tried to manually reply this program and the whole log too, but >> it does not reproduce. This may be related to the fact that filesystem >> accumulates too much global state, so probably first relevant part >> happened long time ago, and then second relevant part happened later >> and triggered the warning. But just re-doing the second part does not >> reproduce the bug. > > It was probably some other process racing with EXT4_IOC_SWAP_BOOT. > The patch I referenced in my previous e-mail protects against > additional scenarios where someone might be trying to punch a whole > into a file that is being swapped into the bootloader ioctl. This > particular ioctl isn't yet being used by anyone, so it had some other > issues as well, such as not interacting well with inline_data-enabled > file systems --- not that any bootloader would be small enough that it > would fit in an inline_data inode, but we're basically proofing the > code against a malicious (or buggy) root-privileged program... such as > syzbot. :-) ... or paving the way to opening all of this to non-root users. Why not if not bugs? ;) Back to this bug, I think we should wait to see if it happens more in future and if syzkaller can come up with a repro. If it won't happen more (perhaps fixed by your patch), then we will close it as obsolete. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: WARNING in ext4_invalidatepage 2018-10-16 14:02 ` Dmitry Vyukov @ 2018-10-16 15:53 ` Theodore Y. Ts'o 2018-10-30 12:09 ` Dmitry Vyukov 2018-10-30 12:16 ` Dmitry Vyukov 0 siblings, 2 replies; 9+ messages in thread From: Theodore Y. Ts'o @ 2018-10-16 15:53 UTC (permalink / raw) To: Dmitry Vyukov; +Cc: syzbot, Andreas Dilger, linux-ext4, LKML, syzkaller-bugs On Tue, Oct 16, 2018 at 04:02:07PM +0200, Dmitry Vyukov wrote: > I am not sure how exactly this should be classified. To significant > degree these "$FOO" discriminations are informational and only really > affect the data format passed in. The problem is that putting something which is simply and plainly *wrong* is going to be actively misleading. You *have* to know what ioctl you are trying to be trying to execute because some of them require input data structures that you have to fill in, or that it requires a file descriptor as opposed to an integer or a pointer to a struct. I assume you're not just picking ioctl numbers purely at random, right? So I assume you had a table that said, this ioctl, 0x6611 takes a file descriptor, and has thus-and-so-a-name. If that name is wrong, I'd say it's pretty clearly a bug, right? (I'll again gently point out that a tiny amount of work in making syzkaller a tiny bit more developer toil would make it much more effective, since all of this extra developer toil --- now I know I can't even trust the $FOO discriminators to be right --- makes syzkaller less scalable by pursuading developers that it's not worthwhile to pay attention...) > > The patch I referenced in my previous e-mail protects against > > additional scenarios where someone might be trying to punch a whole > > into a file that is being swapped into the bootloader ioctl. This > > particular ioctl isn't yet being used by anyone, so it had some other > > issues as well, such as not interacting well with inline_data-enabled > > file systems --- not that any bootloader would be small enough that it > > would fit in an inline_data inode, but we're basically proofing the > > code against a malicious (or buggy) root-privileged program... such as > > syzbot. :-) > > ... or paving the way to opening all of this to non-root users. Why > not if not bugs? ;) The intent behind this particular ioctl is used to install a boot loader. It will *never* be opened to non-root users. It doesn't even make sense to make it available to pseudo-containers-root users. :-) - Ted ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: WARNING in ext4_invalidatepage 2018-10-16 15:53 ` Theodore Y. Ts'o @ 2018-10-30 12:09 ` Dmitry Vyukov 2018-10-30 12:16 ` Dmitry Vyukov 1 sibling, 0 replies; 9+ messages in thread From: Dmitry Vyukov @ 2018-10-30 12:09 UTC (permalink / raw) To: Theodore Y. Ts'o, Dmitry Vyukov, syzbot, Andreas Dilger, linux-ext4, LKML, syzkaller-bugs On Tue, Oct 16, 2018 at 5:53 PM, Theodore Y. Ts'o <tytso@mit.edu> wrote: > On Tue, Oct 16, 2018 at 04:02:07PM +0200, Dmitry Vyukov wrote: >> I am not sure how exactly this should be classified. To significant >> degree these "$FOO" discriminations are informational and only really >> affect the data format passed in. > > The problem is that putting something which is simply and plainly > *wrong* is going to be actively misleading. You *have* to know what > ioctl you are trying to be trying to execute because some of them > require input data structures that you have to fill in, or that it > requires a file descriptor as opposed to an integer or a pointer to a > struct. I assume you're not just picking ioctl numbers purely at > random, right? > > So I assume you had a table that said, this ioctl, 0x6611 takes a file > descriptor, and has thus-and-so-a-name. If that name is wrong, I'd > say it's pretty clearly a bug, right? We have such table and it is correct. But it is legal and fine and actually useful in context of testing to take an ioctl that takes a pointer to one structure and give it pointer to another structure, or plain int or just anything else. $FOO states what data layout is used and actual cmd number and fd type what command is executed. > (I'll again gently point out that a tiny amount of work in making > syzkaller a tiny bit more developer toil would make it much more > effective, since all of this extra developer toil --- now I know I > can't even trust the $FOO discriminators to be right --- makes > syzkaller less scalable by pursuading developers that it's not > worthwhile to pay attention...) Ted, I appreciate your suggestions. And the new table-structured email format that you proposed is great. But there are reasons for the current syzkaller program format. It's not just there is a typo in some table that we need to fix. The $FOO part encodes data layout and treatment of argument values (e.g. if 0xab actually takes a byte or a word, and what's it alignment). It's not something that can be restored from anything else. So if we lose this information, the program will useless and confusing in other ways. For example, consider an ioctl accepts a struct with 2 byte fields; but we actually executed it passing a struct with 2 int fields. Now if you look at {0x1, 0x2} you will assume that these are the byte values, but in reality it's 0x1 int which covered both 1-byte arguments. Also consider that ioctl cmd is actually meaningless in itself, and it's generally not possible to statically know what exactly ioctl will be executed. The crucial piece is fd type. I guess it boils down to the halting problem to statically prove what fd type we will have in a random program. Even worse, we can have a race on fd type, so it's non-deterministic and even not detectable dynamically. Taking this into account any attempts to preserve matching $FOO and cmd number are pointless. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: WARNING in ext4_invalidatepage 2018-10-16 15:53 ` Theodore Y. Ts'o 2018-10-30 12:09 ` Dmitry Vyukov @ 2018-10-30 12:16 ` Dmitry Vyukov 1 sibling, 0 replies; 9+ messages in thread From: Dmitry Vyukov @ 2018-10-30 12:16 UTC (permalink / raw) To: Theodore Y. Ts'o, Dmitry Vyukov, syzbot, Andreas Dilger, linux-ext4, LKML, syzkaller-bugs On Tue, Oct 16, 2018 at 5:53 PM, Theodore Y. Ts'o <tytso@mit.edu> wrote: >> > The patch I referenced in my previous e-mail protects against >> > additional scenarios where someone might be trying to punch a whole >> > into a file that is being swapped into the bootloader ioctl. This >> > particular ioctl isn't yet being used by anyone, so it had some other >> > issues as well, such as not interacting well with inline_data-enabled >> > file systems --- not that any bootloader would be small enough that it >> > would fit in an inline_data inode, but we're basically proofing the >> > code against a malicious (or buggy) root-privileged program... such as >> > syzbot. :-) >> >> ... or paving the way to opening all of this to non-root users. Why >> not if not bugs? ;) > > The intent behind this particular ioctl is used to install a boot > loader. It will *never* be opened to non-root users. It doesn't even > make sense to make it available to pseudo-containers-root users. :-) But it does not have to be executed on the root fs, right? Or is it? For example, if I need to build a bootable image (which I actually need to do for syzkaller). In the end it boils down to just creating a local, private file with some particular contents. Should be nobody's business, right? But for some reason on Linux creation of a local file requires root privileges multiple times along the way. As the result I have run the whole service as root, degrading overall security. And I can't say that I fully trust all code and data involved, but Linux simply does not give me any choice. ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2018-10-30 12:16 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-10-08 16:18 WARNING in ext4_invalidatepage syzbot 2018-10-08 16:29 ` Dmitry Vyukov 2018-10-09 1:34 ` Theodore Y. Ts'o 2018-10-15 13:22 ` Dmitry Vyukov 2018-10-15 18:08 ` Theodore Y. Ts'o 2018-10-16 14:02 ` Dmitry Vyukov 2018-10-16 15:53 ` Theodore Y. Ts'o 2018-10-30 12:09 ` Dmitry Vyukov 2018-10-30 12:16 ` Dmitry Vyukov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).