* [syzbot] [ext4?] kernel panic: EXT4-fs (device loop0): panic forced after error (3) @ 2023-08-16 22:48 syzbot 2023-08-17 14:21 ` Theodore Ts'o 0 siblings, 1 reply; 12+ messages in thread From: syzbot @ 2023-08-16 22:48 UTC (permalink / raw) To: adilger.kernel, linux-ext4, linux-fsdevel, linux-kernel, llvm, nathan, ndesaulniers, syzkaller-bugs, trix, tytso Hello, syzbot found the following issue on: HEAD commit: ae545c3283dc Merge tag 'gpio-fixes-for-v6.5-rc6' of git://.. git tree: upstream console+strace: https://syzkaller.appspot.com/x/log.txt?x=13e5d553a80000 kernel config: https://syzkaller.appspot.com/x/.config?x=171b698bc2e613cf dashboard link: https://syzkaller.appspot.com/bug?extid=27eece6916b914a49ce7 compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13433207a80000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=109cd837a80000 Downloadable assets: disk image: https://storage.googleapis.com/syzbot-assets/3b9bad020898/disk-ae545c32.raw.xz vmlinux: https://storage.googleapis.com/syzbot-assets/4073566d0a4b/vmlinux-ae545c32.xz kernel image: https://storage.googleapis.com/syzbot-assets/b163e2a2c47c/bzImage-ae545c32.xz mounted in repro: https://storage.googleapis.com/syzbot-assets/004395fabe81/mount_0.gz Bisection is inconclusive: the issue happens on the oldest tested release. bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=12890d53a80000 final oops: https://syzkaller.appspot.com/x/report.txt?x=11890d53a80000 console output: https://syzkaller.appspot.com/x/log.txt?x=16890d53a80000 IMPORTANT: if you fix the issue, please add the following tag to the commit: Reported-by: syzbot+27eece6916b914a49ce7@syzkaller.appspotmail.com loop0: detected capacity change from 0 to 512 ext2: Unknown parameter '����' EXT4-fs (loop0): feature flags set on rev 0 fs, running e2fsck is recommended EXT4-fs (loop0): warning: checktime reached, running e2fsck is recommended EXT4-fs warning (device loop0): ext4_update_dynamic_rev:1084: updating to rev 1 because of new feature flag, running e2fsck is recommended EXT4-fs error (device loop0): ext4_validate_block_bitmap:430: comm syz-executor211: bg 0: block 46: invalid block bitmap Kernel panic - not syncing: EXT4-fs (device loop0): panic forced after error CPU: 1 PID: 5061 Comm: syz-executor211 Not tainted 6.5.0-rc5-syzkaller-00353-gae545c3283dc #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/26/2023 Call Trace: <TASK> __dump_stack lib/dump_stack.c:88 [inline] dump_stack_lvl+0x1e7/0x2d0 lib/dump_stack.c:106 panic+0x30f/0x770 kernel/panic.c:340 ext4_handle_error+0x84e/0x8b0 fs/ext4/super.c:685 __ext4_error+0x277/0x3b0 fs/ext4/super.c:776 ext4_validate_block_bitmap+0xdf5/0x1200 fs/ext4/balloc.c:429 ext4_read_block_bitmap+0x40/0x80 fs/ext4/balloc.c:595 ext4_mb_clear_bb fs/ext4/mballoc.c:6503 [inline] ext4_free_blocks+0xfd3/0x2e20 fs/ext4/mballoc.c:6748 ext4_remove_blocks fs/ext4/extents.c:2545 [inline] ext4_ext_rm_leaf fs/ext4/extents.c:2710 [inline] ext4_ext_remove_space+0x216e/0x4d90 fs/ext4/extents.c:2958 ext4_ext_truncate+0x164/0x1f0 fs/ext4/extents.c:4408 ext4_truncate+0xa0a/0x1150 fs/ext4/inode.c:4127 ext4_process_orphan+0x1aa/0x2d0 fs/ext4/orphan.c:339 ext4_orphan_cleanup+0xb71/0x1400 fs/ext4/orphan.c:474 __ext4_fill_super fs/ext4/super.c:5577 [inline] ext4_fill_super+0x63ef/0x6ce0 fs/ext4/super.c:5696 get_tree_bdev+0x468/0x6c0 fs/super.c:1318 vfs_get_tree+0x8c/0x270 fs/super.c:1519 do_new_mount+0x28f/0xae0 fs/namespace.c:3335 do_mount fs/namespace.c:3675 [inline] __do_sys_mount fs/namespace.c:3884 [inline] __se_sys_mount+0x2d9/0x3c0 fs/namespace.c:3861 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x63/0xcd RIP: 0033:0x7f86c9eb1a99 Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 f1 17 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007ffc2eecbce8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5 RAX: ffffffffffffffda RBX: 0030656c69662f2e RCX: 00007f86c9eb1a99 RDX: 00000000200001c0 RSI: 00000000200006c0 RDI: 0000000020000640 RBP: 000000000000ec37 R08: 0000000000000000 R09: 00005555562044c0 R10: 000000003f000000 R11: 0000000000000246 R12: 00007ffc2eecbd10 R13: 00007ffc2eecbcfc R14: 431bde82d7b634db R15: 00007f86c9efa03b </TASK> Kernel Offset: disabled Rebooting in 86400 seconds.. --- This report is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at syzkaller@googlegroups.com. syzbot will keep track of this issue. See: https://goo.gl/tpsmEJ#status for how to communicate with syzbot. For information about bisection process see: https://goo.gl/tpsmEJ#bisection If the bug is already fixed, let syzbot know by replying with: #syz fix: exact-commit-title If you want syzbot to run the reproducer, reply with: #syz test: git://repo/address.git branch-or-commit-hash If you attach or paste a git patch, syzbot will apply it before testing. If you want to overwrite bug's subsystems, reply with: #syz set subsystems: new-subsystem (See the list of subsystem names on the web dashboard) If the bug is a duplicate of another bug, reply with: #syz dup: exact-subject-of-another-report If you want to undo deduplication, reply with: #syz undup ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [syzbot] [ext4?] kernel panic: EXT4-fs (device loop0): panic forced after error (3) 2023-08-16 22:48 [syzbot] [ext4?] kernel panic: EXT4-fs (device loop0): panic forced after error (3) syzbot @ 2023-08-17 14:21 ` Theodore Ts'o 2023-08-17 14:28 ` Aleksandr Nogikh 2023-08-17 14:47 ` Eric Sandeen 0 siblings, 2 replies; 12+ messages in thread From: Theodore Ts'o @ 2023-08-17 14:21 UTC (permalink / raw) To: syzbot Cc: adilger.kernel, linux-ext4, linux-fsdevel, linux-kernel, llvm, nathan, ndesaulniers, syzkaller-bugs, trix On Wed, Aug 16, 2023 at 03:48:49PM -0700, syzbot wrote: > Hello, > > syzbot found the following issue on: > > HEAD commit: ae545c3283dc Merge tag 'gpio-fixes-for-v6.5-rc6' of git://.. > git tree: upstream > console+strace: https://syzkaller.appspot.com/x/log.txt?x=13e5d553a80000 > kernel config: https://syzkaller.appspot.com/x/.config?x=171b698bc2e613cf > dashboard link: https://syzkaller.appspot.com/bug?extid=27eece6916b914a49ce7 > compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13433207a80000 > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=109cd837a80000 > > EXT4-fs error (device loop0): ext4_validate_block_bitmap:430: comm syz-executor211: bg 0: block 46: invalid block bitmap > Kernel panic - not syncing: EXT4-fs (device loop0): panic forced after error #syz invalid This is fundamentally a syzbot bug. The file system is horrifically corrupted, *and* the superblock has the "panic on error" (aka "panic onfile system corruption") bit set. This can be desireable because in a failover situation, if the file system is found to be corrupted, you *want* the primary server to fail, and let the secondary server to take over. This is a technique which is decades old. So this is Working As Intended, and is a classic example of (a) if you are root, you can force the file system to crash, and (b) a classic example of syzbot noise. (Five minutes of my life that I'm never getting back. :-) - Ted ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [syzbot] [ext4?] kernel panic: EXT4-fs (device loop0): panic forced after error (3) 2023-08-17 14:21 ` Theodore Ts'o @ 2023-08-17 14:28 ` Aleksandr Nogikh 2023-08-17 14:45 ` Theodore Ts'o 2023-08-17 14:47 ` Eric Sandeen 1 sibling, 1 reply; 12+ messages in thread From: Aleksandr Nogikh @ 2023-08-17 14:28 UTC (permalink / raw) To: Theodore Ts'o Cc: syzbot, adilger.kernel, linux-ext4, linux-fsdevel, linux-kernel, llvm, nathan, ndesaulniers, syzkaller-bugs, trix On Thu, Aug 17, 2023 at 4:21 PM Theodore Ts'o <tytso@mit.edu> wrote: > > On Wed, Aug 16, 2023 at 03:48:49PM -0700, syzbot wrote: > > Hello, > > > > syzbot found the following issue on: > > > > HEAD commit: ae545c3283dc Merge tag 'gpio-fixes-for-v6.5-rc6' of git://.. > > git tree: upstream > > console+strace: https://syzkaller.appspot.com/x/log.txt?x=13e5d553a80000 > > kernel config: https://syzkaller.appspot.com/x/.config?x=171b698bc2e613cf > > dashboard link: https://syzkaller.appspot.com/bug?extid=27eece6916b914a49ce7 > > compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 > > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13433207a80000 > > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=109cd837a80000 > > > > EXT4-fs error (device loop0): ext4_validate_block_bitmap:430: comm syz-executor211: bg 0: block 46: invalid block bitmap > > Kernel panic - not syncing: EXT4-fs (device loop0): panic forced after error > > #syz invalid > > This is fundamentally a syzbot bug. The file system is horrifically > corrupted, *and* the superblock has the "panic on error" (aka "panic > onfile system corruption") bit set. The console log has the following line: [ 60.708717][ T5061] Kernel panic - not syncing: EXT4-fs (device loop0): panic forced after error Can we consider a "panic forced after error" line to be a reliable indicator that syzbot must ignore the report? > > This can be desireable because in a failover situation, if the file > system is found to be corrupted, you *want* the primary server to > fail, and let the secondary server to take over. This is a technique > which is decades old. > > So this is Working As Intended, and is a classic example of (a) if you > are root, you can force the file system to crash, and (b) a classic > example of syzbot noise. (Five minutes of my life that I'm never > getting back. :-) > > - Ted > > -- > You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group. > To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/20230817142103.GA2247938%40mit.edu. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [syzbot] [ext4?] kernel panic: EXT4-fs (device loop0): panic forced after error (3) 2023-08-17 14:28 ` Aleksandr Nogikh @ 2023-08-17 14:45 ` Theodore Ts'o 2023-08-18 11:43 ` Aleksandr Nogikh 0 siblings, 1 reply; 12+ messages in thread From: Theodore Ts'o @ 2023-08-17 14:45 UTC (permalink / raw) To: Aleksandr Nogikh Cc: syzbot, adilger.kernel, linux-ext4, linux-fsdevel, linux-kernel, llvm, nathan, ndesaulniers, syzkaller-bugs, trix On Thu, Aug 17, 2023 at 04:28:33PM +0200, Aleksandr Nogikh wrote: > The console log has the following line: > > [ 60.708717][ T5061] Kernel panic - not syncing: EXT4-fs (device > loop0): panic forced after error > > Can we consider a "panic forced after error" line to be a reliable > indicator that syzbot must ignore the report? Yes. And the file system image that generated this bug should be discarded, because otherwise successive mutations will generate a large number of crashes that syzbot will then need to ignore, thus consuming syzbot resources. Alternatively, you can do the moral equivalent of "tune2fs -e continue foo.img" on any mutated file system seed, which will clear the "panic on error". (The other alternative is "tune2fs -e remount-ro", but given syzbot's desire to find kernel crashes, "tune2fs -e continue" is more likely find ways in which the kernel will find itself into trouble. Some sysadmins will want to chose "remount-ro", however, since that is more likely to limit file system damage once the file system is discovered to be corrupted.) - Ted ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [syzbot] [ext4?] kernel panic: EXT4-fs (device loop0): panic forced after error (3) 2023-08-17 14:45 ` Theodore Ts'o @ 2023-08-18 11:43 ` Aleksandr Nogikh 2023-08-18 16:46 ` Aleksandr Nogikh 0 siblings, 1 reply; 12+ messages in thread From: Aleksandr Nogikh @ 2023-08-18 11:43 UTC (permalink / raw) To: Theodore Ts'o Cc: syzbot, adilger.kernel, linux-ext4, linux-fsdevel, linux-kernel, llvm, nathan, ndesaulniers, syzkaller-bugs, trix I've taken a closer look at the issue. Documentation/filesystems/ext4.txt says that the "errors=" mount parameter "override the errors behavior specified in the superblock". So syzbot can prevent it by passing "errors=continue" as a mount argument and there's no need to filter out such reports. Syzkaller actually already does that in the C reproducer. It just seems that this time the tool has mutated the mount options so much that the simple patching no longer worked (most likely because of \0 characters in between). I'll update the syz_mount_image() code. On Thu, Aug 17, 2023 at 4:45 PM Theodore Ts'o <tytso@mit.edu> wrote: > > On Thu, Aug 17, 2023 at 04:28:33PM +0200, Aleksandr Nogikh wrote: > > The console log has the following line: > > > > [ 60.708717][ T5061] Kernel panic - not syncing: EXT4-fs (device > > loop0): panic forced after error > > > > Can we consider a "panic forced after error" line to be a reliable > > indicator that syzbot must ignore the report? > > Yes. And the file system image that generated this bug should be > discarded, because otherwise successive mutations will generate a > large number of crashes that syzbot will then need to ignore, thus > consuming syzbot resources. > > Alternatively, you can do the moral equivalent of "tune2fs -e continue > foo.img" on any mutated file system seed, which will clear the "panic > on error". > > (The other alternative is "tune2fs -e remount-ro", but given syzbot's > desire to find kernel crashes, "tune2fs -e continue" is more likely > find ways in which the kernel will find itself into trouble. Some > sysadmins will want to chose "remount-ro", however, since that is more > likely to limit file system damage once the file system is discovered > to be corrupted.) > > - Ted > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [syzbot] [ext4?] kernel panic: EXT4-fs (device loop0): panic forced after error (3) 2023-08-18 11:43 ` Aleksandr Nogikh @ 2023-08-18 16:46 ` Aleksandr Nogikh 0 siblings, 0 replies; 12+ messages in thread From: Aleksandr Nogikh @ 2023-08-18 16:46 UTC (permalink / raw) To: Theodore Ts'o Cc: syzbot, adilger.kernel, linux-ext4, linux-fsdevel, linux-kernel, llvm, nathan, ndesaulniers, syzkaller-bugs, trix On Fri, Aug 18, 2023 at 1:43 PM Aleksandr Nogikh <nogikh@google.com> wrote: > > I've taken a closer look at the issue. > > Documentation/filesystems/ext4.txt says that the "errors=" mount > parameter "override the errors behavior specified in the superblock". > So syzbot can prevent it by passing "errors=continue" as a mount > argument and there's no need to filter out such reports. > > Syzkaller actually already does that in the C reproducer. It just > seems that this time the tool has mutated the mount options so much > that the simple patching no longer worked (most likely because of \0 > characters in between). I'll update the syz_mount_image() code. Ah, it's a bit trickier -- the syz_mount_image() code is fine. The reproducer first mounts an ext4 image via syz_mount_image(), which appends "errors=continue" to the options and it doesn't lead to the panic. But then the reproducer does a direct mount() call for the loop device previously created in syz_mount_image(), this time _without_ mount options. I've sent https://github.com/google/syzkaller/pull/4143 to sanitize plain mount() calls. > > > On Thu, Aug 17, 2023 at 4:45 PM Theodore Ts'o <tytso@mit.edu> wrote: > > > > On Thu, Aug 17, 2023 at 04:28:33PM +0200, Aleksandr Nogikh wrote: > > > The console log has the following line: > > > > > > [ 60.708717][ T5061] Kernel panic - not syncing: EXT4-fs (device > > > loop0): panic forced after error > > > > > > Can we consider a "panic forced after error" line to be a reliable > > > indicator that syzbot must ignore the report? > > > > Yes. And the file system image that generated this bug should be > > discarded, because otherwise successive mutations will generate a > > large number of crashes that syzbot will then need to ignore, thus > > consuming syzbot resources. > > > > Alternatively, you can do the moral equivalent of "tune2fs -e continue > > foo.img" on any mutated file system seed, which will clear the "panic > > on error". > > > > (The other alternative is "tune2fs -e remount-ro", but given syzbot's > > desire to find kernel crashes, "tune2fs -e continue" is more likely > > find ways in which the kernel will find itself into trouble. Some > > sysadmins will want to chose "remount-ro", however, since that is more > > likely to limit file system damage once the file system is discovered > > to be corrupted.) > > > > - Ted > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [syzbot] [ext4?] kernel panic: EXT4-fs (device loop0): panic forced after error (3) 2023-08-17 14:21 ` Theodore Ts'o 2023-08-17 14:28 ` Aleksandr Nogikh @ 2023-08-17 14:47 ` Eric Sandeen 2023-08-17 16:11 ` Theodore Ts'o 1 sibling, 1 reply; 12+ messages in thread From: Eric Sandeen @ 2023-08-17 14:47 UTC (permalink / raw) To: Theodore Ts'o, syzbot Cc: adilger.kernel, linux-ext4, linux-fsdevel, linux-kernel, llvm, nathan, ndesaulniers, syzkaller-bugs, trix On 8/17/23 9:21 AM, Theodore Ts'o wrote: > On Wed, Aug 16, 2023 at 03:48:49PM -0700, syzbot wrote: >> Hello, >> >> syzbot found the following issue on: >> >> HEAD commit: ae545c3283dc Merge tag 'gpio-fixes-for-v6.5-rc6' of git://.. >> git tree: upstream >> console+strace: https://syzkaller.appspot.com/x/log.txt?x=13e5d553a80000 >> kernel config: https://syzkaller.appspot.com/x/.config?x=171b698bc2e613cf >> dashboard link: https://syzkaller.appspot.com/bug?extid=27eece6916b914a49ce7 >> compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 >> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13433207a80000 >> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=109cd837a80000 >> >> EXT4-fs error (device loop0): ext4_validate_block_bitmap:430: comm syz-executor211: bg 0: block 46: invalid block bitmap >> Kernel panic - not syncing: EXT4-fs (device loop0): panic forced after error > > #syz invalid > > This is fundamentally a syzbot bug. The file system is horrifically > corrupted, *and* the superblock has the "panic on error" (aka "panic > onfile system corruption") bit set. > > This can be desireable because in a failover situation, if the file > system is found to be corrupted, you *want* the primary server to > fail, and let the secondary server to take over. This is a technique > which is decades old. Just to play devil's advocate here - (sorry) - I don't see this as any different from any other "malicious" filesystem image. I've never been a fan of the idea that malicious images are real security threats, but whether the parking lot USB stick paniced the box in an unexpected way or "on purpose," the result is the same ... I wonder if it might make sense to put EXT4_MOUNT_ERRORS_PANIC under a sysctl or something, so that admins can enable it only when needed. Sorry for stealing another 5 minutes of your life. -Eric > So this is Working As Intended, and is a classic example of (a) if you > are root, you can force the file system to crash, and (b) a classic > example of syzbot noise. (Five minutes of my life that I'm never > getting back. :-) > > - Ted > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [syzbot] [ext4?] kernel panic: EXT4-fs (device loop0): panic forced after error (3) 2023-08-17 14:47 ` Eric Sandeen @ 2023-08-17 16:11 ` Theodore Ts'o 2023-08-17 16:47 ` Eric Biggers 0 siblings, 1 reply; 12+ messages in thread From: Theodore Ts'o @ 2023-08-17 16:11 UTC (permalink / raw) To: sandeen Cc: syzbot, adilger.kernel, linux-ext4, linux-fsdevel, linux-kernel, llvm, nathan, ndesaulniers, syzkaller-bugs, trix On Thu, Aug 17, 2023 at 09:47:48AM -0500, Eric Sandeen wrote: > > Just to play devil's advocate here - (sorry) - I don't see this as any > different from any other "malicious" filesystem image. > > I've never been a fan of the idea that malicious images are real security > threats, but whether the parking lot USB stick paniced the box in an > unexpected way or "on purpose," the result is the same ... > > I wonder if it might make sense to put EXT4_MOUNT_ERRORS_PANIC under a > sysctl or something, so that admins can enable it only when needed. Well, if someone is stupid enough to plug in a parking lot USB stick into their system, they get everything they deserve. And a forced panic isn't going to lead a more privilege escalation attack, so I really don't see a problem if a file system which is marked "panic on error", well, causes a panic. It's a good way of (harmlessly) punishing stupid user tricks. :-) The other way of thinking about it is that if your threat model includes an attacker with physical access to the server with a USB port, attacks include a cable which has a USB port on one side, and a 120V/240V AC mains plug on the the other. This will very likely cause a system shutdown, even if they don't have automount enabled. :-) - Ted ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [syzbot] [ext4?] kernel panic: EXT4-fs (device loop0): panic forced after error (3) 2023-08-17 16:11 ` Theodore Ts'o @ 2023-08-17 16:47 ` Eric Biggers 2023-08-18 2:10 ` Theodore Ts'o 0 siblings, 1 reply; 12+ messages in thread From: Eric Biggers @ 2023-08-17 16:47 UTC (permalink / raw) To: Theodore Ts'o Cc: sandeen, syzbot, adilger.kernel, linux-ext4, linux-fsdevel, linux-kernel, llvm, nathan, ndesaulniers, syzkaller-bugs, trix On Thu, Aug 17, 2023 at 12:11:18PM -0400, Theodore Ts'o wrote: > On Thu, Aug 17, 2023 at 09:47:48AM -0500, Eric Sandeen wrote: > > > > Just to play devil's advocate here - (sorry) - I don't see this as any > > different from any other "malicious" filesystem image. > > > > I've never been a fan of the idea that malicious images are real security > > threats, but whether the parking lot USB stick paniced the box in an > > unexpected way or "on purpose," the result is the same ... > > > > I wonder if it might make sense to put EXT4_MOUNT_ERRORS_PANIC under a > > sysctl or something, so that admins can enable it only when needed. > > Well, if someone is stupid enough to plug in a parking lot USB stick > into their system, they get everything they deserve. And a forced > panic isn't going to lead a more privilege escalation attack, so I > really don't see a problem if a file system which is marked "panic on > error", well, causes a panic. It's a good way of (harmlessly) > punishing stupid user tricks. :-) > > The other way of thinking about it is that if your threat model > includes an attacker with physical access to the server with a USB > port, attacks include a cable which has a USB port on one side, and a > 120V/240V AC mains plug on the the other. This will very likely cause > a system shutdown, even if they don't have automount enabled. :-) > Eric S. is correct that for a filesystem image to enable panic on error, support for panic on error should have to be properly consented to by the kernel configuration, for example through an fs.allow_panic_on_error sysctl. It can be argued that this not important, or not worth implementing when the default will need to remain 1 for backwards compatibility. Or even that syzkaller should work around it in the mean time. But it is incorrect to write "This is fundamentally a syzbot bug." - Eric ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [syzbot] [ext4?] kernel panic: EXT4-fs (device loop0): panic forced after error (3) 2023-08-17 16:47 ` Eric Biggers @ 2023-08-18 2:10 ` Theodore Ts'o 2023-08-18 2:52 ` Eric Biggers 0 siblings, 1 reply; 12+ messages in thread From: Theodore Ts'o @ 2023-08-18 2:10 UTC (permalink / raw) To: Eric Biggers Cc: sandeen, syzbot, adilger.kernel, linux-ext4, linux-fsdevel, linux-kernel, llvm, nathan, ndesaulniers, syzkaller-bugs, trix On Thu, Aug 17, 2023 at 09:47:39AM -0700, Eric Biggers wrote: > > Eric S. is correct that for a filesystem image to enable panic on error, support > for panic on error should have to be properly consented to by the kernel > configuration, for example through an fs.allow_panic_on_error sysctl. I disagree. It's up to the system administrator, not the kernel --- and the system adminsitrator is perfectly free to run e2fsck on a random file system, or to use tune2fs to adjust the panic on error setting on the file system, befure using their root powers to mount the file system. Root can do many things that cause the system to reboot. For example, the system adminsirtator could run /sbin/reboot. Should the kernel "consent" by setting fs.allow_reboot_system_call_to_work before the root user can run the /sbin/reboot binary? Hopefully it's obvious why this makes absolutely no sense. > It can be argued that this not important, or not worth implementing when the > default will need to remain 1 for backwards compatibility. Or even that > syzkaller should work around it in the mean time. But it is incorrect to write > "This is fundamentally a syzbot bug." Well, the current behaviour is Working as Intended. And if syzbot is going about whining about things that are Working as Intended, it's not fit for the upostream developers' purpose. As another example, root can set a real-time priority of a process to be at a level where it will prempt all other processes, including kernel threads. Do enough of these, and you *will* lock up the kernel. Again, should there be a sysctl that allows real-time priorities to work? Or do we teach syzbot that doing things that are documented to cause the kernel to lock up are not something that's worthy of a report. In the past, syzbot issued a *huge* amount of noise caused by precisely to this. Upstream developers complained that it was a false positive, and syzbot was adjusted to Stop Doing That. - Ted ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [syzbot] [ext4?] kernel panic: EXT4-fs (device loop0): panic forced after error (3) 2023-08-18 2:10 ` Theodore Ts'o @ 2023-08-18 2:52 ` Eric Biggers 2023-08-18 14:25 ` Theodore Ts'o 0 siblings, 1 reply; 12+ messages in thread From: Eric Biggers @ 2023-08-18 2:52 UTC (permalink / raw) To: Theodore Ts'o Cc: sandeen, syzbot, adilger.kernel, linux-ext4, linux-fsdevel, linux-kernel, llvm, nathan, ndesaulniers, syzkaller-bugs, trix On Thu, Aug 17, 2023 at 10:10:38PM -0400, Theodore Ts'o wrote: > On Thu, Aug 17, 2023 at 09:47:39AM -0700, Eric Biggers wrote: > > > > Eric S. is correct that for a filesystem image to enable panic on error, support > > for panic on error should have to be properly consented to by the kernel > > configuration, for example through an fs.allow_panic_on_error sysctl. > > I disagree. It's up to the system administrator, not the kernel --- > and the system adminsitrator is perfectly free to run e2fsck on a > random file system, or to use tune2fs to adjust the panic on error > setting on the file system, befure using their root powers to mount > the file system. > > Root can do many things that cause the system to reboot. For example, > the system adminsirtator could run /sbin/reboot. Should the kernel > "consent" by setting fs.allow_reboot_system_call_to_work before the > root user can run the /sbin/reboot binary? Hopefully it's obvious why > this makes absolutely no sense. > > > It can be argued that this not important, or not worth implementing when the > > default will need to remain 1 for backwards compatibility. Or even that > > syzkaller should work around it in the mean time. But it is incorrect to write > > "This is fundamentally a syzbot bug." > > Well, the current behaviour is Working as Intended. And if syzbot is > going about whining about things that are Working as Intended, it's > not fit for the upostream developers' purpose. > > As another example, root can set a real-time priority of a process to > be at a level where it will prempt all other processes, including > kernel threads. Do enough of these, and you *will* lock up the > kernel. Again, should there be a sysctl that allows real-time > priorities to work? Or do we teach syzbot that doing things that are > documented to cause the kernel to lock up are not something that's > worthy of a report. In the past, syzbot issued a *huge* amount of > noise caused by precisely to this. Upstream developers complained > that it was a false positive, and syzbot was adjusted to Stop Doing > That. Obviously it's up to the system administrator; that should have been clear since I suggested a sysctl. Sorry if I wasn't clear. The point is that there are certain conventions for what is allowed to break the safety guarantees that the kernel provides to userspace, which includes causing a kernel panic. Panics on various problems are configured by /proc/sys/kernel/panic_*. So having to opt-in to panic-on-error, or at least being able to opt-out, by setting a sysctl seems natural. Whereas having mount() being able to automatically panic the kernel with no way to opt-out seems like a violation of broader kernel conventions, even if it happens to be "working as intended" in the ext4 context. Anyway, I'm not actually saying this issue is important. I just get frustrated by the total denial that it could even possibly be considered something that could be improved in the kernel... - Eric ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [syzbot] [ext4?] kernel panic: EXT4-fs (device loop0): panic forced after error (3) 2023-08-18 2:52 ` Eric Biggers @ 2023-08-18 14:25 ` Theodore Ts'o 0 siblings, 0 replies; 12+ messages in thread From: Theodore Ts'o @ 2023-08-18 14:25 UTC (permalink / raw) To: Eric Biggers Cc: sandeen, syzbot, adilger.kernel, linux-ext4, linux-fsdevel, linux-kernel, llvm, nathan, ndesaulniers, syzkaller-bugs, trix On Thu, Aug 17, 2023 at 07:52:55PM -0700, Eric Biggers wrote: > Obviously it's up to the system administrator; that should have been clear since > I suggested a sysctl. Sorry if I wasn't clear. The point is that there are > certain conventions for what is allowed to break the safety guarantees that the > kernel provides to userspace, which includes causing a kernel panic. Panics on > various problems are configured by /proc/sys/kernel/panic_*. So having to > opt-in to panic-on-error, or at least being able to opt-out, by setting a sysctl > seems natural. Whereas having mount() being able to automatically panic the > kernel with no way to opt-out seems like a violation of broader kernel > conventions, even if it happens to be "working as intended" in the ext4 context. The reason why a sysctl isn't really great is because the system administrator might want to configure the behavior on a per-file system basis. And you *can* configure it as a mount option, via "mount -o errors=continue" or "mount -o "errors=panic". The superblock setting is just the default if something isn't explicitly specified as a mount option (either on the command line or in /etc/fstab). So mount does not "automatically" panic the kernel, and there are *plenty* of ways to opt-out. You can use the mount option; you can run "tune2fs -e continue"; you can just !@#!?! run fsck.ext4 before mounting the file system. There are all ways of "opting out." Some of them, such as the last, is even considered best practice --- just as picking up a USB stick, or worse, a firewire drive, in a parking lot, and *not* plugging it into your laptop is considered best practice. - Ted ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2023-08-18 16:47 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-08-16 22:48 [syzbot] [ext4?] kernel panic: EXT4-fs (device loop0): panic forced after error (3) syzbot 2023-08-17 14:21 ` Theodore Ts'o 2023-08-17 14:28 ` Aleksandr Nogikh 2023-08-17 14:45 ` Theodore Ts'o 2023-08-18 11:43 ` Aleksandr Nogikh 2023-08-18 16:46 ` Aleksandr Nogikh 2023-08-17 14:47 ` Eric Sandeen 2023-08-17 16:11 ` Theodore Ts'o 2023-08-17 16:47 ` Eric Biggers 2023-08-18 2:10 ` Theodore Ts'o 2023-08-18 2:52 ` Eric Biggers 2023-08-18 14:25 ` Theodore Ts'o
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.