* bcachefs filesystem corruption
@ 2022-03-16 22:58 Eric Wheeler
2022-03-17 3:44 ` Kent Overstreet
0 siblings, 1 reply; 4+ messages in thread
From: Eric Wheeler @ 2022-03-16 22:58 UTC (permalink / raw)
To: Kent Overstreet; +Cc: linux-bcachefs
Hi Kent,
The MySQL+snapshots deployment we has become corrupt and InnoDB is
crashing with a stack trace and lots of borken database noise.
We are going to rebuild it from other replicas and see if it happens
again. We started using your new-allocator code (commit
3985f5f4eff86ee667da7da6e36ca2c346f56be3) after the original deployment
and we haven't re-loaded since you fixed the snapshot iterator issue, or
maybe it broke during the allocator conversion, can't be certain.
There isn't any especially obvious problem but there are some WARNING's
below that you might look at.
There were lots of these:
WARNING: CPU: 0 PID: 7796 at fs/bcachefs/fs-io.c:1290
bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
and lots of hung processes as seen below with a `grep -A10 hung`. Further
down you will find the whole `dmesg` output.
Will let you know if the corruption happens again after we rebuild the
replicas on bcachefs.
-Eric
~]$ grep hung /tmp/pico.79315 -A10
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/u4:3 state:D stack: 0 pid:10501 ppid: 2 flags:0x00004000
Workqueue: writeback wb_workfn (flush-bcachefs-1)
Call Trace:
<TASK>
__schedule (kernel/sched/core.c:4975 kernel/sched/core.c:6253)
schedule (kernel/sched/core.c:6327 (discriminator 1))
schedule_timeout (kernel/time/timer.c:1858)
? __mod_memcg_lruvec_state (mm/memcontrol.c:635 mm/memcontrol.c:708)
__down (kernel/locking/semaphore.c:225 kernel/locking/semaphore.c:241)
down (kernel/locking/semaphore.c:62)
--
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:mysqld state:D stack: 0 pid:16761 ppid: 16251 flags:0x00000000
Call Trace:
<TASK>
__schedule (kernel/sched/core.c:4975 kernel/sched/core.c:6253)
schedule (kernel/sched/core.c:6327 (discriminator 1))
io_schedule (kernel/sched/core.c:8346 kernel/sched/core.c:8372)
folio_wait_bit (mm/filemap.c:1363 mm/filemap.c:1431)
? filemap_alloc_folio (mm/filemap.c:1143)
folio_wait_writeback (./arch/x86/include/asm/bitops.h:207 ./include/asm-generic/bitops/instrumented-non-atomic.h:135 ./include/linux/page-flags.h:464 mm/page-writeback.c:2923)
__filemap_fdatawait_range (./include/linux/page-flags.h:198 (discriminator 3) ./include/linux/page-flags.h:419 (discriminator 3) mm/filemap.c:544 (discriminator 3))
--
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:mysqld state:D stack: 0 pid:18048 ppid: 17541 flags:0x00000000
Call Trace:
<TASK>
__schedule (kernel/sched/core.c:4975 kernel/sched/core.c:6253)
schedule (kernel/sched/core.c:6327 (discriminator 1))
schedule_timeout (kernel/time/timer.c:1858)
? write_cache_pages (mm/page-writeback.c:2209)
? bch2_truncate_pages (fs/bcachefs/fs-io.c:1361) bcachefs
__down (kernel/locking/semaphore.c:225 kernel/locking/semaphore.c:241)
? preempt_count_add (./include/linux/ftrace.h:894 kernel/sched/core.c:5413 kernel/sched/core.c:5410 kernel/sched/core.c:5438)
--
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:mysqld state:D stack: 0 pid:18340 ppid: 17541 flags:0x00000000
Call Trace:
<TASK>
__schedule (kernel/sched/core.c:4975 kernel/sched/core.c:6253)
schedule (kernel/sched/core.c:6327 (discriminator 1))
schedule_timeout (kernel/time/timer.c:1858)
? write_cache_pages (mm/page-writeback.c:2209)
? bch2_truncate_pages (fs/bcachefs/fs-io.c:1361) bcachefs
__down (kernel/locking/semaphore.c:225 kernel/locking/semaphore.c:241)
? preempt_count_add (./include/linux/ftrace.h:894 kernel/sched/core.c:5413 kernel/sched/core.c:5410 kernel/sched/core.c:5438)
--
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:mysqld state:D stack: 0 pid:18387 ppid: 18206 flags:0x00004000
Call Trace:
<TASK>
__schedule (kernel/sched/core.c:4975 kernel/sched/core.c:6253)
schedule (kernel/sched/core.c:6327 (discriminator 1))
schedule_timeout (kernel/time/timer.c:1858)
? __cond_resched (kernel/sched/core.c:8133)
wait_for_completion (kernel/sched/completion.c:86 kernel/sched/completion.c:106 kernel/sched/completion.c:117 kernel/sched/completion.c:138)
bch2_dio_write_loop (fs/bcachefs/fs-io.c:2207) bcachefs
bch2_direct_write (fs/bcachefs/fs-io.c:2340) bcachefs
--
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:mysqld state:D stack: 0 pid:18394 ppid: 18206 flags:0x00000000
Call Trace:
<TASK>
__schedule (kernel/sched/core.c:4975 kernel/sched/core.c:6253)
schedule (kernel/sched/core.c:6327 (discriminator 1))
schedule_timeout (kernel/time/timer.c:1858)
? __mod_memcg_lruvec_state (mm/memcontrol.c:635 mm/memcontrol.c:708)
__down (kernel/locking/semaphore.c:225 kernel/locking/semaphore.c:241)
? preempt_count_add (./include/linux/ftrace.h:894 kernel/sched/core.c:5413 kernel/sched/core.c:5410 kernel/sched/core.c:5438)
down (kernel/locking/semaphore.c:62)
--
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:mysqld state:D stack: 0 pid:18407 ppid: 18206 flags:0x00000000
Call Trace:
<TASK>
__schedule (kernel/sched/core.c:4975 kernel/sched/core.c:6253)
? __remove_hrtimer (kernel/time/hrtimer.c:1116)
schedule (kernel/sched/core.c:6327 (discriminator 1))
schedule_timeout (kernel/time/timer.c:1858)
? __mod_memcg_lruvec_state (mm/memcontrol.c:635 mm/memcontrol.c:708)
__down (kernel/locking/semaphore.c:225 kernel/locking/semaphore.c:241)
? preempt_count_add (./include/linux/ftrace.h:894 kernel/sched/core.c:5413 kernel/sched/core.c:5410 kernel/sched/core.c:5438)
--
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/u4:3 state:D stack: 0 pid:10501 ppid: 2 flags:0x00004000
Workqueue: writeback wb_workfn (flush-bcachefs-1)
Call Trace:
<TASK>
__schedule (kernel/sched/core.c:4975 kernel/sched/core.c:6253)
schedule (kernel/sched/core.c:6327 (discriminator 1))
schedule_timeout (kernel/time/timer.c:1858)
? __mod_memcg_lruvec_state (mm/memcontrol.c:635 mm/memcontrol.c:708)
__down (kernel/locking/semaphore.c:225 kernel/locking/semaphore.c:241)
down (kernel/locking/semaphore.c:62)
--
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:mysqld state:D stack: 0 pid:16761 ppid: 16251 flags:0x00000000
Call Trace:
<TASK>
__schedule (kernel/sched/core.c:4975 kernel/sched/core.c:6253)
schedule (kernel/sched/core.c:6327 (discriminator 1))
io_schedule (kernel/sched/core.c:8346 kernel/sched/core.c:8372)
folio_wait_bit (mm/filemap.c:1363 mm/filemap.c:1431)
? filemap_alloc_folio (mm/filemap.c:1143)
folio_wait_writeback (./arch/x86/include/asm/bitops.h:207 ./include/asm-generic/bitops/instrumented-non-atomic.h:135 ./include/linux/page-flags.h:464 mm/page-writeback.c:2923)
__filemap_fdatawait_range (./include/linux/page-flags.h:198 (discriminator 3) ./include/linux/page-flags.h:419 (discriminator 3) mm/filemap.c:544 (discriminator 3))
--
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:mysqld state:D stack: 0 pid:18048 ppid: 17541 flags:0x00000000
Call Trace:
<TASK>
__schedule (kernel/sched/core.c:4975 kernel/sched/core.c:6253)
schedule (kernel/sched/core.c:6327 (discriminator 1))
schedule_timeout (kernel/time/timer.c:1858)
? write_cache_pages (mm/page-writeback.c:2209)
? bch2_truncate_pages (fs/bcachefs/fs-io.c:1361) bcachefs
__down (kernel/locking/semaphore.c:225 kernel/locking/semaphore.c:241)
? preempt_count_add (./include/linux/ftrace.h:894 kernel/sched/core.c:5413 kernel/sched/core.c:5410 kernel/sched/core.c:5438)
Linux version 5.16.0+ (root@rl8-dev.ewheeler.net) (gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-4), GNU ld version 2.30-108.el8_5.1) #1 SMP PREEMPT Tue Mar 15 10:05:37 PDT 2022
Command line: BOOT_IMAGE=/vmlinuz-5.16.0+ root=/dev/mapper/centos_sbo-root ro rd.lvm.lv=centos_sbo/root rd.lvm.lv=centos_sbo/swap rd.lvm.lv=centos_sbo/usr
[...]
bcachefs (dm-6): initializing new filesystem
bcachefs (dm-6): going read-write
bucket 0:10 gen 0 different types of data in same bucket: btree, journal
while marking journal
bcachefs (dm-6): inconsistency detected - emergency read only
bcachefs: bch2_fs_initialize() Error initializing new filesystem: error marking superblock and journal (-5)
bcachefs (dm-6): IO error
bcachefs (dm-6): recovering from clean shutdown, journal seq 420
bcachefs (dm-6): going read-write
bcachefs (dm-6): mounted with opts: noinodes_use_key_cache
------------[ cut here ]------------
WARNING: CPU: 0 PID: 7796 at fs/bcachefs/fs-io.c:1290 bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Modules linked in: bcachefs lz4_compress crc64 xor raid6_pq zstd_compress libcrc32c xt_comment iptable_filter binfmt_misc rfkill cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr sg i2c_piix4 virtio_balloon floppy i2c_core ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi ata_piix libata virtio_net net_failover failover serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 7796 Comm: kworker/0:2 Not tainted 5.16.0+ #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: bcachefs bch2_write_index [bcachefs]
RIP: 0010:bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Code: 80 64 24 28 f0 e8 58 6d 23 f6 0f b7 83 a0 03 00 00 44 39 f8 0f 8f 63 ff ff ff 48 8b b3 48 01 00 00 48 85 f6 0f 8e 82 fd ff ff <0f> 0b e9 7b fd ff ff 48 83 c5 40 b9 00 10 00 00 e9 bf fd ff ff 8b
All code
========
0: 80 64 24 28 f0 andb $0xf0,0x28(%rsp)
5: e8 58 6d 23 f6 callq 0xfffffffff6236d62
a: 0f b7 83 a0 03 00 00 movzwl 0x3a0(%rbx),%eax
11: 44 39 f8 cmp %r15d,%eax
14: 0f 8f 63 ff ff ff jg 0xffffffffffffff7d
1a: 48 8b b3 48 01 00 00 mov 0x148(%rbx),%rsi
21: 48 85 f6 test %rsi,%rsi
24: 0f 8e 82 fd ff ff jle 0xfffffffffffffdac
2a:* 0f 0b ud2 <-- trapping instruction
2c: e9 7b fd ff ff jmpq 0xfffffffffffffdac
31: 48 83 c5 40 add $0x40,%rbp
35: b9 00 10 00 00 mov $0x1000,%ecx
3a: e9 bf fd ff ff jmpq 0xfffffffffffffdfe
3f: 8b .byte 0x8b
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: e9 7b fd ff ff jmpq 0xfffffffffffffd82
7: 48 83 c5 40 add $0x40,%rbp
b: b9 00 10 00 00 mov $0x1000,%ecx
10: e9 bf fd ff ff jmpq 0xfffffffffffffdd4
15: 8b .byte 0x8b
RSP: 0018:ffffa6c4418d3e68 EFLAGS: 00010202
RAX: 0000000080000000 RBX: ffff8c2cc6604000 RCX: 0000000000000001
RDX: 0000000080000001 RSI: 0000000000000008 RDI: 00000000ffffffff
RBP: ffff8c2cff415540 R08: 000000000007c12a R09: 0000000000000000
R10: 0000000000000000 R11: 00000000ffffffff R12: ffffc6c43fc08f00
R13: 0000000000000020 R14: ffff8c2d7ca2ee00 R15: ffffc6c43fc08f05
FS: 0000000000000000(0000) GS:ffff8c2d7ca00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9b5c3504b1 CR3: 0000000004b5e000 CR4: 00000000000006f0
Call Trace:
<TASK>
process_one_work (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:212 ./include/trace/events/workqueue.h:108 kernel/workqueue.c:2303)
worker_thread (./include/linux/list.h:284 kernel/workqueue.c:2446)
? process_one_work (kernel/workqueue.c:2388)
kthread (kernel/kthread.c:327)
? set_kthread_struct (kernel/kthread.c:272)
ret_from_fork (arch/x86/entry/entry_64.S:301)
</TASK>
---[ end trace ae1f1e0405dedf14 ]---
------------[ cut here ]------------
WARNING: CPU: 0 PID: 7796 at fs/bcachefs/fs-io.c:1290 bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Modules linked in: bcachefs lz4_compress crc64 xor raid6_pq zstd_compress libcrc32c xt_comment iptable_filter binfmt_misc rfkill cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr sg i2c_piix4 virtio_balloon floppy i2c_core ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi ata_piix libata virtio_net net_failover failover serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 7796 Comm: kworker/0:2 Tainted: G W 5.16.0+ #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: bcachefs bch2_write_index [bcachefs]
RIP: 0010:bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Code: 80 64 24 28 f0 e8 58 6d 23 f6 0f b7 83 a0 03 00 00 44 39 f8 0f 8f 63 ff ff ff 48 8b b3 48 01 00 00 48 85 f6 0f 8e 82 fd ff ff <0f> 0b e9 7b fd ff ff 48 83 c5 40 b9 00 10 00 00 e9 bf fd ff ff 8b
All code
========
0: 80 64 24 28 f0 andb $0xf0,0x28(%rsp)
5: e8 58 6d 23 f6 callq 0xfffffffff6236d62
a: 0f b7 83 a0 03 00 00 movzwl 0x3a0(%rbx),%eax
11: 44 39 f8 cmp %r15d,%eax
14: 0f 8f 63 ff ff ff jg 0xffffffffffffff7d
1a: 48 8b b3 48 01 00 00 mov 0x148(%rbx),%rsi
21: 48 85 f6 test %rsi,%rsi
24: 0f 8e 82 fd ff ff jle 0xfffffffffffffdac
2a:* 0f 0b ud2 <-- trapping instruction
2c: e9 7b fd ff ff jmpq 0xfffffffffffffdac
31: 48 83 c5 40 add $0x40,%rbp
35: b9 00 10 00 00 mov $0x1000,%ecx
3a: e9 bf fd ff ff jmpq 0xfffffffffffffdfe
3f: 8b .byte 0x8b
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: e9 7b fd ff ff jmpq 0xfffffffffffffd82
7: 48 83 c5 40 add $0x40,%rbp
b: b9 00 10 00 00 mov $0x1000,%ecx
10: e9 bf fd ff ff jmpq 0xfffffffffffffdd4
15: 8b .byte 0x8b
RSP: 0018:ffffa6c4418d3e68 EFLAGS: 00010202
RAX: 0000000080000000 RBX: ffff8c2cfda3e000 RCX: 0000000000000001
RDX: 0000000080000001 RSI: 0000000000000008 RDI: 00000000ffffffff
RBP: ffff8c2cff415540 R08: 000000000007c159 R09: 0000000000000000
R10: 0000000000000000 R11: 00000000ffffffff R12: ffffc6c43fc08f00
R13: 0000000000000020 R14: ffff8c2d7ca2ee00 R15: ffffc6c43fc08f05
FS: 0000000000000000(0000) GS:ffff8c2d7ca00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8f454f5800 CR3: 0000000004b5e000 CR4: 00000000000006f0
Call Trace:
<TASK>
process_one_work (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:212 ./include/trace/events/workqueue.h:108 kernel/workqueue.c:2303)
worker_thread (./include/linux/list.h:284 kernel/workqueue.c:2446)
? process_one_work (kernel/workqueue.c:2388)
kthread (kernel/kthread.c:327)
? set_kthread_struct (kernel/kthread.c:272)
ret_from_fork (arch/x86/entry/entry_64.S:301)
</TASK>
---[ end trace ae1f1e0405dedf15 ]---
INFO: task kworker/u4:3:10501 blocked for more than 122 seconds.
Tainted: G W 5.16.0+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/u4:3 state:D stack: 0 pid:10501 ppid: 2 flags:0x00004000
Workqueue: writeback wb_workfn (flush-bcachefs-1)
Call Trace:
<TASK>
__schedule (kernel/sched/core.c:4975 kernel/sched/core.c:6253)
schedule (kernel/sched/core.c:6327 (discriminator 1))
schedule_timeout (kernel/time/timer.c:1858)
? __mod_memcg_lruvec_state (mm/memcontrol.c:635 mm/memcontrol.c:708)
__down (kernel/locking/semaphore.c:225 kernel/locking/semaphore.c:241)
down (kernel/locking/semaphore.c:62)
bch2_writepage_do_io.isra.70 (fs/bcachefs/fs-io.c:1321) bcachefs
__bch2_writepage (fs/bcachefs/fs-io.c:1465) bcachefs
? percpu_counter_add_batch (./arch/x86/include/asm/preempt.h:103 (discriminator 82) lib/percpu_counter.c:97 (discriminator 82))
write_cache_pages (mm/page-writeback.c:2256)
? bch2_truncate_pages (fs/bcachefs/fs-io.c:1361) bcachefs
bch2_writepages (fs/bcachefs/fs-io.c:1502) bcachefs
do_writepages (mm/page-writeback.c:2367)
__writeback_single_inode (fs/fs-writeback.c:1616)
writeback_sb_inodes (fs/fs-writeback.c:1883)
__writeback_inodes_wb (fs/fs-writeback.c:1952)
wb_writeback (fs/fs-writeback.c:2056)
[78646.329608] wb_workfn (fs/fs-writeback.c:2156 fs/fs-writeback.c:2209 fs/fs-writeback.c:2238)
? _raw_spin_unlock (./arch/x86/include/asm/preempt.h:103 ./include/linux/spinlock_api_smp.h:143 kernel/locking/spinlock.c:186)
process_one_work (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:212 ./include/trace/events/workqueue.h:108 kernel/workqueue.c:2303)
worker_thread (./include/linux/list.h:284 kernel/workqueue.c:2446)
? process_one_work (kernel/workqueue.c:2388)
kthread (kernel/kthread.c:327)
? set_kthread_struct (kernel/kthread.c:272)
ret_from_fork (arch/x86/entry/entry_64.S:301)
</TASK>
INFO: task mysqld:16761 blocked for more than 122 seconds.
Tainted: G W 5.16.0+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:mysqld state:D stack: 0 pid:16761 ppid: 16251 flags:0x00000000
Call Trace:
<TASK>
__schedule (kernel/sched/core.c:4975 kernel/sched/core.c:6253)
schedule (kernel/sched/core.c:6327 (discriminator 1))
io_schedule (kernel/sched/core.c:8346 kernel/sched/core.c:8372)
folio_wait_bit (mm/filemap.c:1363 mm/filemap.c:1431)
? filemap_alloc_folio (mm/filemap.c:1143)
folio_wait_writeback (./arch/x86/include/asm/bitops.h:207 ./include/asm-generic/bitops/instrumented-non-atomic.h:135 ./include/linux/page-flags.h:464 mm/page-writeback.c:2923)
__filemap_fdatawait_range (./include/linux/page-flags.h:198 (discriminator 3) ./include/linux/page-flags.h:419 (discriminator 3) mm/filemap.c:544 (discriminator 3))
file_write_and_wait_range (mm/filemap.c:830)
bch2_fsync (fs/bcachefs/fs-io.c:2419) bcachefs
do_fsync (./include/linux/file.h:45 fs/sync.c:207)
__x64_sys_fsync (fs/sync.c:214 fs/sync.c:212 fs/sync.c:212)
do_syscall_64 (arch/x86/entry/common.c:50 arch/x86/entry/common.c:80)
entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:113)
RIP: 0033:0x7f0eb7663d7d
RSP: 002b:00007f0e997f9620 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
RAX: ffffffffffffffda RBX: 00007f0e997f9640 RCX: 00007f0eb7663d7d
RDX: 0000af3600000000 RSI: 0000000000000000 RDI: 000000000000000b
RBP: 00007f0e997f97e0 R08: 00007f0e9c2e4360 R09: 00000000ffffffef
R10: 0000000000000002 R11: 0000000000000293 R12: 0000000000000000
R13: 000000000000000b R14: 00007f0e997f96a0 R15: 00007f0e9c4b7a78
</TASK>
INFO: task mysqld:18048 blocked for more than 122 seconds.
Tainted: G W 5.16.0+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:mysqld state:D stack: 0 pid:18048 ppid: 17541 flags:0x00000000
Call Trace:
<TASK>
__schedule (kernel/sched/core.c:4975 kernel/sched/core.c:6253)
schedule (kernel/sched/core.c:6327 (discriminator 1))
schedule_timeout (kernel/time/timer.c:1858)
? write_cache_pages (mm/page-writeback.c:2209)
? bch2_truncate_pages (fs/bcachefs/fs-io.c:1361) bcachefs
__down (kernel/locking/semaphore.c:225 kernel/locking/semaphore.c:241)
? preempt_count_add (./include/linux/ftrace.h:894 kernel/sched/core.c:5413 kernel/sched/core.c:5410 kernel/sched/core.c:5438)
down (kernel/locking/semaphore.c:62)
bch2_writepage_do_io.isra.70 (fs/bcachefs/fs-io.c:1321) bcachefs
bch2_writepages (fs/bcachefs/fs-io.c:1504) bcachefs
do_writepages (mm/page-writeback.c:2367)
filemap_fdatawrite_wbc (mm/filemap.c:408 mm/filemap.c:397)
file_write_and_wait_range (mm/filemap.c:440 mm/filemap.c:824)
bch2_fsync (fs/bcachefs/fs-io.c:2419) bcachefs
do_fsync (./include/linux/file.h:45 fs/sync.c:207)
__x64_sys_fdatasync (fs/sync.c:219 fs/sync.c:217 fs/sync.c:217)
do_syscall_64 (arch/x86/entry/common.c:50 arch/x86/entry/common.c:80)
entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:113)
RIP: 0033:0x7f367975dc6d
RSP: 002b:00007f35c07fa560 EFLAGS: 00000293 ORIG_RAX: 000000000000004b
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f367975dc6d
RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000009
RBP: 00007f35c07fa590 R08: 0000000000000800 R09: 000000105269934b
R10: 0000000000000019 R11: 0000000000000293 R12: 0000000000000009
R13: 0000000051eb851f R14: 00007f35cf2953f8 R15: 00000000000023fc
</TASK>
INFO: task mysqld:18340 blocked for more than 122 seconds.
Tainted: G W 5.16.0+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:mysqld state:D stack: 0 pid:18340 ppid: 17541 flags:0x00000000
Call Trace:
<TASK>
__schedule (kernel/sched/core.c:4975 kernel/sched/core.c:6253)
schedule (kernel/sched/core.c:6327 (discriminator 1))
schedule_timeout (kernel/time/timer.c:1858)
? write_cache_pages (mm/page-writeback.c:2209)
? bch2_truncate_pages (fs/bcachefs/fs-io.c:1361) bcachefs
__down (kernel/locking/semaphore.c:225 kernel/locking/semaphore.c:241)
? preempt_count_add (./include/linux/ftrace.h:894 kernel/sched/core.c:5413 kernel/sched/core.c:5410 kernel/sched/core.c:5438)
down (kernel/locking/semaphore.c:62)
bch2_writepage_do_io.isra.70 (fs/bcachefs/fs-io.c:1321) bcachefs
bch2_writepages (fs/bcachefs/fs-io.c:1504) bcachefs
do_writepages (mm/page-writeback.c:2367)
filemap_fdatawrite_wbc (mm/filemap.c:408 mm/filemap.c:397)
file_write_and_wait_range (mm/filemap.c:440 mm/filemap.c:824)
bch2_fsync (fs/bcachefs/fs-io.c:2419) bcachefs
do_fsync (./include/linux/file.h:45 fs/sync.c:207)
__x64_sys_fdatasync (fs/sync.c:219 fs/sync.c:217 fs/sync.c:217)
do_syscall_64 (arch/x86/entry/common.c:50 arch/x86/entry/common.c:80)
entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:113)
RIP: 0033:0x7f367975dc6d
RSP: 002b:00007f367aa357d0 EFLAGS: 00000293 ORIG_RAX: 000000000000004b
RAX: ffffffffffffffda RBX: 00007f35cf3e92f8 RCX: 00007f367975dc6d
RDX: 00007f367aa36a50 RSI: 0000000000000001 RDI: 000000000000002f
RBP: 00007f367aa35810 R08: 0000000000000000 R09: 00007f35cf3e8648
R10: 0000000000000001 R11: 0000000000000293 R12: 0000000000000000
R13: 000000000000002f R14: 0000000000000010 R15: 00007f35cf3e8000
</TASK>
INFO: task mysqld:18387 blocked for more than 122 seconds.
Tainted: G W 5.16.0+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:mysqld state:D stack: 0 pid:18387 ppid: 18206 flags:0x00004000
Call Trace:
<TASK>
__schedule (kernel/sched/core.c:4975 kernel/sched/core.c:6253)
schedule (kernel/sched/core.c:6327 (discriminator 1))
schedule_timeout (kernel/time/timer.c:1858)
? __cond_resched (kernel/sched/core.c:8133)
wait_for_completion (kernel/sched/completion.c:86 kernel/sched/completion.c:106 kernel/sched/completion.c:117 kernel/sched/completion.c:138)
bch2_dio_write_loop (fs/bcachefs/fs-io.c:2207) bcachefs
bch2_direct_write (fs/bcachefs/fs-io.c:2340) bcachefs
new_sync_write (fs/read_write.c:504 (discriminator 1))
vfs_write (fs/read_write.c:590 fs/read_write.c:570)
ksys_pwrite64 (./include/linux/file.h:45 fs/read_write.c:698)
? syscall_trace_enter.isra.20 (./include/linux/audit.h:350 kernel/entry/common.c:40 kernel/entry/common.c:79)
do_syscall_64 (arch/x86/entry/common.c:50 arch/x86/entry/common.c:80)
entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:113)
RIP: 0033:0x7f515a9b9023
RSP: 002b:00007f513dff4130 EFLAGS: 00000293 ORIG_RAX: 0000000000000012
RAX: ffffffffffffffda RBX: 00000000000e0000 RCX: 00007f515a9b9023
RDX: 00000000000e0000 RSI: 00007f513d6f4000 RDI: 0000000000000006
RBP: 00007f513dff4320 R08: 0000000000200000 R09: 00007f513dff4340
R10: 0000000000200000 R11: 0000000000000293 R12: 0000000000000000
R13: 00007f513d6f4000 R14: 00000000000e0000 R15: 0000000000200000
</TASK>
WARNING: CPU: 0 PID: 24621 at fs/bcachefs/fs-io.c:1290 bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Modules linked in: bcachefs lz4_compress crc64 xor raid6_pq zstd_compress libcrc32c xt_comment iptable_filter binfmt_misc rfkill cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr sg i2c_piix4 virtio_balloon floppy i2c_core ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi ata_piix libata virtio_net net_failover failover serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 24621 Comm: kworker/0:0 Tainted: G W 5.16.0+ #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: bcachefs bch2_write_index [bcachefs]
RIP: 0010:bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Code: 80 64 24 28 f0 e8 58 6d 23 f6 0f b7 83 a0 03 00 00 44 39 f8 0f 8f 63 ff ff ff 48 8b b3 48 01 00 00 48 85 f6 0f 8e 82 fd ff ff <0f> 0b e9 7b fd ff ff 48 83 c5 40 b9 00 10 00 00 e9 bf fd ff ff 8b
All code
========
0: 80 64 24 28 f0 andb $0xf0,0x28(%rsp)
5: e8 58 6d 23 f6 callq 0xfffffffff6236d62
a: 0f b7 83 a0 03 00 00 movzwl 0x3a0(%rbx),%eax
11: 44 39 f8 cmp %r15d,%eax
14: 0f 8f 63 ff ff ff jg 0xffffffffffffff7d
1a: 48 8b b3 48 01 00 00 mov 0x148(%rbx),%rsi
21: 48 85 f6 test %rsi,%rsi
24: 0f 8e 82 fd ff ff jle 0xfffffffffffffdac
2a:* 0f 0b ud2 <-- trapping instruction
2c: e9 7b fd ff ff jmpq 0xfffffffffffffdac
31: 48 83 c5 40 add $0x40,%rbp
35: b9 00 10 00 00 mov $0x1000,%ecx
3a: e9 bf fd ff ff jmpq 0xfffffffffffffdfe
3f: 8b .byte 0x8b
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: e9 7b fd ff ff jmpq 0xfffffffffffffd82
7: 48 83 c5 40 add $0x40,%rbp
b: b9 00 10 00 00 mov $0x1000,%ecx
10: e9 bf fd ff ff jmpq 0xfffffffffffffdd4
15: 8b .byte 0x8b
RSP: 0018:ffffa6c447f2fe68 EFLAGS: 00010202
RAX: 0000000080000000 RBX: ffff8c2cc469e000 RCX: 0000000000000001
RDX: 0000000080000001 RSI: 0000000000000008 RDI: 00000000ffffffff
RBP: ffff8c2cc46f2540 R08: 0000000000047670 R09: 0000000000000001
R10: 0000000000000000 R11: 00000000ffffffff R12: ffffc6c43fc08f00
R13: 0000000000000020 R14: ffff8c2d7ca2ee00 R15: ffffc6c43fc08f05
FS: 0000000000000000(0000) GS:ffff8c2d7ca00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000020f8000 CR3: 000000003f6e2000 CR4: 00000000000006f0
Call Trace:
<TASK>
process_one_work (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:212 ./include/trace/events/workqueue.h:108 kernel/workqueue.c:2303)
worker_thread (./include/linux/list.h:284 kernel/workqueue.c:2446)
? process_one_work (kernel/workqueue.c:2388)
kthread (kernel/kthread.c:327)
? set_kthread_struct (kernel/kthread.c:272)
ret_from_fork (arch/x86/entry/entry_64.S:301)
</TASK>
---[ end trace ae1f1e0405dedf16 ]---
------------[ cut here ]------------
WARNING: CPU: 0 PID: 21693 at fs/bcachefs/fs-io.c:1290 bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Modules linked in: bcachefs lz4_compress crc64 xor raid6_pq zstd_compress libcrc32c xt_comment iptable_filter binfmt_misc rfkill cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr sg i2c_piix4 virtio_balloon floppy i2c_core ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi ata_piix libata virtio_net net_failover failover serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 21693 Comm: kworker/0:3 Tainted: G W 5.16.0+ #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: bcachefs bch2_write_index [bcachefs]
RIP: 0010:bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Code: 80 64 24 28 f0 e8 58 6d 23 f6 0f b7 83 a0 03 00 00 44 39 f8 0f 8f 63 ff ff ff 48 8b b3 48 01 00 00 48 85 f6 0f 8e 82 fd ff ff <0f> 0b e9 7b fd ff ff 48 83 c5 40 b9 00 10 00 00 e9 bf fd ff ff 8b
All code
========
0: 80 64 24 28 f0 andb $0xf0,0x28(%rsp)
5: e8 58 6d 23 f6 callq 0xfffffffff6236d62
a: 0f b7 83 a0 03 00 00 movzwl 0x3a0(%rbx),%eax
11: 44 39 f8 cmp %r15d,%eax
14: 0f 8f 63 ff ff ff jg 0xffffffffffffff7d
1a: 48 8b b3 48 01 00 00 mov 0x148(%rbx),%rsi
21: 48 85 f6 test %rsi,%rsi
24: 0f 8e 82 fd ff ff jle 0xfffffffffffffdac
2a:* 0f 0b ud2 <-- trapping instruction
2c: e9 7b fd ff ff jmpq 0xfffffffffffffdac
31: 48 83 c5 40 add $0x40,%rbp
35: b9 00 10 00 00 mov $0x1000,%ecx
3a: e9 bf fd ff ff jmpq 0xfffffffffffffdfe
3f: 8b .byte 0x8b
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: e9 7b fd ff ff jmpq 0xfffffffffffffd82
7: 48 83 c5 40 add $0x40,%rbp
b: b9 00 10 00 00 mov $0x1000,%ecx
10: e9 bf fd ff ff jmpq 0xfffffffffffffdd4
15: 8b .byte 0x8b
RSP: 0018:ffffa6c445b97e68 EFLAGS: 00010202
RAX: 0000000080000000 RBX: ffff8c2cff0b7c00 RCX: 0000000000000001
RDX: 0000000080000001 RSI: 0000000000000020 RDI: 00000000ffffffff
RBP: ffff8c2cc46f20c0 R08: 000000000007e0fa R09: 0000000000000000
R10: 0000000000000000 R11: 00000000ffffffff R12: ffffc6c43fc08f00
R13: 0000000000000020 R14: ffff8c2d7ca2ee00 R15: ffffc6c43fc08f05
FS: 0000000000000000(0000) GS:ffff8c2d7ca00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2eaff77148 CR3: 000000003f6e2000 CR4: 00000000000006f0
Call Trace:
<TASK>
process_one_work (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:212 ./include/trace/events/workqueue.h:108 kernel/workqueue.c:2303)
worker_thread (./include/linux/list.h:284 kernel/workqueue.c:2446)
? process_one_work (kernel/workqueue.c:2388)
kthread (kernel/kthread.c:327)
? set_kthread_struct (kernel/kthread.c:272)
ret_from_fork (arch/x86/entry/entry_64.S:301)
</TASK>
---[ end trace ae1f1e0405dedf17 ]---
------------[ cut here ]------------
WARNING: CPU: 0 PID: 5734 at fs/bcachefs/fs-io.c:1290 bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Modules linked in: bcachefs lz4_compress crc64 xor raid6_pq zstd_compress libcrc32c xt_comment iptable_filter binfmt_misc rfkill cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr sg i2c_piix4 virtio_balloon floppy i2c_core ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi ata_piix libata virtio_net net_failover failover serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 5734 Comm: kworker/0:4 Tainted: G W 5.16.0+ #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: bcachefs bch2_write_index [bcachefs]
RIP: 0010:bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Code: 80 64 24 28 f0 e8 58 6d 23 f6 0f b7 83 a0 03 00 00 44 39 f8 0f 8f 63 ff ff ff 48 8b b3 48 01 00 00 48 85 f6 0f 8e 82 fd ff ff <0f> 0b e9 7b fd ff ff 48 83 c5 40 b9 00 10 00 00 e9 bf fd ff ff 8b
All code
========
0: 80 64 24 28 f0 andb $0xf0,0x28(%rsp)
5: e8 58 6d 23 f6 callq 0xfffffffff6236d62
a: 0f b7 83 a0 03 00 00 movzwl 0x3a0(%rbx),%eax
11: 44 39 f8 cmp %r15d,%eax
14: 0f 8f 63 ff ff ff jg 0xffffffffffffff7d
1a: 48 8b b3 48 01 00 00 mov 0x148(%rbx),%rsi
21: 48 85 f6 test %rsi,%rsi
24: 0f 8e 82 fd ff ff jle 0xfffffffffffffdac
2a:* 0f 0b ud2 <-- trapping instruction
2c: e9 7b fd ff ff jmpq 0xfffffffffffffdac
31: 48 83 c5 40 add $0x40,%rbp
35: b9 00 10 00 00 mov $0x1000,%ecx
3a: e9 bf fd ff ff jmpq 0xfffffffffffffdfe
3f: 8b .byte 0x8b
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: e9 7b fd ff ff jmpq 0xfffffffffffffd82
7: 48 83 c5 40 add $0x40,%rbp
b: b9 00 10 00 00 mov $0x1000,%ecx
10: e9 bf fd ff ff jmpq 0xfffffffffffffdd4
15: 8b .byte 0x8b
RSP: 0018:ffffa6c440fbbe68 EFLAGS: 00010202
RAX: 0000000080000000 RBX: ffff8c2cff1b2800 RCX: 0000000000000001
RDX: 0000000080000001 RSI: 0000000000000020 RDI: 00000000ffffffff
RBP: ffff8c2cc46f2f00 R08: 000000000007e340 R09: 0000000000000000
R10: 0000000000000000 R11: 00000000ffffffff R12: ffffc6c43fc08f00
R13: 0000000000000020 R14: ffff8c2d7ca2ee00 R15: ffffc6c43fc08f05
FS: 0000000000000000(0000) GS:ffff8c2d7ca00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd3c6b36000 CR3: 000000003f6e2000 CR4: 00000000000006f0
Call Trace:
<TASK>
process_one_work (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:212 ./include/trace/events/workqueue.h:108 kernel/workqueue.c:2303)
worker_thread (./include/linux/list.h:284 kernel/workqueue.c:2446)
? process_one_work (kernel/workqueue.c:2388)
kthread (kernel/kthread.c:327)
? set_kthread_struct (kernel/kthread.c:272)
ret_from_fork (arch/x86/entry/entry_64.S:301)
</TASK>
---[ end trace ae1f1e0405dedf18 ]---
------------[ cut here ]------------
WARNING: CPU: 0 PID: 21693 at fs/bcachefs/fs-io.c:1290 bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Modules linked in: bcachefs lz4_compress crc64 xor raid6_pq zstd_compress libcrc32c xt_comment iptable_filter binfmt_misc rfkill cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr sg i2c_piix4 virtio_balloon floppy i2c_core ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi ata_piix libata virtio_net net_failover failover serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 21693 Comm: kworker/0:3 Tainted: G W 5.16.0+ #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: bcachefs bch2_write_index [bcachefs]
RIP: 0010:bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Code: 80 64 24 28 f0 e8 58 6d 23 f6 0f b7 83 a0 03 00 00 44 39 f8 0f 8f 63 ff ff ff 48 8b b3 48 01 00 00 48 85 f6 0f 8e 82 fd ff ff <0f> 0b e9 7b fd ff ff 48 83 c5 40 b9 00 10 00 00 e9 bf fd ff ff 8b
All code
========
0: 80 64 24 28 f0 andb $0xf0,0x28(%rsp)
5: e8 58 6d 23 f6 callq 0xfffffffff6236d62
a: 0f b7 83 a0 03 00 00 movzwl 0x3a0(%rbx),%eax
11: 44 39 f8 cmp %r15d,%eax
14: 0f 8f 63 ff ff ff jg 0xffffffffffffff7d
1a: 48 8b b3 48 01 00 00 mov 0x148(%rbx),%rsi
21: 48 85 f6 test %rsi,%rsi
24: 0f 8e 82 fd ff ff jle 0xfffffffffffffdac
2a:* 0f 0b ud2 <-- trapping instruction
2c: e9 7b fd ff ff jmpq 0xfffffffffffffdac
31: 48 83 c5 40 add $0x40,%rbp
35: b9 00 10 00 00 mov $0x1000,%ecx
3a: e9 bf fd ff ff jmpq 0xfffffffffffffdfe
3f: 8b .byte 0x8b
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: e9 7b fd ff ff jmpq 0xfffffffffffffd82
7: 48 83 c5 40 add $0x40,%rbp
b: b9 00 10 00 00 mov $0x1000,%ecx
10: e9 bf fd ff ff jmpq 0xfffffffffffffdd4
15: 8b .byte 0x8b
RSP: 0018:ffffa6c445b97e68 EFLAGS: 00010202
RAX: 0000000080000000 RBX: ffff8c2cff1b2c00 RCX: 0000000000000001
RDX: 0000000080000001 RSI: 0000000000000020 RDI: 00000000ffffffff
RBP: ffff8c2cc46f20c0 R08: 000000000007e33f R09: 0000000000000000
R10: 0000000000000000 R11: 00000000ffffffff R12: ffffc6c43fc08f00
R13: 0000000000000020 R14: ffff8c2d7ca2ee00 R15: ffffc6c43fc08f05
FS: 0000000000000000(0000) GS:ffff8c2d7ca00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd3c6b36000 CR3: 000000003f6e2000 CR4: 00000000000006f0
Call Trace:
<TASK>
process_one_work (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:212 ./include/trace/events/workqueue.h:108 kernel/workqueue.c:2303)
worker_thread (./include/linux/list.h:284 kernel/workqueue.c:2446)
? process_one_work (kernel/workqueue.c:2388)
kthread (kernel/kthread.c:327)
? set_kthread_struct (kernel/kthread.c:272)
ret_from_fork (arch/x86/entry/entry_64.S:301)
</TASK>
---[ end trace ae1f1e0405dedf19 ]---
------------[ cut here ]------------
WARNING: CPU: 1 PID: 16837 at fs/bcachefs/fs-io.c:1290 bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Modules linked in: bcachefs lz4_compress crc64 xor raid6_pq zstd_compress libcrc32c xt_comment iptable_filter binfmt_misc rfkill cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr sg i2c_piix4 virtio_balloon floppy i2c_core ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi ata_piix libata virtio_net net_failover failover serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 16837 Comm: kworker/1:0 Tainted: G W 5.16.0+ #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: bcachefs bch2_write_index [bcachefs]
RIP: 0010:bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Code: 80 64 24 28 f0 e8 58 6d 23 f6 0f b7 83 a0 03 00 00 44 39 f8 0f 8f 63 ff ff ff 48 8b b3 48 01 00 00 48 85 f6 0f 8e 82 fd ff ff <0f> 0b e9 7b fd ff ff 48 83 c5 40 b9 00 10 00 00 e9 bf fd ff ff 8b
All code
========
0: 80 64 24 28 f0 andb $0xf0,0x28(%rsp)
5: e8 58 6d 23 f6 callq 0xfffffffff6236d62
a: 0f b7 83 a0 03 00 00 movzwl 0x3a0(%rbx),%eax
11: 44 39 f8 cmp %r15d,%eax
14: 0f 8f 63 ff ff ff jg 0xffffffffffffff7d
1a: 48 8b b3 48 01 00 00 mov 0x148(%rbx),%rsi
21: 48 85 f6 test %rsi,%rsi
24: 0f 8e 82 fd ff ff jle 0xfffffffffffffdac
2a:* 0f 0b ud2 <-- trapping instruction
2c: e9 7b fd ff ff jmpq 0xfffffffffffffdac
31: 48 83 c5 40 add $0x40,%rbp
35: b9 00 10 00 00 mov $0x1000,%ecx
3a: e9 bf fd ff ff jmpq 0xfffffffffffffdfe
3f: 8b .byte 0x8b
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: e9 7b fd ff ff jmpq 0xfffffffffffffd82
7: 48 83 c5 40 add $0x40,%rbp
b: b9 00 10 00 00 mov $0x1000,%ecx
10: e9 bf fd ff ff jmpq 0xfffffffffffffdd4
15: 8b .byte 0x8b
RSP: 0018:ffffa6c441aa3e68 EFLAGS: 00010202
RAX: 0000000080000000 RBX: ffff8c2cc8773400 RCX: 0000000000000001
RDX: 0000000080000001 RSI: 0000000000000020 RDI: 00000000ffffffff
RBP: ffff8c2cc83d7e40 R08: 000000000007e6c0 R09: 0000000000000000
R10: 0000000000000000 R11: 00000000ffffffff R12: ffffc6c43fd08f00
R13: 0000000000000020 R14: ffff8c2d7cb2ee00 R15: ffffc6c43fd08f05
FS: 0000000000000000(0000) GS:ffff8c2d7cb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2eaff77148 CR3: 000000003db44000 CR4: 00000000000006e0
Call Trace:
<TASK>
process_one_work (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:212 ./include/trace/events/workqueue.h:108 kernel/workqueue.c:2303)
worker_thread (./include/linux/list.h:284 kernel/workqueue.c:2446)
? process_one_work (kernel/workqueue.c:2388)
kthread (kernel/kthread.c:327)
? set_kthread_struct (kernel/kthread.c:272)
ret_from_fork (arch/x86/entry/entry_64.S:301)
</TASK>
---[ end trace ae1f1e0405dedf1a ]---
------------[ cut here ]------------
WARNING: CPU: 1 PID: 16837 at fs/bcachefs/fs-io.c:1290 bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Modules linked in: bcachefs lz4_compress crc64 xor raid6_pq zstd_compress libcrc32c xt_comment iptable_filter binfmt_misc rfkill cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr sg i2c_piix4 virtio_balloon floppy i2c_core ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi ata_piix libata virtio_net net_failover failover serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 16837 Comm: kworker/1:0 Tainted: G W 5.16.0+ #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: bcachefs bch2_write_index [bcachefs]
RIP: 0010:bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Code: 80 64 24 28 f0 e8 58 6d 23 f6 0f b7 83 a0 03 00 00 44 39 f8 0f 8f 63 ff ff ff 48 8b b3 48 01 00 00 48 85 f6 0f 8e 82 fd ff ff <0f> 0b e9 7b fd ff ff 48 83 c5 40 b9 00 10 00 00 e9 bf fd ff ff 8b
All code
========
0: 80 64 24 28 f0 andb $0xf0,0x28(%rsp)
5: e8 58 6d 23 f6 callq 0xfffffffff6236d62
a: 0f b7 83 a0 03 00 00 movzwl 0x3a0(%rbx),%eax
11: 44 39 f8 cmp %r15d,%eax
14: 0f 8f 63 ff ff ff jg 0xffffffffffffff7d
1a: 48 8b b3 48 01 00 00 mov 0x148(%rbx),%rsi
21: 48 85 f6 test %rsi,%rsi
24: 0f 8e 82 fd ff ff jle 0xfffffffffffffdac
2a:* 0f 0b ud2 <-- trapping instruction
2c: e9 7b fd ff ff jmpq 0xfffffffffffffdac
31: 48 83 c5 40 add $0x40,%rbp
35: b9 00 10 00 00 mov $0x1000,%ecx
3a: e9 bf fd ff ff jmpq 0xfffffffffffffdfe
3f: 8b .byte 0x8b
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: e9 7b fd ff ff jmpq 0xfffffffffffffd82
7: 48 83 c5 40 add $0x40,%rbp
b: b9 00 10 00 00 mov $0x1000,%ecx
10: e9 bf fd ff ff jmpq 0xfffffffffffffdd4
15: 8b .byte 0x8b
RSP: 0018:ffffa6c441aa3e68 EFLAGS: 00010202
RAX: 0000000080000000 RBX: ffff8c2cc86f2400 RCX: 0000000000000001
RDX: 0000000080000001 RSI: 0000000000000008 RDI: 00000000ffffffff
RBP: ffff8c2cc83d7e40 R08: 000000000007e874 R09: 0000000000000000
R10: 0000000000000000 R11: 00000000ffffffff R12: ffffc6c43fd08f00
R13: 0000000000000020 R14: ffff8c2d7cb2ee00 R15: ffffc6c43fd08f05
FS: 0000000000000000(0000) GS:ffff8c2d7cb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0e91d94006 CR3: 000000003f6e2000 CR4: 00000000000006e0
Call Trace:
<TASK>
process_one_work (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:212 ./include/trace/events/workqueue.h:108 kernel/workqueue.c:2303)
worker_thread (./include/linux/list.h:284 kernel/workqueue.c:2446)
? process_one_work (kernel/workqueue.c:2388)
kthread (kernel/kthread.c:327)
? set_kthread_struct (kernel/kthread.c:272)
ret_from_fork (arch/x86/entry/entry_64.S:301)
</TASK>
---[ end trace ae1f1e0405dedf1b ]---
------------[ cut here ]------------
WARNING: CPU: 1 PID: 29215 at fs/bcachefs/fs-io.c:1290 bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Modules linked in: bcachefs lz4_compress crc64 xor raid6_pq zstd_compress libcrc32c xt_comment iptable_filter binfmt_misc rfkill cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr sg i2c_piix4 virtio_balloon floppy i2c_core ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi ata_piix libata virtio_net net_failover failover serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 29215 Comm: kworker/1:2 Tainted: G W 5.16.0+ #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: bcachefs bch2_write_index [bcachefs]
RIP: 0010:bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Code: 80 64 24 28 f0 e8 58 6d 23 f6 0f b7 83 a0 03 00 00 44 39 f8 0f 8f 63 ff ff ff 48 8b b3 48 01 00 00 48 85 f6 0f 8e 82 fd ff ff <0f> 0b e9 7b fd ff ff 48 83 c5 40 b9 00 10 00 00 e9 bf fd ff ff 8b
All code
========
0: 80 64 24 28 f0 andb $0xf0,0x28(%rsp)
5: e8 58 6d 23 f6 callq 0xfffffffff6236d62
a: 0f b7 83 a0 03 00 00 movzwl 0x3a0(%rbx),%eax
11: 44 39 f8 cmp %r15d,%eax
14: 0f 8f 63 ff ff ff jg 0xffffffffffffff7d
1a: 48 8b b3 48 01 00 00 mov 0x148(%rbx),%rsi
21: 48 85 f6 test %rsi,%rsi
24: 0f 8e 82 fd ff ff jle 0xfffffffffffffdac
2a:* 0f 0b ud2 <-- trapping instruction
2c: e9 7b fd ff ff jmpq 0xfffffffffffffdac
31: 48 83 c5 40 add $0x40,%rbp
35: b9 00 10 00 00 mov $0x1000,%ecx
3a: e9 bf fd ff ff jmpq 0xfffffffffffffdfe
3f: 8b .byte 0x8b
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: e9 7b fd ff ff jmpq 0xfffffffffffffd82
7: 48 83 c5 40 add $0x40,%rbp
b: b9 00 10 00 00 mov $0x1000,%ecx
10: e9 bf fd ff ff jmpq 0xfffffffffffffdd4
15: 8b .byte 0x8b
RSP: 0018:ffffa6c4414bbe68 EFLAGS: 00010202
RAX: 0000000080000000 RBX: ffff8c2cc86f2400 RCX: 0000000000000001
RDX: 0000000080000001 RSI: 0000000000000020 RDI: 00000000ffffffff
RBP: ffff8c2cc11b66c0 R08: 000000000007ec72 R09: 0000000000000000
R10: 0000000000000000 R11: 00000000ffffffff R12: ffffc6c43fd08f00
R13: 0000000000000020 R14: ffff8c2d7cb2ee00 R15: ffffc6c43fd08f05
FS: 0000000000000000(0000) GS:ffff8c2d7cb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000020f8000 CR3: 000000003f6e2000 CR4: 00000000000006e0
Call Trace:
<TASK>
process_one_work (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:212 ./include/trace/events/workqueue.h:108 kernel/workqueue.c:2303)
worker_thread (./include/linux/list.h:284 kernel/workqueue.c:2446)
? process_one_work (kernel/workqueue.c:2388)
kthread (kernel/kthread.c:327)
? set_kthread_struct (kernel/kthread.c:272)
ret_from_fork (arch/x86/entry/entry_64.S:301)
</TASK>
---[ end trace ae1f1e0405dedf1c ]---
------------[ cut here ]------------
WARNING: CPU: 0 PID: 21693 at fs/bcachefs/fs-io.c:1290 bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Modules linked in: bcachefs lz4_compress crc64 xor raid6_pq zstd_compress libcrc32c xt_comment iptable_filter binfmt_misc rfkill cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr sg i2c_piix4 virtio_balloon floppy i2c_core ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi ata_piix libata virtio_net net_failover failover serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 21693 Comm: kworker/0:3 Tainted: G W 5.16.0+ #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: bcachefs bch2_write_index [bcachefs]
RIP: 0010:bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Code: 80 64 24 28 f0 e8 58 6d 23 f6 0f b7 83 a0 03 00 00 44 39 f8 0f 8f 63 ff ff ff 48 8b b3 48 01 00 00 48 85 f6 0f 8e 82 fd ff ff <0f> 0b e9 7b fd ff ff 48 83 c5 40 b9 00 10 00 00 e9 bf fd ff ff 8b
All code
========
0: 80 64 24 28 f0 andb $0xf0,0x28(%rsp)
5: e8 58 6d 23 f6 callq 0xfffffffff6236d62
a: 0f b7 83 a0 03 00 00 movzwl 0x3a0(%rbx),%eax
11: 44 39 f8 cmp %r15d,%eax
14: 0f 8f 63 ff ff ff jg 0xffffffffffffff7d
1a: 48 8b b3 48 01 00 00 mov 0x148(%rbx),%rsi
21: 48 85 f6 test %rsi,%rsi
24: 0f 8e 82 fd ff ff jle 0xfffffffffffffdac
2a:* 0f 0b ud2 <-- trapping instruction
2c: e9 7b fd ff ff jmpq 0xfffffffffffffdac
31: 48 83 c5 40 add $0x40,%rbp
35: b9 00 10 00 00 mov $0x1000,%ecx
3a: e9 bf fd ff ff jmpq 0xfffffffffffffdfe
3f: 8b .byte 0x8b
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: e9 7b fd ff ff jmpq 0xfffffffffffffd82
7: 48 83 c5 40 add $0x40,%rbp
b: b9 00 10 00 00 mov $0x1000,%ecx
10: e9 bf fd ff ff jmpq 0xfffffffffffffdd4
15: 8b .byte 0x8b
RSP: 0000:ffffa6c445b97e68 EFLAGS: 00010202
RAX: 0000000080000000 RBX: ffff8c2cc469cc00 RCX: 0000000000000001
RDX: 0000000080000001 RSI: 0000000000000008 RDI: 00000000ffffffff
RBP: ffff8c2cc46f20c0 R08: 000000000007ee28 R09: 0000000000000000
R10: 0000000000000000 R11: 00000000ffffffff R12: ffffc6c43fc08f00
R13: 0000000000000020 R14: ffff8c2d7ca2ee00 R15: ffffc6c43fc08f05
FS: 0000000000000000(0000) GS:ffff8c2d7ca00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000020f8000 CR3: 000000000868e000 CR4: 00000000000006f0
Call Trace:
<TASK>
process_one_work (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:212 ./include/trace/events/workqueue.h:108 kernel/workqueue.c:2303)
worker_thread (./include/linux/list.h:284 kernel/workqueue.c:2446)
? process_one_work (kernel/workqueue.c:2388)
kthread (kernel/kthread.c:327)
? set_kthread_struct (kernel/kthread.c:272)
ret_from_fork (arch/x86/entry/entry_64.S:301)
</TASK>
---[ end trace ae1f1e0405dedf1d ]---
------------[ cut here ]------------
WARNING: CPU: 0 PID: 21693 at fs/bcachefs/fs-io.c:1290 bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Modules linked in: bcachefs lz4_compress crc64 xor raid6_pq zstd_compress libcrc32c xt_comment iptable_filter binfmt_misc rfkill cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr sg i2c_piix4 virtio_balloon floppy i2c_core ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi ata_piix libata virtio_net net_failover failover serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 21693 Comm: kworker/0:3 Tainted: G W 5.16.0+ #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: bcachefs bch2_write_index [bcachefs]
RIP: 0010:bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Code: 80 64 24 28 f0 e8 58 6d 23 f6 0f b7 83 a0 03 00 00 44 39 f8 0f 8f 63 ff ff ff 48 8b b3 48 01 00 00 48 85 f6 0f 8e 82 fd ff ff <0f> 0b e9 7b fd ff ff 48 83 c5 40 b9 00 10 00 00 e9 bf fd ff ff 8b
All code
========
0: 80 64 24 28 f0 andb $0xf0,0x28(%rsp)
5: e8 58 6d 23 f6 callq 0xfffffffff6236d62
a: 0f b7 83 a0 03 00 00 movzwl 0x3a0(%rbx),%eax
11: 44 39 f8 cmp %r15d,%eax
14: 0f 8f 63 ff ff ff jg 0xffffffffffffff7d
1a: 48 8b b3 48 01 00 00 mov 0x148(%rbx),%rsi
21: 48 85 f6 test %rsi,%rsi
24: 0f 8e 82 fd ff ff jle 0xfffffffffffffdac
2a:* 0f 0b ud2 <-- trapping instruction
2c: e9 7b fd ff ff jmpq 0xfffffffffffffdac
31: 48 83 c5 40 add $0x40,%rbp
35: b9 00 10 00 00 mov $0x1000,%ecx
3a: e9 bf fd ff ff jmpq 0xfffffffffffffdfe
3f: 8b .byte 0x8b
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: e9 7b fd ff ff jmpq 0xfffffffffffffd82
7: 48 83 c5 40 add $0x40,%rbp
b: b9 00 10 00 00 mov $0x1000,%ecx
10: e9 bf fd ff ff jmpq 0xfffffffffffffdd4
15: 8b .byte 0x8b
RSP: 0018:ffffa6c445b97e68 EFLAGS: 00010202
RAX: 0000000080000000 RBX: ffff8c2cc469e800 RCX: 0000000000000001
RDX: 0000000080000001 RSI: 0000000000000020 RDI: 00000000ffffffff
RBP: ffff8c2cc46f20c0 R08: 000000000007ef64 R09: 0000000000000000
R10: 0000000000000000 R11: 00000000ffffffff R12: ffffc6c43fc08f00
R13: 0000000000000020 R14: ffff8c2d7ca2ee00 R15: ffffc6c43fc08f05
FS: 0000000000000000(0000) GS:ffff8c2d7ca00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc228173838 CR3: 000000003db44000 CR4: 00000000000006f0
Call Trace:
<TASK>
process_one_work (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:212 ./include/trace/events/workqueue.h:108 kernel/workqueue.c:2303)
worker_thread (./include/linux/list.h:284 kernel/workqueue.c:2446)
? process_one_work (kernel/workqueue.c:2388)
kthread (kernel/kthread.c:327)
? set_kthread_struct (kernel/kthread.c:272)
ret_from_fork (arch/x86/entry/entry_64.S:301)
</TASK>
---[ end trace ae1f1e0405dedf1e ]---
------------[ cut here ]------------
WARNING: CPU: 1 PID: 16837 at fs/bcachefs/fs-io.c:1290 bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Modules linked in: bcachefs lz4_compress crc64 xor raid6_pq zstd_compress libcrc32c xt_comment iptable_filter binfmt_misc rfkill cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr sg i2c_piix4 virtio_balloon floppy i2c_core ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi ata_piix libata virtio_net net_failover failover serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 16837 Comm: kworker/1:0 Tainted: G W 5.16.0+ #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: bcachefs bch2_write_index [bcachefs]
RIP: 0010:bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Code: 80 64 24 28 f0 e8 58 6d 23 f6 0f b7 83 a0 03 00 00 44 39 f8 0f 8f 63 ff ff ff 48 8b b3 48 01 00 00 48 85 f6 0f 8e 82 fd ff ff <0f> 0b e9 7b fd ff ff 48 83 c5 40 b9 00 10 00 00 e9 bf fd ff ff 8b
All code
========
0: 80 64 24 28 f0 andb $0xf0,0x28(%rsp)
5: e8 58 6d 23 f6 callq 0xfffffffff6236d62
a: 0f b7 83 a0 03 00 00 movzwl 0x3a0(%rbx),%eax
11: 44 39 f8 cmp %r15d,%eax
14: 0f 8f 63 ff ff ff jg 0xffffffffffffff7d
1a: 48 8b b3 48 01 00 00 mov 0x148(%rbx),%rsi
21: 48 85 f6 test %rsi,%rsi
24: 0f 8e 82 fd ff ff jle 0xfffffffffffffdac
2a:* 0f 0b ud2 <-- trapping instruction
2c: e9 7b fd ff ff jmpq 0xfffffffffffffdac
31: 48 83 c5 40 add $0x40,%rbp
35: b9 00 10 00 00 mov $0x1000,%ecx
3a: e9 bf fd ff ff jmpq 0xfffffffffffffdfe
3f: 8b .byte 0x8b
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: e9 7b fd ff ff jmpq 0xfffffffffffffd82
7: 48 83 c5 40 add $0x40,%rbp
b: b9 00 10 00 00 mov $0x1000,%ecx
10: e9 bf fd ff ff jmpq 0xfffffffffffffdd4
15: 8b .byte 0x8b
RSP: 0018:ffffa6c441aa3e68 EFLAGS: 00010202
RAX: 0000000080000000 RBX: ffff8c2cc541d000 RCX: 0000000000000001
RDX: 0000000080000001 RSI: 0000000000000020 RDI: 00000000ffffffff
RBP: ffff8c2cc83d7e40 R08: 000000000007f72c R09: 0000000000000000
R10: 0000000000000000 R11: 00000000ffffffff R12: ffffc6c43fd08f00
R13: 0000000000000020 R14: ffff8c2d7cb2ee00 R15: ffffc6c43fd08f05
FS: 0000000000000000(0000) GS:ffff8c2d7cb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0e9210d006 CR3: 000000003db44000 CR4: 00000000000006e0
Call Trace:
<TASK>
process_one_work (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:212 ./include/trace/events/workqueue.h:108 kernel/workqueue.c:2303)
worker_thread (./include/linux/list.h:284 kernel/workqueue.c:2446)
? process_one_work (kernel/workqueue.c:2388)
kthread (kernel/kthread.c:327)
? set_kthread_struct (kernel/kthread.c:272)
ret_from_fork (arch/x86/entry/entry_64.S:301)
</TASK>
---[ end trace ae1f1e0405dedf1f ]---
------------[ cut here ]------------
WARNING: CPU: 0 PID: 5699 at fs/bcachefs/fs-io.c:1290 bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Modules linked in: bcachefs lz4_compress crc64 xor raid6_pq zstd_compress libcrc32c xt_comment iptable_filter binfmt_misc rfkill cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr sg i2c_piix4 virtio_balloon floppy i2c_core ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi ata_piix libata virtio_net net_failover failover serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 5699 Comm: kworker/0:1 Tainted: G W 5.16.0+ #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: bcachefs bch2_write_index [bcachefs]
RIP: 0010:bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Code: 80 64 24 28 f0 e8 58 6d 23 f6 0f b7 83 a0 03 00 00 44 39 f8 0f 8f 63 ff ff ff 48 8b b3 48 01 00 00 48 85 f6 0f 8e 82 fd ff ff <0f> 0b e9 7b fd ff ff 48 83 c5 40 b9 00 10 00 00 e9 bf fd ff ff 8b
All code
========
0: 80 64 24 28 f0 andb $0xf0,0x28(%rsp)
5: e8 58 6d 23 f6 callq 0xfffffffff6236d62
a: 0f b7 83 a0 03 00 00 movzwl 0x3a0(%rbx),%eax
11: 44 39 f8 cmp %r15d,%eax
14: 0f 8f 63 ff ff ff jg 0xffffffffffffff7d
1a: 48 8b b3 48 01 00 00 mov 0x148(%rbx),%rsi
21: 48 85 f6 test %rsi,%rsi
24: 0f 8e 82 fd ff ff jle 0xfffffffffffffdac
2a:* 0f 0b ud2 <-- trapping instruction
2c: e9 7b fd ff ff jmpq 0xfffffffffffffdac
31: 48 83 c5 40 add $0x40,%rbp
35: b9 00 10 00 00 mov $0x1000,%ecx
3a: e9 bf fd ff ff jmpq 0xfffffffffffffdfe
3f: 8b .byte 0x8b
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: e9 7b fd ff ff jmpq 0xfffffffffffffd82
7: 48 83 c5 40 add $0x40,%rbp
b: b9 00 10 00 00 mov $0x1000,%ecx
10: e9 bf fd ff ff jmpq 0xfffffffffffffdd4
15: 8b .byte 0x8b
RSP: 0000:ffffa6c440f13e68 EFLAGS: 00010202
RAX: 0000000080000000 RBX: ffff8c2cc6941800 RCX: 0000000000000001
RDX: 0000000080000001 RSI: 0000000000000020 RDI: 00000000ffffffff
RBP: ffff8c2cc83d7d80 R08: 000000000007f814 R09: 0000000000000000
R10: 0000000000000000 R11: 00000000ffffffff R12: ffffc6c43fc08f00
R13: 0000000000000020 R14: ffff8c2d7ca2ee00 R15: ffffc6c43fc08f05
FS: 0000000000000000(0000) GS:ffff8c2d7ca00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1b3f9bf8c0 CR3: 000000003db44000 CR4: 00000000000006f0
Call Trace:
<TASK>
process_one_work (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:212 ./include/trace/events/workqueue.h:108 kernel/workqueue.c:2303)
worker_thread (./include/linux/list.h:284 kernel/workqueue.c:2446)
? process_one_work (kernel/workqueue.c:2388)
kthread (kernel/kthread.c:327)
? set_kthread_struct (kernel/kthread.c:272)
ret_from_fork (arch/x86/entry/entry_64.S:301)
</TASK>
---[ end trace ae1f1e0405dedf20 ]---
------------[ cut here ]------------
WARNING: CPU: 0 PID: 5699 at fs/bcachefs/fs-io.c:1290 bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Modules linked in: bcachefs lz4_compress crc64 xor raid6_pq zstd_compress libcrc32c xt_comment iptable_filter binfmt_misc rfkill cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr sg i2c_piix4 virtio_balloon floppy i2c_core ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi ata_piix libata virtio_net net_failover failover serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 5699 Comm: kworker/0:1 Tainted: G W 5.16.0+ #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: bcachefs bch2_write_index [bcachefs]
RIP: 0010:bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Code: 80 64 24 28 f0 e8 58 6d 23 f6 0f b7 83 a0 03 00 00 44 39 f8 0f 8f 63 ff ff ff 48 8b b3 48 01 00 00 48 85 f6 0f 8e 82 fd ff ff <0f> 0b e9 7b fd ff ff 48 83 c5 40 b9 00 10 00 00 e9 bf fd ff ff 8b
All code
========
0: 80 64 24 28 f0 andb $0xf0,0x28(%rsp)
5: e8 58 6d 23 f6 callq 0xfffffffff6236d62
a: 0f b7 83 a0 03 00 00 movzwl 0x3a0(%rbx),%eax
11: 44 39 f8 cmp %r15d,%eax
14: 0f 8f 63 ff ff ff jg 0xffffffffffffff7d
1a: 48 8b b3 48 01 00 00 mov 0x148(%rbx),%rsi
21: 48 85 f6 test %rsi,%rsi
24: 0f 8e 82 fd ff ff jle 0xfffffffffffffdac
2a:* 0f 0b ud2 <-- trapping instruction
2c: e9 7b fd ff ff jmpq 0xfffffffffffffdac
31: 48 83 c5 40 add $0x40,%rbp
35: b9 00 10 00 00 mov $0x1000,%ecx
3a: e9 bf fd ff ff jmpq 0xfffffffffffffdfe
3f: 8b .byte 0x8b
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: e9 7b fd ff ff jmpq 0xfffffffffffffd82
7: 48 83 c5 40 add $0x40,%rbp
b: b9 00 10 00 00 mov $0x1000,%ecx
10: e9 bf fd ff ff jmpq 0xfffffffffffffdd4
15: 8b .byte 0x8b
RSP: 0000:ffffa6c440f13e68 EFLAGS: 00010202
RAX: 0000000080000000 RBX: ffff8c2cc6943400 RCX: 0000000000000001
RDX: 0000000080000001 RSI: 0000000000000020 RDI: 00000000ffffffff
RBP: ffff8c2cc83d7d80 R08: 000000000007f886 R09: 0000000000000000
R10: 0000000000000000 R11: 00000000ffffffff R12: ffffc6c43fc08f00
R13: 0000000000000020 R14: ffff8c2d7ca2ee00 R15: ffffc6c43fc08f05
FS: 0000000000000000(0000) GS:ffff8c2d7ca00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f366f93c004 CR3: 000000003db44000 CR4: 00000000000006f0
Call Trace:
<TASK>
process_one_work (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:212 ./include/trace/events/workqueue.h:108 kernel/workqueue.c:2303)
worker_thread (./include/linux/list.h:284 kernel/workqueue.c:2446)
? process_one_work (kernel/workqueue.c:2388)
kthread (kernel/kthread.c:327)
? set_kthread_struct (kernel/kthread.c:272)
ret_from_fork (arch/x86/entry/entry_64.S:301)
</TASK>
---[ end trace ae1f1e0405dedf21 ]---
------------[ cut here ]------------
WARNING: CPU: 1 PID: 29215 at fs/bcachefs/fs-io.c:1290 bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Modules linked in: bcachefs lz4_compress crc64 xor raid6_pq zstd_compress libcrc32c xt_comment iptable_filter binfmt_misc rfkill cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr sg i2c_piix4 virtio_balloon floppy i2c_core ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi ata_piix libata virtio_net net_failover failover serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 29215 Comm: kworker/1:2 Tainted: G W 5.16.0+ #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: bcachefs bch2_write_index [bcachefs]
RIP: 0010:bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Code: 80 64 24 28 f0 e8 58 6d 23 f6 0f b7 83 a0 03 00 00 44 39 f8 0f 8f 63 ff ff ff 48 8b b3 48 01 00 00 48 85 f6 0f 8e 82 fd ff ff <0f> 0b e9 7b fd ff ff 48 83 c5 40 b9 00 10 00 00 e9 bf fd ff ff 8b
All code
========
0: 80 64 24 28 f0 andb $0xf0,0x28(%rsp)
5: e8 58 6d 23 f6 callq 0xfffffffff6236d62
a: 0f b7 83 a0 03 00 00 movzwl 0x3a0(%rbx),%eax
11: 44 39 f8 cmp %r15d,%eax
14: 0f 8f 63 ff ff ff jg 0xffffffffffffff7d
1a: 48 8b b3 48 01 00 00 mov 0x148(%rbx),%rsi
21: 48 85 f6 test %rsi,%rsi
24: 0f 8e 82 fd ff ff jle 0xfffffffffffffdac
2a:* 0f 0b ud2 <-- trapping instruction
2c: e9 7b fd ff ff jmpq 0xfffffffffffffdac
31: 48 83 c5 40 add $0x40,%rbp
35: b9 00 10 00 00 mov $0x1000,%ecx
3a: e9 bf fd ff ff jmpq 0xfffffffffffffdfe
3f: 8b .byte 0x8b
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: e9 7b fd ff ff jmpq 0xfffffffffffffd82
7: 48 83 c5 40 add $0x40,%rbp
b: b9 00 10 00 00 mov $0x1000,%ecx
10: e9 bf fd ff ff jmpq 0xfffffffffffffdd4
15: 8b .byte 0x8b
RSP: 0018:ffffa6c4414bbe68 EFLAGS: 00010202
RAX: 0000000080000000 RBX: ffff8c2cc469dc00 RCX: 0000000000000001
RDX: 0000000080000001 RSI: 0000000000000020 RDI: 00000000ffffffff
RBP: ffff8c2cc11b66c0 R08: 000000000007f977 R09: 0000000000000000
R10: 0000000000000000 R11: 00000000ffffffff R12: ffffc6c43fd08f00
R13: 0000000000000020 R14: ffff8c2d7cb2ee00 R15: ffffc6c43fd08f05
FS: 0000000000000000(0000) GS:ffff8c2d7cb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0e924e8006 CR3: 0000000005318000 CR4: 00000000000006e0
Call Trace:
<TASK>
process_one_work (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:212 ./include/trace/events/workqueue.h:108 kernel/workqueue.c:2303)
worker_thread (./include/linux/list.h:284 kernel/workqueue.c:2446)
? process_one_work (kernel/workqueue.c:2388)
kthread (kernel/kthread.c:327)
? set_kthread_struct (kernel/kthread.c:272)
ret_from_fork (arch/x86/entry/entry_64.S:301)
</TASK>
---[ end trace ae1f1e0405dedf22 ]---
------------[ cut here ]------------
WARNING: CPU: 1 PID: 29215 at fs/bcachefs/fs-io.c:1290 bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Modules linked in: bcachefs lz4_compress crc64 xor raid6_pq zstd_compress libcrc32c xt_comment iptable_filter binfmt_misc rfkill cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr sg i2c_piix4 virtio_balloon floppy i2c_core ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi ata_piix libata virtio_net net_failover failover serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 29215 Comm: kworker/1:2 Tainted: G W 5.16.0+ #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: bcachefs bch2_write_index [bcachefs]
RIP: 0010:bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Code: 80 64 24 28 f0 e8 58 6d 23 f6 0f b7 83 a0 03 00 00 44 39 f8 0f 8f 63 ff ff ff 48 8b b3 48 01 00 00 48 85 f6 0f 8e 82 fd ff ff <0f> 0b e9 7b fd ff ff 48 83 c5 40 b9 00 10 00 00 e9 bf fd ff ff 8b
All code
========
0: 80 64 24 28 f0 andb $0xf0,0x28(%rsp)
5: e8 58 6d 23 f6 callq 0xfffffffff6236d62
a: 0f b7 83 a0 03 00 00 movzwl 0x3a0(%rbx),%eax
11: 44 39 f8 cmp %r15d,%eax
14: 0f 8f 63 ff ff ff jg 0xffffffffffffff7d
1a: 48 8b b3 48 01 00 00 mov 0x148(%rbx),%rsi
21: 48 85 f6 test %rsi,%rsi
24: 0f 8e 82 fd ff ff jle 0xfffffffffffffdac
2a:* 0f 0b ud2 <-- trapping instruction
2c: e9 7b fd ff ff jmpq 0xfffffffffffffdac
31: 48 83 c5 40 add $0x40,%rbp
35: b9 00 10 00 00 mov $0x1000,%ecx
3a: e9 bf fd ff ff jmpq 0xfffffffffffffdfe
3f: 8b .byte 0x8b
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: e9 7b fd ff ff jmpq 0xfffffffffffffd82
7: 48 83 c5 40 add $0x40,%rbp
b: b9 00 10 00 00 mov $0x1000,%ecx
10: e9 bf fd ff ff jmpq 0xfffffffffffffdd4
15: 8b .byte 0x8b
RSP: 0018:ffffa6c4414bbe68 EFLAGS: 00010202
RAX: 0000000080000000 RBX: ffff8c2cc469c000 RCX: 0000000000000001
RDX: 0000000080000001 RSI: 0000000000000008 RDI: 00000000ffffffff
RBP: ffff8c2cc11b66c0 R08: 000000000007fa67 R09: 0000000000000000
R10: 0000000000000000 R11: 00000000ffffffff R12: ffffc6c43fd08f00
R13: 0000000000000020 R14: ffff8c2d7cb2ee00 R15: ffffc6c43fd08f05
FS: 0000000000000000(0000) GS:ffff8c2d7cb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0e924eb006 CR3: 000000003f6e2000 CR4: 00000000000006e0
Call Trace:
<TASK>
process_one_work (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:212 ./include/trace/events/workqueue.h:108 kernel/workqueue.c:2303)
worker_thread (./include/linux/list.h:284 kernel/workqueue.c:2446)
? process_one_work (kernel/workqueue.c:2388)
kthread (kernel/kthread.c:327)
? set_kthread_struct (kernel/kthread.c:272)
ret_from_fork (arch/x86/entry/entry_64.S:301)
</TASK>
---[ end trace ae1f1e0405dedf23 ]---
traps: mysqld[3711] general protection fault ip:557e4d219d79 sp:7fc2782c48f0 error:0 in mysqld[557e4c800000+13af000]
------------[ cut here ]------------
WARNING: CPU: 1 PID: 9529 at fs/bcachefs/fs-io.c:1290 bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Modules linked in: bcachefs lz4_compress crc64 xor raid6_pq zstd_compress libcrc32c xt_comment iptable_filter binfmt_misc rfkill cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr sg i2c_piix4 virtio_balloon floppy i2c_core ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi ata_piix libata virtio_net net_failover failover serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 9529 Comm: kworker/1:1 Tainted: G W 5.16.0+ #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: bcachefs bch2_write_index [bcachefs]
RIP: 0010:bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Code: 80 64 24 28 f0 e8 58 6d 23 f6 0f b7 83 a0 03 00 00 44 39 f8 0f 8f 63 ff ff ff 48 8b b3 48 01 00 00 48 85 f6 0f 8e 82 fd ff ff <0f> 0b e9 7b fd ff ff 48 83 c5 40 b9 00 10 00 00 e9 bf fd ff ff 8b
All code
========
0: 80 64 24 28 f0 andb $0xf0,0x28(%rsp)
5: e8 58 6d 23 f6 callq 0xfffffffff6236d62
a: 0f b7 83 a0 03 00 00 movzwl 0x3a0(%rbx),%eax
11: 44 39 f8 cmp %r15d,%eax
14: 0f 8f 63 ff ff ff jg 0xffffffffffffff7d
1a: 48 8b b3 48 01 00 00 mov 0x148(%rbx),%rsi
21: 48 85 f6 test %rsi,%rsi
24: 0f 8e 82 fd ff ff jle 0xfffffffffffffdac
2a:* 0f 0b ud2 <-- trapping instruction
2c: e9 7b fd ff ff jmpq 0xfffffffffffffdac
31: 48 83 c5 40 add $0x40,%rbp
35: b9 00 10 00 00 mov $0x1000,%ecx
3a: e9 bf fd ff ff jmpq 0xfffffffffffffdfe
3f: 8b .byte 0x8b
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: e9 7b fd ff ff jmpq 0xfffffffffffffd82
7: 48 83 c5 40 add $0x40,%rbp
b: b9 00 10 00 00 mov $0x1000,%ecx
10: e9 bf fd ff ff jmpq 0xfffffffffffffdd4
15: 8b .byte 0x8b
RSP: 0018:ffffa6c440dd7e68 EFLAGS: 00010206
RAX: 0000000080000000 RBX: ffff8c2cc8770800 RCX: 0000000000000001
RDX: 0000000080000001 RSI: 0000000000000018 RDI: 00000000ffffffff
RBP: ffff8c2cc7276300 R08: 00000000000800ef R09: 0000000000000000
R10: 0000000000000000 R11: 00000000ffffffff R12: ffffc6c43fd08f00
R13: 0000000000000020 R14: ffff8c2d7cb2ee00 R15: ffffc6c43fd08f05
FS: 0000000000000000(0000) GS:ffff8c2d7cb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2eaff77148 CR3: 000000003f6e2000 CR4: 00000000000006e0
Call Trace:
<TASK>
process_one_work (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:212 ./include/trace/events/workqueue.h:108 kernel/workqueue.c:2303)
worker_thread (./include/linux/list.h:284 kernel/workqueue.c:2446)
? process_one_work (kernel/workqueue.c:2388)
kthread (kernel/kthread.c:327)
? set_kthread_struct (kernel/kthread.c:272)
ret_from_fork (arch/x86/entry/entry_64.S:301)
</TASK>
---[ end trace ae1f1e0405dedf24 ]---
------------[ cut here ]------------
WARNING: CPU: 0 PID: 29770 at fs/bcachefs/fs-io.c:1290 bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Modules linked in: bcachefs lz4_compress crc64 xor raid6_pq zstd_compress libcrc32c xt_comment iptable_filter binfmt_misc rfkill cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr sg i2c_piix4 virtio_balloon floppy i2c_core ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi ata_piix libata virtio_net net_failover failover serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 29770 Comm: kworker/0:0 Tainted: G W 5.16.0+ #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: bcachefs bch2_write_index [bcachefs]
RIP: 0010:bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Code: 80 64 24 28 f0 e8 58 6d 23 f6 0f b7 83 a0 03 00 00 44 39 f8 0f 8f 63 ff ff ff 48 8b b3 48 01 00 00 48 85 f6 0f 8e 82 fd ff ff <0f> 0b e9 7b fd ff ff 48 83 c5 40 b9 00 10 00 00 e9 bf fd ff ff 8b
All code
========
0: 80 64 24 28 f0 andb $0xf0,0x28(%rsp)
5: e8 58 6d 23 f6 callq 0xfffffffff6236d62
a: 0f b7 83 a0 03 00 00 movzwl 0x3a0(%rbx),%eax
11: 44 39 f8 cmp %r15d,%eax
14: 0f 8f 63 ff ff ff jg 0xffffffffffffff7d
1a: 48 8b b3 48 01 00 00 mov 0x148(%rbx),%rsi
21: 48 85 f6 test %rsi,%rsi
24: 0f 8e 82 fd ff ff jle 0xfffffffffffffdac
2a:* 0f 0b ud2 <-- trapping instruction
2c: e9 7b fd ff ff jmpq 0xfffffffffffffdac
31: 48 83 c5 40 add $0x40,%rbp
35: b9 00 10 00 00 mov $0x1000,%ecx
3a: e9 bf fd ff ff jmpq 0xfffffffffffffdfe
3f: 8b .byte 0x8b
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: e9 7b fd ff ff jmpq 0xfffffffffffffd82
7: 48 83 c5 40 add $0x40,%rbp
b: b9 00 10 00 00 mov $0x1000,%ecx
10: e9 bf fd ff ff jmpq 0xfffffffffffffdd4
15: 8b .byte 0x8b
RSP: 0000:ffffa6c440523e68 EFLAGS: 00010202
RAX: 0000000080000000 RBX: ffff8c2cc6942800 RCX: 0000000000000001
RDX: 0000000080000001 RSI: 0000000000000020 RDI: 00000000ffffffff
RBP: ffff8c2cff4159c0 R08: 0000000000080c28 R09: 0000000000000000
R10: 0000000000000000 R11: 00000000ffffffff R12: ffffc6c43fc08f00
R13: 0000000000000020 R14: ffff8c2d7ca2ee00 R15: ffffc6c43fc08f05
FS: 0000000000000000(0000) GS:ffff8c2d7ca00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f19ed64a0a0 CR3: 000000004f40a000 CR4: 00000000000006f0
Call Trace:
<TASK>
process_one_work (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:212 ./include/trace/events/workqueue.h:108 kernel/workqueue.c:2303)
worker_thread (./include/linux/list.h:284 kernel/workqueue.c:2446)
? process_one_work (kernel/workqueue.c:2388)
kthread (kernel/kthread.c:327)
? set_kthread_struct (kernel/kthread.c:272)
ret_from_fork (arch/x86/entry/entry_64.S:301)
</TASK>
---[ end trace ae1f1e0405dedf25 ]---
------------[ cut here ]------------
WARNING: CPU: 1 PID: 9529 at fs/bcachefs/fs-io.c:1290 bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Modules linked in: bcachefs lz4_compress crc64 xor raid6_pq zstd_compress libcrc32c xt_comment iptable_filter binfmt_misc rfkill cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr sg i2c_piix4 virtio_balloon floppy i2c_core ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi ata_piix libata virtio_net net_failover failover serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 9529 Comm: kworker/1:1 Tainted: G W 5.16.0+ #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: bcachefs bch2_write_index [bcachefs]
RIP: 0010:bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Code: 80 64 24 28 f0 e8 58 6d 23 f6 0f b7 83 a0 03 00 00 44 39 f8 0f 8f 63 ff ff ff 48 8b b3 48 01 00 00 48 85 f6 0f 8e 82 fd ff ff <0f> 0b e9 7b fd ff ff 48 83 c5 40 b9 00 10 00 00 e9 bf fd ff ff 8b
All code
========
0: 80 64 24 28 f0 andb $0xf0,0x28(%rsp)
5: e8 58 6d 23 f6 callq 0xfffffffff6236d62
a: 0f b7 83 a0 03 00 00 movzwl 0x3a0(%rbx),%eax
11: 44 39 f8 cmp %r15d,%eax
14: 0f 8f 63 ff ff ff jg 0xffffffffffffff7d
1a: 48 8b b3 48 01 00 00 mov 0x148(%rbx),%rsi
21: 48 85 f6 test %rsi,%rsi
24: 0f 8e 82 fd ff ff jle 0xfffffffffffffdac
2a:* 0f 0b ud2 <-- trapping instruction
2c: e9 7b fd ff ff jmpq 0xfffffffffffffdac
31: 48 83 c5 40 add $0x40,%rbp
35: b9 00 10 00 00 mov $0x1000,%ecx
3a: e9 bf fd ff ff jmpq 0xfffffffffffffdfe
3f: 8b .byte 0x8b
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: e9 7b fd ff ff jmpq 0xfffffffffffffd82
7: 48 83 c5 40 add $0x40,%rbp
b: b9 00 10 00 00 mov $0x1000,%ecx
10: e9 bf fd ff ff jmpq 0xfffffffffffffdd4
15: 8b .byte 0x8b
RSP: 0018:ffffa6c440dd7e68 EFLAGS: 00010202
RAX: 0000000080000000 RBX: ffff8c2cff1b3400 RCX: 0000000000000001
RDX: 0000000080000001 RSI: 0000000000000008 RDI: 00000000ffffffff
RBP: ffff8c2cc7276300 R08: 0000000000081660 R09: 0000000000000000
R10: 0000000000000000 R11: 00000000ffffffff R12: ffffc6c43fd08f00
R13: 0000000000000020 R14: ffff8c2d7cb2ee00 R15: ffffc6c43fd08f05
FS: 0000000000000000(0000) GS:ffff8c2d7cb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f56491f80a0 CR3: 000000003f6e2000 CR4: 00000000000006e0
Call Trace:
<TASK>
process_one_work (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:212 ./include/trace/events/workqueue.h:108 kernel/workqueue.c:2303)
worker_thread (./include/linux/list.h:284 kernel/workqueue.c:2446)
? process_one_work (kernel/workqueue.c:2388)
kthread (kernel/kthread.c:327)
? set_kthread_struct (kernel/kthread.c:272)
ret_from_fork (arch/x86/entry/entry_64.S:301)
</TASK>
---[ end trace ae1f1e0405dedf26 ]---
------------[ cut here ]------------
WARNING: CPU: 1 PID: 9529 at fs/bcachefs/fs-io.c:1290 bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Modules linked in: bcachefs lz4_compress crc64 xor raid6_pq zstd_compress libcrc32c xt_comment iptable_filter binfmt_misc rfkill cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr sg i2c_piix4 virtio_balloon floppy i2c_core ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi ata_piix libata virtio_net net_failover failover serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 9529 Comm: kworker/1:1 Tainted: G W 5.16.0+ #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: bcachefs bch2_write_index [bcachefs]
RIP: 0010:bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Code: 80 64 24 28 f0 e8 58 6d 23 f6 0f b7 83 a0 03 00 00 44 39 f8 0f 8f 63 ff ff ff 48 8b b3 48 01 00 00 48 85 f6 0f 8e 82 fd ff ff <0f> 0b e9 7b fd ff ff 48 83 c5 40 b9 00 10 00 00 e9 bf fd ff ff 8b
All code
========
0: 80 64 24 28 f0 andb $0xf0,0x28(%rsp)
5: e8 58 6d 23 f6 callq 0xfffffffff6236d62
a: 0f b7 83 a0 03 00 00 movzwl 0x3a0(%rbx),%eax
11: 44 39 f8 cmp %r15d,%eax
14: 0f 8f 63 ff ff ff jg 0xffffffffffffff7d
1a: 48 8b b3 48 01 00 00 mov 0x148(%rbx),%rsi
21: 48 85 f6 test %rsi,%rsi
24: 0f 8e 82 fd ff ff jle 0xfffffffffffffdac
2a:* 0f 0b ud2 <-- trapping instruction
2c: e9 7b fd ff ff jmpq 0xfffffffffffffdac
31: 48 83 c5 40 add $0x40,%rbp
35: b9 00 10 00 00 mov $0x1000,%ecx
3a: e9 bf fd ff ff jmpq 0xfffffffffffffdfe
3f: 8b .byte 0x8b
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: e9 7b fd ff ff jmpq 0xfffffffffffffd82
7: 48 83 c5 40 add $0x40,%rbp
b: b9 00 10 00 00 mov $0x1000,%ecx
10: e9 bf fd ff ff jmpq 0xfffffffffffffdd4
15: 8b .byte 0x8b
RSP: 0018:ffffa6c440dd7e68 EFLAGS: 00010202
RAX: 0000000080000000 RBX: ffff8c2cff1b1400 RCX: 0000000000000001
RDX: 0000000080000001 RSI: 0000000000000008 RDI: 00000000ffffffff
RBP: ffff8c2cc7276300 R08: 0000000000081666 R09: 0000000000000000
R10: 0000000000000000 R11: 00000000ffffffff R12: ffffc6c43fd08f00
R13: 0000000000000020 R14: ffff8c2d7cb2ee00 R15: ffffc6c43fd08f05
FS: 0000000000000000(0000) GS:ffff8c2d7cb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa4284ca000 CR3: 000000003f6e2000 CR4: 00000000000006e0
Call Trace:
<TASK>
process_one_work (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:212 ./include/trace/events/workqueue.h:108 kernel/workqueue.c:2303)
worker_thread (./include/linux/list.h:284 kernel/workqueue.c:2446)
? process_one_work (kernel/workqueue.c:2388)
kthread (kernel/kthread.c:327)
? set_kthread_struct (kernel/kthread.c:272)
ret_from_fork (arch/x86/entry/entry_64.S:301)
</TASK>
---[ end trace ae1f1e0405dedf27 ]---
------------[ cut here ]------------
WARNING: CPU: 1 PID: 29215 at fs/bcachefs/fs-io.c:1290 bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Modules linked in: bcachefs lz4_compress crc64 xor raid6_pq zstd_compress libcrc32c xt_comment iptable_filter binfmt_misc rfkill cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr sg i2c_piix4 virtio_balloon floppy i2c_core ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi ata_piix libata virtio_net net_failover failover serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 29215 Comm: kworker/1:2 Tainted: G W 5.16.0+ #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: bcachefs bch2_write_index [bcachefs]
RIP: 0010:bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Code: 80 64 24 28 f0 e8 58 6d 23 f6 0f b7 83 a0 03 00 00 44 39 f8 0f 8f 63 ff ff ff 48 8b b3 48 01 00 00 48 85 f6 0f 8e 82 fd ff ff <0f> 0b e9 7b fd ff ff 48 83 c5 40 b9 00 10 00 00 e9 bf fd ff ff 8b
All code
========
0: 80 64 24 28 f0 andb $0xf0,0x28(%rsp)
5: e8 58 6d 23 f6 callq 0xfffffffff6236d62
a: 0f b7 83 a0 03 00 00 movzwl 0x3a0(%rbx),%eax
11: 44 39 f8 cmp %r15d,%eax
14: 0f 8f 63 ff ff ff jg 0xffffffffffffff7d
1a: 48 8b b3 48 01 00 00 mov 0x148(%rbx),%rsi
21: 48 85 f6 test %rsi,%rsi
24: 0f 8e 82 fd ff ff jle 0xfffffffffffffdac
2a:* 0f 0b ud2 <-- trapping instruction
2c: e9 7b fd ff ff jmpq 0xfffffffffffffdac
31: 48 83 c5 40 add $0x40,%rbp
35: b9 00 10 00 00 mov $0x1000,%ecx
3a: e9 bf fd ff ff jmpq 0xfffffffffffffdfe
3f: 8b .byte 0x8b
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: e9 7b fd ff ff jmpq 0xfffffffffffffd82
7: 48 83 c5 40 add $0x40,%rbp
b: b9 00 10 00 00 mov $0x1000,%ecx
10: e9 bf fd ff ff jmpq 0xfffffffffffffdd4
15: 8b .byte 0x8b
RSP: 0018:ffffa6c4414bbe68 EFLAGS: 00010202
RAX: 0000000080000000 RBX: ffff8c2cc86f3800 RCX: 0000000000000001
RDX: 0000000080000001 RSI: 0000000000000020 RDI: 00000000ffffffff
RBP: ffff8c2cc11b66c0 R08: 0000000000082056 R09: 0000000000000000
R10: 0000000000000000 R11: 00000000ffffffff R12: ffffc6c43fd08f00
R13: 0000000000000020 R14: ffff8c2d7cb2ee00 R15: ffffc6c43fd08f05
FS: 0000000000000000(0000) GS:ffff8c2d7cb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2a7cec30a0 CR3: 000000003f6e2000 CR4: 00000000000006e0
Call Trace:
<TASK>
process_one_work (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:212 ./include/trace/events/workqueue.h:108 kernel/workqueue.c:2303)
worker_thread (./include/linux/list.h:284 kernel/workqueue.c:2446)
? process_one_work (kernel/workqueue.c:2388)
kthread (kernel/kthread.c:327)
? set_kthread_struct (kernel/kthread.c:272)
ret_from_fork (arch/x86/entry/entry_64.S:301)
</TASK>
---[ end trace ae1f1e0405dedf28 ]---
------------[ cut here ]------------
WARNING: CPU: 1 PID: 29215 at fs/bcachefs/fs-io.c:1290 bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Modules linked in: bcachefs lz4_compress crc64 xor raid6_pq zstd_compress libcrc32c xt_comment iptable_filter binfmt_misc rfkill cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr sg i2c_piix4 virtio_balloon floppy i2c_core ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi ata_piix libata virtio_net net_failover failover serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 29215 Comm: kworker/1:2 Tainted: G W 5.16.0+ #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: bcachefs bch2_write_index [bcachefs]
RIP: 0010:bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
Code: 80 64 24 28 f0 e8 58 6d 23 f6 0f b7 83 a0 03 00 00 44 39 f8 0f 8f 63 ff ff ff 48 8b b3 48 01 00 00 48 85 f6 0f 8e 82 fd ff ff <0f> 0b e9 7b fd ff ff 48 83 c5 40 b9 00 10 00 00 e9 bf fd ff ff 8b
All code
========
0: 80 64 24 28 f0 andb $0xf0,0x28(%rsp)
5: e8 58 6d 23 f6 callq 0xfffffffff6236d62
a: 0f b7 83 a0 03 00 00 movzwl 0x3a0(%rbx),%eax
11: 44 39 f8 cmp %r15d,%eax
14: 0f 8f 63 ff ff ff jg 0xffffffffffffff7d
1a: 48 8b b3 48 01 00 00 mov 0x148(%rbx),%rsi
21: 48 85 f6 test %rsi,%rsi
24: 0f 8e 82 fd ff ff jle 0xfffffffffffffdac
2a:* 0f 0b ud2 <-- trapping instruction
2c: e9 7b fd ff ff jmpq 0xfffffffffffffdac
31: 48 83 c5 40 add $0x40,%rbp
35: b9 00 10 00 00 mov $0x1000,%ecx
3a: e9 bf fd ff ff jmpq 0xfffffffffffffdfe
3f: 8b .byte 0x8b
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: e9 7b fd ff ff jmpq 0xfffffffffffffd82
7: 48 83 c5 40 add $0x40,%rbp
b: b9 00 10 00 00 mov $0x1000,%ecx
10: e9 bf fd ff ff jmpq 0xfffffffffffffdd4
15: 8b .byte 0x8b
RSP: 0000:ffffa6c4414bbe68 EFLAGS: 00010202
RAX: 0000000080000000 RBX: ffff8c2cff1b1000 RCX: 0000000000000001
RDX: 0000000080000001 RSI: 0000000000000008 RDI: 00000000ffffffff
RBP: ffff8c2cc11b66c0 R08: 00000000000820a9 R09: 0000000000000000
R10: 0000000000000000 R11: 00000000ffffffff R12: ffffc6c43fd08f00
R13: 0000000000000020 R14: ffff8c2d7cb2ee00 R15: ffffc6c43fd08f05
FS: 0000000000000000(0000) GS:ffff8c2d7cb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd3c6b36000 CR3: 0000000006d2c000 CR4: 00000000000006e0
Call Trace:
<TASK>
process_one_work (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:212 ./include/trace/events/workqueue.h:108 kernel/workqueue.c:2303)
worker_thread (./include/linux/list.h:284 kernel/workqueue.c:2446)
? process_one_work (kernel/workqueue.c:2388)
kthread (kernel/kthread.c:327)
? set_kthread_struct (kernel/kthread.c:272)
ret_from_fork (arch/x86/entry/entry_64.S:301)
</TASK>
---[ end trace ae1f1e0405dedf29 ]---
--
Eric Wheeler
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: bcachefs filesystem corruption
2022-03-16 22:58 bcachefs filesystem corruption Eric Wheeler
@ 2022-03-17 3:44 ` Kent Overstreet
2022-03-17 23:40 ` Eric Wheeler
0 siblings, 1 reply; 4+ messages in thread
From: Kent Overstreet @ 2022-03-17 3:44 UTC (permalink / raw)
To: Eric Wheeler; +Cc: linux-bcachefs
On Wed, Mar 16, 2022 at 03:58:34PM -0700, Eric Wheeler wrote:
> Hi Kent,
>
> The MySQL+snapshots deployment we has become corrupt and InnoDB is
> crashing with a stack trace and lots of borken database noise.
>
> We are going to rebuild it from other replicas and see if it happens
> again. We started using your new-allocator code (commit
> 3985f5f4eff86ee667da7da6e36ca2c346f56be3) after the original deployment
> and we haven't re-loaded since you fixed the snapshot iterator issue, or
> maybe it broke during the allocator conversion, can't be certain.
>
> There isn't any especially obvious problem but there are some WARNING's
> below that you might look at.
>
> There were lots of these:
> WARNING: CPU: 0 PID: 7796 at fs/bcachefs/fs-io.c:1290
> bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
So the warnings are just a symptom, and not terribly interesting. I'm changing
them to WARN_ON_ONCE().
> bcachefs (dm-6): initializing new filesystem
> bcachefs (dm-6): going read-write
> bucket 0:10 gen 0 different types of data in same bucket: btree, journal
> while marking journal
^ That's probably the real culprit - we're using the same bucket for both
journal and btree, no wonder things got hosed.
Interesting bug, something strange must have happened for us to not catch this
right away.
Would it be possible for you to dump the filesystem metadata for me, with the
bcachefs dump tool? I might get some clues from examining it.
Also, how long was the system in use before this happened? How many reboots?
Were you by chance running the resize-journal command?
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: bcachefs filesystem corruption
2022-03-17 3:44 ` Kent Overstreet
@ 2022-03-17 23:40 ` Eric Wheeler
2022-03-18 23:22 ` Eric Wheeler
0 siblings, 1 reply; 4+ messages in thread
From: Eric Wheeler @ 2022-03-17 23:40 UTC (permalink / raw)
To: Kent Overstreet; +Cc: linux-bcachefs
On Wed, 16 Mar 2022, Kent Overstreet wrote:
> On Wed, Mar 16, 2022 at 03:58:34PM -0700, Eric Wheeler wrote:
> > Hi Kent,
> >
> > The MySQL+snapshots deployment we has become corrupt and InnoDB is
> > crashing with a stack trace and lots of borken database noise.
> >
> > We are going to rebuild it from other replicas and see if it happens
> > again. We started using your new-allocator code (commit
> > 3985f5f4eff86ee667da7da6e36ca2c346f56be3) after the original deployment
> > and we haven't re-loaded since you fixed the snapshot iterator issue, or
> > maybe it broke during the allocator conversion, can't be certain.
> >
> > There isn't any especially obvious problem but there are some WARNING's
> > below that you might look at.
> >
> > There were lots of these:
> > WARNING: CPU: 0 PID: 7796 at fs/bcachefs/fs-io.c:1290
> > bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
>
> So the warnings are just a symptom, and not terribly interesting. I'm changing
> them to WARN_ON_ONCE().
>
> > bcachefs (dm-6): initializing new filesystem
> > bcachefs (dm-6): going read-write
> > bucket 0:10 gen 0 different types of data in same bucket: btree, journal
> > while marking journal
>
> ^ That's probably the real culprit - we're using the same bucket for both
> journal and btree, no wonder things got hosed.
>
> Interesting bug, something strange must have happened for us to not catch this
> right away.
The deployment would have been like this:
1. Make lots of subvolumes
2. Deploy mysql instances with their --datadir pointing to the subvols
3. Snapshot every 4 hrs
4. Crash because of iterator issue
5. Reboot with iterator+new allocator. This would have done the allocator
upgrade to add the new structures.
6. The rest is in this email thread.
Also we have a weird problem where mkfs hangs, not sure if it is
relavent, maybe mkfs isn't finishing, but we think it is because it mounts
wihout error:
We have to run mkfs then ctrl-c. It seems to finish and mount, not sure
what the problem is there. These are the RPM versions of the libraries
reported by `ldd bcachefs` that we are using in el7 build:
bzip2-libs-1.0.6-13.el7.x86_64
elfutils-libelf-0.176-5.el7.x86_64
elfutils-libs-0.176-5.el7.x86_64
glibc-2.17-325.el7_9.x86_64
keyutils-libs-1.5.8-3.el7.x86_64
libaio-0.3.109-13.el7.x86_64
libattr-2.4.46-13.el7.x86_64
libblkid-2.23.2-65.el7.x86_64
libcap-2.22-11.el7.x86_64
libgcc-4.8.5-44.el7.x86_64
libsodium-1.0.18-1.el7.x86_64
libuuid-2.23.2-65.el7.x86_64
libzstd-1.5.0-1.el7.x86_64
lz4-1.8.3-1.el7.x86_64
systemd-libs-219-78.el7.x86_64
userspace-rcu-0.10.1-4.el7.x86_64
xz-libs-5.2.2-1.el7.x86_64
zlib-1.2.7-19.el7_9.x86_64
Haven't investigated with gdb, not sure where it is stuck.
> Would it be possible for you to dump the filesystem metadata for me, with the
> bcachefs dump tool? I might get some clues from examining it.
Does bcachefs dump include any user data? Metadata is fine, but the
content needs to stay private.
Please confirm that there is no metadata and I'll send the meta.
> Also, how long was the system in use before this happened? How many reboots?
Only 1 or 2 reboots. Basically we set it up and it crashed because of the
iterator issue. Then we rebooted into the new kernel with both your
iterator fix and the new allocater update. I think that's about it
though.
> Were you by chance running the resize-journal command?
Nope.
--
Eric Wheeler
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: bcachefs filesystem corruption
2022-03-17 23:40 ` Eric Wheeler
@ 2022-03-18 23:22 ` Eric Wheeler
0 siblings, 0 replies; 4+ messages in thread
From: Eric Wheeler @ 2022-03-18 23:22 UTC (permalink / raw)
To: Kent Overstreet; +Cc: linux-bcachefs
On Thu, 17 Mar 2022, Eric Wheeler wrote:
> On Wed, 16 Mar 2022, Kent Overstreet wrote:
> > On Wed, Mar 16, 2022 at 03:58:34PM -0700, Eric Wheeler wrote:
> > > Hi Kent,
> > >
> > > The MySQL+snapshots deployment we has become corrupt and InnoDB is
> > > crashing with a stack trace and lots of borken database noise.
> > >
> > > We are going to rebuild it from other replicas and see if it happens
> > > again. We started using your new-allocator code (commit
> > > 3985f5f4eff86ee667da7da6e36ca2c346f56be3) after the original deployment
> > > and we haven't re-loaded since you fixed the snapshot iterator issue, or
> > > maybe it broke during the allocator conversion, can't be certain.
> > >
> > > There isn't any especially obvious problem but there are some WARNING's
> > > below that you might look at.
> > >
> > > There were lots of these:
> > > WARNING: CPU: 0 PID: 7796 at fs/bcachefs/fs-io.c:1290
> > > bch2_writepage_io_done (fs/bcachefs/fs-io.c:1290 (discriminator 1)) bcachefs
> >
> > So the warnings are just a symptom, and not terribly interesting. I'm changing
> > them to WARN_ON_ONCE().
> >
> > > bcachefs (dm-6): initializing new filesystem
> > > bcachefs (dm-6): going read-write
> > > bucket 0:10 gen 0 different types of data in same bucket: btree, journal
> > > while marking journal
> >
> > ^ That's probably the real culprit - we're using the same bucket for both
> > journal and btree, no wonder things got hosed.
Could that have happened from an in-place upgrade of the on-disk
structures from the new allocator code you added between
5490c9c529770aa18b2571bd98f5416ed9ae24c6..3985f5f4eff86ee667da7da6e36ca2c346f56be3 ?
> >
> > Interesting bug, something strange must have happened for us to not catch this
> > right away.
>
> The deployment would have been like this:
>
> 1. Make lots of subvolumes
> 2. Deploy mysql instances with their --datadir pointing to the subvols
> 3. Snapshot every 4 hrs
> 4. Crash because of iterator issue
> 5. Reboot with iterator+new allocator. This would have done the allocator
> upgrade to add the new structures.
> 6. The rest is in this email thread.
>
> Also we have a weird problem where mkfs hangs, not sure if it is
> relavent, maybe mkfs isn't finishing, but we think it is because it mounts
> wihout error.
Ok, so we reproduced what appears to be a userspace deadlock in `bcachefs
format`. We are building with the latest version (I think) at commit
3765483ff0cf9abd0243fcafe11aebd0f9beb03d. Stack traces are below.
Here is how we reproduce the problem:
lvcreate -n bcachefstest -L 200g data
gdb bcachefs format -ex 'run format /dev/data/bcachefstest'
Then it will spin at 100% CPU. We hit CTRL-C when it spins for too long
and and get a backtrace of each thread using `gdb`.
Interestingly we were able to get it to finish the format reliably with
`taskset -c 1` and `chrt -f 1` to pin it on one CPU and make it SCHED_FIFO
to prevent thread interleaving. Definitely a hack, and I wasn't sure if
it would make things worse or solve the deadlock, but this _does_ work
reliably:
chrt -f 1 taskset -c 1 gdb bcachefs -ex 'run format /dev/data/bcachefstest'
There are quite a few thread traces, most of are not interesting.
Here is a full stacktrace: https://pastebin.com/0fuv8aHu
And these are the ones that look relevant:
====== Test 1 ======
Starting program: /usr/local/bin/bcachefs format /dev/data/bcachefstest
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff7fe3700 (LWP 18865)]
[New Thread 0x7ffff7fda700 (LWP 18866)]
[New Thread 0x7ffff7fd1700 (LWP 18867)]
[New Thread 0x7ffff7fc8700 (LWP 18868)]
[New Thread 0x7ffff7fbf700 (LWP 18869)]
[New Thread 0x7ffff7fb6700 (LWP 18870)]
[New Thread 0x7ffff7fad700 (LWP 18871)]
External UUID: fd0a5313-6cbd-4175-9f99-96b8b02c2d74
Internal UUID: 06c376af-29f7-4e28-b224-47f0c116d018
Device index: 0
Label:
Version: 19
Oldest version on disk: 19
Created: Fri Mar 18 15:27:58 2022
Sequence number: 0
Superblock size: 816
Clean: 0
Devices: 1
Sections: members
Features: new_siphash,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes
Compat features:
Options:
block_size: 512
btree_node_size: 256k
errors: continue [ro] panic
metadata_replicas: 1
data_replicas: 1
metadata_replicas_required: 1
data_replicas_required: 1
encoded_extent_max: 64k
metadata_checksum: none [crc32c] crc64 xxhash
data_checksum: none [crc32c] crc64 xxhash
compression: [none] lz4 gzip zstd
background_compression: [none] lz4 gzip zstd
str_hash: crc32c crc64 [siphash]
metadata_target: none
foreground_target: none
background_target: none
promote_target: none
erasure_code: 0
inodes_32bit: 1
shard_inode_numbers: 1
inodes_use_key_cache: 1
gc_reserve_percent: 8
gc_reserve_bytes: 0
root_reserve_percent: 0
wide_macs: 0
acl: 1
usrquota: 0
grpquota: 0
prjquota: 0
journal_flush_delay: 1000
journal_flush_disabled: 0
journal_reclaim_delay: 100
journal_transaction_names: 1
members (size 64):
Device: 0
UUID: 50ae6b14-eefc-41a9-8e49-ea62d0d70c15
Size: 200G
Bucket size: 256k
First bucket: 0
Buckets: 819200
Last mount: (never)
State: rw
Group: (none)
Data allowed: journal,btree,user
Has data: (none)
Discard: 0
Freespace initialized: 0
bch2_fs_open()
bch2_read_super()
bch2_read_super() ret 0
bch2_fs_alloc()
[New Thread 0x7ffff7f61700 (LWP 18873)]
[New Thread 0x7ffff7f58700 (LWP 18875)]
[New Thread 0x7ffff7f4f700 (LWP 18876)]
[New Thread 0x7ffff7f46700 (LWP 18877)]
bch2_fs_journal_init()
bch2_fs_journal_init() ret 0
bch2_fs_btree_cache_init()
bch2_fs_btree_cache_init() ret 0
[New Thread 0x7ffff7e05700 (LWP 18878)]
bch2_fs_encryption_init()
bch2_fs_encryption_init() ret 0
__bch2_fs_compress_init()
__bch2_fs_compress_init() ret 0
bch2_dev_alloc()
bch2_dev_alloc() ret 0
bch2_fs_alloc() ret 0
initializing new filesystem
[New Thread 0x7ffff3ff1700 (LWP 18879)]
going read-write
marking superblocks
[New Thread 0x7ffff3fe8700 (LWP 18880)]
^C
Program received signal SIGINT, Interrupt.
0x00007ffff670dd62 in pthread_mutex_lock () from /lib64/libpthread.so.0
### Note that the stdio output above is from a _different_ run than than for the
### traces below. However, I don't think the stdio output is important, really
### only the UUIDs change.
### Spinning 100% CPU for a long time so pressed CTRL-C and ran backtraces ###
(gdb) thr a a bt
Thread 1 (Thread 0x7ffff7fe4b80 (LWP 23493)):
#0 0x00007ffff5c1cd19 in syscall () from /lib64/libc.so.6
#1 0x0000000000421a32 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=<optimized out>, op=128, uaddr=<optimized out>)
at /usr/include/urcu/futex.h:65
#2 schedule () at linux/sched.c:41
#3 0x00000000004186e3 in __six_lock_type_slowpath (lock=0x843200, type=type@entry=SIX_LOCK_intent,
should_sleep_fn=0x42fb40 <lock_root_check_fn>, should_sleep_fn@entry=0x7ffffff, p=0x7ffff7f638b0, p@entry=0xffffffffffffffff)
at linux/six.c:501
#4 0x0000000000418d58 in __six_lock_type (p=p@entry=0xffffffffffffffff, should_sleep_fn=should_sleep_fn@entry=0x7ffffff,
type=SIX_LOCK_intent, lock=lock@entry=0xfffffffff0000000) at linux/six.c:531
#5 six_lock_intent (lock=lock@entry=0x843200, should_sleep_fn=should_sleep_fn@entry=0x42fb40 <lock_root_check_fn>, p=p@entry=0x7ffff7f638b0)
at linux/six.c:606
#6 0x00000000004315c3 in six_lock_type (p=0x7ffff7f638b0, should_sleep_fn=0x42fb40 <lock_root_check_fn>, type=(unknown: 4294954432),
lock=0x843200) at include/linux/six.h:182
#7 btree_node_lock_type (p=0x7ffff7f638b0, should_sleep_fn=0x42fb40 <lock_root_check_fn>, type=(unknown: 4294954432), level=0, pos=...,
b=0x843200, path=0x91d080, trans=0x7fffffffd400) at libbcachefs/btree_locking.h:157
#8 __bch2_btree_node_lock (trans=trans@entry=0x7fffffffd400, path=path@entry=0x91d080, b=b@entry=0x843200, pos=..., level=0,
type=type@entry=SIX_LOCK_intent, should_sleep_fn=should_sleep_fn@entry=0x42fb40 <lock_root_check_fn>, p=0x7ffff7f638b0, ip=4430528)
at libbcachefs/btree_iter.c:372
#9 0x0000000000432b7e in btree_node_lock (should_sleep_fn=0x42fb40 <lock_root_check_fn>, ip=4430528, p=0x7ffff7f638b0, type=SIX_LOCK_intent,
level=0, pos=..., b=0x843200, path=0x91d080, trans=0x7fffffffd400) at libbcachefs/btree_locking.h:204
#10 btree_path_lock_root (trace_ip=<optimized out>, depth_want=<optimized out>, path=0x91d080, trans=0x7fffffffd400)
at libbcachefs/btree_iter.c:1230
#11 btree_path_traverse_one (trans=trans@entry=0x7fffffffd400, path=path@entry=0x91d080, flags=flags@entry=0, trace_ip=trace_ip@entry=4430528)
at libbcachefs/btree_iter.c:1607
#12 0x0000000000439ad4 in bch2_btree_path_traverse_all (trans=0x7fffffffd400) at libbcachefs/btree_iter.c:1488
#13 bch2_trans_begin (trans=trans@entry=0x7fffffffd400) at libbcachefs/btree_iter.c:3085
#14 0x0000000000461c39 in bch2_trans_mark_metadata_bucket (trans=trans@entry=0x7fffffffd400, ca=ca@entry=0x914800, b=4781,
type=type@entry=BCH_DATA_journal, sectors=512) at libbcachefs/buckets.c:1905
#15 0x0000000000461f80 in __bch2_trans_mark_dev_sb (ca=0x914800, trans=0x7fffffffd400) at libbcachefs/buckets.c:1971
#16 bch2_trans_mark_dev_sb (c=c@entry=0x7ffff7f63000, ca=ca@entry=0x914800) at libbcachefs/buckets.c:1983
#17 0x000000000049a544 in bch2_fs_initialize (c=0x7ffff7f63000) at libbcachefs/recovery.c:1394
#18 0x00000000004768c3 in bch2_fs_start (c=c@entry=0x7ffff7f63000) at libbcachefs/super.c:898
#19 0x0000000000478ee2 in bch2_fs_open (devices=devices@entry=0x8288d0, nr_devices=nr_devices@entry=1, opts=...) at libbcachefs/super.c:1905
#20 0x0000000000412f92 in cmd_format (argc=argc@entry=2, argv=argv@entry=0x7fffffffe418) at cmd_format.c:268
#21 0x0000000000407ff7 in main (argc=2, argv=0x7fffffffe418) at bcachefs.c:195
...
Thread 14 (Thread 0x7ffff3ff1700 (LWP 23512)):
#0 0x00007ffff670ef0e in pthread_mutex_unlock () from /lib64/libpthread.so.0
#1 0x000000000041c4a0 in del_timer (timer=timer@entry=0x7ffff7f813f0) at linux/timer.c:166
#2 0x00000000004172f8 in grab_pending (work=work@entry=0x7ffff7f813d0, is_dwork=is_dwork@entry=true) at linux/workqueue.c:118
#3 0x0000000000417722 in cancel_delayed_work (dwork=dwork@entry=0x7ffff7f813d0) at linux/workqueue.c:210
#4 0x00000000004b1b8e in __journal_entry_close (j=0x7ffff7f81078, closed_val=<optimized out>) at libbcachefs/journal.c:140
#5 0x00000000004b511d in __journal_res_get (j=j@entry=0x7ffff7f81078, res=res@entry=0x7ffff3ff08f0, flags=flags@entry=5)
at libbcachefs/journal.c:379
#6 0x00000000004b5959 in bch2_journal_res_get_slowpath (j=0x7ffff7f81078, res=res@entry=0x7ffff3ff08f0, flags=flags@entry=5)
at libbcachefs/journal.c:454
#7 0x00000000004af5a0 in bch2_journal_res_get (flags=5, u64s=<optimized out>, res=0x7ffff3ff08f0, j=<optimized out>)
at libbcachefs/journal.h:360
#8 bch2_trans_journal_res_get (flags=5, trans=0x7ffff3ff0810) at libbcachefs/btree_update_leaf.c:301
#9 bch2_trans_commit_write_locked (trace_ip=<optimized out>, stopped_at=<synthetic pointer>, trans=0x7ffff3ff0810)
at libbcachefs/btree_update_leaf.c:686
#10 do_bch2_trans_commit (trace_ip=<optimized out>, stopped_at=<synthetic pointer>, trans=0x7ffff3ff0810)
at libbcachefs/btree_update_leaf.c:917
#11 __bch2_trans_commit (trans=trans@entry=0x7ffff3ff0810) at libbcachefs/btree_update_leaf.c:1149
#12 0x000000000049e5d0 in bch2_trans_commit (flags=<optimized out>, journal_seq=0x0, disk_res=0x0, trans=0x7ffff3ff0810)
at libbcachefs/btree_update.h:105
#13 btree_key_cache_flush_pos (trans=trans@entry=0x7ffff3ff0810, key=..., journal_seq=journal_seq@entry=3,
commit_flags=commit_flags@entry=64, evict=evict@entry=false) at libbcachefs/btree_key_cache.c:418
#14 0x000000000049e8dc in bch2_btree_key_cache_journal_flush (j=0x7ffff7f81078, pin=0x9ef1c0, seq=3) at libbcachefs/btree_key_cache.c:492
#15 0x0000000000494b37 in journal_flush_pins (j=j@entry=0x7ffff7f81078, seq_to_flush=seq_to_flush@entry=0, min_any=0, min_key_cache=0,
min_key_cache@entry=1) at libbcachefs/journal_reclaim.c:506
#16 0x0000000000495dac in __bch2_journal_reclaim (j=j@entry=0x7ffff7f81078, direct=direct@entry=false) at libbcachefs/journal_reclaim.c:645
#17 0x0000000000495f33 in bch2_journal_reclaim_thread (arg=0x7ffff7f81078) at libbcachefs/journal_reclaim.c:684
#18 0x00000000004216c3 in kthread_start_fn (data=0x832f40) at linux/kthread.c:25
#19 0x00007ffff670bea5 in start_thread () from /lib64/libpthread.so.0
#20 0x00007ffff5c229fd in clone () from /lib64/libc.so.6
====== Test 2 ======
(gdb) thr a a bt
Thread 1 (Thread 0x7ffff7fe4b80 (LWP 27067)):
#0 0x00007ffff671254d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007ffff670de9b in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00007ffff670dd68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x0000000000496239 in raw_spin_lock (lock=0x7ffff7f81328) at include/linux/spinlock.h:20
#4 bch2_journal_pin_set (j=0x7ffff7f81078, seq=7, pin=0xbd6fc0, flush_fn=0x49e800 <bch2_btree_key_cache_journal_flush>)
at libbcachefs/journal_reclaim.c:380
#5 0x00000000004a0d43 in bch2_journal_pin_update (flush_fn=0x49e800 <bch2_btree_key_cache_journal_flush>, pin=<optimized out>,
seq=<optimized out>, j=0x7ffff7f81078) at libbcachefs/journal_reclaim.h:66
#6 bch2_btree_insert_key_cached (trans=trans@entry=0x7fffffffd400, path=<optimized out>, insert=<optimized out>)
at libbcachefs/btree_key_cache.c:550
#7 0x00000000004af900 in do_btree_insert_one (i=0x91f000, trans=0x7fffffffd400) at libbcachefs/btree_update_leaf.c:410
#8 bch2_trans_commit_write_locked (trace_ip=<optimized out>, stopped_at=<synthetic pointer>, trans=0x7fffffffd400)
at libbcachefs/btree_update_leaf.c:738
#9 do_bch2_trans_commit (trace_ip=<optimized out>, stopped_at=<synthetic pointer>, trans=0x7fffffffd400)
at libbcachefs/btree_update_leaf.c:917
#10 __bch2_trans_commit (trans=trans@entry=0x7fffffffd400) at libbcachefs/btree_update_leaf.c:1149
#11 0x0000000000461c28 in bch2_trans_commit (flags=0, journal_seq=0x0, disk_res=0x0, trans=0x7fffffffd400) at libbcachefs/btree_update.h:105
#12 bch2_trans_mark_metadata_bucket (trans=trans@entry=0x7fffffffd400, ca=ca@entry=0x914800, b=4780, type=type@entry=BCH_DATA_journal,
sectors=512) at libbcachefs/buckets.c:1905
#13 0x0000000000461f80 in __bch2_trans_mark_dev_sb (ca=0x914800, trans=0x7fffffffd400) at libbcachefs/buckets.c:1971
#14 bch2_trans_mark_dev_sb (c=c@entry=0x7ffff7f63000, ca=ca@entry=0x914800) at libbcachefs/buckets.c:1983
#15 0x000000000049a544 in bch2_fs_initialize (c=0x7ffff7f63000) at libbcachefs/recovery.c:1394
#16 0x00000000004768c3 in bch2_fs_start (c=c@entry=0x7ffff7f63000) at libbcachefs/super.c:898
#17 0x0000000000478ee2 in bch2_fs_open (devices=devices@entry=0x8288d0, nr_devices=nr_devices@entry=1, opts=...) at libbcachefs/super.c:1905
#18 0x0000000000412f92 in cmd_format (argc=argc@entry=2, argv=argv@entry=0x7fffffffe418) at cmd_format.c:268
#19 0x0000000000407ff7 in main (argc=2, argv=0x7fffffffe418) at bcachefs.c:195
...
Thread 14 (Thread 0x7ffff3ff1700 (LWP 27169)):
#0 0x00007ffff670ef0e in pthread_mutex_unlock () from /lib64/libpthread.so.0
#1 0x000000000041c4a0 in del_timer (timer=timer@entry=0x7ffff7f813f0) at linux/timer.c:166
#2 0x00000000004172f8 in grab_pending (work=work@entry=0x7ffff7f813d0, is_dwork=is_dwork@entry=true) at linux/workqueue.c:118
#3 0x0000000000417722 in cancel_delayed_work (dwork=dwork@entry=0x7ffff7f813d0) at linux/workqueue.c:210
#4 0x00000000004b1b8e in __journal_entry_close (j=0x7ffff7f81078, closed_val=<optimized out>) at libbcachefs/journal.c:140
#5 0x00000000004b511d in __journal_res_get (j=j@entry=0x7ffff7f81078, res=res@entry=0x7ffff3ff08f0, flags=flags@entry=5)
at libbcachefs/journal.c:379
#6 0x00000000004b5959 in bch2_journal_res_get_slowpath (j=0x7ffff7f81078, res=res@entry=0x7ffff3ff08f0, flags=flags@entry=5)
at libbcachefs/journal.c:454
#7 0x00000000004af5a0 in bch2_journal_res_get (flags=5, u64s=<optimized out>, res=0x7ffff3ff08f0, j=<optimized out>)
at libbcachefs/journal.h:360
#8 bch2_trans_journal_res_get (flags=5, trans=0x7ffff3ff0810) at libbcachefs/btree_update_leaf.c:301
#9 bch2_trans_commit_write_locked (trace_ip=<optimized out>, stopped_at=<synthetic pointer>, trans=0x7ffff3ff0810)
at libbcachefs/btree_update_leaf.c:686
#10 do_bch2_trans_commit (trace_ip=<optimized out>, stopped_at=<synthetic pointer>, trans=0x7ffff3ff0810)
at libbcachefs/btree_update_leaf.c:917
#11 __bch2_trans_commit (trans=trans@entry=0x7ffff3ff0810) at libbcachefs/btree_update_leaf.c:1149
#12 0x000000000049e5d0 in bch2_trans_commit (flags=<optimized out>, journal_seq=0x0, disk_res=0x0, trans=0x7ffff3ff0810)
at libbcachefs/btree_update.h:105
#13 btree_key_cache_flush_pos (trans=trans@entry=0x7ffff3ff0810, key=..., journal_seq=journal_seq@entry=3,
commit_flags=commit_flags@entry=64, evict=evict@entry=false) at libbcachefs/btree_key_cache.c:418
#14 0x000000000049e8dc in bch2_btree_key_cache_journal_flush (j=0x7ffff7f81078, pin=0x9ef2c0, seq=3) at libbcachefs/btree_key_cache.c:492
#15 0x0000000000494b37 in journal_flush_pins (j=j@entry=0x7ffff7f81078, seq_to_flush=seq_to_flush@entry=0, min_any=0, min_key_cache=0,
min_key_cache@entry=1) at libbcachefs/journal_reclaim.c:506
#16 0x0000000000495dac in __bch2_journal_reclaim (j=j@entry=0x7ffff7f81078, direct=direct@entry=false) at libbcachefs/journal_reclaim.c:645
#17 0x0000000000495f33 in bch2_journal_reclaim_thread (arg=0x7ffff7f81078) at libbcachefs/journal_reclaim.c:684
#18 0x00000000004216c3 in kthread_start_fn (data=0x832f40) at linux/kthread.c:25
#19 0x00007ffff670bea5 in start_thread () from /lib64/libpthread.so.0
#20 0x00007ffff5c229fd in clone () from /lib64/libc.so.6
> > Also, how long was the system in use before this happened? How many reboots?
>
> Only 1 or 2 reboots. Basically we set it up and it crashed because of the
> iterator issue. Then we rebooted into the new kernel with both your
> iterator fix and the new allocater update. I think that's about it
> though.
>
> > Were you by chance running the resize-journal command?
>
> Nope.
-Eric
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2022-03-18 23:22 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-03-16 22:58 bcachefs filesystem corruption Eric Wheeler
2022-03-17 3:44 ` Kent Overstreet
2022-03-17 23:40 ` Eric Wheeler
2022-03-18 23:22 ` Eric Wheeler
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.