* kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 with scsi_mod.use_blk_mq=y @ 2019-09-25 18:40 ` Eric Wheeler 0 siblings, 0 replies; 43+ messages in thread From: Eric Wheeler @ 2019-09-25 18:40 UTC (permalink / raw) To: dm-devel, linux-block, linux-bcache, lvm-devel Hello, We are using the 4.19.75 stable tree with dm-thin and multi-queue scsi. We have been using the 4.19 branch for months without issue; we just switched to MQ and we seem to have hit this BUG_ON. Whether or not MQ is related to the issue, I don't know, maybe coincidence: static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) { int r; enum allocation_event ev; struct sm_disk *smd = container_of(sm, struct sm_disk, sm); /* FIXME: we should loop round a couple of times */ r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); if (r) return r; smd->begin = *b + 1; r = sm_ll_inc(&smd->ll, *b, &ev); if (!r) { BUG_ON(ev != SM_ALLOC); <<<<<<<<<<<<<<<<<<<<<<<<<<<<< smd->nr_allocated_this_transaction++; } return r; } This is a brand-new thin pool created about 12 hours ago: lvcreate -c 64k -L 12t --type thin-pool --thinpool data-pool --poolmetadatasize 16G data /dev/bcache0 We are using bcache, but I don't see any bcache code in the backtraces. The metadata is also on the bcache volume. We were transferring data to the new thin volumes and it ran for about 12 hours and then gave the trace below. So far it has only happened once and I don't have a way to reproduce it. Any idea what this BUG_ON would indicate and how we might contrive a fix? -Eric [199391.677689] ------------[ cut here ]------------ [199391.678437] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! [199391.679183] invalid opcode: 0000 [#1] SMP NOPTI [199391.679941] CPU: 4 PID: 31359 Comm: kworker/u16:4 Not tainted 4.19.75 #1 [199391.680683] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 [199391.681446] Workqueue: dm-thin do_worker [dm_thin_pool] [199391.682187] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] [199391.682929] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 [199391.684432] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 [199391.685186] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 [199391.685936] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 [199391.686659] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 [199391.687379] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 [199391.688120] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 [199391.688843] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 [199391.689571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [199391.690253] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 [199391.690984] Call Trace: [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] [199391.693852] ? sort+0x17b/0x270 [199391.694527] ? u32_swap+0x10/0x10 [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] [199391.695890] process_one_work+0x171/0x370 [199391.696640] worker_thread+0x49/0x3f0 [199391.697332] kthread+0xf8/0x130 [199391.697988] ? max_active_store+0x80/0x80 [199391.698659] ? kthread_bind+0x10/0x10 [199391.699281] ret_from_fork+0x1f/0x40 [199391.699930] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_bit drm_kms_helper [199391.705631] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod [199391.708083] ---[ end trace c31536d98046e8ec ]--- [199391.866776] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] [199391.867960] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 [199391.870379] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 [199391.871524] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 [199391.872364] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 [199391.873173] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 [199391.873871] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 [199391.874550] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 [199391.875231] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 [199391.875941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [199391.876633] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 [199391.877317] Kernel panic - not syncing: Fatal exception [199391.878006] Kernel Offset: disabled [199392.032304] ---[ end Kernel panic - not syncing: Fatal exception ]--- [199392.032962] unchecked MSR access error: WRMSR to 0x83f (tried to write 0x00000000000000f6) at rIP: 0xffffffff81067af4 (native_write_msr+0x4/0x20) [199392.034277] Call Trace: [199392.034929] <IRQ> [199392.035576] native_apic_msr_write+0x2e/0x40 [199392.036228] arch_irq_work_raise+0x28/0x40 [199392.036877] irq_work_queue_on+0x83/0xa0 [199392.037518] irq_work_run_list+0x4c/0x70 [199392.038149] irq_work_run+0x14/0x40 [199392.038771] smp_call_function_single_interrupt+0x3a/0xd0 [199392.039393] call_function_single_interrupt+0xf/0x20 [199392.040011] </IRQ> [199392.040624] RIP: 0010:panic+0x209/0x25c [199392.041234] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de [199392.042518] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 [199392.043174] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 [199392.043833] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 [199392.044493] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 [199392.045155] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c [199392.045820] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 [199392.046486] oops_end+0xc1/0xd0 [199392.047149] do_trap+0x13d/0x150 [199392.047795] do_error_trap+0xd5/0x130 [199392.048427] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] [199392.049048] invalid_op+0x14/0x20 [199392.049650] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] [199392.050245] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 [199392.051434] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 [199392.052010] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 [199392.052580] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 [199392.053150] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 [199392.053715] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 [199392.054266] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 [199392.054807] ? __wake_up_common_lock+0x87/0xc0 [199392.055342] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] [199392.055877] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] [199392.056410] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] [199392.056947] process_cell+0x2a3/0x550 [dm_thin_pool] [199392.057484] ? sort+0x17b/0x270 [199392.058016] ? u32_swap+0x10/0x10 [199392.058538] do_worker+0x268/0x9a0 [dm_thin_pool] [199392.059060] process_one_work+0x171/0x370 [199392.059576] worker_thread+0x49/0x3f0 [199392.060083] kthread+0xf8/0x130 [199392.060587] ? max_active_store+0x80/0x80 [199392.061086] ? kthread_bind+0x10/0x10 [199392.061569] ret_from_fork+0x1f/0x40 [199392.062038] ------------[ cut here ]------------ [199392.062508] sched: Unexpected reschedule of offline CPU#1! [199392.062989] WARNING: CPU: 4 PID: 31359 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40 [199392.063485] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_bit drm_kms_helper [199392.067463] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod [199392.068729] CPU: 4 PID: 31359 Comm: kworker/u16:4 Tainted: G D 4.19.75 #1 [199392.069356] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 [199392.069982] Workqueue: dm-thin do_worker [dm_thin_pool] [199392.070604] RIP: 0010:native_smp_send_reschedule+0x39/0x40 [199392.071229] Code: 0f 92 c0 84 c0 74 15 48 8b 05 93 f1 eb 00 be fd 00 00 00 48 8b 40 30 e9 85 f4 9a 00 89 fe 48 c7 c7 08 23 e5 81 e8 c7 9a 05 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48 [199392.072528] RSP: 0018:ffff88880fb03dc0 EFLAGS: 00010082 [199392.073186] RAX: 0000000000000000 RBX: ffff88880fa625c0 RCX: 0000000000000006 [199392.073847] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff88880fb168b0 [199392.074512] RBP: ffff88880fa625c0 R08: 0000000000000001 R09: 00000000000174e5 [199392.075182] R10: ffff8881f2512300 R11: 0000000000000000 R12: ffff888804541640 [199392.075849] R13: ffff88880fb03e08 R14: 0000000000000000 R15: 0000000000000001 [199392.076512] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 [199392.077179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [199392.077833] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 [199392.078481] Call Trace: [199392.079117] <IRQ> [199392.079745] check_preempt_curr+0x6b/0x90 [199392.080373] ttwu_do_wakeup+0x19/0x130 [199392.080999] try_to_wake_up+0x1e2/0x460 [199392.081623] __wake_up_common+0x8f/0x160 [199392.082246] ep_poll_callback+0x1af/0x300 [199392.082860] __wake_up_common+0x8f/0x160 [199392.083470] __wake_up_common_lock+0x7a/0xc0 [199392.084074] irq_work_run_list+0x4c/0x70 [199392.084675] smp_call_function_single_interrupt+0x3a/0xd0 [199392.085277] call_function_single_interrupt+0xf/0x20 [199392.085879] </IRQ> [199392.086477] RIP: 0010:panic+0x209/0x25c [199392.087079] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de [199392.088341] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 [199392.088988] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 [199392.089639] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 [199392.090291] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 [199392.090947] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c [199392.091601] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 [199392.092255] oops_end+0xc1/0xd0 [199392.092894] do_trap+0x13d/0x150 [199392.093516] do_error_trap+0xd5/0x130 [199392.094122] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] [199392.094718] invalid_op+0x14/0x20 [199392.095296] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] [199392.095866] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 [199392.097006] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 [199392.097558] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 [199392.098103] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 [199392.098644] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 [199392.099176] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 [199392.099703] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 [199392.100229] ? __wake_up_common_lock+0x87/0xc0 [199392.100744] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] [199392.101251] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] [199392.101752] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] [199392.102252] process_cell+0x2a3/0x550 [dm_thin_pool] [199392.102750] ? sort+0x17b/0x270 [199392.103242] ? u32_swap+0x10/0x10 [199392.103733] do_worker+0x268/0x9a0 [dm_thin_pool] [199392.104228] process_one_work+0x171/0x370 [199392.104714] worker_thread+0x49/0x3f0 [199392.105193] kthread+0xf8/0x130 [199392.105665] ? max_active_store+0x80/0x80 [199392.106132] ? kthread_bind+0x10/0x10 [199392.106601] ret_from_fork+0x1f/0x40 [199392.107069] ---[ end trace c31536d98046e8ed ]--- [199392.107544] ------------[ cut here ]------------ [199392.108017] sched: Unexpected reschedule of offline CPU#7! [199392.108497] WARNING: CPU: 4 PID: 31359 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40 [199392.108996] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_bit drm_kms_helper [199392.112967] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod [199392.114203] CPU: 4 PID: 31359 Comm: kworker/u16:4 Tainted: G D W 4.19.75 #1 [199392.114819] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 [199392.115439] Workqueue: dm-thin do_worker [dm_thin_pool] [199392.116061] RIP: 0010:native_smp_send_reschedule+0x39/0x40 [199392.116683] Code: 0f 92 c0 84 c0 74 15 48 8b 05 93 f1 eb 00 be fd 00 00 00 48 8b 40 30 e9 85 f4 9a 00 89 fe 48 c7 c7 08 23 e5 81 e8 c7 9a 05 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48 [199392.117982] RSP: 0018:ffff88880fb03ee0 EFLAGS: 00010082 [199392.118632] RAX: 0000000000000000 RBX: ffff8887d093ac80 RCX: 0000000000000006 [199392.119295] RDX: 0000000000000000 RSI: 0000000000000092 RDI: ffff88880fb168b0 [199392.119961] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000017529 [199392.120623] R10: ffff8881438ba800 R11: 0000000000000000 R12: ffffc9000a147988 [199392.121283] R13: ffffffff8113f9a0 R14: 0000000000000002 R15: ffff88880fb1cff8 [199392.121938] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 [199392.122589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [199392.123229] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 [199392.123867] Call Trace: [199392.124495] <IRQ> [199392.125116] update_process_times+0x40/0x50 [199392.125742] tick_sched_handle+0x25/0x60 [199392.126367] tick_sched_timer+0x37/0x70 [199392.126987] __hrtimer_run_queues+0xfb/0x270 [199392.127601] hrtimer_interrupt+0x122/0x270 [199392.128211] smp_apic_timer_interrupt+0x6a/0x140 [199392.128819] apic_timer_interrupt+0xf/0x20 [199392.129423] </IRQ> [199392.130020] RIP: 0010:panic+0x209/0x25c [199392.130619] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de [199392.131883] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [199392.132530] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 [199392.133180] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 [199392.133830] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 [199392.134482] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c [199392.135136] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 [199392.135792] oops_end+0xc1/0xd0 [199392.136444] do_trap+0x13d/0x150 [199392.137094] do_error_trap+0xd5/0x130 [199392.137740] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] [199392.138383] invalid_op+0x14/0x20 [199392.139010] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] [199392.139630] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 [199392.140870] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 [199392.141472] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 [199392.142066] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 [199392.142647] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 [199392.143211] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 [199392.143759] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 [199392.144301] ? __wake_up_common_lock+0x87/0xc0 [199392.144838] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] [199392.145374] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] [199392.145909] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] [199392.146435] process_cell+0x2a3/0x550 [dm_thin_pool] [199392.146949] ? sort+0x17b/0x270 [199392.147450] ? u32_swap+0x10/0x10 [199392.147944] do_worker+0x268/0x9a0 [dm_thin_pool] [199392.148441] process_one_work+0x171/0x370 [199392.148937] worker_thread+0x49/0x3f0 [199392.149430] kthread+0xf8/0x130 [199392.149922] ? max_active_store+0x80/0x80 [199392.150406] ? kthread_bind+0x10/0x10 [199392.150883] ret_from_fork+0x1f/0x40 [199392.151353] ---[ end trace c31536d98046e8ee ]--- -- Eric Wheeler ^ permalink raw reply [flat|nested] 43+ messages in thread
* kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 with scsi_mod.use_blk_mq=y @ 2019-09-25 18:40 ` Eric Wheeler 0 siblings, 0 replies; 43+ messages in thread From: Eric Wheeler @ 2019-09-25 18:40 UTC (permalink / raw) To: lvm-devel Hello, We are using the 4.19.75 stable tree with dm-thin and multi-queue scsi. We have been using the 4.19 branch for months without issue; we just switched to MQ and we seem to have hit this BUG_ON. Whether or not MQ is related to the issue, I don't know, maybe coincidence: static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) { int r; enum allocation_event ev; struct sm_disk *smd = container_of(sm, struct sm_disk, sm); /* FIXME: we should loop round a couple of times */ r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); if (r) return r; smd->begin = *b + 1; r = sm_ll_inc(&smd->ll, *b, &ev); if (!r) { BUG_ON(ev != SM_ALLOC); <<<<<<<<<<<<<<<<<<<<<<<<<<<<< smd->nr_allocated_this_transaction++; } return r; } This is a brand-new thin pool created about 12 hours ago: lvcreate -c 64k -L 12t --type thin-pool --thinpool data-pool --poolmetadatasize 16G data /dev/bcache0 We are using bcache, but I don't see any bcache code in the backtraces. The metadata is also on the bcache volume. We were transferring data to the new thin volumes and it ran for about 12 hours and then gave the trace below. So far it has only happened once and I don't have a way to reproduce it. Any idea what this BUG_ON would indicate and how we might contrive a fix? -Eric [199391.677689] ------------[ cut here ]------------ [199391.678437] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! [199391.679183] invalid opcode: 0000 [#1] SMP NOPTI [199391.679941] CPU: 4 PID: 31359 Comm: kworker/u16:4 Not tainted 4.19.75 #1 [199391.680683] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 [199391.681446] Workqueue: dm-thin do_worker [dm_thin_pool] [199391.682187] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] [199391.682929] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 [199391.684432] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 [199391.685186] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 [199391.685936] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 [199391.686659] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 [199391.687379] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 [199391.688120] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 [199391.688843] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 [199391.689571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [199391.690253] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 [199391.690984] Call Trace: [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] [199391.693852] ? sort+0x17b/0x270 [199391.694527] ? u32_swap+0x10/0x10 [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] [199391.695890] process_one_work+0x171/0x370 [199391.696640] worker_thread+0x49/0x3f0 [199391.697332] kthread+0xf8/0x130 [199391.697988] ? max_active_store+0x80/0x80 [199391.698659] ? kthread_bind+0x10/0x10 [199391.699281] ret_from_fork+0x1f/0x40 [199391.699930] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_bit drm_kms_helper [199391.705631] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod [199391.708083] ---[ end trace c31536d98046e8ec ]--- [199391.866776] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] [199391.867960] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 [199391.870379] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 [199391.871524] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 [199391.872364] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 [199391.873173] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 [199391.873871] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 [199391.874550] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 [199391.875231] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 [199391.875941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [199391.876633] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 [199391.877317] Kernel panic - not syncing: Fatal exception [199391.878006] Kernel Offset: disabled [199392.032304] ---[ end Kernel panic - not syncing: Fatal exception ]--- [199392.032962] unchecked MSR access error: WRMSR to 0x83f (tried to write 0x00000000000000f6) at rIP: 0xffffffff81067af4 (native_write_msr+0x4/0x20) [199392.034277] Call Trace: [199392.034929] <IRQ> [199392.035576] native_apic_msr_write+0x2e/0x40 [199392.036228] arch_irq_work_raise+0x28/0x40 [199392.036877] irq_work_queue_on+0x83/0xa0 [199392.037518] irq_work_run_list+0x4c/0x70 [199392.038149] irq_work_run+0x14/0x40 [199392.038771] smp_call_function_single_interrupt+0x3a/0xd0 [199392.039393] call_function_single_interrupt+0xf/0x20 [199392.040011] </IRQ> [199392.040624] RIP: 0010:panic+0x209/0x25c [199392.041234] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de [199392.042518] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 [199392.043174] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 [199392.043833] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 [199392.044493] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 [199392.045155] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c [199392.045820] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 [199392.046486] oops_end+0xc1/0xd0 [199392.047149] do_trap+0x13d/0x150 [199392.047795] do_error_trap+0xd5/0x130 [199392.048427] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] [199392.049048] invalid_op+0x14/0x20 [199392.049650] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] [199392.050245] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 [199392.051434] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 [199392.052010] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 [199392.052580] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 [199392.053150] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 [199392.053715] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 [199392.054266] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 [199392.054807] ? __wake_up_common_lock+0x87/0xc0 [199392.055342] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] [199392.055877] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] [199392.056410] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] [199392.056947] process_cell+0x2a3/0x550 [dm_thin_pool] [199392.057484] ? sort+0x17b/0x270 [199392.058016] ? u32_swap+0x10/0x10 [199392.058538] do_worker+0x268/0x9a0 [dm_thin_pool] [199392.059060] process_one_work+0x171/0x370 [199392.059576] worker_thread+0x49/0x3f0 [199392.060083] kthread+0xf8/0x130 [199392.060587] ? max_active_store+0x80/0x80 [199392.061086] ? kthread_bind+0x10/0x10 [199392.061569] ret_from_fork+0x1f/0x40 [199392.062038] ------------[ cut here ]------------ [199392.062508] sched: Unexpected reschedule of offline CPU#1! [199392.062989] WARNING: CPU: 4 PID: 31359 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40 [199392.063485] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_bit drm_kms_helper [199392.067463] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod [199392.068729] CPU: 4 PID: 31359 Comm: kworker/u16:4 Tainted: G D 4.19.75 #1 [199392.069356] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 [199392.069982] Workqueue: dm-thin do_worker [dm_thin_pool] [199392.070604] RIP: 0010:native_smp_send_reschedule+0x39/0x40 [199392.071229] Code: 0f 92 c0 84 c0 74 15 48 8b 05 93 f1 eb 00 be fd 00 00 00 48 8b 40 30 e9 85 f4 9a 00 89 fe 48 c7 c7 08 23 e5 81 e8 c7 9a 05 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48 [199392.072528] RSP: 0018:ffff88880fb03dc0 EFLAGS: 00010082 [199392.073186] RAX: 0000000000000000 RBX: ffff88880fa625c0 RCX: 0000000000000006 [199392.073847] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff88880fb168b0 [199392.074512] RBP: ffff88880fa625c0 R08: 0000000000000001 R09: 00000000000174e5 [199392.075182] R10: ffff8881f2512300 R11: 0000000000000000 R12: ffff888804541640 [199392.075849] R13: ffff88880fb03e08 R14: 0000000000000000 R15: 0000000000000001 [199392.076512] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 [199392.077179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [199392.077833] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 [199392.078481] Call Trace: [199392.079117] <IRQ> [199392.079745] check_preempt_curr+0x6b/0x90 [199392.080373] ttwu_do_wakeup+0x19/0x130 [199392.080999] try_to_wake_up+0x1e2/0x460 [199392.081623] __wake_up_common+0x8f/0x160 [199392.082246] ep_poll_callback+0x1af/0x300 [199392.082860] __wake_up_common+0x8f/0x160 [199392.083470] __wake_up_common_lock+0x7a/0xc0 [199392.084074] irq_work_run_list+0x4c/0x70 [199392.084675] smp_call_function_single_interrupt+0x3a/0xd0 [199392.085277] call_function_single_interrupt+0xf/0x20 [199392.085879] </IRQ> [199392.086477] RIP: 0010:panic+0x209/0x25c [199392.087079] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de [199392.088341] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 [199392.088988] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 [199392.089639] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 [199392.090291] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 [199392.090947] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c [199392.091601] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 [199392.092255] oops_end+0xc1/0xd0 [199392.092894] do_trap+0x13d/0x150 [199392.093516] do_error_trap+0xd5/0x130 [199392.094122] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] [199392.094718] invalid_op+0x14/0x20 [199392.095296] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] [199392.095866] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 [199392.097006] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 [199392.097558] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 [199392.098103] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 [199392.098644] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 [199392.099176] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 [199392.099703] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 [199392.100229] ? __wake_up_common_lock+0x87/0xc0 [199392.100744] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] [199392.101251] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] [199392.101752] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] [199392.102252] process_cell+0x2a3/0x550 [dm_thin_pool] [199392.102750] ? sort+0x17b/0x270 [199392.103242] ? u32_swap+0x10/0x10 [199392.103733] do_worker+0x268/0x9a0 [dm_thin_pool] [199392.104228] process_one_work+0x171/0x370 [199392.104714] worker_thread+0x49/0x3f0 [199392.105193] kthread+0xf8/0x130 [199392.105665] ? max_active_store+0x80/0x80 [199392.106132] ? kthread_bind+0x10/0x10 [199392.106601] ret_from_fork+0x1f/0x40 [199392.107069] ---[ end trace c31536d98046e8ed ]--- [199392.107544] ------------[ cut here ]------------ [199392.108017] sched: Unexpected reschedule of offline CPU#7! [199392.108497] WARNING: CPU: 4 PID: 31359 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40 [199392.108996] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_bit drm_kms_helper [199392.112967] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod [199392.114203] CPU: 4 PID: 31359 Comm: kworker/u16:4 Tainted: G D W 4.19.75 #1 [199392.114819] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 [199392.115439] Workqueue: dm-thin do_worker [dm_thin_pool] [199392.116061] RIP: 0010:native_smp_send_reschedule+0x39/0x40 [199392.116683] Code: 0f 92 c0 84 c0 74 15 48 8b 05 93 f1 eb 00 be fd 00 00 00 48 8b 40 30 e9 85 f4 9a 00 89 fe 48 c7 c7 08 23 e5 81 e8 c7 9a 05 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48 [199392.117982] RSP: 0018:ffff88880fb03ee0 EFLAGS: 00010082 [199392.118632] RAX: 0000000000000000 RBX: ffff8887d093ac80 RCX: 0000000000000006 [199392.119295] RDX: 0000000000000000 RSI: 0000000000000092 RDI: ffff88880fb168b0 [199392.119961] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000017529 [199392.120623] R10: ffff8881438ba800 R11: 0000000000000000 R12: ffffc9000a147988 [199392.121283] R13: ffffffff8113f9a0 R14: 0000000000000002 R15: ffff88880fb1cff8 [199392.121938] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 [199392.122589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [199392.123229] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 [199392.123867] Call Trace: [199392.124495] <IRQ> [199392.125116] update_process_times+0x40/0x50 [199392.125742] tick_sched_handle+0x25/0x60 [199392.126367] tick_sched_timer+0x37/0x70 [199392.126987] __hrtimer_run_queues+0xfb/0x270 [199392.127601] hrtimer_interrupt+0x122/0x270 [199392.128211] smp_apic_timer_interrupt+0x6a/0x140 [199392.128819] apic_timer_interrupt+0xf/0x20 [199392.129423] </IRQ> [199392.130020] RIP: 0010:panic+0x209/0x25c [199392.130619] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de [199392.131883] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [199392.132530] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 [199392.133180] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 [199392.133830] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 [199392.134482] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c [199392.135136] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 [199392.135792] oops_end+0xc1/0xd0 [199392.136444] do_trap+0x13d/0x150 [199392.137094] do_error_trap+0xd5/0x130 [199392.137740] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] [199392.138383] invalid_op+0x14/0x20 [199392.139010] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] [199392.139630] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 [199392.140870] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 [199392.141472] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 [199392.142066] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 [199392.142647] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 [199392.143211] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 [199392.143759] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 [199392.144301] ? __wake_up_common_lock+0x87/0xc0 [199392.144838] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] [199392.145374] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] [199392.145909] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] [199392.146435] process_cell+0x2a3/0x550 [dm_thin_pool] [199392.146949] ? sort+0x17b/0x270 [199392.147450] ? u32_swap+0x10/0x10 [199392.147944] do_worker+0x268/0x9a0 [dm_thin_pool] [199392.148441] process_one_work+0x171/0x370 [199392.148937] worker_thread+0x49/0x3f0 [199392.149430] kthread+0xf8/0x130 [199392.149922] ? max_active_store+0x80/0x80 [199392.150406] ? kthread_bind+0x10/0x10 [199392.150883] ret_from_fork+0x1f/0x40 [199392.151353] ---[ end trace c31536d98046e8ee ]--- -- Eric Wheeler ^ permalink raw reply [flat|nested] 43+ messages in thread
* kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 with scsi_mod.use_blk_mq=y @ 2019-09-25 18:40 ` Eric Wheeler 0 siblings, 0 replies; 43+ messages in thread From: Eric Wheeler @ 2019-09-25 18:40 UTC (permalink / raw) To: dm-devel, linux-block, linux-bcache, lvm-devel Hello, We are using the 4.19.75 stable tree with dm-thin and multi-queue scsi. We have been using the 4.19 branch for months without issue; we just switched to MQ and we seem to have hit this BUG_ON. Whether or not MQ is related to the issue, I don't know, maybe coincidence: static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) { int r; enum allocation_event ev; struct sm_disk *smd = container_of(sm, struct sm_disk, sm); /* FIXME: we should loop round a couple of times */ r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); if (r) return r; smd->begin = *b + 1; r = sm_ll_inc(&smd->ll, *b, &ev); if (!r) { BUG_ON(ev != SM_ALLOC); <<<<<<<<<<<<<<<<<<<<<<<<<<<<< smd->nr_allocated_this_transaction++; } return r; } This is a brand-new thin pool created about 12 hours ago: lvcreate -c 64k -L 12t --type thin-pool --thinpool data-pool --poolmetadatasize 16G data /dev/bcache0 We are using bcache, but I don't see any bcache code in the backtraces. The metadata is also on the bcache volume. We were transferring data to the new thin volumes and it ran for about 12 hours and then gave the trace below. So far it has only happened once and I don't have a way to reproduce it. Any idea what this BUG_ON would indicate and how we might contrive a fix? -Eric [199391.677689] ------------[ cut here ]------------ [199391.678437] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! [199391.679183] invalid opcode: 0000 [#1] SMP NOPTI [199391.679941] CPU: 4 PID: 31359 Comm: kworker/u16:4 Not tainted 4.19.75 #1 [199391.680683] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 [199391.681446] Workqueue: dm-thin do_worker [dm_thin_pool] [199391.682187] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] [199391.682929] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 [199391.684432] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 [199391.685186] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 [199391.685936] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 [199391.686659] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 [199391.687379] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 [199391.688120] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 [199391.688843] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 [199391.689571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [199391.690253] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 [199391.690984] Call Trace: [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] [199391.693852] ? sort+0x17b/0x270 [199391.694527] ? u32_swap+0x10/0x10 [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] [199391.695890] process_one_work+0x171/0x370 [199391.696640] worker_thread+0x49/0x3f0 [199391.697332] kthread+0xf8/0x130 [199391.697988] ? max_active_store+0x80/0x80 [199391.698659] ? kthread_bind+0x10/0x10 [199391.699281] ret_from_fork+0x1f/0x40 [199391.699930] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_ bit drm_kms_helper [199391.705631] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod [199391.708083] ---[ end trace c31536d98046e8ec ]--- [199391.866776] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] [199391.867960] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 [199391.870379] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 [199391.871524] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 [199391.872364] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 [199391.873173] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 [199391.873871] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 [199391.874550] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 [199391.875231] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 [199391.875941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [199391.876633] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 [199391.877317] Kernel panic - not syncing: Fatal exception [199391.878006] Kernel Offset: disabled [199392.032304] ---[ end Kernel panic - not syncing: Fatal exception ]--- [199392.032962] unchecked MSR access error: WRMSR to 0x83f (tried to write 0x00000000000000f6) at rIP: 0xffffffff81067af4 (native_write_msr+0x4/0x20) [199392.034277] Call Trace: [199392.034929] <IRQ> [199392.035576] native_apic_msr_write+0x2e/0x40 [199392.036228] arch_irq_work_raise+0x28/0x40 [199392.036877] irq_work_queue_on+0x83/0xa0 [199392.037518] irq_work_run_list+0x4c/0x70 [199392.038149] irq_work_run+0x14/0x40 [199392.038771] smp_call_function_single_interrupt+0x3a/0xd0 [199392.039393] call_function_single_interrupt+0xf/0x20 [199392.040011] </IRQ> [199392.040624] RIP: 0010:panic+0x209/0x25c [199392.041234] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de [199392.042518] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 [199392.043174] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 [199392.043833] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 [199392.044493] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 [199392.045155] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c [199392.045820] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 [199392.046486] oops_end+0xc1/0xd0 [199392.047149] do_trap+0x13d/0x150 [199392.047795] do_error_trap+0xd5/0x130 [199392.048427] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] [199392.049048] invalid_op+0x14/0x20 [199392.049650] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] [199392.050245] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 [199392.051434] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 [199392.052010] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 [199392.052580] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 [199392.053150] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 [199392.053715] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 [199392.054266] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 [199392.054807] ? __wake_up_common_lock+0x87/0xc0 [199392.055342] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] [199392.055877] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] [199392.056410] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] [199392.056947] process_cell+0x2a3/0x550 [dm_thin_pool] [199392.057484] ? sort+0x17b/0x270 [199392.058016] ? u32_swap+0x10/0x10 [199392.058538] do_worker+0x268/0x9a0 [dm_thin_pool] [199392.059060] process_one_work+0x171/0x370 [199392.059576] worker_thread+0x49/0x3f0 [199392.060083] kthread+0xf8/0x130 [199392.060587] ? max_active_store+0x80/0x80 [199392.061086] ? kthread_bind+0x10/0x10 [199392.061569] ret_from_fork+0x1f/0x40 [199392.062038] ------------[ cut here ]------------ [199392.062508] sched: Unexpected reschedule of offline CPU#1! [199392.062989] WARNING: CPU: 4 PID: 31359 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40 [199392.063485] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_ bit drm_kms_helper [199392.067463] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod [199392.068729] CPU: 4 PID: 31359 Comm: kworker/u16:4 Tainted: G D 4.19.75 #1 [199392.069356] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 [199392.069982] Workqueue: dm-thin do_worker [dm_thin_pool] [199392.070604] RIP: 0010:native_smp_send_reschedule+0x39/0x40 [199392.071229] Code: 0f 92 c0 84 c0 74 15 48 8b 05 93 f1 eb 00 be fd 00 00 00 48 8b 40 30 e9 85 f4 9a 00 89 fe 48 c7 c7 08 23 e5 81 e8 c7 9a 05 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48 [199392.072528] RSP: 0018:ffff88880fb03dc0 EFLAGS: 00010082 [199392.073186] RAX: 0000000000000000 RBX: ffff88880fa625c0 RCX: 0000000000000006 [199392.073847] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff88880fb168b0 [199392.074512] RBP: ffff88880fa625c0 R08: 0000000000000001 R09: 00000000000174e5 [199392.075182] R10: ffff8881f2512300 R11: 0000000000000000 R12: ffff888804541640 [199392.075849] R13: ffff88880fb03e08 R14: 0000000000000000 R15: 0000000000000001 [199392.076512] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 [199392.077179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [199392.077833] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 [199392.078481] Call Trace: [199392.079117] <IRQ> [199392.079745] check_preempt_curr+0x6b/0x90 [199392.080373] ttwu_do_wakeup+0x19/0x130 [199392.080999] try_to_wake_up+0x1e2/0x460 [199392.081623] __wake_up_common+0x8f/0x160 [199392.082246] ep_poll_callback+0x1af/0x300 [199392.082860] __wake_up_common+0x8f/0x160 [199392.083470] __wake_up_common_lock+0x7a/0xc0 [199392.084074] irq_work_run_list+0x4c/0x70 [199392.084675] smp_call_function_single_interrupt+0x3a/0xd0 [199392.085277] call_function_single_interrupt+0xf/0x20 [199392.085879] </IRQ> [199392.086477] RIP: 0010:panic+0x209/0x25c [199392.087079] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de [199392.088341] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 [199392.088988] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 [199392.089639] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 [199392.090291] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 [199392.090947] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c [199392.091601] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 [199392.092255] oops_end+0xc1/0xd0 [199392.092894] do_trap+0x13d/0x150 [199392.093516] do_error_trap+0xd5/0x130 [199392.094122] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] [199392.094718] invalid_op+0x14/0x20 [199392.095296] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] [199392.095866] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 [199392.097006] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 [199392.097558] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 [199392.098103] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 [199392.098644] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 [199392.099176] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 [199392.099703] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 [199392.100229] ? __wake_up_common_lock+0x87/0xc0 [199392.100744] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] [199392.101251] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] [199392.101752] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] [199392.102252] process_cell+0x2a3/0x550 [dm_thin_pool] [199392.102750] ? sort+0x17b/0x270 [199392.103242] ? u32_swap+0x10/0x10 [199392.103733] do_worker+0x268/0x9a0 [dm_thin_pool] [199392.104228] process_one_work+0x171/0x370 [199392.104714] worker_thread+0x49/0x3f0 [199392.105193] kthread+0xf8/0x130 [199392.105665] ? max_active_store+0x80/0x80 [199392.106132] ? kthread_bind+0x10/0x10 [199392.106601] ret_from_fork+0x1f/0x40 [199392.107069] ---[ end trace c31536d98046e8ed ]--- [199392.107544] ------------[ cut here ]------------ [199392.108017] sched: Unexpected reschedule of offline CPU#7! [199392.108497] WARNING: CPU: 4 PID: 31359 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40 [199392.108996] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_ bit drm_kms_helper [199392.112967] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod [199392.114203] CPU: 4 PID: 31359 Comm: kworker/u16:4 Tainted: G D W 4.19.75 #1 [199392.114819] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 [199392.115439] Workqueue: dm-thin do_worker [dm_thin_pool] [199392.116061] RIP: 0010:native_smp_send_reschedule+0x39/0x40 [199392.116683] Code: 0f 92 c0 84 c0 74 15 48 8b 05 93 f1 eb 00 be fd 00 00 00 48 8b 40 30 e9 85 f4 9a 00 89 fe 48 c7 c7 08 23 e5 81 e8 c7 9a 05 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48 [199392.117982] RSP: 0018:ffff88880fb03ee0 EFLAGS: 00010082 [199392.118632] RAX: 0000000000000000 RBX: ffff8887d093ac80 RCX: 0000000000000006 [199392.119295] RDX: 0000000000000000 RSI: 0000000000000092 RDI: ffff88880fb168b0 [199392.119961] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000017529 [199392.120623] R10: ffff8881438ba800 R11: 0000000000000000 R12: ffffc9000a147988 [199392.121283] R13: ffffffff8113f9a0 R14: 0000000000000002 R15: ffff88880fb1cff8 [199392.121938] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 [199392.122589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [199392.123229] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 [199392.123867] Call Trace: [199392.124495] <IRQ> [199392.125116] update_process_times+0x40/0x50 [199392.125742] tick_sched_handle+0x25/0x60 [199392.126367] tick_sched_timer+0x37/0x70 [199392.126987] __hrtimer_run_queues+0xfb/0x270 [199392.127601] hrtimer_interrupt+0x122/0x270 [199392.128211] smp_apic_timer_interrupt+0x6a/0x140 [199392.128819] apic_timer_interrupt+0xf/0x20 [199392.129423] </IRQ> [199392.130020] RIP: 0010:panic+0x209/0x25c [199392.130619] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de [199392.131883] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [199392.132530] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 [199392.133180] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 [199392.133830] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 [199392.134482] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c [199392.135136] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 [199392.135792] oops_end+0xc1/0xd0 [199392.136444] do_trap+0x13d/0x150 [199392.137094] do_error_trap+0xd5/0x130 [199392.137740] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] [199392.138383] invalid_op+0x14/0x20 [199392.139010] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] [199392.139630] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 [199392.140870] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 [199392.141472] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 [199392.142066] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 [199392.142647] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 [199392.143211] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 [199392.143759] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 [199392.144301] ? __wake_up_common_lock+0x87/0xc0 [199392.144838] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] [199392.145374] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] [199392.145909] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] [199392.146435] process_cell+0x2a3/0x550 [dm_thin_pool] [199392.146949] ? sort+0x17b/0x270 [199392.147450] ? u32_swap+0x10/0x10 [199392.147944] do_worker+0x268/0x9a0 [dm_thin_pool] [199392.148441] process_one_work+0x171/0x370 [199392.148937] worker_thread+0x49/0x3f0 [199392.149430] kthread+0xf8/0x130 [199392.149922] ? max_active_store+0x80/0x80 [199392.150406] ? kthread_bind+0x10/0x10 [199392.150883] ret_from_fork+0x1f/0x40 [199392.151353] ---[ end trace c31536d98046e8ee ]--- -- Eric Wheeler ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 with scsi_mod.use_blk_mq=y 2019-09-25 18:40 ` Eric Wheeler (?) @ 2019-09-25 20:01 ` Mike Snitzer -1 siblings, 0 replies; 43+ messages in thread From: Mike Snitzer @ 2019-09-25 20:01 UTC (permalink / raw) To: Eric Wheeler, ejt; +Cc: dm-devel, linux-block, linux-bcache, lvm-devel On Wed, Sep 25 2019 at 2:40pm -0400, Eric Wheeler <dm-devel@lists.ewheeler.net> wrote: > Hello, > > We are using the 4.19.75 stable tree with dm-thin and multi-queue scsi. > We have been using the 4.19 branch for months without issue; we just > switched to MQ and we seem to have hit this BUG_ON. Whether or not MQ is > related to the issue, I don't know, maybe coincidence: > > static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > { > int r; > enum allocation_event ev; > struct sm_disk *smd = container_of(sm, struct sm_disk, sm); > > /* FIXME: we should loop round a couple of times */ > r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); > if (r) > return r; > > smd->begin = *b + 1; > r = sm_ll_inc(&smd->ll, *b, &ev); > if (!r) { > BUG_ON(ev != SM_ALLOC); <<<<<<<<<<<<<<<<<<<<<<<<<<<<< > smd->nr_allocated_this_transaction++; > } > > return r; > } > > This is a brand-new thin pool created about 12 hours ago: > > lvcreate -c 64k -L 12t --type thin-pool --thinpool data-pool --poolmetadatasize 16G data /dev/bcache0 > > We are using bcache, but I don't see any bcache code in the backtraces. > The metadata is also on the bcache volume. So bcache is be used for both data and metadata. > We were transferring data to the new thin volumes and it ran for about 12 > hours and then gave the trace below. So far it has only happened once > and I don't have a way to reproduce it. > > Any idea what this BUG_ON would indicate and how we might contrive a fix? > > [199391.677689] ------------[ cut here ]------------ > [199391.678437] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! > [199391.679183] invalid opcode: 0000 [#1] SMP NOPTI > [199391.679941] CPU: 4 PID: 31359 Comm: kworker/u16:4 Not tainted 4.19.75 #1 > [199391.680683] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > [199391.681446] Workqueue: dm-thin do_worker [dm_thin_pool] > [199391.682187] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199391.682929] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > [199391.684432] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > [199391.685186] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > [199391.685936] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > [199391.686659] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > [199391.687379] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > [199391.688120] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > [199391.688843] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > [199391.689571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [199391.690253] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > [199391.690984] Call Trace: > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > [199391.693852] ? sort+0x17b/0x270 > [199391.694527] ? u32_swap+0x10/0x10 > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > [199391.695890] process_one_work+0x171/0x370 > [199391.696640] worker_thread+0x49/0x3f0 > [199391.697332] kthread+0xf8/0x130 > [199391.697988] ? max_active_store+0x80/0x80 > [199391.698659] ? kthread_bind+0x10/0x10 > [199391.699281] ret_from_fork+0x1f/0x40 The stack shows the call to sm_disk_new_block() is due to dm_pool_alloc_data_block(). sm_disk_new_block()'s BUG_ON(ev != SM_ALLOC) indicates that somehow it is getting called without the passed 'ev' being set to SM_ALLOC. Only drivers/md/persistent-dat/dm-space-map-common.c:sm_ll_mutate() sets SM_ALLOC. sm_disk_new_block() is indirectly calling sm_ll_mutate() sm_ll_mutate() will only return 0 if ll->save_ie() does, the ll_disk *ll should be ll_disk, and so disk_ll_save_ie()'s call to dm_btree_insert() returns 0 -- which simply means success. And on success sm_disk_new_block() assumes ev was set to SM_ALLOC (by sm_ll_mutate). sm_ll_mutate() decided to _not_ set SM_ALLOC because either: 1) ref_count wasn't set or 2) old was identified So all said: somehow a new data block was found to already be in use. _WHY_ that is the case isn't clear from this stack... But it does speak to the possibility of data block allocation racing with other operations to the same block. Which implies missing locking. But that's all I've got so far... I'll review past dm-thinp changes with all this in mind and see what turns up. But Joe Thornber (ejt) likely needs to have a look at this too. But could it be that bcache is the source of the data device race (same block used concurrently)? And DM thinp is acting as the canary in the coal mine? Thanks, Mike ^ permalink raw reply [flat|nested] 43+ messages in thread
* kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 with scsi_mod.use_blk_mq=y @ 2019-09-25 20:01 ` Mike Snitzer 0 siblings, 0 replies; 43+ messages in thread From: Mike Snitzer @ 2019-09-25 20:01 UTC (permalink / raw) To: lvm-devel On Wed, Sep 25 2019 at 2:40pm -0400, Eric Wheeler <dm-devel@lists.ewheeler.net> wrote: > Hello, > > We are using the 4.19.75 stable tree with dm-thin and multi-queue scsi. > We have been using the 4.19 branch for months without issue; we just > switched to MQ and we seem to have hit this BUG_ON. Whether or not MQ is > related to the issue, I don't know, maybe coincidence: > > static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > { > int r; > enum allocation_event ev; > struct sm_disk *smd = container_of(sm, struct sm_disk, sm); > > /* FIXME: we should loop round a couple of times */ > r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); > if (r) > return r; > > smd->begin = *b + 1; > r = sm_ll_inc(&smd->ll, *b, &ev); > if (!r) { > BUG_ON(ev != SM_ALLOC); <<<<<<<<<<<<<<<<<<<<<<<<<<<<< > smd->nr_allocated_this_transaction++; > } > > return r; > } > > This is a brand-new thin pool created about 12 hours ago: > > lvcreate -c 64k -L 12t --type thin-pool --thinpool data-pool --poolmetadatasize 16G data /dev/bcache0 > > We are using bcache, but I don't see any bcache code in the backtraces. > The metadata is also on the bcache volume. So bcache is be used for both data and metadata. > We were transferring data to the new thin volumes and it ran for about 12 > hours and then gave the trace below. So far it has only happened once > and I don't have a way to reproduce it. > > Any idea what this BUG_ON would indicate and how we might contrive a fix? > > [199391.677689] ------------[ cut here ]------------ > [199391.678437] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! > [199391.679183] invalid opcode: 0000 [#1] SMP NOPTI > [199391.679941] CPU: 4 PID: 31359 Comm: kworker/u16:4 Not tainted 4.19.75 #1 > [199391.680683] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > [199391.681446] Workqueue: dm-thin do_worker [dm_thin_pool] > [199391.682187] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199391.682929] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > [199391.684432] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > [199391.685186] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > [199391.685936] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > [199391.686659] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > [199391.687379] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > [199391.688120] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > [199391.688843] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > [199391.689571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [199391.690253] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > [199391.690984] Call Trace: > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > [199391.693852] ? sort+0x17b/0x270 > [199391.694527] ? u32_swap+0x10/0x10 > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > [199391.695890] process_one_work+0x171/0x370 > [199391.696640] worker_thread+0x49/0x3f0 > [199391.697332] kthread+0xf8/0x130 > [199391.697988] ? max_active_store+0x80/0x80 > [199391.698659] ? kthread_bind+0x10/0x10 > [199391.699281] ret_from_fork+0x1f/0x40 The stack shows the call to sm_disk_new_block() is due to dm_pool_alloc_data_block(). sm_disk_new_block()'s BUG_ON(ev != SM_ALLOC) indicates that somehow it is getting called without the passed 'ev' being set to SM_ALLOC. Only drivers/md/persistent-dat/dm-space-map-common.c:sm_ll_mutate() sets SM_ALLOC. sm_disk_new_block() is indirectly calling sm_ll_mutate() sm_ll_mutate() will only return 0 if ll->save_ie() does, the ll_disk *ll should be ll_disk, and so disk_ll_save_ie()'s call to dm_btree_insert() returns 0 -- which simply means success. And on success sm_disk_new_block() assumes ev was set to SM_ALLOC (by sm_ll_mutate). sm_ll_mutate() decided to _not_ set SM_ALLOC because either: 1) ref_count wasn't set or 2) old was identified So all said: somehow a new data block was found to already be in use. _WHY_ that is the case isn't clear from this stack... But it does speak to the possibility of data block allocation racing with other operations to the same block. Which implies missing locking. But that's all I've got so far... I'll review past dm-thinp changes with all this in mind and see what turns up. But Joe Thornber (ejt) likely needs to have a look at this too. But could it be that bcache is the source of the data device race (same block used concurrently)? And DM thinp is acting as the canary in the coal mine? Thanks, Mike ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 with scsi_mod.use_blk_mq=y @ 2019-09-25 20:01 ` Mike Snitzer 0 siblings, 0 replies; 43+ messages in thread From: Mike Snitzer @ 2019-09-25 20:01 UTC (permalink / raw) To: Eric Wheeler, ejt; +Cc: linux-block, dm-devel, linux-bcache, lvm-devel On Wed, Sep 25 2019 at 2:40pm -0400, Eric Wheeler <dm-devel@lists.ewheeler.net> wrote: > Hello, > > We are using the 4.19.75 stable tree with dm-thin and multi-queue scsi. > We have been using the 4.19 branch for months without issue; we just > switched to MQ and we seem to have hit this BUG_ON. Whether or not MQ is > related to the issue, I don't know, maybe coincidence: > > static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > { > int r; > enum allocation_event ev; > struct sm_disk *smd = container_of(sm, struct sm_disk, sm); > > /* FIXME: we should loop round a couple of times */ > r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); > if (r) > return r; > > smd->begin = *b + 1; > r = sm_ll_inc(&smd->ll, *b, &ev); > if (!r) { > BUG_ON(ev != SM_ALLOC); <<<<<<<<<<<<<<<<<<<<<<<<<<<<< > smd->nr_allocated_this_transaction++; > } > > return r; > } > > This is a brand-new thin pool created about 12 hours ago: > > lvcreate -c 64k -L 12t --type thin-pool --thinpool data-pool --poolmetadatasize 16G data /dev/bcache0 > > We are using bcache, but I don't see any bcache code in the backtraces. > The metadata is also on the bcache volume. So bcache is be used for both data and metadata. > We were transferring data to the new thin volumes and it ran for about 12 > hours and then gave the trace below. So far it has only happened once > and I don't have a way to reproduce it. > > Any idea what this BUG_ON would indicate and how we might contrive a fix? > > [199391.677689] ------------[ cut here ]------------ > [199391.678437] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! > [199391.679183] invalid opcode: 0000 [#1] SMP NOPTI > [199391.679941] CPU: 4 PID: 31359 Comm: kworker/u16:4 Not tainted 4.19.75 #1 > [199391.680683] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > [199391.681446] Workqueue: dm-thin do_worker [dm_thin_pool] > [199391.682187] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199391.682929] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > [199391.684432] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > [199391.685186] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > [199391.685936] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > [199391.686659] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > [199391.687379] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > [199391.688120] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > [199391.688843] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > [199391.689571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [199391.690253] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > [199391.690984] Call Trace: > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > [199391.693852] ? sort+0x17b/0x270 > [199391.694527] ? u32_swap+0x10/0x10 > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > [199391.695890] process_one_work+0x171/0x370 > [199391.696640] worker_thread+0x49/0x3f0 > [199391.697332] kthread+0xf8/0x130 > [199391.697988] ? max_active_store+0x80/0x80 > [199391.698659] ? kthread_bind+0x10/0x10 > [199391.699281] ret_from_fork+0x1f/0x40 The stack shows the call to sm_disk_new_block() is due to dm_pool_alloc_data_block(). sm_disk_new_block()'s BUG_ON(ev != SM_ALLOC) indicates that somehow it is getting called without the passed 'ev' being set to SM_ALLOC. Only drivers/md/persistent-dat/dm-space-map-common.c:sm_ll_mutate() sets SM_ALLOC. sm_disk_new_block() is indirectly calling sm_ll_mutate() sm_ll_mutate() will only return 0 if ll->save_ie() does, the ll_disk *ll should be ll_disk, and so disk_ll_save_ie()'s call to dm_btree_insert() returns 0 -- which simply means success. And on success sm_disk_new_block() assumes ev was set to SM_ALLOC (by sm_ll_mutate). sm_ll_mutate() decided to _not_ set SM_ALLOC because either: 1) ref_count wasn't set or 2) old was identified So all said: somehow a new data block was found to already be in use. _WHY_ that is the case isn't clear from this stack... But it does speak to the possibility of data block allocation racing with other operations to the same block. Which implies missing locking. But that's all I've got so far... I'll review past dm-thinp changes with all this in mind and see what turns up. But Joe Thornber (ejt) likely needs to have a look at this too. But could it be that bcache is the source of the data device race (same block used concurrently)? And DM thinp is acting as the canary in the coal mine? Thanks, Mike ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 with scsi_mod.use_blk_mq=y 2019-09-25 20:01 ` Mike Snitzer (?) @ 2019-09-25 20:33 ` Eric Wheeler -1 siblings, 0 replies; 43+ messages in thread From: Eric Wheeler @ 2019-09-25 20:33 UTC (permalink / raw) To: Coly Li; +Cc: Mike Snitzer, ejt, dm-devel, linux-block, linux-bcache, lvm-devel On Wed, 25 Sep 2019, Mike Snitzer wrote: > On Wed, Sep 25 2019 at 2:40pm -0400, Eric Wheeler <dm-devel@lists.ewheeler.net> wrote: > > > Hello, > > > > We are using the 4.19.75 stable tree with dm-thin and multi-queue scsi. > > We have been using the 4.19 branch for months without issue; we just > > switched to MQ and we seem to have hit this BUG_ON. Whether or not MQ is > > related to the issue, I don't know, maybe coincidence: > > > > static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > > { > > int r; > > enum allocation_event ev; > > struct sm_disk *smd = container_of(sm, struct sm_disk, sm); > > > > /* FIXME: we should loop round a couple of times */ > > r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); > > if (r) > > return r; > > > > smd->begin = *b + 1; > > r = sm_ll_inc(&smd->ll, *b, &ev); > > if (!r) { > > BUG_ON(ev != SM_ALLOC); <<<<<<<<<<<<<<<<<<<<<<<<<<<<< > > smd->nr_allocated_this_transaction++; > > } > > > > return r; > > } > > > > This is a brand-new thin pool created about 12 hours ago: > > > > lvcreate -c 64k -L 12t --type thin-pool --thinpool data-pool --poolmetadatasize 16G data /dev/bcache0 > > > > We are using bcache, but I don't see any bcache code in the backtraces. > > The metadata is also on the bcache volume. > > So bcache is be used for both data and metadata. Correct. > > We were transferring data to the new thin volumes and it ran for about 12 > > hours and then gave the trace below. So far it has only happened once > > and I don't have a way to reproduce it. > > > > Any idea what this BUG_ON would indicate and how we might contrive a fix? > > > > [199391.677689] ------------[ cut here ]------------ > > [199391.678437] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! > > [199391.679183] invalid opcode: 0000 [#1] SMP NOPTI > > [199391.679941] CPU: 4 PID: 31359 Comm: kworker/u16:4 Not tainted 4.19.75 #1 > > [199391.680683] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > > [199391.681446] Workqueue: dm-thin do_worker [dm_thin_pool] > > [199391.682187] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199391.682929] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > [199391.684432] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > [199391.685186] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > [199391.685936] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > [199391.686659] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > [199391.687379] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > [199391.688120] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > [199391.688843] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > [199391.689571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [199391.690253] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > [199391.690984] Call Trace: > > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > > [199391.693852] ? sort+0x17b/0x270 > > [199391.694527] ? u32_swap+0x10/0x10 > > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > > [199391.695890] process_one_work+0x171/0x370 > > [199391.696640] worker_thread+0x49/0x3f0 > > [199391.697332] kthread+0xf8/0x130 > > [199391.697988] ? max_active_store+0x80/0x80 > > [199391.698659] ? kthread_bind+0x10/0x10 > > [199391.699281] ret_from_fork+0x1f/0x40 > > The stack shows the call to sm_disk_new_block() is due to > dm_pool_alloc_data_block(). > > sm_disk_new_block()'s BUG_ON(ev != SM_ALLOC) indicates that somehow it is > getting called without the passed 'ev' being set to SM_ALLOC. Only > drivers/md/persistent-dat/dm-space-map-common.c:sm_ll_mutate() sets > SM_ALLOC. sm_disk_new_block() is indirectly calling sm_ll_mutate() > > sm_ll_mutate() will only return 0 if ll->save_ie() does, the ll_disk *ll > should be ll_disk, and so disk_ll_save_ie()'s call to dm_btree_insert() > returns 0 -- which simply means success. And on success > sm_disk_new_block() assumes ev was set to SM_ALLOC (by sm_ll_mutate). > > sm_ll_mutate() decided to _not_ set SM_ALLOC because either: > 1) ref_count wasn't set > or > 2) old was identified > > So all said: somehow a new data block was found to already be in use. > _WHY_ that is the case isn't clear from this stack... > > But it does speak to the possibility of data block allocation racing > with other operations to the same block. Which implies missing locking. > > But that's all I've got so far... I'll review past dm-thinp changes with > all this in mind and see what turns up. But Joe Thornber (ejt) likely > needs to have a look at this too. > > But could it be that bcache is the source of the data device race (same > block used concurrently)? And DM thinp is acting as the canary in the > coal mine? Hi Mike, thanks for the detail. Coly, any idea on the possible bcache interaction here? -- Eric Wheeler ^ permalink raw reply [flat|nested] 43+ messages in thread
* kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 with scsi_mod.use_blk_mq=y @ 2019-09-25 20:33 ` Eric Wheeler 0 siblings, 0 replies; 43+ messages in thread From: Eric Wheeler @ 2019-09-25 20:33 UTC (permalink / raw) To: lvm-devel On Wed, 25 Sep 2019, Mike Snitzer wrote: > On Wed, Sep 25 2019 at 2:40pm -0400, Eric Wheeler <dm-devel@lists.ewheeler.net> wrote: > > > Hello, > > > > We are using the 4.19.75 stable tree with dm-thin and multi-queue scsi. > > We have been using the 4.19 branch for months without issue; we just > > switched to MQ and we seem to have hit this BUG_ON. Whether or not MQ is > > related to the issue, I don't know, maybe coincidence: > > > > static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > > { > > int r; > > enum allocation_event ev; > > struct sm_disk *smd = container_of(sm, struct sm_disk, sm); > > > > /* FIXME: we should loop round a couple of times */ > > r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); > > if (r) > > return r; > > > > smd->begin = *b + 1; > > r = sm_ll_inc(&smd->ll, *b, &ev); > > if (!r) { > > BUG_ON(ev != SM_ALLOC); <<<<<<<<<<<<<<<<<<<<<<<<<<<<< > > smd->nr_allocated_this_transaction++; > > } > > > > return r; > > } > > > > This is a brand-new thin pool created about 12 hours ago: > > > > lvcreate -c 64k -L 12t --type thin-pool --thinpool data-pool --poolmetadatasize 16G data /dev/bcache0 > > > > We are using bcache, but I don't see any bcache code in the backtraces. > > The metadata is also on the bcache volume. > > So bcache is be used for both data and metadata. Correct. > > We were transferring data to the new thin volumes and it ran for about 12 > > hours and then gave the trace below. So far it has only happened once > > and I don't have a way to reproduce it. > > > > Any idea what this BUG_ON would indicate and how we might contrive a fix? > > > > [199391.677689] ------------[ cut here ]------------ > > [199391.678437] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! > > [199391.679183] invalid opcode: 0000 [#1] SMP NOPTI > > [199391.679941] CPU: 4 PID: 31359 Comm: kworker/u16:4 Not tainted 4.19.75 #1 > > [199391.680683] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > > [199391.681446] Workqueue: dm-thin do_worker [dm_thin_pool] > > [199391.682187] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199391.682929] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > [199391.684432] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > [199391.685186] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > [199391.685936] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > [199391.686659] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > [199391.687379] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > [199391.688120] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > [199391.688843] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > [199391.689571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [199391.690253] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > [199391.690984] Call Trace: > > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > > [199391.693852] ? sort+0x17b/0x270 > > [199391.694527] ? u32_swap+0x10/0x10 > > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > > [199391.695890] process_one_work+0x171/0x370 > > [199391.696640] worker_thread+0x49/0x3f0 > > [199391.697332] kthread+0xf8/0x130 > > [199391.697988] ? max_active_store+0x80/0x80 > > [199391.698659] ? kthread_bind+0x10/0x10 > > [199391.699281] ret_from_fork+0x1f/0x40 > > The stack shows the call to sm_disk_new_block() is due to > dm_pool_alloc_data_block(). > > sm_disk_new_block()'s BUG_ON(ev != SM_ALLOC) indicates that somehow it is > getting called without the passed 'ev' being set to SM_ALLOC. Only > drivers/md/persistent-dat/dm-space-map-common.c:sm_ll_mutate() sets > SM_ALLOC. sm_disk_new_block() is indirectly calling sm_ll_mutate() > > sm_ll_mutate() will only return 0 if ll->save_ie() does, the ll_disk *ll > should be ll_disk, and so disk_ll_save_ie()'s call to dm_btree_insert() > returns 0 -- which simply means success. And on success > sm_disk_new_block() assumes ev was set to SM_ALLOC (by sm_ll_mutate). > > sm_ll_mutate() decided to _not_ set SM_ALLOC because either: > 1) ref_count wasn't set > or > 2) old was identified > > So all said: somehow a new data block was found to already be in use. > _WHY_ that is the case isn't clear from this stack... > > But it does speak to the possibility of data block allocation racing > with other operations to the same block. Which implies missing locking. > > But that's all I've got so far... I'll review past dm-thinp changes with > all this in mind and see what turns up. But Joe Thornber (ejt) likely > needs to have a look at this too. > > But could it be that bcache is the source of the data device race (same > block used concurrently)? And DM thinp is acting as the canary in the > coal mine? Hi Mike, thanks for the detail. Coly, any idea on the possible bcache interaction here? -- Eric Wheeler ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 with scsi_mod.use_blk_mq=y @ 2019-09-25 20:33 ` Eric Wheeler 0 siblings, 0 replies; 43+ messages in thread From: Eric Wheeler @ 2019-09-25 20:33 UTC (permalink / raw) To: Coly Li; +Cc: Mike Snitzer, dm-devel, linux-bcache, lvm-devel, linux-block, ejt On Wed, 25 Sep 2019, Mike Snitzer wrote: > On Wed, Sep 25 2019 at 2:40pm -0400, Eric Wheeler <dm-devel@lists.ewheeler.net> wrote: > > > Hello, > > > > We are using the 4.19.75 stable tree with dm-thin and multi-queue scsi. > > We have been using the 4.19 branch for months without issue; we just > > switched to MQ and we seem to have hit this BUG_ON. Whether or not MQ is > > related to the issue, I don't know, maybe coincidence: > > > > static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > > { > > int r; > > enum allocation_event ev; > > struct sm_disk *smd = container_of(sm, struct sm_disk, sm); > > > > /* FIXME: we should loop round a couple of times */ > > r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); > > if (r) > > return r; > > > > smd->begin = *b + 1; > > r = sm_ll_inc(&smd->ll, *b, &ev); > > if (!r) { > > BUG_ON(ev != SM_ALLOC); <<<<<<<<<<<<<<<<<<<<<<<<<<<<< > > smd->nr_allocated_this_transaction++; > > } > > > > return r; > > } > > > > This is a brand-new thin pool created about 12 hours ago: > > > > lvcreate -c 64k -L 12t --type thin-pool --thinpool data-pool --poolmetadatasize 16G data /dev/bcache0 > > > > We are using bcache, but I don't see any bcache code in the backtraces. > > The metadata is also on the bcache volume. > > So bcache is be used for both data and metadata. Correct. > > We were transferring data to the new thin volumes and it ran for about 12 > > hours and then gave the trace below. So far it has only happened once > > and I don't have a way to reproduce it. > > > > Any idea what this BUG_ON would indicate and how we might contrive a fix? > > > > [199391.677689] ------------[ cut here ]------------ > > [199391.678437] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! > > [199391.679183] invalid opcode: 0000 [#1] SMP NOPTI > > [199391.679941] CPU: 4 PID: 31359 Comm: kworker/u16:4 Not tainted 4.19.75 #1 > > [199391.680683] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > > [199391.681446] Workqueue: dm-thin do_worker [dm_thin_pool] > > [199391.682187] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199391.682929] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > [199391.684432] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > [199391.685186] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > [199391.685936] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > [199391.686659] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > [199391.687379] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > [199391.688120] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > [199391.688843] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > [199391.689571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [199391.690253] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > [199391.690984] Call Trace: > > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > > [199391.693852] ? sort+0x17b/0x270 > > [199391.694527] ? u32_swap+0x10/0x10 > > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > > [199391.695890] process_one_work+0x171/0x370 > > [199391.696640] worker_thread+0x49/0x3f0 > > [199391.697332] kthread+0xf8/0x130 > > [199391.697988] ? max_active_store+0x80/0x80 > > [199391.698659] ? kthread_bind+0x10/0x10 > > [199391.699281] ret_from_fork+0x1f/0x40 > > The stack shows the call to sm_disk_new_block() is due to > dm_pool_alloc_data_block(). > > sm_disk_new_block()'s BUG_ON(ev != SM_ALLOC) indicates that somehow it is > getting called without the passed 'ev' being set to SM_ALLOC. Only > drivers/md/persistent-dat/dm-space-map-common.c:sm_ll_mutate() sets > SM_ALLOC. sm_disk_new_block() is indirectly calling sm_ll_mutate() > > sm_ll_mutate() will only return 0 if ll->save_ie() does, the ll_disk *ll > should be ll_disk, and so disk_ll_save_ie()'s call to dm_btree_insert() > returns 0 -- which simply means success. And on success > sm_disk_new_block() assumes ev was set to SM_ALLOC (by sm_ll_mutate). > > sm_ll_mutate() decided to _not_ set SM_ALLOC because either: > 1) ref_count wasn't set > or > 2) old was identified > > So all said: somehow a new data block was found to already be in use. > _WHY_ that is the case isn't clear from this stack... > > But it does speak to the possibility of data block allocation racing > with other operations to the same block. Which implies missing locking. > > But that's all I've got so far... I'll review past dm-thinp changes with > all this in mind and see what turns up. But Joe Thornber (ejt) likely > needs to have a look at this too. > > But could it be that bcache is the source of the data device race (same > block used concurrently)? And DM thinp is acting as the canary in the > coal mine? Hi Mike, thanks for the detail. Coly, any idea on the possible bcache interaction here? -- Eric Wheeler ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 with scsi_mod.use_blk_mq=y 2019-09-25 20:01 ` Mike Snitzer (?) @ 2019-09-26 18:27 ` Eric Wheeler -1 siblings, 0 replies; 43+ messages in thread From: Eric Wheeler @ 2019-09-26 18:27 UTC (permalink / raw) To: Mike Snitzer; +Cc: ejt, Coly Li, dm-devel, linux-block, linux-bcache, lvm-devel On Wed, 25 Sep 2019, Mike Snitzer wrote: > On Wed, Sep 25 2019 at 2:40pm -0400, > Eric Wheeler <dm-devel@lists.ewheeler.net> wrote: > > > Hello, > > > > We are using the 4.19.75 stable tree with dm-thin and multi-queue scsi. > > We have been using the 4.19 branch for months without issue; we just > > switched to MQ and we seem to have hit this BUG_ON. Whether or not MQ is > > related to the issue, I don't know, maybe coincidence: > > > > static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > > { > > int r; > > enum allocation_event ev; > > struct sm_disk *smd = container_of(sm, struct sm_disk, sm); > > > > /* FIXME: we should loop round a couple of times */ > > r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); > > if (r) > > return r; > > > > smd->begin = *b + 1; > > r = sm_ll_inc(&smd->ll, *b, &ev); > > if (!r) { > > BUG_ON(ev != SM_ALLOC); <<<<<<<<<<<<<<<<<<<<<<<<<<<<< > > smd->nr_allocated_this_transaction++; > > } > > > > return r; > > } > > > > This is a brand-new thin pool created about 12 hours ago: > > > > lvcreate -c 64k -L 12t --type thin-pool --thinpool data-pool --poolmetadatasize 16G data /dev/bcache0 > > > > We are using bcache, but I don't see any bcache code in the backtraces. > > The metadata is also on the bcache volume. > > So bcache is be used for both data and metadata. Hi Mike, I pvmoved the tmeta to an SSD logical volume (dm-linear) on a non-bcache volume and we got the same trace this morning, so while the tdata still passes through bcache, all meta operations are direct to an SSD. This is still using multi-queue scsi, but dm_mod.use_blk_mq=N. Since bcache is no longer involved with metadata operations, and since this appears to be a metadata issue, are there any other reasons to suspect bcache? Since we seem to hit this every night, I can try any patches that you would like for testing. I appreciate your help, hopefully we can solve this quickly. -Eric > > We were transferring data to the new thin volumes and it ran for about 12 > > hours and then gave the trace below. So far it has only happened once > > and I don't have a way to reproduce it. > > > > Any idea what this BUG_ON would indicate and how we might contrive a fix? > > > > [199391.677689] ------------[ cut here ]------------ > > [199391.678437] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! > > [199391.679183] invalid opcode: 0000 [#1] SMP NOPTI > > [199391.679941] CPU: 4 PID: 31359 Comm: kworker/u16:4 Not tainted 4.19.75 #1 > > [199391.680683] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > > [199391.681446] Workqueue: dm-thin do_worker [dm_thin_pool] > > [199391.682187] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199391.682929] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > [199391.684432] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > [199391.685186] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > [199391.685936] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > [199391.686659] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > [199391.687379] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > [199391.688120] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > [199391.688843] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > [199391.689571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [199391.690253] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > [199391.690984] Call Trace: > > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > > [199391.693852] ? sort+0x17b/0x270 > > [199391.694527] ? u32_swap+0x10/0x10 > > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > > [199391.695890] process_one_work+0x171/0x370 > > [199391.696640] worker_thread+0x49/0x3f0 > > [199391.697332] kthread+0xf8/0x130 > > [199391.697988] ? max_active_store+0x80/0x80 > > [199391.698659] ? kthread_bind+0x10/0x10 > > [199391.699281] ret_from_fork+0x1f/0x40 > > The stack shows the call to sm_disk_new_block() is due to > dm_pool_alloc_data_block(). > > sm_disk_new_block()'s BUG_ON(ev != SM_ALLOC) indicates that somehow it is > getting called without the passed 'ev' being set to SM_ALLOC. Only > drivers/md/persistent-dat/dm-space-map-common.c:sm_ll_mutate() sets > SM_ALLOC. sm_disk_new_block() is indirectly calling sm_ll_mutate() > > sm_ll_mutate() will only return 0 if ll->save_ie() does, the ll_disk *ll > should be ll_disk, and so disk_ll_save_ie()'s call to dm_btree_insert() > returns 0 -- which simply means success. And on success > sm_disk_new_block() assumes ev was set to SM_ALLOC (by sm_ll_mutate). > > sm_ll_mutate() decided to _not_ set SM_ALLOC because either: > 1) ref_count wasn't set > or > 2) old was identified > > So all said: somehow a new data block was found to already be in use. > _WHY_ that is the case isn't clear from this stack... > > But it does speak to the possibility of data block allocation racing > with other operations to the same block. Which implies missing locking. > > But that's all I've got so far... I'll review past dm-thinp changes with > all this in mind and see what turns up. But Joe Thornber (ejt) likely > needs to have a look at this too. > > But could it be that bcache is the source of the data device race (same > block used concurrently)? And DM thinp is acting as the canary in the > coal mine? > > Thanks, > Mike > ^ permalink raw reply [flat|nested] 43+ messages in thread
* kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 with scsi_mod.use_blk_mq=y @ 2019-09-26 18:27 ` Eric Wheeler 0 siblings, 0 replies; 43+ messages in thread From: Eric Wheeler @ 2019-09-26 18:27 UTC (permalink / raw) To: lvm-devel On Wed, 25 Sep 2019, Mike Snitzer wrote: > On Wed, Sep 25 2019 at 2:40pm -0400, > Eric Wheeler <dm-devel@lists.ewheeler.net> wrote: > > > Hello, > > > > We are using the 4.19.75 stable tree with dm-thin and multi-queue scsi. > > We have been using the 4.19 branch for months without issue; we just > > switched to MQ and we seem to have hit this BUG_ON. Whether or not MQ is > > related to the issue, I don't know, maybe coincidence: > > > > static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > > { > > int r; > > enum allocation_event ev; > > struct sm_disk *smd = container_of(sm, struct sm_disk, sm); > > > > /* FIXME: we should loop round a couple of times */ > > r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); > > if (r) > > return r; > > > > smd->begin = *b + 1; > > r = sm_ll_inc(&smd->ll, *b, &ev); > > if (!r) { > > BUG_ON(ev != SM_ALLOC); <<<<<<<<<<<<<<<<<<<<<<<<<<<<< > > smd->nr_allocated_this_transaction++; > > } > > > > return r; > > } > > > > This is a brand-new thin pool created about 12 hours ago: > > > > lvcreate -c 64k -L 12t --type thin-pool --thinpool data-pool --poolmetadatasize 16G data /dev/bcache0 > > > > We are using bcache, but I don't see any bcache code in the backtraces. > > The metadata is also on the bcache volume. > > So bcache is be used for both data and metadata. Hi Mike, I pvmoved the tmeta to an SSD logical volume (dm-linear) on a non-bcache volume and we got the same trace this morning, so while the tdata still passes through bcache, all meta operations are direct to an SSD. This is still using multi-queue scsi, but dm_mod.use_blk_mq=N. Since bcache is no longer involved with metadata operations, and since this appears to be a metadata issue, are there any other reasons to suspect bcache? Since we seem to hit this every night, I can try any patches that you would like for testing. I appreciate your help, hopefully we can solve this quickly. -Eric > > We were transferring data to the new thin volumes and it ran for about 12 > > hours and then gave the trace below. So far it has only happened once > > and I don't have a way to reproduce it. > > > > Any idea what this BUG_ON would indicate and how we might contrive a fix? > > > > [199391.677689] ------------[ cut here ]------------ > > [199391.678437] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! > > [199391.679183] invalid opcode: 0000 [#1] SMP NOPTI > > [199391.679941] CPU: 4 PID: 31359 Comm: kworker/u16:4 Not tainted 4.19.75 #1 > > [199391.680683] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > > [199391.681446] Workqueue: dm-thin do_worker [dm_thin_pool] > > [199391.682187] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199391.682929] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > [199391.684432] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > [199391.685186] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > [199391.685936] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > [199391.686659] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > [199391.687379] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > [199391.688120] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > [199391.688843] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > [199391.689571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [199391.690253] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > [199391.690984] Call Trace: > > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > > [199391.693852] ? sort+0x17b/0x270 > > [199391.694527] ? u32_swap+0x10/0x10 > > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > > [199391.695890] process_one_work+0x171/0x370 > > [199391.696640] worker_thread+0x49/0x3f0 > > [199391.697332] kthread+0xf8/0x130 > > [199391.697988] ? max_active_store+0x80/0x80 > > [199391.698659] ? kthread_bind+0x10/0x10 > > [199391.699281] ret_from_fork+0x1f/0x40 > > The stack shows the call to sm_disk_new_block() is due to > dm_pool_alloc_data_block(). > > sm_disk_new_block()'s BUG_ON(ev != SM_ALLOC) indicates that somehow it is > getting called without the passed 'ev' being set to SM_ALLOC. Only > drivers/md/persistent-dat/dm-space-map-common.c:sm_ll_mutate() sets > SM_ALLOC. sm_disk_new_block() is indirectly calling sm_ll_mutate() > > sm_ll_mutate() will only return 0 if ll->save_ie() does, the ll_disk *ll > should be ll_disk, and so disk_ll_save_ie()'s call to dm_btree_insert() > returns 0 -- which simply means success. And on success > sm_disk_new_block() assumes ev was set to SM_ALLOC (by sm_ll_mutate). > > sm_ll_mutate() decided to _not_ set SM_ALLOC because either: > 1) ref_count wasn't set > or > 2) old was identified > > So all said: somehow a new data block was found to already be in use. > _WHY_ that is the case isn't clear from this stack... > > But it does speak to the possibility of data block allocation racing > with other operations to the same block. Which implies missing locking. > > But that's all I've got so far... I'll review past dm-thinp changes with > all this in mind and see what turns up. But Joe Thornber (ejt) likely > needs to have a look at this too. > > But could it be that bcache is the source of the data device race (same > block used concurrently)? And DM thinp is acting as the canary in the > coal mine? > > Thanks, > Mike > ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 with scsi_mod.use_blk_mq=y @ 2019-09-26 18:27 ` Eric Wheeler 0 siblings, 0 replies; 43+ messages in thread From: Eric Wheeler @ 2019-09-26 18:27 UTC (permalink / raw) To: Mike Snitzer; +Cc: dm-devel, linux-bcache, lvm-devel, linux-block, ejt, Coly Li On Wed, 25 Sep 2019, Mike Snitzer wrote: > On Wed, Sep 25 2019 at 2:40pm -0400, > Eric Wheeler <dm-devel@lists.ewheeler.net> wrote: > > > Hello, > > > > We are using the 4.19.75 stable tree with dm-thin and multi-queue scsi. > > We have been using the 4.19 branch for months without issue; we just > > switched to MQ and we seem to have hit this BUG_ON. Whether or not MQ is > > related to the issue, I don't know, maybe coincidence: > > > > static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > > { > > int r; > > enum allocation_event ev; > > struct sm_disk *smd = container_of(sm, struct sm_disk, sm); > > > > /* FIXME: we should loop round a couple of times */ > > r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); > > if (r) > > return r; > > > > smd->begin = *b + 1; > > r = sm_ll_inc(&smd->ll, *b, &ev); > > if (!r) { > > BUG_ON(ev != SM_ALLOC); <<<<<<<<<<<<<<<<<<<<<<<<<<<<< > > smd->nr_allocated_this_transaction++; > > } > > > > return r; > > } > > > > This is a brand-new thin pool created about 12 hours ago: > > > > lvcreate -c 64k -L 12t --type thin-pool --thinpool data-pool --poolmetadatasize 16G data /dev/bcache0 > > > > We are using bcache, but I don't see any bcache code in the backtraces. > > The metadata is also on the bcache volume. > > So bcache is be used for both data and metadata. Hi Mike, I pvmoved the tmeta to an SSD logical volume (dm-linear) on a non-bcache volume and we got the same trace this morning, so while the tdata still passes through bcache, all meta operations are direct to an SSD. This is still using multi-queue scsi, but dm_mod.use_blk_mq=N. Since bcache is no longer involved with metadata operations, and since this appears to be a metadata issue, are there any other reasons to suspect bcache? Since we seem to hit this every night, I can try any patches that you would like for testing. I appreciate your help, hopefully we can solve this quickly. -Eric > > We were transferring data to the new thin volumes and it ran for about 12 > > hours and then gave the trace below. So far it has only happened once > > and I don't have a way to reproduce it. > > > > Any idea what this BUG_ON would indicate and how we might contrive a fix? > > > > [199391.677689] ------------[ cut here ]------------ > > [199391.678437] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! > > [199391.679183] invalid opcode: 0000 [#1] SMP NOPTI > > [199391.679941] CPU: 4 PID: 31359 Comm: kworker/u16:4 Not tainted 4.19.75 #1 > > [199391.680683] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > > [199391.681446] Workqueue: dm-thin do_worker [dm_thin_pool] > > [199391.682187] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199391.682929] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > [199391.684432] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > [199391.685186] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > [199391.685936] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > [199391.686659] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > [199391.687379] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > [199391.688120] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > [199391.688843] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > [199391.689571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [199391.690253] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > [199391.690984] Call Trace: > > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > > [199391.693852] ? sort+0x17b/0x270 > > [199391.694527] ? u32_swap+0x10/0x10 > > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > > [199391.695890] process_one_work+0x171/0x370 > > [199391.696640] worker_thread+0x49/0x3f0 > > [199391.697332] kthread+0xf8/0x130 > > [199391.697988] ? max_active_store+0x80/0x80 > > [199391.698659] ? kthread_bind+0x10/0x10 > > [199391.699281] ret_from_fork+0x1f/0x40 > > The stack shows the call to sm_disk_new_block() is due to > dm_pool_alloc_data_block(). > > sm_disk_new_block()'s BUG_ON(ev != SM_ALLOC) indicates that somehow it is > getting called without the passed 'ev' being set to SM_ALLOC. Only > drivers/md/persistent-dat/dm-space-map-common.c:sm_ll_mutate() sets > SM_ALLOC. sm_disk_new_block() is indirectly calling sm_ll_mutate() > > sm_ll_mutate() will only return 0 if ll->save_ie() does, the ll_disk *ll > should be ll_disk, and so disk_ll_save_ie()'s call to dm_btree_insert() > returns 0 -- which simply means success. And on success > sm_disk_new_block() assumes ev was set to SM_ALLOC (by sm_ll_mutate). > > sm_ll_mutate() decided to _not_ set SM_ALLOC because either: > 1) ref_count wasn't set > or > 2) old was identified > > So all said: somehow a new data block was found to already be in use. > _WHY_ that is the case isn't clear from this stack... > > But it does speak to the possibility of data block allocation racing > with other operations to the same block. Which implies missing locking. > > But that's all I've got so far... I'll review past dm-thinp changes with > all this in mind and see what turns up. But Joe Thornber (ejt) likely > needs to have a look at this too. > > But could it be that bcache is the source of the data device race (same > block used concurrently)? And DM thinp is acting as the canary in the > coal mine? > > Thanks, > Mike > ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 with scsi_mod.use_blk_mq=y 2019-09-26 18:27 ` Eric Wheeler (?) @ 2019-09-27 8:32 ` Joe Thornber -1 siblings, 0 replies; 43+ messages in thread From: Joe Thornber @ 2019-09-27 8:32 UTC (permalink / raw) To: Eric Wheeler Cc: Mike Snitzer, ejt, Coly Li, dm-devel, linux-block, linux-bcache, lvm-devel, joe.thornber Hi Eric, On Thu, Sep 26, 2019 at 06:27:09PM +0000, Eric Wheeler wrote: > I pvmoved the tmeta to an SSD logical volume (dm-linear) on a non-bcache > volume and we got the same trace this morning, so while the tdata still > passes through bcache, all meta operations are direct to an SSD. This is > still using multi-queue scsi, but dm_mod.use_blk_mq=N. > > Since bcache is no longer involved with metadata operations, and since > this appears to be a metadata issue, are there any other reasons to > suspect bcache? Did you recreate the pool, or are you just using the existing pool but with a different IO path? If it's the latter then there could still be something wrong with the metadata, introduced while bcache was in the stack. Would it be possible to send me a copy of the metadata device please so I can double check the space maps (I presume you've run thin_check on it)? [Assuming you're using the existing pool] Another useful experiment would be to thump_dump and then thin_restore the metadata, which will create totally fresh metadata and see if you can still reproduce the issue. Thanks, - Joe ^ permalink raw reply [flat|nested] 43+ messages in thread
* kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 with scsi_mod.use_blk_mq=y @ 2019-09-27 8:32 ` Joe Thornber 0 siblings, 0 replies; 43+ messages in thread From: Joe Thornber @ 2019-09-27 8:32 UTC (permalink / raw) To: lvm-devel Hi Eric, On Thu, Sep 26, 2019 at 06:27:09PM +0000, Eric Wheeler wrote: > I pvmoved the tmeta to an SSD logical volume (dm-linear) on a non-bcache > volume and we got the same trace this morning, so while the tdata still > passes through bcache, all meta operations are direct to an SSD. This is > still using multi-queue scsi, but dm_mod.use_blk_mq=N. > > Since bcache is no longer involved with metadata operations, and since > this appears to be a metadata issue, are there any other reasons to > suspect bcache? Did you recreate the pool, or are you just using the existing pool but with a different IO path? If it's the latter then there could still be something wrong with the metadata, introduced while bcache was in the stack. Would it be possible to send me a copy of the metadata device please so I can double check the space maps (I presume you've run thin_check on it)? [Assuming you're using the existing pool] Another useful experiment would be to thump_dump and then thin_restore the metadata, which will create totally fresh metadata and see if you can still reproduce the issue. Thanks, - Joe ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 with scsi_mod.use_blk_mq=y @ 2019-09-27 8:32 ` Joe Thornber 0 siblings, 0 replies; 43+ messages in thread From: Joe Thornber @ 2019-09-27 8:32 UTC (permalink / raw) To: Eric Wheeler Cc: joe.thornber, Mike Snitzer, dm-devel, linux-bcache, Coly Li, linux-block, ejt, lvm-devel Hi Eric, On Thu, Sep 26, 2019 at 06:27:09PM +0000, Eric Wheeler wrote: > I pvmoved the tmeta to an SSD logical volume (dm-linear) on a non-bcache > volume and we got the same trace this morning, so while the tdata still > passes through bcache, all meta operations are direct to an SSD. This is > still using multi-queue scsi, but dm_mod.use_blk_mq=N. > > Since bcache is no longer involved with metadata operations, and since > this appears to be a metadata issue, are there any other reasons to > suspect bcache? Did you recreate the pool, or are you just using the existing pool but with a different IO path? If it's the latter then there could still be something wrong with the metadata, introduced while bcache was in the stack. Would it be possible to send me a copy of the metadata device please so I can double check the space maps (I presume you've run thin_check on it)? [Assuming you're using the existing pool] Another useful experiment would be to thump_dump and then thin_restore the metadata, which will create totally fresh metadata and see if you can still reproduce the issue. Thanks, - Joe ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 with scsi_mod.use_blk_mq=y 2019-09-27 8:32 ` Joe Thornber (?) @ 2019-09-27 18:45 ` Eric Wheeler -1 siblings, 0 replies; 43+ messages in thread From: Eric Wheeler @ 2019-09-27 18:45 UTC (permalink / raw) To: Joe Thornber Cc: Mike Snitzer, ejt, Coly Li, dm-devel, linux-block, linux-bcache, lvm-devel, joe.thornber On Fri, 27 Sep 2019, Joe Thornber wrote: > Hi Eric, > > On Thu, Sep 26, 2019 at 06:27:09PM +0000, Eric Wheeler wrote: > > I pvmoved the tmeta to an SSD logical volume (dm-linear) on a non-bcache > > volume and we got the same trace this morning, so while the tdata still > > passes through bcache, all meta operations are direct to an SSD. This is > > still using multi-queue scsi, but dm_mod.use_blk_mq=N. > > > > Since bcache is no longer involved with metadata operations, and since > > this appears to be a metadata issue, are there any other reasons to > > suspect bcache? > > Did you recreate the pool, or are you just using the existing pool but with > a different IO path? If it's the latter then there could still be something > wrong with the metadata, introduced while bcache was in the stack. We did not create the pool after the initial problem, though the pool was new just before the problem occurred. > Would it be possible to send me a copy of the metadata device please so > I can double check the space maps (I presume you've run thin_check on it)? ~]# /usr/local/bin/thin_check /dev/mapper/data-data--pool_tmeta examining superblock TRANSACTION_ID=2347 METADATA_FREE_BLOCKS=4145151 examining devices tree examining mapping tree checking space map counts ~]# echo $? 0 ~]# /usr/local/bin/thin_check -V 0.8.5 > [Assuming you're using the existing pool] Another useful experiment would be to > thump_dump and then thin_restore the metadata, which will create totally fresh > metadata and see if you can still reproduce the issue. It didn't lockup last night, but I'll keep working to reproduce the problem and let you know what we find. Mike said it could be a race: > The stack shows the call to sm_disk_new_block() is due to > dm_pool_alloc_data_block(). > > sm_disk_new_block()'s BUG_ON(ev != SM_ALLOC) indicates that somehow it is > getting called without the passed 'ev' being set to SM_ALLOC. Only > drivers/md/persistent-dat/dm-space-map-common.c:sm_ll_mutate() sets > SM_ALLOC. sm_disk_new_block() is indirectly calling sm_ll_mutate() > > sm_ll_mutate() will only return 0 if ll->save_ie() does, the ll_disk *ll > should be ll_disk, and so disk_ll_save_ie()'s call to dm_btree_insert() > returns 0 -- which simply means success. And on success > sm_disk_new_block() assumes ev was set to SM_ALLOC (by sm_ll_mutate). > > sm_ll_mutate() decided to _not_ set SM_ALLOC because either: > 1) ref_count wasn't set > or > 2) old was identified > > So all said: somehow a new data block was found to already be in use. > _WHY_ that is the case isn't clear from this stack... > > But it does speak to the possibility of data block allocation racing > with other operations to the same block. Which implies missing locking. Would a spinlock on the block solve the issue? Where might such a spinlock be added? -- Eric Wheeler > > Thanks, > > - Joe > ^ permalink raw reply [flat|nested] 43+ messages in thread
* kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 with scsi_mod.use_blk_mq=y @ 2019-09-27 18:45 ` Eric Wheeler 0 siblings, 0 replies; 43+ messages in thread From: Eric Wheeler @ 2019-09-27 18:45 UTC (permalink / raw) To: lvm-devel On Fri, 27 Sep 2019, Joe Thornber wrote: > Hi Eric, > > On Thu, Sep 26, 2019 at 06:27:09PM +0000, Eric Wheeler wrote: > > I pvmoved the tmeta to an SSD logical volume (dm-linear) on a non-bcache > > volume and we got the same trace this morning, so while the tdata still > > passes through bcache, all meta operations are direct to an SSD. This is > > still using multi-queue scsi, but dm_mod.use_blk_mq=N. > > > > Since bcache is no longer involved with metadata operations, and since > > this appears to be a metadata issue, are there any other reasons to > > suspect bcache? > > Did you recreate the pool, or are you just using the existing pool but with > a different IO path? If it's the latter then there could still be something > wrong with the metadata, introduced while bcache was in the stack. We did not create the pool after the initial problem, though the pool was new just before the problem occurred. > Would it be possible to send me a copy of the metadata device please so > I can double check the space maps (I presume you've run thin_check on it)? ~]# /usr/local/bin/thin_check /dev/mapper/data-data--pool_tmeta examining superblock TRANSACTION_ID=2347 METADATA_FREE_BLOCKS=4145151 examining devices tree examining mapping tree checking space map counts ~]# echo $? 0 ~]# /usr/local/bin/thin_check -V 0.8.5 > [Assuming you're using the existing pool] Another useful experiment would be to > thump_dump and then thin_restore the metadata, which will create totally fresh > metadata and see if you can still reproduce the issue. It didn't lockup last night, but I'll keep working to reproduce the problem and let you know what we find. Mike said it could be a race: > The stack shows the call to sm_disk_new_block() is due to > dm_pool_alloc_data_block(). > > sm_disk_new_block()'s BUG_ON(ev != SM_ALLOC) indicates that somehow it is > getting called without the passed 'ev' being set to SM_ALLOC. Only > drivers/md/persistent-dat/dm-space-map-common.c:sm_ll_mutate() sets > SM_ALLOC. sm_disk_new_block() is indirectly calling sm_ll_mutate() > > sm_ll_mutate() will only return 0 if ll->save_ie() does, the ll_disk *ll > should be ll_disk, and so disk_ll_save_ie()'s call to dm_btree_insert() > returns 0 -- which simply means success. And on success > sm_disk_new_block() assumes ev was set to SM_ALLOC (by sm_ll_mutate). > > sm_ll_mutate() decided to _not_ set SM_ALLOC because either: > 1) ref_count wasn't set > or > 2) old was identified > > So all said: somehow a new data block was found to already be in use. > _WHY_ that is the case isn't clear from this stack... > > But it does speak to the possibility of data block allocation racing > with other operations to the same block. Which implies missing locking. Would a spinlock on the block solve the issue? Where might such a spinlock be added? -- Eric Wheeler > > Thanks, > > - Joe > ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 with scsi_mod.use_blk_mq=y @ 2019-09-27 18:45 ` Eric Wheeler 0 siblings, 0 replies; 43+ messages in thread From: Eric Wheeler @ 2019-09-27 18:45 UTC (permalink / raw) To: Joe Thornber Cc: joe.thornber, Mike Snitzer, dm-devel, linux-bcache, Coly Li, linux-block, ejt, lvm-devel On Fri, 27 Sep 2019, Joe Thornber wrote: > Hi Eric, > > On Thu, Sep 26, 2019 at 06:27:09PM +0000, Eric Wheeler wrote: > > I pvmoved the tmeta to an SSD logical volume (dm-linear) on a non-bcache > > volume and we got the same trace this morning, so while the tdata still > > passes through bcache, all meta operations are direct to an SSD. This is > > still using multi-queue scsi, but dm_mod.use_blk_mq=N. > > > > Since bcache is no longer involved with metadata operations, and since > > this appears to be a metadata issue, are there any other reasons to > > suspect bcache? > > Did you recreate the pool, or are you just using the existing pool but with > a different IO path? If it's the latter then there could still be something > wrong with the metadata, introduced while bcache was in the stack. We did not create the pool after the initial problem, though the pool was new just before the problem occurred. > Would it be possible to send me a copy of the metadata device please so > I can double check the space maps (I presume you've run thin_check on it)? ~]# /usr/local/bin/thin_check /dev/mapper/data-data--pool_tmeta examining superblock TRANSACTION_ID=2347 METADATA_FREE_BLOCKS=4145151 examining devices tree examining mapping tree checking space map counts ~]# echo $? 0 ~]# /usr/local/bin/thin_check -V 0.8.5 > [Assuming you're using the existing pool] Another useful experiment would be to > thump_dump and then thin_restore the metadata, which will create totally fresh > metadata and see if you can still reproduce the issue. It didn't lockup last night, but I'll keep working to reproduce the problem and let you know what we find. Mike said it could be a race: > The stack shows the call to sm_disk_new_block() is due to > dm_pool_alloc_data_block(). > > sm_disk_new_block()'s BUG_ON(ev != SM_ALLOC) indicates that somehow it is > getting called without the passed 'ev' being set to SM_ALLOC. Only > drivers/md/persistent-dat/dm-space-map-common.c:sm_ll_mutate() sets > SM_ALLOC. sm_disk_new_block() is indirectly calling sm_ll_mutate() > > sm_ll_mutate() will only return 0 if ll->save_ie() does, the ll_disk *ll > should be ll_disk, and so disk_ll_save_ie()'s call to dm_btree_insert() > returns 0 -- which simply means success. And on success > sm_disk_new_block() assumes ev was set to SM_ALLOC (by sm_ll_mutate). > > sm_ll_mutate() decided to _not_ set SM_ALLOC because either: > 1) ref_count wasn't set > or > 2) old was identified > > So all said: somehow a new data block was found to already be in use. > _WHY_ that is the case isn't clear from this stack... > > But it does speak to the possibility of data block allocation racing > with other operations to the same block. Which implies missing locking. Would a spinlock on the block solve the issue? Where might such a spinlock be added? -- Eric Wheeler > > Thanks, > > - Joe > -- lvm-devel mailing list lvm-devel@redhat.com https://www.redhat.com/mailman/listinfo/lvm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 with scsi_mod.use_blk_mq=y 2019-09-25 18:40 ` Eric Wheeler (?) @ 2019-12-20 19:54 ` Eric Wheeler -1 siblings, 0 replies; 43+ messages in thread From: Eric Wheeler @ 2019-12-20 19:54 UTC (permalink / raw) To: ejt Cc: Mike Snitzer, joe.thornber, dm-devel, linux-block, lvm-devel, markus.schade On Wed, 25 Sep 2019, Eric Wheeler wrote: > We are using the 4.19.75 stable tree with dm-thin and multi-queue scsi. > We have been using the 4.19 branch for months without issue; we just > switched to MQ and we seem to have hit this BUG_ON. Whether or not MQ is > related to the issue, I don't know, maybe coincidence: > > static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > { > int r; > enum allocation_event ev; > struct sm_disk *smd = container_of(sm, struct sm_disk, sm); > > /* FIXME: we should loop round a couple of times */ > r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); > if (r) > return r; > > smd->begin = *b + 1; > r = sm_ll_inc(&smd->ll, *b, &ev); > if (!r) { > BUG_ON(ev != SM_ALLOC); <<<<<<<<<<<<<<<<<<<<<<<<<<<<< > smd->nr_allocated_this_transaction++; > } Hello all, We hit this BUG_ON again, this time with 4.19.86 with scsi_mod.use_blk_mq=y, and it is known to be present as of 5.1.2 as additionally reported by Markus Schade: https://www.redhat.com/archives/dm-devel/2019-June/msg00116.html and https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1777398 In our case, the most latest trace (below) is from a different system that has been stable for years on Linux 4.1 with tmeta direct on the SSD. We updated to 4.19.86 a few weeks ago and just hit this, what Mike Snitzer explains to be an allocator race: On Wed, 25 Sep 2019, Mike Snitzer wrote: > > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > > [199391.693852] ? sort+0x17b/0x270 > > [199391.694527] ? u32_swap+0x10/0x10 > > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > > [199391.695890] process_one_work+0x171/0x370 > > [199391.696640] worker_thread+0x49/0x3f0 > > [199391.697332] kthread+0xf8/0x130 > > [199391.697988] ? max_active_store+0x80/0x80 > > [199391.698659] ? kthread_bind+0x10/0x10 > > [199391.699281] ret_from_fork+0x1f/0x40 > > The stack shows the call to sm_disk_new_block() is due to > dm_pool_alloc_data_block(). > > sm_disk_new_block()'s BUG_ON(ev != SM_ALLOC) indicates that somehow it is > getting called without the passed 'ev' being set to SM_ALLOC. Only > drivers/md/persistent-dat/dm-space-map-common.c:sm_ll_mutate() sets > SM_ALLOC. sm_disk_new_block() is indirectly calling sm_ll_mutate() > > sm_ll_mutate() will only return 0 if ll->save_ie() does, the ll_disk *ll > should be ll_disk, and so disk_ll_save_ie()'s call to dm_btree_insert() > returns 0 -- which simply means success. And on success > sm_disk_new_block() assumes ev was set to SM_ALLOC (by sm_ll_mutate). > > sm_ll_mutate() decided to _not_ set SM_ALLOC because either: > 1) ref_count wasn't set > or > 2) old was identified > > So all said: somehow a new data block was found to already be in use. > _WHY_ that is the case isn't clear from this stack... > > But it does speak to the possibility of data block allocation racing > with other operations to the same block. Which implies missing locking. Where would you look to add locking do you think? > But that's all I've got so far... I'll review past dm-thinp changes with > all this in mind and see what turns up. But Joe Thornber (ejt) likely > needs to have a look at this too. > > But could it be that bcache is the source of the data device race (same > block used concurrently)? And DM thinp is acting as the canary in the > coal mine? As Marcus has shown, this bug triggers without bcache. Other questions: 1) Can sm_disk_new_block() fail more gracefully than BUG_ON? For example: + spin_lock(&lock); /* protect smd->begin */ smd->begin = *b + 1; r = sm_ll_inc(&smd->ll, *b, &ev); if (!r) { - BUG_ON(ev != SM_ALLOC); smd->nr_allocated_this_transaction++; } + else { + r = -ENOSPC; + smd->begin = *b - 1; + } + spin_unlock(&lock); The lock might protect smd->begin, but I'm not sure how &smd->ll might have been modified by sm_ll_inc(). However, since ll->save_ie() failed in sm_ll_mutate() then perhaps this is safe. What do you think? Putting the pool into PM_OUT_OF_DATA_SPACE isn't ideal since it isn't out of space, but I would take it over a BUG_ON. 2) If example #1 above returned -EAGAIN, how might alloc_data_block be taught retry? This bug shows up weeks or months apart, even on heavily loaded systems with ~100 live thin volumes, so retrying would be fine IMHO. 3) In the thread from June, Marcus says: "Judging from the call trace, my guess is that there is somewhere a race condition, when a new block needs to be allocated which has still to be discarded." Is this discard sitation possible? Wouldn't the bio prison prevent this? -- Eric Wheeler www.datawall.us Here is the new trace, old trace below: kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! invalid opcode: 0000 [#1] SMP NOPTI CPU: 11 PID: 22939 Comm: kworker/u48:1 Not tainted 4.19.86 #1 Hardware name: Supermicro SYS-2026T-6RFT+/X8DTU-6+, BIOS 2.1c 11/30/2012 Workqueue: dm-thin do_worker [dm_thin_pool] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 b0 a5 a9 e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 RSP: 0018:ffffc90007237c78 EFLAGS: 00010297 RAX: 0000000000000000 RBX: ffff88861d7ac000 RCX: 0000000000000000 RDX: ffff8885f3d13f00 RSI: 0000000000000246 RDI: ffff888620e38c00 RBP: 0000000000000000 R08: 0000000000000000 R09: ffff888620e38c98 R10: ffffffff810f9177 R11: 0000000000000000 R12: ffffc90007237d48 R13: ffffc90007237d48 R14: 00000000ffffffc3 R15: 00000000000d9e3a FS: 0000000000000000(0000) GS:ffff8886279c0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000081ceff0 CR3: 000000000200a004 CR4: 00000000000226e0 Call Trace: dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] process_cell+0x2a3/0x550 [dm_thin_pool] ? mempool_alloc+0x6f/0x180 ? sort+0x17b/0x270 ? u32_swap+0x10/0x10 process_deferred_bios+0x1af/0x870 [dm_thin_pool] do_worker+0x94/0xe0 [dm_thin_pool] process_one_work+0x171/0x370 worker_thread+0x49/0x3f0 kthread+0xf8/0x130 ? max_active_store+0x80/0x80 ? kthread_bind+0x10/0x10 ret_from_fork+0x1f/0x40 > > [199391.677689] ------------[ cut here ]------------ > [199391.678437] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! > [199391.679183] invalid opcode: 0000 [#1] SMP NOPTI > [199391.679941] CPU: 4 PID: 31359 Comm: kworker/u16:4 Not tainted 4.19.75 #1 > [199391.680683] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > [199391.681446] Workqueue: dm-thin do_worker [dm_thin_pool] > [199391.682187] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199391.682929] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > [199391.684432] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > [199391.685186] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > [199391.685936] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > [199391.686659] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > [199391.687379] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > [199391.688120] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > [199391.688843] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > [199391.689571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [199391.690253] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > [199391.690984] Call Trace: > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > [199391.693852] ? sort+0x17b/0x270 > [199391.694527] ? u32_swap+0x10/0x10 > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > [199391.695890] process_one_work+0x171/0x370 > [199391.696640] worker_thread+0x49/0x3f0 > [199391.697332] kthread+0xf8/0x130 > [199391.697988] ? max_active_store+0x80/0x80 > [199391.698659] ? kthread_bind+0x10/0x10 > [199391.699281] ret_from_fork+0x1f/0x40 > [199391.699930] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_bit drm_kms_helper > [199391.705631] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > [199391.708083] ---[ end trace c31536d98046e8ec ]--- > [199391.866776] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199391.867960] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > [199391.870379] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > [199391.871524] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > [199391.872364] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > [199391.873173] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > [199391.873871] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > [199391.874550] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > [199391.875231] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > [199391.875941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [199391.876633] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > [199391.877317] Kernel panic - not syncing: Fatal exception > [199391.878006] Kernel Offset: disabled > [199392.032304] ---[ end Kernel panic - not syncing: Fatal exception ]--- > [199392.032962] unchecked MSR access error: WRMSR to 0x83f (tried to write 0x00000000000000f6) at rIP: 0xffffffff81067af4 (native_write_msr+0x4/0x20) > [199392.034277] Call Trace: > [199392.034929] <IRQ> > [199392.035576] native_apic_msr_write+0x2e/0x40 > [199392.036228] arch_irq_work_raise+0x28/0x40 > [199392.036877] irq_work_queue_on+0x83/0xa0 > [199392.037518] irq_work_run_list+0x4c/0x70 > [199392.038149] irq_work_run+0x14/0x40 > [199392.038771] smp_call_function_single_interrupt+0x3a/0xd0 > [199392.039393] call_function_single_interrupt+0xf/0x20 > [199392.040011] </IRQ> > [199392.040624] RIP: 0010:panic+0x209/0x25c > [199392.041234] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > [199392.042518] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 > [199392.043174] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > [199392.043833] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > [199392.044493] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > [199392.045155] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > [199392.045820] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > [199392.046486] oops_end+0xc1/0xd0 > [199392.047149] do_trap+0x13d/0x150 > [199392.047795] do_error_trap+0xd5/0x130 > [199392.048427] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199392.049048] invalid_op+0x14/0x20 > [199392.049650] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199392.050245] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > [199392.051434] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > [199392.052010] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > [199392.052580] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > [199392.053150] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > [199392.053715] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > [199392.054266] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > [199392.054807] ? __wake_up_common_lock+0x87/0xc0 > [199392.055342] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > [199392.055877] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > [199392.056410] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > [199392.056947] process_cell+0x2a3/0x550 [dm_thin_pool] > [199392.057484] ? sort+0x17b/0x270 > [199392.058016] ? u32_swap+0x10/0x10 > [199392.058538] do_worker+0x268/0x9a0 [dm_thin_pool] > [199392.059060] process_one_work+0x171/0x370 > [199392.059576] worker_thread+0x49/0x3f0 > [199392.060083] kthread+0xf8/0x130 > [199392.060587] ? max_active_store+0x80/0x80 > [199392.061086] ? kthread_bind+0x10/0x10 > [199392.061569] ret_from_fork+0x1f/0x40 > [199392.062038] ------------[ cut here ]------------ > [199392.062508] sched: Unexpected reschedule of offline CPU#1! > [199392.062989] WARNING: CPU: 4 PID: 31359 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40 > [199392.063485] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_bit drm_kms_helper > [199392.067463] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > [199392.068729] CPU: 4 PID: 31359 Comm: kworker/u16:4 Tainted: G D 4.19.75 #1 > [199392.069356] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > [199392.069982] Workqueue: dm-thin do_worker [dm_thin_pool] > [199392.070604] RIP: 0010:native_smp_send_reschedule+0x39/0x40 > [199392.071229] Code: 0f 92 c0 84 c0 74 15 48 8b 05 93 f1 eb 00 be fd 00 00 00 48 8b 40 30 e9 85 f4 9a 00 89 fe 48 c7 c7 08 23 e5 81 e8 c7 9a 05 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48 > [199392.072528] RSP: 0018:ffff88880fb03dc0 EFLAGS: 00010082 > [199392.073186] RAX: 0000000000000000 RBX: ffff88880fa625c0 RCX: 0000000000000006 > [199392.073847] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff88880fb168b0 > [199392.074512] RBP: ffff88880fa625c0 R08: 0000000000000001 R09: 00000000000174e5 > [199392.075182] R10: ffff8881f2512300 R11: 0000000000000000 R12: ffff888804541640 > [199392.075849] R13: ffff88880fb03e08 R14: 0000000000000000 R15: 0000000000000001 > [199392.076512] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > [199392.077179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [199392.077833] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > [199392.078481] Call Trace: > [199392.079117] <IRQ> > [199392.079745] check_preempt_curr+0x6b/0x90 > [199392.080373] ttwu_do_wakeup+0x19/0x130 > [199392.080999] try_to_wake_up+0x1e2/0x460 > [199392.081623] __wake_up_common+0x8f/0x160 > [199392.082246] ep_poll_callback+0x1af/0x300 > [199392.082860] __wake_up_common+0x8f/0x160 > [199392.083470] __wake_up_common_lock+0x7a/0xc0 > [199392.084074] irq_work_run_list+0x4c/0x70 > [199392.084675] smp_call_function_single_interrupt+0x3a/0xd0 > [199392.085277] call_function_single_interrupt+0xf/0x20 > [199392.085879] </IRQ> > [199392.086477] RIP: 0010:panic+0x209/0x25c > [199392.087079] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > [199392.088341] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 > [199392.088988] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > [199392.089639] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > [199392.090291] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > [199392.090947] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > [199392.091601] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > [199392.092255] oops_end+0xc1/0xd0 > [199392.092894] do_trap+0x13d/0x150 > [199392.093516] do_error_trap+0xd5/0x130 > [199392.094122] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199392.094718] invalid_op+0x14/0x20 > [199392.095296] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199392.095866] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > [199392.097006] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > [199392.097558] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > [199392.098103] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > [199392.098644] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > [199392.099176] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > [199392.099703] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > [199392.100229] ? __wake_up_common_lock+0x87/0xc0 > [199392.100744] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > [199392.101251] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > [199392.101752] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > [199392.102252] process_cell+0x2a3/0x550 [dm_thin_pool] > [199392.102750] ? sort+0x17b/0x270 > [199392.103242] ? u32_swap+0x10/0x10 > [199392.103733] do_worker+0x268/0x9a0 [dm_thin_pool] > [199392.104228] process_one_work+0x171/0x370 > [199392.104714] worker_thread+0x49/0x3f0 > [199392.105193] kthread+0xf8/0x130 > [199392.105665] ? max_active_store+0x80/0x80 > [199392.106132] ? kthread_bind+0x10/0x10 > [199392.106601] ret_from_fork+0x1f/0x40 > [199392.107069] ---[ end trace c31536d98046e8ed ]--- > [199392.107544] ------------[ cut here ]------------ > [199392.108017] sched: Unexpected reschedule of offline CPU#7! > [199392.108497] WARNING: CPU: 4 PID: 31359 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40 > [199392.108996] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_bit drm_kms_helper > [199392.112967] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > [199392.114203] CPU: 4 PID: 31359 Comm: kworker/u16:4 Tainted: G D W 4.19.75 #1 > [199392.114819] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > [199392.115439] Workqueue: dm-thin do_worker [dm_thin_pool] > [199392.116061] RIP: 0010:native_smp_send_reschedule+0x39/0x40 > [199392.116683] Code: 0f 92 c0 84 c0 74 15 48 8b 05 93 f1 eb 00 be fd 00 00 00 48 8b 40 30 e9 85 f4 9a 00 89 fe 48 c7 c7 08 23 e5 81 e8 c7 9a 05 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48 > [199392.117982] RSP: 0018:ffff88880fb03ee0 EFLAGS: 00010082 > [199392.118632] RAX: 0000000000000000 RBX: ffff8887d093ac80 RCX: 0000000000000006 > [199392.119295] RDX: 0000000000000000 RSI: 0000000000000092 RDI: ffff88880fb168b0 > [199392.119961] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000017529 > [199392.120623] R10: ffff8881438ba800 R11: 0000000000000000 R12: ffffc9000a147988 > [199392.121283] R13: ffffffff8113f9a0 R14: 0000000000000002 R15: ffff88880fb1cff8 > [199392.121938] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > [199392.122589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [199392.123229] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > [199392.123867] Call Trace: > [199392.124495] <IRQ> > [199392.125116] update_process_times+0x40/0x50 > [199392.125742] tick_sched_handle+0x25/0x60 > [199392.126367] tick_sched_timer+0x37/0x70 > [199392.126987] __hrtimer_run_queues+0xfb/0x270 > [199392.127601] hrtimer_interrupt+0x122/0x270 > [199392.128211] smp_apic_timer_interrupt+0x6a/0x140 > [199392.128819] apic_timer_interrupt+0xf/0x20 > [199392.129423] </IRQ> > [199392.130020] RIP: 0010:panic+0x209/0x25c > [199392.130619] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > [199392.131883] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 > [199392.132530] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > [199392.133180] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > [199392.133830] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > [199392.134482] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > [199392.135136] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > [199392.135792] oops_end+0xc1/0xd0 > [199392.136444] do_trap+0x13d/0x150 > [199392.137094] do_error_trap+0xd5/0x130 > [199392.137740] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199392.138383] invalid_op+0x14/0x20 > [199392.139010] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199392.139630] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > [199392.140870] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > [199392.141472] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > [199392.142066] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > [199392.142647] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > [199392.143211] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > [199392.143759] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > [199392.144301] ? __wake_up_common_lock+0x87/0xc0 > [199392.144838] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > [199392.145374] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > [199392.145909] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > [199392.146435] process_cell+0x2a3/0x550 [dm_thin_pool] > [199392.146949] ? sort+0x17b/0x270 > [199392.147450] ? u32_swap+0x10/0x10 > [199392.147944] do_worker+0x268/0x9a0 [dm_thin_pool] > [199392.148441] process_one_work+0x171/0x370 > [199392.148937] worker_thread+0x49/0x3f0 > [199392.149430] kthread+0xf8/0x130 > [199392.149922] ? max_active_store+0x80/0x80 > [199392.150406] ? kthread_bind+0x10/0x10 > [199392.150883] ret_from_fork+0x1f/0x40 > [199392.151353] ---[ end trace c31536d98046e8ee ]--- > > > -- > Eric Wheeler > > -- > dm-devel mailing list > dm-devel@redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel > ^ permalink raw reply [flat|nested] 43+ messages in thread
* [dm-devel] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 with scsi_mod.use_blk_mq=y @ 2019-12-20 19:54 ` Eric Wheeler 0 siblings, 0 replies; 43+ messages in thread From: Eric Wheeler @ 2019-12-20 19:54 UTC (permalink / raw) To: lvm-devel On Wed, 25 Sep 2019, Eric Wheeler wrote: > We are using the 4.19.75 stable tree with dm-thin and multi-queue scsi. > We have been using the 4.19 branch for months without issue; we just > switched to MQ and we seem to have hit this BUG_ON. Whether or not MQ is > related to the issue, I don't know, maybe coincidence: > > static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > { > int r; > enum allocation_event ev; > struct sm_disk *smd = container_of(sm, struct sm_disk, sm); > > /* FIXME: we should loop round a couple of times */ > r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); > if (r) > return r; > > smd->begin = *b + 1; > r = sm_ll_inc(&smd->ll, *b, &ev); > if (!r) { > BUG_ON(ev != SM_ALLOC); <<<<<<<<<<<<<<<<<<<<<<<<<<<<< > smd->nr_allocated_this_transaction++; > } Hello all, We hit this BUG_ON again, this time with 4.19.86 with scsi_mod.use_blk_mq=y, and it is known to be present as of 5.1.2 as additionally reported by Markus Schade: https://www.redhat.com/archives/dm-devel/2019-June/msg00116.html and https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1777398 In our case, the most latest trace (below) is from a different system that has been stable for years on Linux 4.1 with tmeta direct on the SSD. We updated to 4.19.86 a few weeks ago and just hit this, what Mike Snitzer explains to be an allocator race: On Wed, 25 Sep 2019, Mike Snitzer wrote: > > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > > [199391.693852] ? sort+0x17b/0x270 > > [199391.694527] ? u32_swap+0x10/0x10 > > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > > [199391.695890] process_one_work+0x171/0x370 > > [199391.696640] worker_thread+0x49/0x3f0 > > [199391.697332] kthread+0xf8/0x130 > > [199391.697988] ? max_active_store+0x80/0x80 > > [199391.698659] ? kthread_bind+0x10/0x10 > > [199391.699281] ret_from_fork+0x1f/0x40 > > The stack shows the call to sm_disk_new_block() is due to > dm_pool_alloc_data_block(). > > sm_disk_new_block()'s BUG_ON(ev != SM_ALLOC) indicates that somehow it is > getting called without the passed 'ev' being set to SM_ALLOC. Only > drivers/md/persistent-dat/dm-space-map-common.c:sm_ll_mutate() sets > SM_ALLOC. sm_disk_new_block() is indirectly calling sm_ll_mutate() > > sm_ll_mutate() will only return 0 if ll->save_ie() does, the ll_disk *ll > should be ll_disk, and so disk_ll_save_ie()'s call to dm_btree_insert() > returns 0 -- which simply means success. And on success > sm_disk_new_block() assumes ev was set to SM_ALLOC (by sm_ll_mutate). > > sm_ll_mutate() decided to _not_ set SM_ALLOC because either: > 1) ref_count wasn't set > or > 2) old was identified > > So all said: somehow a new data block was found to already be in use. > _WHY_ that is the case isn't clear from this stack... > > But it does speak to the possibility of data block allocation racing > with other operations to the same block. Which implies missing locking. Where would you look to add locking do you think? > But that's all I've got so far... I'll review past dm-thinp changes with > all this in mind and see what turns up. But Joe Thornber (ejt) likely > needs to have a look at this too. > > But could it be that bcache is the source of the data device race (same > block used concurrently)? And DM thinp is acting as the canary in the > coal mine? As Marcus has shown, this bug triggers without bcache. Other questions: 1) Can sm_disk_new_block() fail more gracefully than BUG_ON? For example: + spin_lock(&lock); /* protect smd->begin */ smd->begin = *b + 1; r = sm_ll_inc(&smd->ll, *b, &ev); if (!r) { - BUG_ON(ev != SM_ALLOC); smd->nr_allocated_this_transaction++; } + else { + r = -ENOSPC; + smd->begin = *b - 1; + } + spin_unlock(&lock); The lock might protect smd->begin, but I'm not sure how &smd->ll might have been modified by sm_ll_inc(). However, since ll->save_ie() failed in sm_ll_mutate() then perhaps this is safe. What do you think? Putting the pool into PM_OUT_OF_DATA_SPACE isn't ideal since it isn't out of space, but I would take it over a BUG_ON. 2) If example #1 above returned -EAGAIN, how might alloc_data_block be taught retry? This bug shows up weeks or months apart, even on heavily loaded systems with ~100 live thin volumes, so retrying would be fine IMHO. 3) In the thread from June, Marcus says: "Judging from the call trace, my guess is that there is somewhere a race condition, when a new block needs to be allocated which has still to be discarded." Is this discard sitation possible? Wouldn't the bio prison prevent this? -- Eric Wheeler www.datawall.us Here is the new trace, old trace below: kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! invalid opcode: 0000 [#1] SMP NOPTI CPU: 11 PID: 22939 Comm: kworker/u48:1 Not tainted 4.19.86 #1 Hardware name: Supermicro SYS-2026T-6RFT+/X8DTU-6+, BIOS 2.1c 11/30/2012 Workqueue: dm-thin do_worker [dm_thin_pool] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 b0 a5 a9 e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 RSP: 0018:ffffc90007237c78 EFLAGS: 00010297 RAX: 0000000000000000 RBX: ffff88861d7ac000 RCX: 0000000000000000 RDX: ffff8885f3d13f00 RSI: 0000000000000246 RDI: ffff888620e38c00 RBP: 0000000000000000 R08: 0000000000000000 R09: ffff888620e38c98 R10: ffffffff810f9177 R11: 0000000000000000 R12: ffffc90007237d48 R13: ffffc90007237d48 R14: 00000000ffffffc3 R15: 00000000000d9e3a FS: 0000000000000000(0000) GS:ffff8886279c0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000081ceff0 CR3: 000000000200a004 CR4: 00000000000226e0 Call Trace: dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] process_cell+0x2a3/0x550 [dm_thin_pool] ? mempool_alloc+0x6f/0x180 ? sort+0x17b/0x270 ? u32_swap+0x10/0x10 process_deferred_bios+0x1af/0x870 [dm_thin_pool] do_worker+0x94/0xe0 [dm_thin_pool] process_one_work+0x171/0x370 worker_thread+0x49/0x3f0 kthread+0xf8/0x130 ? max_active_store+0x80/0x80 ? kthread_bind+0x10/0x10 ret_from_fork+0x1f/0x40 > > [199391.677689] ------------[ cut here ]------------ > [199391.678437] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! > [199391.679183] invalid opcode: 0000 [#1] SMP NOPTI > [199391.679941] CPU: 4 PID: 31359 Comm: kworker/u16:4 Not tainted 4.19.75 #1 > [199391.680683] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > [199391.681446] Workqueue: dm-thin do_worker [dm_thin_pool] > [199391.682187] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199391.682929] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > [199391.684432] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > [199391.685186] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > [199391.685936] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > [199391.686659] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > [199391.687379] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > [199391.688120] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > [199391.688843] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > [199391.689571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [199391.690253] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > [199391.690984] Call Trace: > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > [199391.693852] ? sort+0x17b/0x270 > [199391.694527] ? u32_swap+0x10/0x10 > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > [199391.695890] process_one_work+0x171/0x370 > [199391.696640] worker_thread+0x49/0x3f0 > [199391.697332] kthread+0xf8/0x130 > [199391.697988] ? max_active_store+0x80/0x80 > [199391.698659] ? kthread_bind+0x10/0x10 > [199391.699281] ret_from_fork+0x1f/0x40 > [199391.699930] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_bit drm_kms_helper > [199391.705631] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > [199391.708083] ---[ end trace c31536d98046e8ec ]--- > [199391.866776] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199391.867960] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > [199391.870379] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > [199391.871524] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > [199391.872364] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > [199391.873173] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > [199391.873871] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > [199391.874550] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > [199391.875231] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > [199391.875941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [199391.876633] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > [199391.877317] Kernel panic - not syncing: Fatal exception > [199391.878006] Kernel Offset: disabled > [199392.032304] ---[ end Kernel panic - not syncing: Fatal exception ]--- > [199392.032962] unchecked MSR access error: WRMSR to 0x83f (tried to write 0x00000000000000f6) at rIP: 0xffffffff81067af4 (native_write_msr+0x4/0x20) > [199392.034277] Call Trace: > [199392.034929] <IRQ> > [199392.035576] native_apic_msr_write+0x2e/0x40 > [199392.036228] arch_irq_work_raise+0x28/0x40 > [199392.036877] irq_work_queue_on+0x83/0xa0 > [199392.037518] irq_work_run_list+0x4c/0x70 > [199392.038149] irq_work_run+0x14/0x40 > [199392.038771] smp_call_function_single_interrupt+0x3a/0xd0 > [199392.039393] call_function_single_interrupt+0xf/0x20 > [199392.040011] </IRQ> > [199392.040624] RIP: 0010:panic+0x209/0x25c > [199392.041234] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > [199392.042518] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 > [199392.043174] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > [199392.043833] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > [199392.044493] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > [199392.045155] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > [199392.045820] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > [199392.046486] oops_end+0xc1/0xd0 > [199392.047149] do_trap+0x13d/0x150 > [199392.047795] do_error_trap+0xd5/0x130 > [199392.048427] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199392.049048] invalid_op+0x14/0x20 > [199392.049650] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199392.050245] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > [199392.051434] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > [199392.052010] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > [199392.052580] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > [199392.053150] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > [199392.053715] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > [199392.054266] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > [199392.054807] ? __wake_up_common_lock+0x87/0xc0 > [199392.055342] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > [199392.055877] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > [199392.056410] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > [199392.056947] process_cell+0x2a3/0x550 [dm_thin_pool] > [199392.057484] ? sort+0x17b/0x270 > [199392.058016] ? u32_swap+0x10/0x10 > [199392.058538] do_worker+0x268/0x9a0 [dm_thin_pool] > [199392.059060] process_one_work+0x171/0x370 > [199392.059576] worker_thread+0x49/0x3f0 > [199392.060083] kthread+0xf8/0x130 > [199392.060587] ? max_active_store+0x80/0x80 > [199392.061086] ? kthread_bind+0x10/0x10 > [199392.061569] ret_from_fork+0x1f/0x40 > [199392.062038] ------------[ cut here ]------------ > [199392.062508] sched: Unexpected reschedule of offline CPU#1! > [199392.062989] WARNING: CPU: 4 PID: 31359 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40 > [199392.063485] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_bit drm_kms_helper > [199392.067463] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > [199392.068729] CPU: 4 PID: 31359 Comm: kworker/u16:4 Tainted: G D 4.19.75 #1 > [199392.069356] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > [199392.069982] Workqueue: dm-thin do_worker [dm_thin_pool] > [199392.070604] RIP: 0010:native_smp_send_reschedule+0x39/0x40 > [199392.071229] Code: 0f 92 c0 84 c0 74 15 48 8b 05 93 f1 eb 00 be fd 00 00 00 48 8b 40 30 e9 85 f4 9a 00 89 fe 48 c7 c7 08 23 e5 81 e8 c7 9a 05 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48 > [199392.072528] RSP: 0018:ffff88880fb03dc0 EFLAGS: 00010082 > [199392.073186] RAX: 0000000000000000 RBX: ffff88880fa625c0 RCX: 0000000000000006 > [199392.073847] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff88880fb168b0 > [199392.074512] RBP: ffff88880fa625c0 R08: 0000000000000001 R09: 00000000000174e5 > [199392.075182] R10: ffff8881f2512300 R11: 0000000000000000 R12: ffff888804541640 > [199392.075849] R13: ffff88880fb03e08 R14: 0000000000000000 R15: 0000000000000001 > [199392.076512] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > [199392.077179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [199392.077833] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > [199392.078481] Call Trace: > [199392.079117] <IRQ> > [199392.079745] check_preempt_curr+0x6b/0x90 > [199392.080373] ttwu_do_wakeup+0x19/0x130 > [199392.080999] try_to_wake_up+0x1e2/0x460 > [199392.081623] __wake_up_common+0x8f/0x160 > [199392.082246] ep_poll_callback+0x1af/0x300 > [199392.082860] __wake_up_common+0x8f/0x160 > [199392.083470] __wake_up_common_lock+0x7a/0xc0 > [199392.084074] irq_work_run_list+0x4c/0x70 > [199392.084675] smp_call_function_single_interrupt+0x3a/0xd0 > [199392.085277] call_function_single_interrupt+0xf/0x20 > [199392.085879] </IRQ> > [199392.086477] RIP: 0010:panic+0x209/0x25c > [199392.087079] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > [199392.088341] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 > [199392.088988] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > [199392.089639] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > [199392.090291] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > [199392.090947] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > [199392.091601] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > [199392.092255] oops_end+0xc1/0xd0 > [199392.092894] do_trap+0x13d/0x150 > [199392.093516] do_error_trap+0xd5/0x130 > [199392.094122] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199392.094718] invalid_op+0x14/0x20 > [199392.095296] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199392.095866] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > [199392.097006] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > [199392.097558] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > [199392.098103] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > [199392.098644] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > [199392.099176] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > [199392.099703] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > [199392.100229] ? __wake_up_common_lock+0x87/0xc0 > [199392.100744] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > [199392.101251] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > [199392.101752] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > [199392.102252] process_cell+0x2a3/0x550 [dm_thin_pool] > [199392.102750] ? sort+0x17b/0x270 > [199392.103242] ? u32_swap+0x10/0x10 > [199392.103733] do_worker+0x268/0x9a0 [dm_thin_pool] > [199392.104228] process_one_work+0x171/0x370 > [199392.104714] worker_thread+0x49/0x3f0 > [199392.105193] kthread+0xf8/0x130 > [199392.105665] ? max_active_store+0x80/0x80 > [199392.106132] ? kthread_bind+0x10/0x10 > [199392.106601] ret_from_fork+0x1f/0x40 > [199392.107069] ---[ end trace c31536d98046e8ed ]--- > [199392.107544] ------------[ cut here ]------------ > [199392.108017] sched: Unexpected reschedule of offline CPU#7! > [199392.108497] WARNING: CPU: 4 PID: 31359 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40 > [199392.108996] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_bit drm_kms_helper > [199392.112967] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > [199392.114203] CPU: 4 PID: 31359 Comm: kworker/u16:4 Tainted: G D W 4.19.75 #1 > [199392.114819] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > [199392.115439] Workqueue: dm-thin do_worker [dm_thin_pool] > [199392.116061] RIP: 0010:native_smp_send_reschedule+0x39/0x40 > [199392.116683] Code: 0f 92 c0 84 c0 74 15 48 8b 05 93 f1 eb 00 be fd 00 00 00 48 8b 40 30 e9 85 f4 9a 00 89 fe 48 c7 c7 08 23 e5 81 e8 c7 9a 05 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48 > [199392.117982] RSP: 0018:ffff88880fb03ee0 EFLAGS: 00010082 > [199392.118632] RAX: 0000000000000000 RBX: ffff8887d093ac80 RCX: 0000000000000006 > [199392.119295] RDX: 0000000000000000 RSI: 0000000000000092 RDI: ffff88880fb168b0 > [199392.119961] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000017529 > [199392.120623] R10: ffff8881438ba800 R11: 0000000000000000 R12: ffffc9000a147988 > [199392.121283] R13: ffffffff8113f9a0 R14: 0000000000000002 R15: ffff88880fb1cff8 > [199392.121938] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > [199392.122589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [199392.123229] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > [199392.123867] Call Trace: > [199392.124495] <IRQ> > [199392.125116] update_process_times+0x40/0x50 > [199392.125742] tick_sched_handle+0x25/0x60 > [199392.126367] tick_sched_timer+0x37/0x70 > [199392.126987] __hrtimer_run_queues+0xfb/0x270 > [199392.127601] hrtimer_interrupt+0x122/0x270 > [199392.128211] smp_apic_timer_interrupt+0x6a/0x140 > [199392.128819] apic_timer_interrupt+0xf/0x20 > [199392.129423] </IRQ> > [199392.130020] RIP: 0010:panic+0x209/0x25c > [199392.130619] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > [199392.131883] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 > [199392.132530] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > [199392.133180] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > [199392.133830] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > [199392.134482] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > [199392.135136] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > [199392.135792] oops_end+0xc1/0xd0 > [199392.136444] do_trap+0x13d/0x150 > [199392.137094] do_error_trap+0xd5/0x130 > [199392.137740] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199392.138383] invalid_op+0x14/0x20 > [199392.139010] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199392.139630] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > [199392.140870] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > [199392.141472] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > [199392.142066] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > [199392.142647] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > [199392.143211] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > [199392.143759] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > [199392.144301] ? __wake_up_common_lock+0x87/0xc0 > [199392.144838] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > [199392.145374] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > [199392.145909] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > [199392.146435] process_cell+0x2a3/0x550 [dm_thin_pool] > [199392.146949] ? sort+0x17b/0x270 > [199392.147450] ? u32_swap+0x10/0x10 > [199392.147944] do_worker+0x268/0x9a0 [dm_thin_pool] > [199392.148441] process_one_work+0x171/0x370 > [199392.148937] worker_thread+0x49/0x3f0 > [199392.149430] kthread+0xf8/0x130 > [199392.149922] ? max_active_store+0x80/0x80 > [199392.150406] ? kthread_bind+0x10/0x10 > [199392.150883] ret_from_fork+0x1f/0x40 > [199392.151353] ---[ end trace c31536d98046e8ee ]--- > > > -- > Eric Wheeler > > -- > dm-devel mailing list > dm-devel at redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel > ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 with scsi_mod.use_blk_mq=y @ 2019-12-20 19:54 ` Eric Wheeler 0 siblings, 0 replies; 43+ messages in thread From: Eric Wheeler @ 2019-12-20 19:54 UTC (permalink / raw) To: ejt Cc: Mike Snitzer, markus.schade, lvm-devel, linux-block, dm-devel, joe.thornber On Wed, 25 Sep 2019, Eric Wheeler wrote: > We are using the 4.19.75 stable tree with dm-thin and multi-queue scsi. > We have been using the 4.19 branch for months without issue; we just > switched to MQ and we seem to have hit this BUG_ON. Whether or not MQ is > related to the issue, I don't know, maybe coincidence: > > static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > { > int r; > enum allocation_event ev; > struct sm_disk *smd = container_of(sm, struct sm_disk, sm); > > /* FIXME: we should loop round a couple of times */ > r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); > if (r) > return r; > > smd->begin = *b + 1; > r = sm_ll_inc(&smd->ll, *b, &ev); > if (!r) { > BUG_ON(ev != SM_ALLOC); <<<<<<<<<<<<<<<<<<<<<<<<<<<<< > smd->nr_allocated_this_transaction++; > } Hello all, We hit this BUG_ON again, this time with 4.19.86 with scsi_mod.use_blk_mq=y, and it is known to be present as of 5.1.2 as additionally reported by Markus Schade: https://www.redhat.com/archives/dm-devel/2019-June/msg00116.html and https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1777398 In our case, the most latest trace (below) is from a different system that has been stable for years on Linux 4.1 with tmeta direct on the SSD. We updated to 4.19.86 a few weeks ago and just hit this, what Mike Snitzer explains to be an allocator race: On Wed, 25 Sep 2019, Mike Snitzer wrote: > > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > > [199391.693852] ? sort+0x17b/0x270 > > [199391.694527] ? u32_swap+0x10/0x10 > > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > > [199391.695890] process_one_work+0x171/0x370 > > [199391.696640] worker_thread+0x49/0x3f0 > > [199391.697332] kthread+0xf8/0x130 > > [199391.697988] ? max_active_store+0x80/0x80 > > [199391.698659] ? kthread_bind+0x10/0x10 > > [199391.699281] ret_from_fork+0x1f/0x40 > > The stack shows the call to sm_disk_new_block() is due to > dm_pool_alloc_data_block(). > > sm_disk_new_block()'s BUG_ON(ev != SM_ALLOC) indicates that somehow it is > getting called without the passed 'ev' being set to SM_ALLOC. Only > drivers/md/persistent-dat/dm-space-map-common.c:sm_ll_mutate() sets > SM_ALLOC. sm_disk_new_block() is indirectly calling sm_ll_mutate() > > sm_ll_mutate() will only return 0 if ll->save_ie() does, the ll_disk *ll > should be ll_disk, and so disk_ll_save_ie()'s call to dm_btree_insert() > returns 0 -- which simply means success. And on success > sm_disk_new_block() assumes ev was set to SM_ALLOC (by sm_ll_mutate). > > sm_ll_mutate() decided to _not_ set SM_ALLOC because either: > 1) ref_count wasn't set > or > 2) old was identified > > So all said: somehow a new data block was found to already be in use. > _WHY_ that is the case isn't clear from this stack... > > But it does speak to the possibility of data block allocation racing > with other operations to the same block. Which implies missing locking. Where would you look to add locking do you think? > But that's all I've got so far... I'll review past dm-thinp changes with > all this in mind and see what turns up. But Joe Thornber (ejt) likely > needs to have a look at this too. > > But could it be that bcache is the source of the data device race (same > block used concurrently)? And DM thinp is acting as the canary in the > coal mine? As Marcus has shown, this bug triggers without bcache. Other questions: 1) Can sm_disk_new_block() fail more gracefully than BUG_ON? For example: + spin_lock(&lock); /* protect smd->begin */ smd->begin = *b + 1; r = sm_ll_inc(&smd->ll, *b, &ev); if (!r) { - BUG_ON(ev != SM_ALLOC); smd->nr_allocated_this_transaction++; } + else { + r = -ENOSPC; + smd->begin = *b - 1; + } + spin_unlock(&lock); The lock might protect smd->begin, but I'm not sure how &smd->ll might have been modified by sm_ll_inc(). However, since ll->save_ie() failed in sm_ll_mutate() then perhaps this is safe. What do you think? Putting the pool into PM_OUT_OF_DATA_SPACE isn't ideal since it isn't out of space, but I would take it over a BUG_ON. 2) If example #1 above returned -EAGAIN, how might alloc_data_block be taught retry? This bug shows up weeks or months apart, even on heavily loaded systems with ~100 live thin volumes, so retrying would be fine IMHO. 3) In the thread from June, Marcus says: "Judging from the call trace, my guess is that there is somewhere a race condition, when a new block needs to be allocated which has still to be discarded." Is this discard sitation possible? Wouldn't the bio prison prevent this? -- Eric Wheeler www.datawall.us Here is the new trace, old trace below: kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! invalid opcode: 0000 [#1] SMP NOPTI CPU: 11 PID: 22939 Comm: kworker/u48:1 Not tainted 4.19.86 #1 Hardware name: Supermicro SYS-2026T-6RFT+/X8DTU-6+, BIOS 2.1c 11/30/2012 Workqueue: dm-thin do_worker [dm_thin_pool] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 b0 a5 a9 e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 RSP: 0018:ffffc90007237c78 EFLAGS: 00010297 RAX: 0000000000000000 RBX: ffff88861d7ac000 RCX: 0000000000000000 RDX: ffff8885f3d13f00 RSI: 0000000000000246 RDI: ffff888620e38c00 RBP: 0000000000000000 R08: 0000000000000000 R09: ffff888620e38c98 R10: ffffffff810f9177 R11: 0000000000000000 R12: ffffc90007237d48 R13: ffffc90007237d48 R14: 00000000ffffffc3 R15: 00000000000d9e3a FS: 0000000000000000(0000) GS:ffff8886279c0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000081ceff0 CR3: 000000000200a004 CR4: 00000000000226e0 Call Trace: dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] process_cell+0x2a3/0x550 [dm_thin_pool] ? mempool_alloc+0x6f/0x180 ? sort+0x17b/0x270 ? u32_swap+0x10/0x10 process_deferred_bios+0x1af/0x870 [dm_thin_pool] do_worker+0x94/0xe0 [dm_thin_pool] process_one_work+0x171/0x370 worker_thread+0x49/0x3f0 kthread+0xf8/0x130 ? max_active_store+0x80/0x80 ? kthread_bind+0x10/0x10 ret_from_fork+0x1f/0x40 > > [199391.677689] ------------[ cut here ]------------ > [199391.678437] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! > [199391.679183] invalid opcode: 0000 [#1] SMP NOPTI > [199391.679941] CPU: 4 PID: 31359 Comm: kworker/u16:4 Not tainted 4.19.75 #1 > [199391.680683] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > [199391.681446] Workqueue: dm-thin do_worker [dm_thin_pool] > [199391.682187] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199391.682929] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > [199391.684432] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > [199391.685186] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > [199391.685936] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > [199391.686659] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > [199391.687379] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > [199391.688120] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > [199391.688843] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > [199391.689571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [199391.690253] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > [199391.690984] Call Trace: > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > [199391.693852] ? sort+0x17b/0x270 > [199391.694527] ? u32_swap+0x10/0x10 > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > [199391.695890] process_one_work+0x171/0x370 > [199391.696640] worker_thread+0x49/0x3f0 > [199391.697332] kthread+0xf8/0x130 > [199391.697988] ? max_active_store+0x80/0x80 > [199391.698659] ? kthread_bind+0x10/0x10 > [199391.699281] ret_from_fork+0x1f/0x40 > [199391.699930] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_alg o_bit drm_kms_helper > [199391.705631] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > [199391.708083] ---[ end trace c31536d98046e8ec ]--- > [199391.866776] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199391.867960] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > [199391.870379] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > [199391.871524] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > [199391.872364] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > [199391.873173] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > [199391.873871] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > [199391.874550] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > [199391.875231] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > [199391.875941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [199391.876633] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > [199391.877317] Kernel panic - not syncing: Fatal exception > [199391.878006] Kernel Offset: disabled > [199392.032304] ---[ end Kernel panic - not syncing: Fatal exception ]--- > [199392.032962] unchecked MSR access error: WRMSR to 0x83f (tried to write 0x00000000000000f6) at rIP: 0xffffffff81067af4 (native_write_msr+0x4/0x20) > [199392.034277] Call Trace: > [199392.034929] <IRQ> > [199392.035576] native_apic_msr_write+0x2e/0x40 > [199392.036228] arch_irq_work_raise+0x28/0x40 > [199392.036877] irq_work_queue_on+0x83/0xa0 > [199392.037518] irq_work_run_list+0x4c/0x70 > [199392.038149] irq_work_run+0x14/0x40 > [199392.038771] smp_call_function_single_interrupt+0x3a/0xd0 > [199392.039393] call_function_single_interrupt+0xf/0x20 > [199392.040011] </IRQ> > [199392.040624] RIP: 0010:panic+0x209/0x25c > [199392.041234] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > [199392.042518] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 > [199392.043174] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > [199392.043833] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > [199392.044493] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > [199392.045155] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > [199392.045820] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > [199392.046486] oops_end+0xc1/0xd0 > [199392.047149] do_trap+0x13d/0x150 > [199392.047795] do_error_trap+0xd5/0x130 > [199392.048427] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199392.049048] invalid_op+0x14/0x20 > [199392.049650] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199392.050245] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > [199392.051434] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > [199392.052010] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > [199392.052580] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > [199392.053150] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > [199392.053715] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > [199392.054266] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > [199392.054807] ? __wake_up_common_lock+0x87/0xc0 > [199392.055342] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > [199392.055877] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > [199392.056410] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > [199392.056947] process_cell+0x2a3/0x550 [dm_thin_pool] > [199392.057484] ? sort+0x17b/0x270 > [199392.058016] ? u32_swap+0x10/0x10 > [199392.058538] do_worker+0x268/0x9a0 [dm_thin_pool] > [199392.059060] process_one_work+0x171/0x370 > [199392.059576] worker_thread+0x49/0x3f0 > [199392.060083] kthread+0xf8/0x130 > [199392.060587] ? max_active_store+0x80/0x80 > [199392.061086] ? kthread_bind+0x10/0x10 > [199392.061569] ret_from_fork+0x1f/0x40 > [199392.062038] ------------[ cut here ]------------ > [199392.062508] sched: Unexpected reschedule of offline CPU#1! > [199392.062989] WARNING: CPU: 4 PID: 31359 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40 > [199392.063485] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_alg o_bit drm_kms_helper > [199392.067463] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > [199392.068729] CPU: 4 PID: 31359 Comm: kworker/u16:4 Tainted: G D 4.19.75 #1 > [199392.069356] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > [199392.069982] Workqueue: dm-thin do_worker [dm_thin_pool] > [199392.070604] RIP: 0010:native_smp_send_reschedule+0x39/0x40 > [199392.071229] Code: 0f 92 c0 84 c0 74 15 48 8b 05 93 f1 eb 00 be fd 00 00 00 48 8b 40 30 e9 85 f4 9a 00 89 fe 48 c7 c7 08 23 e5 81 e8 c7 9a 05 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48 > [199392.072528] RSP: 0018:ffff88880fb03dc0 EFLAGS: 00010082 > [199392.073186] RAX: 0000000000000000 RBX: ffff88880fa625c0 RCX: 0000000000000006 > [199392.073847] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff88880fb168b0 > [199392.074512] RBP: ffff88880fa625c0 R08: 0000000000000001 R09: 00000000000174e5 > [199392.075182] R10: ffff8881f2512300 R11: 0000000000000000 R12: ffff888804541640 > [199392.075849] R13: ffff88880fb03e08 R14: 0000000000000000 R15: 0000000000000001 > [199392.076512] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > [199392.077179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [199392.077833] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > [199392.078481] Call Trace: > [199392.079117] <IRQ> > [199392.079745] check_preempt_curr+0x6b/0x90 > [199392.080373] ttwu_do_wakeup+0x19/0x130 > [199392.080999] try_to_wake_up+0x1e2/0x460 > [199392.081623] __wake_up_common+0x8f/0x160 > [199392.082246] ep_poll_callback+0x1af/0x300 > [199392.082860] __wake_up_common+0x8f/0x160 > [199392.083470] __wake_up_common_lock+0x7a/0xc0 > [199392.084074] irq_work_run_list+0x4c/0x70 > [199392.084675] smp_call_function_single_interrupt+0x3a/0xd0 > [199392.085277] call_function_single_interrupt+0xf/0x20 > [199392.085879] </IRQ> > [199392.086477] RIP: 0010:panic+0x209/0x25c > [199392.087079] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > [199392.088341] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 > [199392.088988] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > [199392.089639] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > [199392.090291] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > [199392.090947] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > [199392.091601] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > [199392.092255] oops_end+0xc1/0xd0 > [199392.092894] do_trap+0x13d/0x150 > [199392.093516] do_error_trap+0xd5/0x130 > [199392.094122] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199392.094718] invalid_op+0x14/0x20 > [199392.095296] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199392.095866] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > [199392.097006] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > [199392.097558] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > [199392.098103] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > [199392.098644] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > [199392.099176] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > [199392.099703] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > [199392.100229] ? __wake_up_common_lock+0x87/0xc0 > [199392.100744] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > [199392.101251] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > [199392.101752] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > [199392.102252] process_cell+0x2a3/0x550 [dm_thin_pool] > [199392.102750] ? sort+0x17b/0x270 > [199392.103242] ? u32_swap+0x10/0x10 > [199392.103733] do_worker+0x268/0x9a0 [dm_thin_pool] > [199392.104228] process_one_work+0x171/0x370 > [199392.104714] worker_thread+0x49/0x3f0 > [199392.105193] kthread+0xf8/0x130 > [199392.105665] ? max_active_store+0x80/0x80 > [199392.106132] ? kthread_bind+0x10/0x10 > [199392.106601] ret_from_fork+0x1f/0x40 > [199392.107069] ---[ end trace c31536d98046e8ed ]--- > [199392.107544] ------------[ cut here ]------------ > [199392.108017] sched: Unexpected reschedule of offline CPU#7! > [199392.108497] WARNING: CPU: 4 PID: 31359 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40 > [199392.108996] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_alg o_bit drm_kms_helper > [199392.112967] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > [199392.114203] CPU: 4 PID: 31359 Comm: kworker/u16:4 Tainted: G D W 4.19.75 #1 > [199392.114819] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > [199392.115439] Workqueue: dm-thin do_worker [dm_thin_pool] > [199392.116061] RIP: 0010:native_smp_send_reschedule+0x39/0x40 > [199392.116683] Code: 0f 92 c0 84 c0 74 15 48 8b 05 93 f1 eb 00 be fd 00 00 00 48 8b 40 30 e9 85 f4 9a 00 89 fe 48 c7 c7 08 23 e5 81 e8 c7 9a 05 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48 > [199392.117982] RSP: 0018:ffff88880fb03ee0 EFLAGS: 00010082 > [199392.118632] RAX: 0000000000000000 RBX: ffff8887d093ac80 RCX: 0000000000000006 > [199392.119295] RDX: 0000000000000000 RSI: 0000000000000092 RDI: ffff88880fb168b0 > [199392.119961] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000017529 > [199392.120623] R10: ffff8881438ba800 R11: 0000000000000000 R12: ffffc9000a147988 > [199392.121283] R13: ffffffff8113f9a0 R14: 0000000000000002 R15: ffff88880fb1cff8 > [199392.121938] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > [199392.122589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [199392.123229] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > [199392.123867] Call Trace: > [199392.124495] <IRQ> > [199392.125116] update_process_times+0x40/0x50 > [199392.125742] tick_sched_handle+0x25/0x60 > [199392.126367] tick_sched_timer+0x37/0x70 > [199392.126987] __hrtimer_run_queues+0xfb/0x270 > [199392.127601] hrtimer_interrupt+0x122/0x270 > [199392.128211] smp_apic_timer_interrupt+0x6a/0x140 > [199392.128819] apic_timer_interrupt+0xf/0x20 > [199392.129423] </IRQ> > [199392.130020] RIP: 0010:panic+0x209/0x25c > [199392.130619] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > [199392.131883] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 > [199392.132530] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > [199392.133180] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > [199392.133830] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > [199392.134482] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > [199392.135136] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > [199392.135792] oops_end+0xc1/0xd0 > [199392.136444] do_trap+0x13d/0x150 > [199392.137094] do_error_trap+0xd5/0x130 > [199392.137740] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199392.138383] invalid_op+0x14/0x20 > [199392.139010] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > [199392.139630] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > [199392.140870] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > [199392.141472] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > [199392.142066] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > [199392.142647] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > [199392.143211] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > [199392.143759] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > [199392.144301] ? __wake_up_common_lock+0x87/0xc0 > [199392.144838] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > [199392.145374] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > [199392.145909] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > [199392.146435] process_cell+0x2a3/0x550 [dm_thin_pool] > [199392.146949] ? sort+0x17b/0x270 > [199392.147450] ? u32_swap+0x10/0x10 > [199392.147944] do_worker+0x268/0x9a0 [dm_thin_pool] > [199392.148441] process_one_work+0x171/0x370 > [199392.148937] worker_thread+0x49/0x3f0 > [199392.149430] kthread+0xf8/0x130 > [199392.149922] ? max_active_store+0x80/0x80 > [199392.150406] ? kthread_bind+0x10/0x10 > [199392.150883] ret_from_fork+0x1f/0x40 > [199392.151353] ---[ end trace c31536d98046e8ee ]--- > > > -- > Eric Wheeler > > -- > dm-devel mailing list > dm-devel@redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel > -- lvm-devel mailing list lvm-devel@redhat.com https://www.redhat.com/mailman/listinfo/lvm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 2019-12-20 19:54 ` Eric Wheeler (?) @ 2019-12-27 1:47 ` Eric Wheeler -1 siblings, 0 replies; 43+ messages in thread From: Eric Wheeler @ 2019-12-27 1:47 UTC (permalink / raw) To: lvm-devel Cc: ejt, Mike Snitzer, markus.schade, linux-block, dm-devel, joe.thornber On Fri, 20 Dec 2019, Eric Wheeler wrote: > On Wed, 25 Sep 2019, Eric Wheeler wrote: > > We are using the 4.19.75 stable tree with dm-thin and multi-queue scsi. > > We have been using the 4.19 branch for months without issue; we just > > switched to MQ and we seem to have hit this BUG_ON. Whether or not MQ is > > related to the issue, I don't know, maybe coincidence: > > > > static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > > { > > int r; > > enum allocation_event ev; > > struct sm_disk *smd = container_of(sm, struct sm_disk, sm); > > > > /* FIXME: we should loop round a couple of times */ > > r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); > > if (r) > > return r; > > > > smd->begin = *b + 1; > > r = sm_ll_inc(&smd->ll, *b, &ev); > > if (!r) { > > BUG_ON(ev != SM_ALLOC); <<<<<<<<<<<<<<<<<<<<<<<<<<<<< > > smd->nr_allocated_this_transaction++; > > } > > Hello all, > > We hit this BUG_ON again, this time with 4.19.86 with > scsi_mod.use_blk_mq=y, and it is known to be present as of 5.1.2 as > additionally reported by Markus Schade: > > https://www.redhat.com/archives/dm-devel/2019-June/msg00116.html > and > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1777398 > > In our case, the most latest trace (below) is from a different system that > has been stable for years on Linux 4.1 with tmeta direct on the SSD. > We updated to 4.19.86 a few weeks ago and just hit this, what Mike > Snitzer explains to be an allocator race: > > On Wed, 25 Sep 2019, Mike Snitzer wrote: > > > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > > > [199391.693852] ? sort+0x17b/0x270 > > > [199391.694527] ? u32_swap+0x10/0x10 > > > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > > > [199391.695890] process_one_work+0x171/0x370 > > > [199391.696640] worker_thread+0x49/0x3f0 > > > [199391.697332] kthread+0xf8/0x130 > > > [199391.697988] ? max_active_store+0x80/0x80 > > > [199391.698659] ? kthread_bind+0x10/0x10 > > > [199391.699281] ret_from_fork+0x1f/0x40 > > > > The stack shows the call to sm_disk_new_block() is due to > > dm_pool_alloc_data_block(). > > > > sm_disk_new_block()'s BUG_ON(ev != SM_ALLOC) indicates that somehow it is > > getting called without the passed 'ev' being set to SM_ALLOC. Only > > drivers/md/persistent-dat/dm-space-map-common.c:sm_ll_mutate() sets > > SM_ALLOC. sm_disk_new_block() is indirectly calling sm_ll_mutate() > > > > sm_ll_mutate() will only return 0 if ll->save_ie() does, the ll_disk *ll > > should be ll_disk, and so disk_ll_save_ie()'s call to dm_btree_insert() > > returns 0 -- which simply means success. And on success > > sm_disk_new_block() assumes ev was set to SM_ALLOC (by sm_ll_mutate). > > > > sm_ll_mutate() decided to _not_ set SM_ALLOC because either: > > 1) ref_count wasn't set > > or > > 2) old was identified > > > > So all said: somehow a new data block was found to already be in use. > > _WHY_ that is the case isn't clear from this stack... > > > > But it does speak to the possibility of data block allocation racing > > with other operations to the same block. Which implies missing locking. > > Where would you look to add locking do you think? > > > But that's all I've got so far... I'll review past dm-thinp changes with > > all this in mind and see what turns up. But Joe Thornber (ejt) likely > > needs to have a look at this too. > > > > But could it be that bcache is the source of the data device race (same > > block used concurrently)? And DM thinp is acting as the canary in the > > coal mine? > > As Marcus has shown, this bug triggers without bcache. > > > Other questions: > > 1) Can sm_disk_new_block() fail more gracefully than BUG_ON? For example: > > + spin_lock(&lock); /* protect smd->begin */ > smd->begin = *b + 1; > r = sm_ll_inc(&smd->ll, *b, &ev); > if (!r) { > - BUG_ON(ev != SM_ALLOC); > smd->nr_allocated_this_transaction++; > } > + else { > + r = -ENOSPC; > + smd->begin = *b - 1; > + } > + spin_unlock(&lock); Just hit the bug again without mq-scsi (scsi_mod.use_blk_mq=n), all other times MQ has been turned on. I'm trying the -ENOSPC hack which will flag the pool as being out of space so I can recover more gracefully than a BUG_ON. Here's a first-draft patch, maybe the spinlock will even prevent the issue. Compile tested, I'll try on a real system tomorrow. Comments welcome: diff --git a/drivers/md/persistent-data/dm-space-map-disk.c b/drivers/md/persistent-data/dm-space-map-disk.c index 32adf6b..cb27a20 100644 --- a/drivers/md/persistent-data/dm-space-map-disk.c +++ b/drivers/md/persistent-data/dm-space-map-disk.c @@ -161,6 +161,7 @@ static int sm_disk_dec_block(struct dm_space_map *sm, dm_block_t b) return r; } +static DEFINE_SPINLOCK(smd_lock); static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) { int r; @@ -168,17 +169,30 @@ static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) struct sm_disk *smd = container_of(sm, struct sm_disk, sm); /* FIXME: we should loop round a couple of times */ + spin_lock(&smd_lock); r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); - if (r) + if (r) { + spin_unlock(&smd_lock); return r; + } smd->begin = *b + 1; r = sm_ll_inc(&smd->ll, *b, &ev); if (!r) { - BUG_ON(ev != SM_ALLOC); - smd->nr_allocated_this_transaction++; + if (ev == SM_ALLOC) + smd->nr_allocated_this_transaction++; + else { + /* Not actually out of space, this is a bug: + * https://lore.kernel.org/linux-block/20190925200138.GA20584@redhat.com/ + */ + WARN(ev != SM_ALLOC, "Pool metadata allocation race, marking pool out-of-space."); + r = -ENOSPC; + smd->begin = *b - 1; + } } + spin_unlock(&smd_lock); + return r; } -- Eric Wheeler > > The lock might protect smd->begin, but I'm not sure how &smd->ll might > have been modified by sm_ll_inc(). However, since ll->save_ie() failed in > sm_ll_mutate() then perhaps this is safe. What do you think? > > Putting the pool into PM_OUT_OF_DATA_SPACE isn't ideal since it isn't out > of space, but I would take it over a BUG_ON. > > 2) If example #1 above returned -EAGAIN, how might alloc_data_block be > taught retry? This bug shows up weeks or months apart, even on heavily > loaded systems with ~100 live thin volumes, so retrying would be fine > IMHO. > > 3) In the thread from June, Marcus says: > "Judging from the call trace, my guess is that there is somewhere > a race condition, when a new block needs to be allocated which has > still to be discarded." > > Is this discard sitation possible? Wouldn't the bio prison prevent this? > > -- > Eric Wheeler > www.datawall.us > > > > Here is the new trace, old trace below: > > kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! > invalid opcode: 0000 [#1] SMP NOPTI > CPU: 11 PID: 22939 Comm: kworker/u48:1 Not tainted 4.19.86 #1 > Hardware name: Supermicro SYS-2026T-6RFT+/X8DTU-6+, BIOS 2.1c 11/30/2012 > Workqueue: dm-thin do_worker [dm_thin_pool] > RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 b0 a5 a9 e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 > RSP: 0018:ffffc90007237c78 EFLAGS: 00010297 > RAX: 0000000000000000 RBX: ffff88861d7ac000 RCX: 0000000000000000 > RDX: ffff8885f3d13f00 RSI: 0000000000000246 RDI: ffff888620e38c00 > RBP: 0000000000000000 R08: 0000000000000000 R09: ffff888620e38c98 > R10: ffffffff810f9177 R11: 0000000000000000 R12: ffffc90007237d48 > R13: ffffc90007237d48 R14: 00000000ffffffc3 R15: 00000000000d9e3a > FS: 0000000000000000(0000) GS:ffff8886279c0000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 00000000081ceff0 CR3: 000000000200a004 CR4: 00000000000226e0 > Call Trace: > dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > process_cell+0x2a3/0x550 [dm_thin_pool] > ? mempool_alloc+0x6f/0x180 > ? sort+0x17b/0x270 > ? u32_swap+0x10/0x10 > process_deferred_bios+0x1af/0x870 [dm_thin_pool] > do_worker+0x94/0xe0 [dm_thin_pool] > process_one_work+0x171/0x370 > worker_thread+0x49/0x3f0 > kthread+0xf8/0x130 > ? max_active_store+0x80/0x80 > ? kthread_bind+0x10/0x10 > ret_from_fork+0x1f/0x40 > > > > > [199391.677689] ------------[ cut here ]------------ > > [199391.678437] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! > > [199391.679183] invalid opcode: 0000 [#1] SMP NOPTI > > [199391.679941] CPU: 4 PID: 31359 Comm: kworker/u16:4 Not tainted 4.19.75 #1 > > [199391.680683] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > > [199391.681446] Workqueue: dm-thin do_worker [dm_thin_pool] > > [199391.682187] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199391.682929] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > [199391.684432] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > [199391.685186] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > [199391.685936] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > [199391.686659] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > [199391.687379] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > [199391.688120] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > [199391.688843] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > [199391.689571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [199391.690253] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > [199391.690984] Call Trace: > > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > > [199391.693852] ? sort+0x17b/0x270 > > [199391.694527] ? u32_swap+0x10/0x10 > > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > > [199391.695890] process_one_work+0x171/0x370 > > [199391.696640] worker_thread+0x49/0x3f0 > > [199391.697332] kthread+0xf8/0x130 > > [199391.697988] ? max_active_store+0x80/0x80 > > [199391.698659] ? kthread_bind+0x10/0x10 > > [199391.699281] ret_from_fork+0x1f/0x40 > > [199391.699930] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_bit drm_kms_helper > > [199391.705631] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > > [199391.708083] ---[ end trace c31536d98046e8ec ]--- > > [199391.866776] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199391.867960] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > [199391.870379] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > [199391.871524] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > [199391.872364] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > [199391.873173] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > [199391.873871] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > [199391.874550] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > [199391.875231] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > [199391.875941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [199391.876633] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > [199391.877317] Kernel panic - not syncing: Fatal exception > > [199391.878006] Kernel Offset: disabled > > [199392.032304] ---[ end Kernel panic - not syncing: Fatal exception ]--- > > [199392.032962] unchecked MSR access error: WRMSR to 0x83f (tried to write 0x00000000000000f6) at rIP: 0xffffffff81067af4 (native_write_msr+0x4/0x20) > > [199392.034277] Call Trace: > > [199392.034929] <IRQ> > > [199392.035576] native_apic_msr_write+0x2e/0x40 > > [199392.036228] arch_irq_work_raise+0x28/0x40 > > [199392.036877] irq_work_queue_on+0x83/0xa0 > > [199392.037518] irq_work_run_list+0x4c/0x70 > > [199392.038149] irq_work_run+0x14/0x40 > > [199392.038771] smp_call_function_single_interrupt+0x3a/0xd0 > > [199392.039393] call_function_single_interrupt+0xf/0x20 > > [199392.040011] </IRQ> > > [199392.040624] RIP: 0010:panic+0x209/0x25c > > [199392.041234] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > > [199392.042518] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 > > [199392.043174] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > > [199392.043833] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > > [199392.044493] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > > [199392.045155] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > > [199392.045820] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > > [199392.046486] oops_end+0xc1/0xd0 > > [199392.047149] do_trap+0x13d/0x150 > > [199392.047795] do_error_trap+0xd5/0x130 > > [199392.048427] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199392.049048] invalid_op+0x14/0x20 > > [199392.049650] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199392.050245] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > [199392.051434] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > [199392.052010] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > [199392.052580] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > [199392.053150] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > [199392.053715] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > [199392.054266] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > [199392.054807] ? __wake_up_common_lock+0x87/0xc0 > > [199392.055342] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > > [199392.055877] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > [199392.056410] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > [199392.056947] process_cell+0x2a3/0x550 [dm_thin_pool] > > [199392.057484] ? sort+0x17b/0x270 > > [199392.058016] ? u32_swap+0x10/0x10 > > [199392.058538] do_worker+0x268/0x9a0 [dm_thin_pool] > > [199392.059060] process_one_work+0x171/0x370 > > [199392.059576] worker_thread+0x49/0x3f0 > > [199392.060083] kthread+0xf8/0x130 > > [199392.060587] ? max_active_store+0x80/0x80 > > [199392.061086] ? kthread_bind+0x10/0x10 > > [199392.061569] ret_from_fork+0x1f/0x40 > > [199392.062038] ------------[ cut here ]------------ > > [199392.062508] sched: Unexpected reschedule of offline CPU#1! > > [199392.062989] WARNING: CPU: 4 PID: 31359 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40 > > [199392.063485] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_bit drm_kms_helper > > [199392.067463] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > > [199392.068729] CPU: 4 PID: 31359 Comm: kworker/u16:4 Tainted: G D 4.19.75 #1 > > [199392.069356] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > > [199392.069982] Workqueue: dm-thin do_worker [dm_thin_pool] > > [199392.070604] RIP: 0010:native_smp_send_reschedule+0x39/0x40 > > [199392.071229] Code: 0f 92 c0 84 c0 74 15 48 8b 05 93 f1 eb 00 be fd 00 00 00 48 8b 40 30 e9 85 f4 9a 00 89 fe 48 c7 c7 08 23 e5 81 e8 c7 9a 05 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48 > > [199392.072528] RSP: 0018:ffff88880fb03dc0 EFLAGS: 00010082 > > [199392.073186] RAX: 0000000000000000 RBX: ffff88880fa625c0 RCX: 0000000000000006 > > [199392.073847] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff88880fb168b0 > > [199392.074512] RBP: ffff88880fa625c0 R08: 0000000000000001 R09: 00000000000174e5 > > [199392.075182] R10: ffff8881f2512300 R11: 0000000000000000 R12: ffff888804541640 > > [199392.075849] R13: ffff88880fb03e08 R14: 0000000000000000 R15: 0000000000000001 > > [199392.076512] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > [199392.077179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [199392.077833] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > [199392.078481] Call Trace: > > [199392.079117] <IRQ> > > [199392.079745] check_preempt_curr+0x6b/0x90 > > [199392.080373] ttwu_do_wakeup+0x19/0x130 > > [199392.080999] try_to_wake_up+0x1e2/0x460 > > [199392.081623] __wake_up_common+0x8f/0x160 > > [199392.082246] ep_poll_callback+0x1af/0x300 > > [199392.082860] __wake_up_common+0x8f/0x160 > > [199392.083470] __wake_up_common_lock+0x7a/0xc0 > > [199392.084074] irq_work_run_list+0x4c/0x70 > > [199392.084675] smp_call_function_single_interrupt+0x3a/0xd0 > > [199392.085277] call_function_single_interrupt+0xf/0x20 > > [199392.085879] </IRQ> > > [199392.086477] RIP: 0010:panic+0x209/0x25c > > [199392.087079] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > > [199392.088341] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 > > [199392.088988] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > > [199392.089639] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > > [199392.090291] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > > [199392.090947] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > > [199392.091601] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > > [199392.092255] oops_end+0xc1/0xd0 > > [199392.092894] do_trap+0x13d/0x150 > > [199392.093516] do_error_trap+0xd5/0x130 > > [199392.094122] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199392.094718] invalid_op+0x14/0x20 > > [199392.095296] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199392.095866] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > [199392.097006] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > [199392.097558] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > [199392.098103] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > [199392.098644] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > [199392.099176] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > [199392.099703] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > [199392.100229] ? __wake_up_common_lock+0x87/0xc0 > > [199392.100744] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > > [199392.101251] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > [199392.101752] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > [199392.102252] process_cell+0x2a3/0x550 [dm_thin_pool] > > [199392.102750] ? sort+0x17b/0x270 > > [199392.103242] ? u32_swap+0x10/0x10 > > [199392.103733] do_worker+0x268/0x9a0 [dm_thin_pool] > > [199392.104228] process_one_work+0x171/0x370 > > [199392.104714] worker_thread+0x49/0x3f0 > > [199392.105193] kthread+0xf8/0x130 > > [199392.105665] ? max_active_store+0x80/0x80 > > [199392.106132] ? kthread_bind+0x10/0x10 > > [199392.106601] ret_from_fork+0x1f/0x40 > > [199392.107069] ---[ end trace c31536d98046e8ed ]--- > > [199392.107544] ------------[ cut here ]------------ > > [199392.108017] sched: Unexpected reschedule of offline CPU#7! > > [199392.108497] WARNING: CPU: 4 PID: 31359 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40 > > [199392.108996] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_bit drm_kms_helper > > [199392.112967] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > > [199392.114203] CPU: 4 PID: 31359 Comm: kworker/u16:4 Tainted: G D W 4.19.75 #1 > > [199392.114819] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > > [199392.115439] Workqueue: dm-thin do_worker [dm_thin_pool] > > [199392.116061] RIP: 0010:native_smp_send_reschedule+0x39/0x40 > > [199392.116683] Code: 0f 92 c0 84 c0 74 15 48 8b 05 93 f1 eb 00 be fd 00 00 00 48 8b 40 30 e9 85 f4 9a 00 89 fe 48 c7 c7 08 23 e5 81 e8 c7 9a 05 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48 > > [199392.117982] RSP: 0018:ffff88880fb03ee0 EFLAGS: 00010082 > > [199392.118632] RAX: 0000000000000000 RBX: ffff8887d093ac80 RCX: 0000000000000006 > > [199392.119295] RDX: 0000000000000000 RSI: 0000000000000092 RDI: ffff88880fb168b0 > > [199392.119961] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000017529 > > [199392.120623] R10: ffff8881438ba800 R11: 0000000000000000 R12: ffffc9000a147988 > > [199392.121283] R13: ffffffff8113f9a0 R14: 0000000000000002 R15: ffff88880fb1cff8 > > [199392.121938] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > [199392.122589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [199392.123229] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > [199392.123867] Call Trace: > > [199392.124495] <IRQ> > > [199392.125116] update_process_times+0x40/0x50 > > [199392.125742] tick_sched_handle+0x25/0x60 > > [199392.126367] tick_sched_timer+0x37/0x70 > > [199392.126987] __hrtimer_run_queues+0xfb/0x270 > > [199392.127601] hrtimer_interrupt+0x122/0x270 > > [199392.128211] smp_apic_timer_interrupt+0x6a/0x140 > > [199392.128819] apic_timer_interrupt+0xf/0x20 > > [199392.129423] </IRQ> > > [199392.130020] RIP: 0010:panic+0x209/0x25c > > [199392.130619] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > > [199392.131883] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 > > [199392.132530] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > > [199392.133180] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > > [199392.133830] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > > [199392.134482] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > > [199392.135136] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > > [199392.135792] oops_end+0xc1/0xd0 > > [199392.136444] do_trap+0x13d/0x150 > > [199392.137094] do_error_trap+0xd5/0x130 > > [199392.137740] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199392.138383] invalid_op+0x14/0x20 > > [199392.139010] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199392.139630] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > [199392.140870] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > [199392.141472] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > [199392.142066] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > [199392.142647] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > [199392.143211] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > [199392.143759] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > [199392.144301] ? __wake_up_common_lock+0x87/0xc0 > > [199392.144838] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > > [199392.145374] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > [199392.145909] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > [199392.146435] process_cell+0x2a3/0x550 [dm_thin_pool] > > [199392.146949] ? sort+0x17b/0x270 > > [199392.147450] ? u32_swap+0x10/0x10 > > [199392.147944] do_worker+0x268/0x9a0 [dm_thin_pool] > > [199392.148441] process_one_work+0x171/0x370 > > [199392.148937] worker_thread+0x49/0x3f0 > > [199392.149430] kthread+0xf8/0x130 > > [199392.149922] ? max_active_store+0x80/0x80 > > [199392.150406] ? kthread_bind+0x10/0x10 > > [199392.150883] ret_from_fork+0x1f/0x40 > > [199392.151353] ---[ end trace c31536d98046e8ee ]--- > > > > > > -- > > Eric Wheeler > > > > -- > > dm-devel mailing list > > dm-devel@redhat.com > > https://www.redhat.com/mailman/listinfo/dm-devel > > > > > -- > dm-devel mailing list > dm-devel@redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel > > ^ permalink raw reply related [flat|nested] 43+ messages in thread
* [dm-devel] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 @ 2019-12-27 1:47 ` Eric Wheeler 0 siblings, 0 replies; 43+ messages in thread From: Eric Wheeler @ 2019-12-27 1:47 UTC (permalink / raw) To: lvm-devel On Fri, 20 Dec 2019, Eric Wheeler wrote: > On Wed, 25 Sep 2019, Eric Wheeler wrote: > > We are using the 4.19.75 stable tree with dm-thin and multi-queue scsi. > > We have been using the 4.19 branch for months without issue; we just > > switched to MQ and we seem to have hit this BUG_ON. Whether or not MQ is > > related to the issue, I don't know, maybe coincidence: > > > > static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > > { > > int r; > > enum allocation_event ev; > > struct sm_disk *smd = container_of(sm, struct sm_disk, sm); > > > > /* FIXME: we should loop round a couple of times */ > > r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); > > if (r) > > return r; > > > > smd->begin = *b + 1; > > r = sm_ll_inc(&smd->ll, *b, &ev); > > if (!r) { > > BUG_ON(ev != SM_ALLOC); <<<<<<<<<<<<<<<<<<<<<<<<<<<<< > > smd->nr_allocated_this_transaction++; > > } > > Hello all, > > We hit this BUG_ON again, this time with 4.19.86 with > scsi_mod.use_blk_mq=y, and it is known to be present as of 5.1.2 as > additionally reported by Markus Schade: > > https://www.redhat.com/archives/dm-devel/2019-June/msg00116.html > and > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1777398 > > In our case, the most latest trace (below) is from a different system that > has been stable for years on Linux 4.1 with tmeta direct on the SSD. > We updated to 4.19.86 a few weeks ago and just hit this, what Mike > Snitzer explains to be an allocator race: > > On Wed, 25 Sep 2019, Mike Snitzer wrote: > > > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > > > [199391.693852] ? sort+0x17b/0x270 > > > [199391.694527] ? u32_swap+0x10/0x10 > > > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > > > [199391.695890] process_one_work+0x171/0x370 > > > [199391.696640] worker_thread+0x49/0x3f0 > > > [199391.697332] kthread+0xf8/0x130 > > > [199391.697988] ? max_active_store+0x80/0x80 > > > [199391.698659] ? kthread_bind+0x10/0x10 > > > [199391.699281] ret_from_fork+0x1f/0x40 > > > > The stack shows the call to sm_disk_new_block() is due to > > dm_pool_alloc_data_block(). > > > > sm_disk_new_block()'s BUG_ON(ev != SM_ALLOC) indicates that somehow it is > > getting called without the passed 'ev' being set to SM_ALLOC. Only > > drivers/md/persistent-dat/dm-space-map-common.c:sm_ll_mutate() sets > > SM_ALLOC. sm_disk_new_block() is indirectly calling sm_ll_mutate() > > > > sm_ll_mutate() will only return 0 if ll->save_ie() does, the ll_disk *ll > > should be ll_disk, and so disk_ll_save_ie()'s call to dm_btree_insert() > > returns 0 -- which simply means success. And on success > > sm_disk_new_block() assumes ev was set to SM_ALLOC (by sm_ll_mutate). > > > > sm_ll_mutate() decided to _not_ set SM_ALLOC because either: > > 1) ref_count wasn't set > > or > > 2) old was identified > > > > So all said: somehow a new data block was found to already be in use. > > _WHY_ that is the case isn't clear from this stack... > > > > But it does speak to the possibility of data block allocation racing > > with other operations to the same block. Which implies missing locking. > > Where would you look to add locking do you think? > > > But that's all I've got so far... I'll review past dm-thinp changes with > > all this in mind and see what turns up. But Joe Thornber (ejt) likely > > needs to have a look at this too. > > > > But could it be that bcache is the source of the data device race (same > > block used concurrently)? And DM thinp is acting as the canary in the > > coal mine? > > As Marcus has shown, this bug triggers without bcache. > > > Other questions: > > 1) Can sm_disk_new_block() fail more gracefully than BUG_ON? For example: > > + spin_lock(&lock); /* protect smd->begin */ > smd->begin = *b + 1; > r = sm_ll_inc(&smd->ll, *b, &ev); > if (!r) { > - BUG_ON(ev != SM_ALLOC); > smd->nr_allocated_this_transaction++; > } > + else { > + r = -ENOSPC; > + smd->begin = *b - 1; > + } > + spin_unlock(&lock); Just hit the bug again without mq-scsi (scsi_mod.use_blk_mq=n), all other times MQ has been turned on. I'm trying the -ENOSPC hack which will flag the pool as being out of space so I can recover more gracefully than a BUG_ON. Here's a first-draft patch, maybe the spinlock will even prevent the issue. Compile tested, I'll try on a real system tomorrow. Comments welcome: diff --git a/drivers/md/persistent-data/dm-space-map-disk.c b/drivers/md/persistent-data/dm-space-map-disk.c index 32adf6b..cb27a20 100644 --- a/drivers/md/persistent-data/dm-space-map-disk.c +++ b/drivers/md/persistent-data/dm-space-map-disk.c @@ -161,6 +161,7 @@ static int sm_disk_dec_block(struct dm_space_map *sm, dm_block_t b) return r; } +static DEFINE_SPINLOCK(smd_lock); static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) { int r; @@ -168,17 +169,30 @@ static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) struct sm_disk *smd = container_of(sm, struct sm_disk, sm); /* FIXME: we should loop round a couple of times */ + spin_lock(&smd_lock); r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); - if (r) + if (r) { + spin_unlock(&smd_lock); return r; + } smd->begin = *b + 1; r = sm_ll_inc(&smd->ll, *b, &ev); if (!r) { - BUG_ON(ev != SM_ALLOC); - smd->nr_allocated_this_transaction++; + if (ev == SM_ALLOC) + smd->nr_allocated_this_transaction++; + else { + /* Not actually out of space, this is a bug: + * https://lore.kernel.org/linux-block/20190925200138.GA20584 at redhat.com/ + */ + WARN(ev != SM_ALLOC, "Pool metadata allocation race, marking pool out-of-space."); + r = -ENOSPC; + smd->begin = *b - 1; + } } + spin_unlock(&smd_lock); + return r; } -- Eric Wheeler > > The lock might protect smd->begin, but I'm not sure how &smd->ll might > have been modified by sm_ll_inc(). However, since ll->save_ie() failed in > sm_ll_mutate() then perhaps this is safe. What do you think? > > Putting the pool into PM_OUT_OF_DATA_SPACE isn't ideal since it isn't out > of space, but I would take it over a BUG_ON. > > 2) If example #1 above returned -EAGAIN, how might alloc_data_block be > taught retry? This bug shows up weeks or months apart, even on heavily > loaded systems with ~100 live thin volumes, so retrying would be fine > IMHO. > > 3) In the thread from June, Marcus says: > "Judging from the call trace, my guess is that there is somewhere > a race condition, when a new block needs to be allocated which has > still to be discarded." > > Is this discard sitation possible? Wouldn't the bio prison prevent this? > > -- > Eric Wheeler > www.datawall.us > > > > Here is the new trace, old trace below: > > kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! > invalid opcode: 0000 [#1] SMP NOPTI > CPU: 11 PID: 22939 Comm: kworker/u48:1 Not tainted 4.19.86 #1 > Hardware name: Supermicro SYS-2026T-6RFT+/X8DTU-6+, BIOS 2.1c 11/30/2012 > Workqueue: dm-thin do_worker [dm_thin_pool] > RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 b0 a5 a9 e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 > RSP: 0018:ffffc90007237c78 EFLAGS: 00010297 > RAX: 0000000000000000 RBX: ffff88861d7ac000 RCX: 0000000000000000 > RDX: ffff8885f3d13f00 RSI: 0000000000000246 RDI: ffff888620e38c00 > RBP: 0000000000000000 R08: 0000000000000000 R09: ffff888620e38c98 > R10: ffffffff810f9177 R11: 0000000000000000 R12: ffffc90007237d48 > R13: ffffc90007237d48 R14: 00000000ffffffc3 R15: 00000000000d9e3a > FS: 0000000000000000(0000) GS:ffff8886279c0000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 00000000081ceff0 CR3: 000000000200a004 CR4: 00000000000226e0 > Call Trace: > dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > process_cell+0x2a3/0x550 [dm_thin_pool] > ? mempool_alloc+0x6f/0x180 > ? sort+0x17b/0x270 > ? u32_swap+0x10/0x10 > process_deferred_bios+0x1af/0x870 [dm_thin_pool] > do_worker+0x94/0xe0 [dm_thin_pool] > process_one_work+0x171/0x370 > worker_thread+0x49/0x3f0 > kthread+0xf8/0x130 > ? max_active_store+0x80/0x80 > ? kthread_bind+0x10/0x10 > ret_from_fork+0x1f/0x40 > > > > > [199391.677689] ------------[ cut here ]------------ > > [199391.678437] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! > > [199391.679183] invalid opcode: 0000 [#1] SMP NOPTI > > [199391.679941] CPU: 4 PID: 31359 Comm: kworker/u16:4 Not tainted 4.19.75 #1 > > [199391.680683] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > > [199391.681446] Workqueue: dm-thin do_worker [dm_thin_pool] > > [199391.682187] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199391.682929] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > [199391.684432] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > [199391.685186] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > [199391.685936] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > [199391.686659] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > [199391.687379] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > [199391.688120] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > [199391.688843] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > [199391.689571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [199391.690253] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > [199391.690984] Call Trace: > > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > > [199391.693852] ? sort+0x17b/0x270 > > [199391.694527] ? u32_swap+0x10/0x10 > > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > > [199391.695890] process_one_work+0x171/0x370 > > [199391.696640] worker_thread+0x49/0x3f0 > > [199391.697332] kthread+0xf8/0x130 > > [199391.697988] ? max_active_store+0x80/0x80 > > [199391.698659] ? kthread_bind+0x10/0x10 > > [199391.699281] ret_from_fork+0x1f/0x40 > > [199391.699930] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_bit drm_kms_helper > > [199391.705631] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > > [199391.708083] ---[ end trace c31536d98046e8ec ]--- > > [199391.866776] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199391.867960] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > [199391.870379] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > [199391.871524] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > [199391.872364] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > [199391.873173] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > [199391.873871] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > [199391.874550] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > [199391.875231] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > [199391.875941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [199391.876633] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > [199391.877317] Kernel panic - not syncing: Fatal exception > > [199391.878006] Kernel Offset: disabled > > [199392.032304] ---[ end Kernel panic - not syncing: Fatal exception ]--- > > [199392.032962] unchecked MSR access error: WRMSR to 0x83f (tried to write 0x00000000000000f6) at rIP: 0xffffffff81067af4 (native_write_msr+0x4/0x20) > > [199392.034277] Call Trace: > > [199392.034929] <IRQ> > > [199392.035576] native_apic_msr_write+0x2e/0x40 > > [199392.036228] arch_irq_work_raise+0x28/0x40 > > [199392.036877] irq_work_queue_on+0x83/0xa0 > > [199392.037518] irq_work_run_list+0x4c/0x70 > > [199392.038149] irq_work_run+0x14/0x40 > > [199392.038771] smp_call_function_single_interrupt+0x3a/0xd0 > > [199392.039393] call_function_single_interrupt+0xf/0x20 > > [199392.040011] </IRQ> > > [199392.040624] RIP: 0010:panic+0x209/0x25c > > [199392.041234] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > > [199392.042518] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 > > [199392.043174] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > > [199392.043833] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > > [199392.044493] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > > [199392.045155] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > > [199392.045820] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > > [199392.046486] oops_end+0xc1/0xd0 > > [199392.047149] do_trap+0x13d/0x150 > > [199392.047795] do_error_trap+0xd5/0x130 > > [199392.048427] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199392.049048] invalid_op+0x14/0x20 > > [199392.049650] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199392.050245] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > [199392.051434] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > [199392.052010] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > [199392.052580] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > [199392.053150] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > [199392.053715] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > [199392.054266] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > [199392.054807] ? __wake_up_common_lock+0x87/0xc0 > > [199392.055342] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > > [199392.055877] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > [199392.056410] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > [199392.056947] process_cell+0x2a3/0x550 [dm_thin_pool] > > [199392.057484] ? sort+0x17b/0x270 > > [199392.058016] ? u32_swap+0x10/0x10 > > [199392.058538] do_worker+0x268/0x9a0 [dm_thin_pool] > > [199392.059060] process_one_work+0x171/0x370 > > [199392.059576] worker_thread+0x49/0x3f0 > > [199392.060083] kthread+0xf8/0x130 > > [199392.060587] ? max_active_store+0x80/0x80 > > [199392.061086] ? kthread_bind+0x10/0x10 > > [199392.061569] ret_from_fork+0x1f/0x40 > > [199392.062038] ------------[ cut here ]------------ > > [199392.062508] sched: Unexpected reschedule of offline CPU#1! > > [199392.062989] WARNING: CPU: 4 PID: 31359 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40 > > [199392.063485] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_bit drm_kms_helper > > [199392.067463] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > > [199392.068729] CPU: 4 PID: 31359 Comm: kworker/u16:4 Tainted: G D 4.19.75 #1 > > [199392.069356] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > > [199392.069982] Workqueue: dm-thin do_worker [dm_thin_pool] > > [199392.070604] RIP: 0010:native_smp_send_reschedule+0x39/0x40 > > [199392.071229] Code: 0f 92 c0 84 c0 74 15 48 8b 05 93 f1 eb 00 be fd 00 00 00 48 8b 40 30 e9 85 f4 9a 00 89 fe 48 c7 c7 08 23 e5 81 e8 c7 9a 05 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48 > > [199392.072528] RSP: 0018:ffff88880fb03dc0 EFLAGS: 00010082 > > [199392.073186] RAX: 0000000000000000 RBX: ffff88880fa625c0 RCX: 0000000000000006 > > [199392.073847] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff88880fb168b0 > > [199392.074512] RBP: ffff88880fa625c0 R08: 0000000000000001 R09: 00000000000174e5 > > [199392.075182] R10: ffff8881f2512300 R11: 0000000000000000 R12: ffff888804541640 > > [199392.075849] R13: ffff88880fb03e08 R14: 0000000000000000 R15: 0000000000000001 > > [199392.076512] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > [199392.077179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [199392.077833] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > [199392.078481] Call Trace: > > [199392.079117] <IRQ> > > [199392.079745] check_preempt_curr+0x6b/0x90 > > [199392.080373] ttwu_do_wakeup+0x19/0x130 > > [199392.080999] try_to_wake_up+0x1e2/0x460 > > [199392.081623] __wake_up_common+0x8f/0x160 > > [199392.082246] ep_poll_callback+0x1af/0x300 > > [199392.082860] __wake_up_common+0x8f/0x160 > > [199392.083470] __wake_up_common_lock+0x7a/0xc0 > > [199392.084074] irq_work_run_list+0x4c/0x70 > > [199392.084675] smp_call_function_single_interrupt+0x3a/0xd0 > > [199392.085277] call_function_single_interrupt+0xf/0x20 > > [199392.085879] </IRQ> > > [199392.086477] RIP: 0010:panic+0x209/0x25c > > [199392.087079] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > > [199392.088341] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 > > [199392.088988] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > > [199392.089639] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > > [199392.090291] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > > [199392.090947] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > > [199392.091601] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > > [199392.092255] oops_end+0xc1/0xd0 > > [199392.092894] do_trap+0x13d/0x150 > > [199392.093516] do_error_trap+0xd5/0x130 > > [199392.094122] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199392.094718] invalid_op+0x14/0x20 > > [199392.095296] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199392.095866] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > [199392.097006] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > [199392.097558] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > [199392.098103] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > [199392.098644] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > [199392.099176] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > [199392.099703] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > [199392.100229] ? __wake_up_common_lock+0x87/0xc0 > > [199392.100744] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > > [199392.101251] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > [199392.101752] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > [199392.102252] process_cell+0x2a3/0x550 [dm_thin_pool] > > [199392.102750] ? sort+0x17b/0x270 > > [199392.103242] ? u32_swap+0x10/0x10 > > [199392.103733] do_worker+0x268/0x9a0 [dm_thin_pool] > > [199392.104228] process_one_work+0x171/0x370 > > [199392.104714] worker_thread+0x49/0x3f0 > > [199392.105193] kthread+0xf8/0x130 > > [199392.105665] ? max_active_store+0x80/0x80 > > [199392.106132] ? kthread_bind+0x10/0x10 > > [199392.106601] ret_from_fork+0x1f/0x40 > > [199392.107069] ---[ end trace c31536d98046e8ed ]--- > > [199392.107544] ------------[ cut here ]------------ > > [199392.108017] sched: Unexpected reschedule of offline CPU#7! > > [199392.108497] WARNING: CPU: 4 PID: 31359 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40 > > [199392.108996] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_bit drm_kms_helper > > [199392.112967] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > > [199392.114203] CPU: 4 PID: 31359 Comm: kworker/u16:4 Tainted: G D W 4.19.75 #1 > > [199392.114819] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > > [199392.115439] Workqueue: dm-thin do_worker [dm_thin_pool] > > [199392.116061] RIP: 0010:native_smp_send_reschedule+0x39/0x40 > > [199392.116683] Code: 0f 92 c0 84 c0 74 15 48 8b 05 93 f1 eb 00 be fd 00 00 00 48 8b 40 30 e9 85 f4 9a 00 89 fe 48 c7 c7 08 23 e5 81 e8 c7 9a 05 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48 > > [199392.117982] RSP: 0018:ffff88880fb03ee0 EFLAGS: 00010082 > > [199392.118632] RAX: 0000000000000000 RBX: ffff8887d093ac80 RCX: 0000000000000006 > > [199392.119295] RDX: 0000000000000000 RSI: 0000000000000092 RDI: ffff88880fb168b0 > > [199392.119961] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000017529 > > [199392.120623] R10: ffff8881438ba800 R11: 0000000000000000 R12: ffffc9000a147988 > > [199392.121283] R13: ffffffff8113f9a0 R14: 0000000000000002 R15: ffff88880fb1cff8 > > [199392.121938] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > [199392.122589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [199392.123229] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > [199392.123867] Call Trace: > > [199392.124495] <IRQ> > > [199392.125116] update_process_times+0x40/0x50 > > [199392.125742] tick_sched_handle+0x25/0x60 > > [199392.126367] tick_sched_timer+0x37/0x70 > > [199392.126987] __hrtimer_run_queues+0xfb/0x270 > > [199392.127601] hrtimer_interrupt+0x122/0x270 > > [199392.128211] smp_apic_timer_interrupt+0x6a/0x140 > > [199392.128819] apic_timer_interrupt+0xf/0x20 > > [199392.129423] </IRQ> > > [199392.130020] RIP: 0010:panic+0x209/0x25c > > [199392.130619] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > > [199392.131883] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 > > [199392.132530] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > > [199392.133180] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > > [199392.133830] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > > [199392.134482] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > > [199392.135136] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > > [199392.135792] oops_end+0xc1/0xd0 > > [199392.136444] do_trap+0x13d/0x150 > > [199392.137094] do_error_trap+0xd5/0x130 > > [199392.137740] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199392.138383] invalid_op+0x14/0x20 > > [199392.139010] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199392.139630] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > [199392.140870] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > [199392.141472] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > [199392.142066] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > [199392.142647] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > [199392.143211] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > [199392.143759] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > [199392.144301] ? __wake_up_common_lock+0x87/0xc0 > > [199392.144838] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > > [199392.145374] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > [199392.145909] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > [199392.146435] process_cell+0x2a3/0x550 [dm_thin_pool] > > [199392.146949] ? sort+0x17b/0x270 > > [199392.147450] ? u32_swap+0x10/0x10 > > [199392.147944] do_worker+0x268/0x9a0 [dm_thin_pool] > > [199392.148441] process_one_work+0x171/0x370 > > [199392.148937] worker_thread+0x49/0x3f0 > > [199392.149430] kthread+0xf8/0x130 > > [199392.149922] ? max_active_store+0x80/0x80 > > [199392.150406] ? kthread_bind+0x10/0x10 > > [199392.150883] ret_from_fork+0x1f/0x40 > > [199392.151353] ---[ end trace c31536d98046e8ee ]--- > > > > > > -- > > Eric Wheeler > > > > -- > > dm-devel mailing list > > dm-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/dm-devel > > > > > -- > dm-devel mailing list > dm-devel at redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel > > ^ permalink raw reply related [flat|nested] 43+ messages in thread
* Re: [dm-devel] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 @ 2019-12-27 1:47 ` Eric Wheeler 0 siblings, 0 replies; 43+ messages in thread From: Eric Wheeler @ 2019-12-27 1:47 UTC (permalink / raw) To: lvm-devel Cc: Mike Snitzer, markus.schade, dm-devel, linux-block, ejt, joe.thornber On Fri, 20 Dec 2019, Eric Wheeler wrote: > On Wed, 25 Sep 2019, Eric Wheeler wrote: > > We are using the 4.19.75 stable tree with dm-thin and multi-queue scsi. > > We have been using the 4.19 branch for months without issue; we just > > switched to MQ and we seem to have hit this BUG_ON. Whether or not MQ is > > related to the issue, I don't know, maybe coincidence: > > > > static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > > { > > int r; > > enum allocation_event ev; > > struct sm_disk *smd = container_of(sm, struct sm_disk, sm); > > > > /* FIXME: we should loop round a couple of times */ > > r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); > > if (r) > > return r; > > > > smd->begin = *b + 1; > > r = sm_ll_inc(&smd->ll, *b, &ev); > > if (!r) { > > BUG_ON(ev != SM_ALLOC); <<<<<<<<<<<<<<<<<<<<<<<<<<<<< > > smd->nr_allocated_this_transaction++; > > } > > Hello all, > > We hit this BUG_ON again, this time with 4.19.86 with > scsi_mod.use_blk_mq=y, and it is known to be present as of 5.1.2 as > additionally reported by Markus Schade: > > https://www.redhat.com/archives/dm-devel/2019-June/msg00116.html > and > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1777398 > > In our case, the most latest trace (below) is from a different system that > has been stable for years on Linux 4.1 with tmeta direct on the SSD. > We updated to 4.19.86 a few weeks ago and just hit this, what Mike > Snitzer explains to be an allocator race: > > On Wed, 25 Sep 2019, Mike Snitzer wrote: > > > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > > > [199391.693852] ? sort+0x17b/0x270 > > > [199391.694527] ? u32_swap+0x10/0x10 > > > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > > > [199391.695890] process_one_work+0x171/0x370 > > > [199391.696640] worker_thread+0x49/0x3f0 > > > [199391.697332] kthread+0xf8/0x130 > > > [199391.697988] ? max_active_store+0x80/0x80 > > > [199391.698659] ? kthread_bind+0x10/0x10 > > > [199391.699281] ret_from_fork+0x1f/0x40 > > > > The stack shows the call to sm_disk_new_block() is due to > > dm_pool_alloc_data_block(). > > > > sm_disk_new_block()'s BUG_ON(ev != SM_ALLOC) indicates that somehow it is > > getting called without the passed 'ev' being set to SM_ALLOC. Only > > drivers/md/persistent-dat/dm-space-map-common.c:sm_ll_mutate() sets > > SM_ALLOC. sm_disk_new_block() is indirectly calling sm_ll_mutate() > > > > sm_ll_mutate() will only return 0 if ll->save_ie() does, the ll_disk *ll > > should be ll_disk, and so disk_ll_save_ie()'s call to dm_btree_insert() > > returns 0 -- which simply means success. And on success > > sm_disk_new_block() assumes ev was set to SM_ALLOC (by sm_ll_mutate). > > > > sm_ll_mutate() decided to _not_ set SM_ALLOC because either: > > 1) ref_count wasn't set > > or > > 2) old was identified > > > > So all said: somehow a new data block was found to already be in use. > > _WHY_ that is the case isn't clear from this stack... > > > > But it does speak to the possibility of data block allocation racing > > with other operations to the same block. Which implies missing locking. > > Where would you look to add locking do you think? > > > But that's all I've got so far... I'll review past dm-thinp changes with > > all this in mind and see what turns up. But Joe Thornber (ejt) likely > > needs to have a look at this too. > > > > But could it be that bcache is the source of the data device race (same > > block used concurrently)? And DM thinp is acting as the canary in the > > coal mine? > > As Marcus has shown, this bug triggers without bcache. > > > Other questions: > > 1) Can sm_disk_new_block() fail more gracefully than BUG_ON? For example: > > + spin_lock(&lock); /* protect smd->begin */ > smd->begin = *b + 1; > r = sm_ll_inc(&smd->ll, *b, &ev); > if (!r) { > - BUG_ON(ev != SM_ALLOC); > smd->nr_allocated_this_transaction++; > } > + else { > + r = -ENOSPC; > + smd->begin = *b - 1; > + } > + spin_unlock(&lock); Just hit the bug again without mq-scsi (scsi_mod.use_blk_mq=n), all other times MQ has been turned on. I'm trying the -ENOSPC hack which will flag the pool as being out of space so I can recover more gracefully than a BUG_ON. Here's a first-draft patch, maybe the spinlock will even prevent the issue. Compile tested, I'll try on a real system tomorrow. Comments welcome: diff --git a/drivers/md/persistent-data/dm-space-map-disk.c b/drivers/md/persistent-data/dm-space-map-disk.c index 32adf6b..cb27a20 100644 --- a/drivers/md/persistent-data/dm-space-map-disk.c +++ b/drivers/md/persistent-data/dm-space-map-disk.c @@ -161,6 +161,7 @@ static int sm_disk_dec_block(struct dm_space_map *sm, dm_block_t b) return r; } +static DEFINE_SPINLOCK(smd_lock); static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) { int r; @@ -168,17 +169,30 @@ static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) struct sm_disk *smd = container_of(sm, struct sm_disk, sm); /* FIXME: we should loop round a couple of times */ + spin_lock(&smd_lock); r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); - if (r) + if (r) { + spin_unlock(&smd_lock); return r; + } smd->begin = *b + 1; r = sm_ll_inc(&smd->ll, *b, &ev); if (!r) { - BUG_ON(ev != SM_ALLOC); - smd->nr_allocated_this_transaction++; + if (ev == SM_ALLOC) + smd->nr_allocated_this_transaction++; + else { + /* Not actually out of space, this is a bug: + * https://lore.kernel.org/linux-block/20190925200138.GA20584@redhat.com/ + */ + WARN(ev != SM_ALLOC, "Pool metadata allocation race, marking pool out-of-space."); + r = -ENOSPC; + smd->begin = *b - 1; + } } + spin_unlock(&smd_lock); + return r; } -- Eric Wheeler > > The lock might protect smd->begin, but I'm not sure how &smd->ll might > have been modified by sm_ll_inc(). However, since ll->save_ie() failed in > sm_ll_mutate() then perhaps this is safe. What do you think? > > Putting the pool into PM_OUT_OF_DATA_SPACE isn't ideal since it isn't out > of space, but I would take it over a BUG_ON. > > 2) If example #1 above returned -EAGAIN, how might alloc_data_block be > taught retry? This bug shows up weeks or months apart, even on heavily > loaded systems with ~100 live thin volumes, so retrying would be fine > IMHO. > > 3) In the thread from June, Marcus says: > "Judging from the call trace, my guess is that there is somewhere > a race condition, when a new block needs to be allocated which has > still to be discarded." > > Is this discard sitation possible? Wouldn't the bio prison prevent this? > > -- > Eric Wheeler > www.datawall.us > > > > Here is the new trace, old trace below: > > kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! > invalid opcode: 0000 [#1] SMP NOPTI > CPU: 11 PID: 22939 Comm: kworker/u48:1 Not tainted 4.19.86 #1 > Hardware name: Supermicro SYS-2026T-6RFT+/X8DTU-6+, BIOS 2.1c 11/30/2012 > Workqueue: dm-thin do_worker [dm_thin_pool] > RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 b0 a5 a9 e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 > RSP: 0018:ffffc90007237c78 EFLAGS: 00010297 > RAX: 0000000000000000 RBX: ffff88861d7ac000 RCX: 0000000000000000 > RDX: ffff8885f3d13f00 RSI: 0000000000000246 RDI: ffff888620e38c00 > RBP: 0000000000000000 R08: 0000000000000000 R09: ffff888620e38c98 > R10: ffffffff810f9177 R11: 0000000000000000 R12: ffffc90007237d48 > R13: ffffc90007237d48 R14: 00000000ffffffc3 R15: 00000000000d9e3a > FS: 0000000000000000(0000) GS:ffff8886279c0000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 00000000081ceff0 CR3: 000000000200a004 CR4: 00000000000226e0 > Call Trace: > dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > process_cell+0x2a3/0x550 [dm_thin_pool] > ? mempool_alloc+0x6f/0x180 > ? sort+0x17b/0x270 > ? u32_swap+0x10/0x10 > process_deferred_bios+0x1af/0x870 [dm_thin_pool] > do_worker+0x94/0xe0 [dm_thin_pool] > process_one_work+0x171/0x370 > worker_thread+0x49/0x3f0 > kthread+0xf8/0x130 > ? max_active_store+0x80/0x80 > ? kthread_bind+0x10/0x10 > ret_from_fork+0x1f/0x40 > > > > > [199391.677689] ------------[ cut here ]------------ > > [199391.678437] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! > > [199391.679183] invalid opcode: 0000 [#1] SMP NOPTI > > [199391.679941] CPU: 4 PID: 31359 Comm: kworker/u16:4 Not tainted 4.19.75 #1 > > [199391.680683] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > > [199391.681446] Workqueue: dm-thin do_worker [dm_thin_pool] > > [199391.682187] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199391.682929] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > [199391.684432] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > [199391.685186] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > [199391.685936] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > [199391.686659] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > [199391.687379] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > [199391.688120] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > [199391.688843] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > [199391.689571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [199391.690253] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > [199391.690984] Call Trace: > > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > > [199391.693852] ? sort+0x17b/0x270 > > [199391.694527] ? u32_swap+0x10/0x10 > > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > > [199391.695890] process_one_work+0x171/0x370 > > [199391.696640] worker_thread+0x49/0x3f0 > > [199391.697332] kthread+0xf8/0x130 > > [199391.697988] ? max_active_store+0x80/0x80 > > [199391.698659] ? kthread_bind+0x10/0x10 > > [199391.699281] ret_from_fork+0x1f/0x40 > > [199391.699930] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_a lgo_bit drm_kms_helper > > [199391.705631] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > > [199391.708083] ---[ end trace c31536d98046e8ec ]--- > > [199391.866776] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199391.867960] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > [199391.870379] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > [199391.871524] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > [199391.872364] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > [199391.873173] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > [199391.873871] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > [199391.874550] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > [199391.875231] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > [199391.875941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [199391.876633] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > [199391.877317] Kernel panic - not syncing: Fatal exception > > [199391.878006] Kernel Offset: disabled > > [199392.032304] ---[ end Kernel panic - not syncing: Fatal exception ]--- > > [199392.032962] unchecked MSR access error: WRMSR to 0x83f (tried to write 0x00000000000000f6) at rIP: 0xffffffff81067af4 (native_write_msr+0x4/0x20) > > [199392.034277] Call Trace: > > [199392.034929] <IRQ> > > [199392.035576] native_apic_msr_write+0x2e/0x40 > > [199392.036228] arch_irq_work_raise+0x28/0x40 > > [199392.036877] irq_work_queue_on+0x83/0xa0 > > [199392.037518] irq_work_run_list+0x4c/0x70 > > [199392.038149] irq_work_run+0x14/0x40 > > [199392.038771] smp_call_function_single_interrupt+0x3a/0xd0 > > [199392.039393] call_function_single_interrupt+0xf/0x20 > > [199392.040011] </IRQ> > > [199392.040624] RIP: 0010:panic+0x209/0x25c > > [199392.041234] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > > [199392.042518] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 > > [199392.043174] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > > [199392.043833] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > > [199392.044493] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > > [199392.045155] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > > [199392.045820] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > > [199392.046486] oops_end+0xc1/0xd0 > > [199392.047149] do_trap+0x13d/0x150 > > [199392.047795] do_error_trap+0xd5/0x130 > > [199392.048427] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199392.049048] invalid_op+0x14/0x20 > > [199392.049650] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199392.050245] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > [199392.051434] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > [199392.052010] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > [199392.052580] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > [199392.053150] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > [199392.053715] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > [199392.054266] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > [199392.054807] ? __wake_up_common_lock+0x87/0xc0 > > [199392.055342] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > > [199392.055877] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > [199392.056410] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > [199392.056947] process_cell+0x2a3/0x550 [dm_thin_pool] > > [199392.057484] ? sort+0x17b/0x270 > > [199392.058016] ? u32_swap+0x10/0x10 > > [199392.058538] do_worker+0x268/0x9a0 [dm_thin_pool] > > [199392.059060] process_one_work+0x171/0x370 > > [199392.059576] worker_thread+0x49/0x3f0 > > [199392.060083] kthread+0xf8/0x130 > > [199392.060587] ? max_active_store+0x80/0x80 > > [199392.061086] ? kthread_bind+0x10/0x10 > > [199392.061569] ret_from_fork+0x1f/0x40 > > [199392.062038] ------------[ cut here ]------------ > > [199392.062508] sched: Unexpected reschedule of offline CPU#1! > > [199392.062989] WARNING: CPU: 4 PID: 31359 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40 > > [199392.063485] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_a lgo_bit drm_kms_helper > > [199392.067463] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > > [199392.068729] CPU: 4 PID: 31359 Comm: kworker/u16:4 Tainted: G D 4.19.75 #1 > > [199392.069356] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > > [199392.069982] Workqueue: dm-thin do_worker [dm_thin_pool] > > [199392.070604] RIP: 0010:native_smp_send_reschedule+0x39/0x40 > > [199392.071229] Code: 0f 92 c0 84 c0 74 15 48 8b 05 93 f1 eb 00 be fd 00 00 00 48 8b 40 30 e9 85 f4 9a 00 89 fe 48 c7 c7 08 23 e5 81 e8 c7 9a 05 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48 > > [199392.072528] RSP: 0018:ffff88880fb03dc0 EFLAGS: 00010082 > > [199392.073186] RAX: 0000000000000000 RBX: ffff88880fa625c0 RCX: 0000000000000006 > > [199392.073847] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff88880fb168b0 > > [199392.074512] RBP: ffff88880fa625c0 R08: 0000000000000001 R09: 00000000000174e5 > > [199392.075182] R10: ffff8881f2512300 R11: 0000000000000000 R12: ffff888804541640 > > [199392.075849] R13: ffff88880fb03e08 R14: 0000000000000000 R15: 0000000000000001 > > [199392.076512] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > [199392.077179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [199392.077833] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > [199392.078481] Call Trace: > > [199392.079117] <IRQ> > > [199392.079745] check_preempt_curr+0x6b/0x90 > > [199392.080373] ttwu_do_wakeup+0x19/0x130 > > [199392.080999] try_to_wake_up+0x1e2/0x460 > > [199392.081623] __wake_up_common+0x8f/0x160 > > [199392.082246] ep_poll_callback+0x1af/0x300 > > [199392.082860] __wake_up_common+0x8f/0x160 > > [199392.083470] __wake_up_common_lock+0x7a/0xc0 > > [199392.084074] irq_work_run_list+0x4c/0x70 > > [199392.084675] smp_call_function_single_interrupt+0x3a/0xd0 > > [199392.085277] call_function_single_interrupt+0xf/0x20 > > [199392.085879] </IRQ> > > [199392.086477] RIP: 0010:panic+0x209/0x25c > > [199392.087079] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > > [199392.088341] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 > > [199392.088988] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > > [199392.089639] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > > [199392.090291] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > > [199392.090947] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > > [199392.091601] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > > [199392.092255] oops_end+0xc1/0xd0 > > [199392.092894] do_trap+0x13d/0x150 > > [199392.093516] do_error_trap+0xd5/0x130 > > [199392.094122] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199392.094718] invalid_op+0x14/0x20 > > [199392.095296] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199392.095866] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > [199392.097006] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > [199392.097558] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > [199392.098103] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > [199392.098644] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > [199392.099176] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > [199392.099703] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > [199392.100229] ? __wake_up_common_lock+0x87/0xc0 > > [199392.100744] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > > [199392.101251] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > [199392.101752] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > [199392.102252] process_cell+0x2a3/0x550 [dm_thin_pool] > > [199392.102750] ? sort+0x17b/0x270 > > [199392.103242] ? u32_swap+0x10/0x10 > > [199392.103733] do_worker+0x268/0x9a0 [dm_thin_pool] > > [199392.104228] process_one_work+0x171/0x370 > > [199392.104714] worker_thread+0x49/0x3f0 > > [199392.105193] kthread+0xf8/0x130 > > [199392.105665] ? max_active_store+0x80/0x80 > > [199392.106132] ? kthread_bind+0x10/0x10 > > [199392.106601] ret_from_fork+0x1f/0x40 > > [199392.107069] ---[ end trace c31536d98046e8ed ]--- > > [199392.107544] ------------[ cut here ]------------ > > [199392.108017] sched: Unexpected reschedule of offline CPU#7! > > [199392.108497] WARNING: CPU: 4 PID: 31359 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40 > > [199392.108996] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_a lgo_bit drm_kms_helper > > [199392.112967] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > > [199392.114203] CPU: 4 PID: 31359 Comm: kworker/u16:4 Tainted: G D W 4.19.75 #1 > > [199392.114819] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > > [199392.115439] Workqueue: dm-thin do_worker [dm_thin_pool] > > [199392.116061] RIP: 0010:native_smp_send_reschedule+0x39/0x40 > > [199392.116683] Code: 0f 92 c0 84 c0 74 15 48 8b 05 93 f1 eb 00 be fd 00 00 00 48 8b 40 30 e9 85 f4 9a 00 89 fe 48 c7 c7 08 23 e5 81 e8 c7 9a 05 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48 > > [199392.117982] RSP: 0018:ffff88880fb03ee0 EFLAGS: 00010082 > > [199392.118632] RAX: 0000000000000000 RBX: ffff8887d093ac80 RCX: 0000000000000006 > > [199392.119295] RDX: 0000000000000000 RSI: 0000000000000092 RDI: ffff88880fb168b0 > > [199392.119961] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000017529 > > [199392.120623] R10: ffff8881438ba800 R11: 0000000000000000 R12: ffffc9000a147988 > > [199392.121283] R13: ffffffff8113f9a0 R14: 0000000000000002 R15: ffff88880fb1cff8 > > [199392.121938] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > [199392.122589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [199392.123229] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > [199392.123867] Call Trace: > > [199392.124495] <IRQ> > > [199392.125116] update_process_times+0x40/0x50 > > [199392.125742] tick_sched_handle+0x25/0x60 > > [199392.126367] tick_sched_timer+0x37/0x70 > > [199392.126987] __hrtimer_run_queues+0xfb/0x270 > > [199392.127601] hrtimer_interrupt+0x122/0x270 > > [199392.128211] smp_apic_timer_interrupt+0x6a/0x140 > > [199392.128819] apic_timer_interrupt+0xf/0x20 > > [199392.129423] </IRQ> > > [199392.130020] RIP: 0010:panic+0x209/0x25c > > [199392.130619] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > > [199392.131883] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 > > [199392.132530] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > > [199392.133180] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > > [199392.133830] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > > [199392.134482] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > > [199392.135136] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > > [199392.135792] oops_end+0xc1/0xd0 > > [199392.136444] do_trap+0x13d/0x150 > > [199392.137094] do_error_trap+0xd5/0x130 > > [199392.137740] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199392.138383] invalid_op+0x14/0x20 > > [199392.139010] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > [199392.139630] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > [199392.140870] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > [199392.141472] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > [199392.142066] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > [199392.142647] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > [199392.143211] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > [199392.143759] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > [199392.144301] ? __wake_up_common_lock+0x87/0xc0 > > [199392.144838] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > > [199392.145374] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > [199392.145909] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > [199392.146435] process_cell+0x2a3/0x550 [dm_thin_pool] > > [199392.146949] ? sort+0x17b/0x270 > > [199392.147450] ? u32_swap+0x10/0x10 > > [199392.147944] do_worker+0x268/0x9a0 [dm_thin_pool] > > [199392.148441] process_one_work+0x171/0x370 > > [199392.148937] worker_thread+0x49/0x3f0 > > [199392.149430] kthread+0xf8/0x130 > > [199392.149922] ? max_active_store+0x80/0x80 > > [199392.150406] ? kthread_bind+0x10/0x10 > > [199392.150883] ret_from_fork+0x1f/0x40 > > [199392.151353] ---[ end trace c31536d98046e8ee ]--- > > > > > > -- > > Eric Wheeler > > > > -- > > dm-devel mailing list > > dm-devel@redhat.com > > https://www.redhat.com/mailman/listinfo/dm-devel > > > > > -- > dm-devel mailing list > dm-devel@redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel > > -- lvm-devel mailing list lvm-devel@redhat.com https://www.redhat.com/mailman/listinfo/lvm-devel ^ permalink raw reply related [flat|nested] 43+ messages in thread
* Re: [dm-devel] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 2019-12-27 1:47 ` Eric Wheeler (?) @ 2019-12-28 2:13 ` Eric Wheeler -1 siblings, 0 replies; 43+ messages in thread From: Eric Wheeler @ 2019-12-28 2:13 UTC (permalink / raw) To: lvm-devel Cc: Mike Snitzer, markus.schade, dm-devel, linux-block, ejt, joe.thornber On Fri, 27 Dec 2019, Eric Wheeler wrote: > On Fri, 20 Dec 2019, Eric Wheeler wrote: > > On Wed, 25 Sep 2019, Eric Wheeler wrote: > > > We are using the 4.19.75 stable tree with dm-thin and multi-queue scsi. > > > We have been using the 4.19 branch for months without issue; we just > > > switched to MQ and we seem to have hit this BUG_ON. Whether or not MQ is > > > related to the issue, I don't know, maybe coincidence: > > > > > > static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > > > { > > > int r; > > > enum allocation_event ev; > > > struct sm_disk *smd = container_of(sm, struct sm_disk, sm); > > > > > > /* FIXME: we should loop round a couple of times */ > > > r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); > > > if (r) > > > return r; > > > > > > smd->begin = *b + 1; > > > r = sm_ll_inc(&smd->ll, *b, &ev); > > > if (!r) { > > > BUG_ON(ev != SM_ALLOC); <<<<<<<<<<<<<<<<<<<<<<<<<<<<< > > > smd->nr_allocated_this_transaction++; > > > } > > > > Hello all, > > > > We hit this BUG_ON again, this time with 4.19.86 with > > scsi_mod.use_blk_mq=y, and it is known to be present as of 5.1.2 as > > additionally reported by Markus Schade: > > > > https://www.redhat.com/archives/dm-devel/2019-June/msg00116.html > > and > > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1777398 > > > > In our case, the most latest trace (below) is from a different system that > > has been stable for years on Linux 4.1 with tmeta direct on the SSD. > > We updated to 4.19.86 a few weeks ago and just hit this, what Mike > > Snitzer explains to be an allocator race: > > > > On Wed, 25 Sep 2019, Mike Snitzer wrote: > > > > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > > > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > > > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > > > > [199391.693852] ? sort+0x17b/0x270 > > > > [199391.694527] ? u32_swap+0x10/0x10 > > > > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > > > > [199391.695890] process_one_work+0x171/0x370 > > > > [199391.696640] worker_thread+0x49/0x3f0 > > > > [199391.697332] kthread+0xf8/0x130 > > > > [199391.697988] ? max_active_store+0x80/0x80 > > > > [199391.698659] ? kthread_bind+0x10/0x10 > > > > [199391.699281] ret_from_fork+0x1f/0x40 > > > > > > The stack shows the call to sm_disk_new_block() is due to > > > dm_pool_alloc_data_block(). > > > > > > sm_disk_new_block()'s BUG_ON(ev != SM_ALLOC) indicates that somehow it is > > > getting called without the passed 'ev' being set to SM_ALLOC. Only > > > drivers/md/persistent-dat/dm-space-map-common.c:sm_ll_mutate() sets > > > SM_ALLOC. sm_disk_new_block() is indirectly calling sm_ll_mutate() > > > > > > sm_ll_mutate() will only return 0 if ll->save_ie() does, the ll_disk *ll > > > should be ll_disk, and so disk_ll_save_ie()'s call to dm_btree_insert() > > > returns 0 -- which simply means success. And on success > > > sm_disk_new_block() assumes ev was set to SM_ALLOC (by sm_ll_mutate). > > > > > > sm_ll_mutate() decided to _not_ set SM_ALLOC because either: > > > 1) ref_count wasn't set > > > or > > > 2) old was identified > > > > > > So all said: somehow a new data block was found to already be in use. > > > _WHY_ that is the case isn't clear from this stack... > > > > > > But it does speak to the possibility of data block allocation racing > > > with other operations to the same block. Which implies missing locking. > > > > Where would you look to add locking do you think? > > > > > But that's all I've got so far... I'll review past dm-thinp changes with > > > all this in mind and see what turns up. But Joe Thornber (ejt) likely > > > needs to have a look at this too. > > > > > > But could it be that bcache is the source of the data device race (same > > > block used concurrently)? And DM thinp is acting as the canary in the > > > coal mine? > > > > As Marcus has shown, this bug triggers without bcache. > > > > > > Other questions: > > > > 1) Can sm_disk_new_block() fail more gracefully than BUG_ON? For example: > > > > + spin_lock(&lock); /* protect smd->begin */ > > smd->begin = *b + 1; > > r = sm_ll_inc(&smd->ll, *b, &ev); > > if (!r) { > > - BUG_ON(ev != SM_ALLOC); > > smd->nr_allocated_this_transaction++; > > } > > + else { > > + r = -ENOSPC; > > + smd->begin = *b - 1; > > + } > > + spin_unlock(&lock); > > Just hit the bug again without mq-scsi (scsi_mod.use_blk_mq=n), all other > times MQ has been turned on. > > I'm trying the -ENOSPC hack which will flag the pool as being out of space > so I can recover more gracefully than a BUG_ON. Here's a first-draft > patch, maybe the spinlock will even prevent the issue. > > Compile tested, I'll try on a real system tomorrow. > > Comments welcome: > > diff --git a/drivers/md/persistent-data/dm-space-map-disk.c b/drivers/md/persistent-data/dm-space-map-disk.c > index 32adf6b..cb27a20 100644 > --- a/drivers/md/persistent-data/dm-space-map-disk.c > +++ b/drivers/md/persistent-data/dm-space-map-disk.c > @@ -161,6 +161,7 @@ static int sm_disk_dec_block(struct dm_space_map *sm, dm_block_t b) > return r; > } > > +static DEFINE_SPINLOCK(smd_lock); > static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > { > int r; > @@ -168,17 +169,30 @@ static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > struct sm_disk *smd = container_of(sm, struct sm_disk, sm); > > /* FIXME: we should loop round a couple of times */ > + spin_lock(&smd_lock); > r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); > - if (r) > + if (r) { > + spin_unlock(&smd_lock); > return r; > + } > > smd->begin = *b + 1; > r = sm_ll_inc(&smd->ll, *b, &ev); > if (!r) { > - BUG_ON(ev != SM_ALLOC); > - smd->nr_allocated_this_transaction++; > + if (ev == SM_ALLOC) > + smd->nr_allocated_this_transaction++; > + else { > + /* Not actually out of space, this is a bug: > + * https://lore.kernel.org/linux-block/20190925200138.GA20584@redhat.com/ > + */ > + WARN(ev != SM_ALLOC, "Pool metadata allocation race, marking pool out-of-space."); > + r = -ENOSPC; > + smd->begin = *b - 1; > + } > } > > + spin_unlock(&smd_lock); > + > return r; > } So far, so good. There are 3 systems running with the patch (1 w/ MQ=y, 2 w/ MQ=n), time will tell if we hit the WARN or if the spinlock prevents the race. -- Eric Wheeler > > > > -- > Eric Wheeler > > > > > > The lock might protect smd->begin, but I'm not sure how &smd->ll might > > have been modified by sm_ll_inc(). However, since ll->save_ie() failed in > > sm_ll_mutate() then perhaps this is safe. What do you think? > > > > Putting the pool into PM_OUT_OF_DATA_SPACE isn't ideal since it isn't out > > of space, but I would take it over a BUG_ON. > > > > 2) If example #1 above returned -EAGAIN, how might alloc_data_block be > > taught retry? This bug shows up weeks or months apart, even on heavily > > loaded systems with ~100 live thin volumes, so retrying would be fine > > IMHO. > > > > 3) In the thread from June, Marcus says: > > "Judging from the call trace, my guess is that there is somewhere > > a race condition, when a new block needs to be allocated which has > > still to be discarded." > > > > Is this discard sitation possible? Wouldn't the bio prison prevent this? > > > > -- > > Eric Wheeler > > www.datawall.us > > > > > > > > Here is the new trace, old trace below: > > > > kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! > > invalid opcode: 0000 [#1] SMP NOPTI > > CPU: 11 PID: 22939 Comm: kworker/u48:1 Not tainted 4.19.86 #1 > > Hardware name: Supermicro SYS-2026T-6RFT+/X8DTU-6+, BIOS 2.1c 11/30/2012 > > Workqueue: dm-thin do_worker [dm_thin_pool] > > RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 b0 a5 a9 e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 > > RSP: 0018:ffffc90007237c78 EFLAGS: 00010297 > > RAX: 0000000000000000 RBX: ffff88861d7ac000 RCX: 0000000000000000 > > RDX: ffff8885f3d13f00 RSI: 0000000000000246 RDI: ffff888620e38c00 > > RBP: 0000000000000000 R08: 0000000000000000 R09: ffff888620e38c98 > > R10: ffffffff810f9177 R11: 0000000000000000 R12: ffffc90007237d48 > > R13: ffffc90007237d48 R14: 00000000ffffffc3 R15: 00000000000d9e3a > > FS: 0000000000000000(0000) GS:ffff8886279c0000(0000) knlGS:0000000000000000 > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > CR2: 00000000081ceff0 CR3: 000000000200a004 CR4: 00000000000226e0 > > Call Trace: > > dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > process_cell+0x2a3/0x550 [dm_thin_pool] > > ? mempool_alloc+0x6f/0x180 > > ? sort+0x17b/0x270 > > ? u32_swap+0x10/0x10 > > process_deferred_bios+0x1af/0x870 [dm_thin_pool] > > do_worker+0x94/0xe0 [dm_thin_pool] > > process_one_work+0x171/0x370 > > worker_thread+0x49/0x3f0 > > kthread+0xf8/0x130 > > ? max_active_store+0x80/0x80 > > ? kthread_bind+0x10/0x10 > > ret_from_fork+0x1f/0x40 > > > > > > > > [199391.677689] ------------[ cut here ]------------ > > > [199391.678437] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! > > > [199391.679183] invalid opcode: 0000 [#1] SMP NOPTI > > > [199391.679941] CPU: 4 PID: 31359 Comm: kworker/u16:4 Not tainted 4.19.75 #1 > > > [199391.680683] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > > > [199391.681446] Workqueue: dm-thin do_worker [dm_thin_pool] > > > [199391.682187] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > > [199391.682929] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > > [199391.684432] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > > [199391.685186] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > > [199391.685936] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > > [199391.686659] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > > [199391.687379] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > > [199391.688120] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > > [199391.688843] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > > [199391.689571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [199391.690253] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > > [199391.690984] Call Trace: > > > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > > > [199391.693852] ? sort+0x17b/0x270 > > > [199391.694527] ? u32_swap+0x10/0x10 > > > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > > > [199391.695890] process_one_work+0x171/0x370 > > > [199391.696640] worker_thread+0x49/0x3f0 > > > [199391.697332] kthread+0xf8/0x130 > > > [199391.697988] ? max_active_store+0x80/0x80 > > > [199391.698659] ? kthread_bind+0x10/0x10 > > > [199391.699281] ret_from_fork+0x1f/0x40 > > > [199391.699930] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_bit drm_kms_helper > > > [199391.705631] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > > > [199391.708083] ---[ end trace c31536d98046e8ec ]--- > > > [199391.866776] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > > [199391.867960] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > > [199391.870379] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > > [199391.871524] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > > [199391.872364] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > > [199391.873173] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > > [199391.873871] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > > [199391.874550] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > > [199391.875231] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > > [199391.875941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [199391.876633] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > > [199391.877317] Kernel panic - not syncing: Fatal exception > > > [199391.878006] Kernel Offset: disabled > > > [199392.032304] ---[ end Kernel panic - not syncing: Fatal exception ]--- > > > [199392.032962] unchecked MSR access error: WRMSR to 0x83f (tried to write 0x00000000000000f6) at rIP: 0xffffffff81067af4 (native_write_msr+0x4/0x20) > > > [199392.034277] Call Trace: > > > [199392.034929] <IRQ> > > > [199392.035576] native_apic_msr_write+0x2e/0x40 > > > [199392.036228] arch_irq_work_raise+0x28/0x40 > > > [199392.036877] irq_work_queue_on+0x83/0xa0 > > > [199392.037518] irq_work_run_list+0x4c/0x70 > > > [199392.038149] irq_work_run+0x14/0x40 > > > [199392.038771] smp_call_function_single_interrupt+0x3a/0xd0 > > > [199392.039393] call_function_single_interrupt+0xf/0x20 > > > [199392.040011] </IRQ> > > > [199392.040624] RIP: 0010:panic+0x209/0x25c > > > [199392.041234] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > > > [199392.042518] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 > > > [199392.043174] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > > > [199392.043833] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > > > [199392.044493] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > > > [199392.045155] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > > > [199392.045820] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > > > [199392.046486] oops_end+0xc1/0xd0 > > > [199392.047149] do_trap+0x13d/0x150 > > > [199392.047795] do_error_trap+0xd5/0x130 > > > [199392.048427] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > > [199392.049048] invalid_op+0x14/0x20 > > > [199392.049650] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > > [199392.050245] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > > [199392.051434] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > > [199392.052010] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > > [199392.052580] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > > [199392.053150] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > > [199392.053715] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > > [199392.054266] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > > [199392.054807] ? __wake_up_common_lock+0x87/0xc0 > > > [199392.055342] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > > > [199392.055877] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > > [199392.056410] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > > [199392.056947] process_cell+0x2a3/0x550 [dm_thin_pool] > > > [199392.057484] ? sort+0x17b/0x270 > > > [199392.058016] ? u32_swap+0x10/0x10 > > > [199392.058538] do_worker+0x268/0x9a0 [dm_thin_pool] > > > [199392.059060] process_one_work+0x171/0x370 > > > [199392.059576] worker_thread+0x49/0x3f0 > > > [199392.060083] kthread+0xf8/0x130 > > > [199392.060587] ? max_active_store+0x80/0x80 > > > [199392.061086] ? kthread_bind+0x10/0x10 > > > [199392.061569] ret_from_fork+0x1f/0x40 > > > [199392.062038] ------------[ cut here ]------------ > > > [199392.062508] sched: Unexpected reschedule of offline CPU#1! > > > [199392.062989] WARNING: CPU: 4 PID: 31359 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40 > > > [199392.063485] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_bit drm_kms_helper > > > [199392.067463] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > > > [199392.068729] CPU: 4 PID: 31359 Comm: kworker/u16:4 Tainted: G D 4.19.75 #1 > > > [199392.069356] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > > > [199392.069982] Workqueue: dm-thin do_worker [dm_thin_pool] > > > [199392.070604] RIP: 0010:native_smp_send_reschedule+0x39/0x40 > > > [199392.071229] Code: 0f 92 c0 84 c0 74 15 48 8b 05 93 f1 eb 00 be fd 00 00 00 48 8b 40 30 e9 85 f4 9a 00 89 fe 48 c7 c7 08 23 e5 81 e8 c7 9a 05 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48 > > > [199392.072528] RSP: 0018:ffff88880fb03dc0 EFLAGS: 00010082 > > > [199392.073186] RAX: 0000000000000000 RBX: ffff88880fa625c0 RCX: 0000000000000006 > > > [199392.073847] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff88880fb168b0 > > > [199392.074512] RBP: ffff88880fa625c0 R08: 0000000000000001 R09: 00000000000174e5 > > > [199392.075182] R10: ffff8881f2512300 R11: 0000000000000000 R12: ffff888804541640 > > > [199392.075849] R13: ffff88880fb03e08 R14: 0000000000000000 R15: 0000000000000001 > > > [199392.076512] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > > [199392.077179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [199392.077833] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > > [199392.078481] Call Trace: > > > [199392.079117] <IRQ> > > > [199392.079745] check_preempt_curr+0x6b/0x90 > > > [199392.080373] ttwu_do_wakeup+0x19/0x130 > > > [199392.080999] try_to_wake_up+0x1e2/0x460 > > > [199392.081623] __wake_up_common+0x8f/0x160 > > > [199392.082246] ep_poll_callback+0x1af/0x300 > > > [199392.082860] __wake_up_common+0x8f/0x160 > > > [199392.083470] __wake_up_common_lock+0x7a/0xc0 > > > [199392.084074] irq_work_run_list+0x4c/0x70 > > > [199392.084675] smp_call_function_single_interrupt+0x3a/0xd0 > > > [199392.085277] call_function_single_interrupt+0xf/0x20 > > > [199392.085879] </IRQ> > > > [199392.086477] RIP: 0010:panic+0x209/0x25c > > > [199392.087079] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > > > [199392.088341] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 > > > [199392.088988] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > > > [199392.089639] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > > > [199392.090291] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > > > [199392.090947] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > > > [199392.091601] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > > > [199392.092255] oops_end+0xc1/0xd0 > > > [199392.092894] do_trap+0x13d/0x150 > > > [199392.093516] do_error_trap+0xd5/0x130 > > > [199392.094122] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > > [199392.094718] invalid_op+0x14/0x20 > > > [199392.095296] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > > [199392.095866] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > > [199392.097006] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > > [199392.097558] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > > [199392.098103] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > > [199392.098644] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > > [199392.099176] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > > [199392.099703] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > > [199392.100229] ? __wake_up_common_lock+0x87/0xc0 > > > [199392.100744] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > > > [199392.101251] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > > [199392.101752] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > > [199392.102252] process_cell+0x2a3/0x550 [dm_thin_pool] > > > [199392.102750] ? sort+0x17b/0x270 > > > [199392.103242] ? u32_swap+0x10/0x10 > > > [199392.103733] do_worker+0x268/0x9a0 [dm_thin_pool] > > > [199392.104228] process_one_work+0x171/0x370 > > > [199392.104714] worker_thread+0x49/0x3f0 > > > [199392.105193] kthread+0xf8/0x130 > > > [199392.105665] ? max_active_store+0x80/0x80 > > > [199392.106132] ? kthread_bind+0x10/0x10 > > > [199392.106601] ret_from_fork+0x1f/0x40 > > > [199392.107069] ---[ end trace c31536d98046e8ed ]--- > > > [199392.107544] ------------[ cut here ]------------ > > > [199392.108017] sched: Unexpected reschedule of offline CPU#7! > > > [199392.108497] WARNING: CPU: 4 PID: 31359 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40 > > > [199392.108996] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_bit drm_kms_helper > > > [199392.112967] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > > > [199392.114203] CPU: 4 PID: 31359 Comm: kworker/u16:4 Tainted: G D W 4.19.75 #1 > > > [199392.114819] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > > > [199392.115439] Workqueue: dm-thin do_worker [dm_thin_pool] > > > [199392.116061] RIP: 0010:native_smp_send_reschedule+0x39/0x40 > > > [199392.116683] Code: 0f 92 c0 84 c0 74 15 48 8b 05 93 f1 eb 00 be fd 00 00 00 48 8b 40 30 e9 85 f4 9a 00 89 fe 48 c7 c7 08 23 e5 81 e8 c7 9a 05 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48 > > > [199392.117982] RSP: 0018:ffff88880fb03ee0 EFLAGS: 00010082 > > > [199392.118632] RAX: 0000000000000000 RBX: ffff8887d093ac80 RCX: 0000000000000006 > > > [199392.119295] RDX: 0000000000000000 RSI: 0000000000000092 RDI: ffff88880fb168b0 > > > [199392.119961] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000017529 > > > [199392.120623] R10: ffff8881438ba800 R11: 0000000000000000 R12: ffffc9000a147988 > > > [199392.121283] R13: ffffffff8113f9a0 R14: 0000000000000002 R15: ffff88880fb1cff8 > > > [199392.121938] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > > [199392.122589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [199392.123229] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > > [199392.123867] Call Trace: > > > [199392.124495] <IRQ> > > > [199392.125116] update_process_times+0x40/0x50 > > > [199392.125742] tick_sched_handle+0x25/0x60 > > > [199392.126367] tick_sched_timer+0x37/0x70 > > > [199392.126987] __hrtimer_run_queues+0xfb/0x270 > > > [199392.127601] hrtimer_interrupt+0x122/0x270 > > > [199392.128211] smp_apic_timer_interrupt+0x6a/0x140 > > > [199392.128819] apic_timer_interrupt+0xf/0x20 > > > [199392.129423] </IRQ> > > > [199392.130020] RIP: 0010:panic+0x209/0x25c > > > [199392.130619] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > > > [199392.131883] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 > > > [199392.132530] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > > > [199392.133180] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > > > [199392.133830] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > > > [199392.134482] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > > > [199392.135136] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > > > [199392.135792] oops_end+0xc1/0xd0 > > > [199392.136444] do_trap+0x13d/0x150 > > > [199392.137094] do_error_trap+0xd5/0x130 > > > [199392.137740] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > > [199392.138383] invalid_op+0x14/0x20 > > > [199392.139010] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > > [199392.139630] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > > [199392.140870] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > > [199392.141472] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > > [199392.142066] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > > [199392.142647] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > > [199392.143211] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > > [199392.143759] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > > [199392.144301] ? __wake_up_common_lock+0x87/0xc0 > > > [199392.144838] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > > > [199392.145374] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > > [199392.145909] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > > [199392.146435] process_cell+0x2a3/0x550 [dm_thin_pool] > > > [199392.146949] ? sort+0x17b/0x270 > > > [199392.147450] ? u32_swap+0x10/0x10 > > > [199392.147944] do_worker+0x268/0x9a0 [dm_thin_pool] > > > [199392.148441] process_one_work+0x171/0x370 > > > [199392.148937] worker_thread+0x49/0x3f0 > > > [199392.149430] kthread+0xf8/0x130 > > > [199392.149922] ? max_active_store+0x80/0x80 > > > [199392.150406] ? kthread_bind+0x10/0x10 > > > [199392.150883] ret_from_fork+0x1f/0x40 > > > [199392.151353] ---[ end trace c31536d98046e8ee ]--- > > > > > > > > > -- > > > Eric Wheeler > > > > > > -- > > > dm-devel mailing list > > > dm-devel@redhat.com > > > https://www.redhat.com/mailman/listinfo/dm-devel > > > > > > > > > -- > > dm-devel mailing list > > dm-devel@redhat.com > > https://www.redhat.com/mailman/listinfo/dm-devel > > > > > > > -- > dm-devel mailing list > dm-devel@redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel > > ^ permalink raw reply [flat|nested] 43+ messages in thread
* [dm-devel] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 @ 2019-12-28 2:13 ` Eric Wheeler 0 siblings, 0 replies; 43+ messages in thread From: Eric Wheeler @ 2019-12-28 2:13 UTC (permalink / raw) To: lvm-devel On Fri, 27 Dec 2019, Eric Wheeler wrote: > On Fri, 20 Dec 2019, Eric Wheeler wrote: > > On Wed, 25 Sep 2019, Eric Wheeler wrote: > > > We are using the 4.19.75 stable tree with dm-thin and multi-queue scsi. > > > We have been using the 4.19 branch for months without issue; we just > > > switched to MQ and we seem to have hit this BUG_ON. Whether or not MQ is > > > related to the issue, I don't know, maybe coincidence: > > > > > > static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > > > { > > > int r; > > > enum allocation_event ev; > > > struct sm_disk *smd = container_of(sm, struct sm_disk, sm); > > > > > > /* FIXME: we should loop round a couple of times */ > > > r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); > > > if (r) > > > return r; > > > > > > smd->begin = *b + 1; > > > r = sm_ll_inc(&smd->ll, *b, &ev); > > > if (!r) { > > > BUG_ON(ev != SM_ALLOC); <<<<<<<<<<<<<<<<<<<<<<<<<<<<< > > > smd->nr_allocated_this_transaction++; > > > } > > > > Hello all, > > > > We hit this BUG_ON again, this time with 4.19.86 with > > scsi_mod.use_blk_mq=y, and it is known to be present as of 5.1.2 as > > additionally reported by Markus Schade: > > > > https://www.redhat.com/archives/dm-devel/2019-June/msg00116.html > > and > > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1777398 > > > > In our case, the most latest trace (below) is from a different system that > > has been stable for years on Linux 4.1 with tmeta direct on the SSD. > > We updated to 4.19.86 a few weeks ago and just hit this, what Mike > > Snitzer explains to be an allocator race: > > > > On Wed, 25 Sep 2019, Mike Snitzer wrote: > > > > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > > > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > > > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > > > > [199391.693852] ? sort+0x17b/0x270 > > > > [199391.694527] ? u32_swap+0x10/0x10 > > > > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > > > > [199391.695890] process_one_work+0x171/0x370 > > > > [199391.696640] worker_thread+0x49/0x3f0 > > > > [199391.697332] kthread+0xf8/0x130 > > > > [199391.697988] ? max_active_store+0x80/0x80 > > > > [199391.698659] ? kthread_bind+0x10/0x10 > > > > [199391.699281] ret_from_fork+0x1f/0x40 > > > > > > The stack shows the call to sm_disk_new_block() is due to > > > dm_pool_alloc_data_block(). > > > > > > sm_disk_new_block()'s BUG_ON(ev != SM_ALLOC) indicates that somehow it is > > > getting called without the passed 'ev' being set to SM_ALLOC. Only > > > drivers/md/persistent-dat/dm-space-map-common.c:sm_ll_mutate() sets > > > SM_ALLOC. sm_disk_new_block() is indirectly calling sm_ll_mutate() > > > > > > sm_ll_mutate() will only return 0 if ll->save_ie() does, the ll_disk *ll > > > should be ll_disk, and so disk_ll_save_ie()'s call to dm_btree_insert() > > > returns 0 -- which simply means success. And on success > > > sm_disk_new_block() assumes ev was set to SM_ALLOC (by sm_ll_mutate). > > > > > > sm_ll_mutate() decided to _not_ set SM_ALLOC because either: > > > 1) ref_count wasn't set > > > or > > > 2) old was identified > > > > > > So all said: somehow a new data block was found to already be in use. > > > _WHY_ that is the case isn't clear from this stack... > > > > > > But it does speak to the possibility of data block allocation racing > > > with other operations to the same block. Which implies missing locking. > > > > Where would you look to add locking do you think? > > > > > But that's all I've got so far... I'll review past dm-thinp changes with > > > all this in mind and see what turns up. But Joe Thornber (ejt) likely > > > needs to have a look at this too. > > > > > > But could it be that bcache is the source of the data device race (same > > > block used concurrently)? And DM thinp is acting as the canary in the > > > coal mine? > > > > As Marcus has shown, this bug triggers without bcache. > > > > > > Other questions: > > > > 1) Can sm_disk_new_block() fail more gracefully than BUG_ON? For example: > > > > + spin_lock(&lock); /* protect smd->begin */ > > smd->begin = *b + 1; > > r = sm_ll_inc(&smd->ll, *b, &ev); > > if (!r) { > > - BUG_ON(ev != SM_ALLOC); > > smd->nr_allocated_this_transaction++; > > } > > + else { > > + r = -ENOSPC; > > + smd->begin = *b - 1; > > + } > > + spin_unlock(&lock); > > Just hit the bug again without mq-scsi (scsi_mod.use_blk_mq=n), all other > times MQ has been turned on. > > I'm trying the -ENOSPC hack which will flag the pool as being out of space > so I can recover more gracefully than a BUG_ON. Here's a first-draft > patch, maybe the spinlock will even prevent the issue. > > Compile tested, I'll try on a real system tomorrow. > > Comments welcome: > > diff --git a/drivers/md/persistent-data/dm-space-map-disk.c b/drivers/md/persistent-data/dm-space-map-disk.c > index 32adf6b..cb27a20 100644 > --- a/drivers/md/persistent-data/dm-space-map-disk.c > +++ b/drivers/md/persistent-data/dm-space-map-disk.c > @@ -161,6 +161,7 @@ static int sm_disk_dec_block(struct dm_space_map *sm, dm_block_t b) > return r; > } > > +static DEFINE_SPINLOCK(smd_lock); > static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > { > int r; > @@ -168,17 +169,30 @@ static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > struct sm_disk *smd = container_of(sm, struct sm_disk, sm); > > /* FIXME: we should loop round a couple of times */ > + spin_lock(&smd_lock); > r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); > - if (r) > + if (r) { > + spin_unlock(&smd_lock); > return r; > + } > > smd->begin = *b + 1; > r = sm_ll_inc(&smd->ll, *b, &ev); > if (!r) { > - BUG_ON(ev != SM_ALLOC); > - smd->nr_allocated_this_transaction++; > + if (ev == SM_ALLOC) > + smd->nr_allocated_this_transaction++; > + else { > + /* Not actually out of space, this is a bug: > + * https://lore.kernel.org/linux-block/20190925200138.GA20584 at redhat.com/ > + */ > + WARN(ev != SM_ALLOC, "Pool metadata allocation race, marking pool out-of-space."); > + r = -ENOSPC; > + smd->begin = *b - 1; > + } > } > > + spin_unlock(&smd_lock); > + > return r; > } So far, so good. There are 3 systems running with the patch (1 w/ MQ=y, 2 w/ MQ=n), time will tell if we hit the WARN or if the spinlock prevents the race. -- Eric Wheeler > > > > -- > Eric Wheeler > > > > > > The lock might protect smd->begin, but I'm not sure how &smd->ll might > > have been modified by sm_ll_inc(). However, since ll->save_ie() failed in > > sm_ll_mutate() then perhaps this is safe. What do you think? > > > > Putting the pool into PM_OUT_OF_DATA_SPACE isn't ideal since it isn't out > > of space, but I would take it over a BUG_ON. > > > > 2) If example #1 above returned -EAGAIN, how might alloc_data_block be > > taught retry? This bug shows up weeks or months apart, even on heavily > > loaded systems with ~100 live thin volumes, so retrying would be fine > > IMHO. > > > > 3) In the thread from June, Marcus says: > > "Judging from the call trace, my guess is that there is somewhere > > a race condition, when a new block needs to be allocated which has > > still to be discarded." > > > > Is this discard sitation possible? Wouldn't the bio prison prevent this? > > > > -- > > Eric Wheeler > > www.datawall.us > > > > > > > > Here is the new trace, old trace below: > > > > kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! > > invalid opcode: 0000 [#1] SMP NOPTI > > CPU: 11 PID: 22939 Comm: kworker/u48:1 Not tainted 4.19.86 #1 > > Hardware name: Supermicro SYS-2026T-6RFT+/X8DTU-6+, BIOS 2.1c 11/30/2012 > > Workqueue: dm-thin do_worker [dm_thin_pool] > > RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 b0 a5 a9 e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 > > RSP: 0018:ffffc90007237c78 EFLAGS: 00010297 > > RAX: 0000000000000000 RBX: ffff88861d7ac000 RCX: 0000000000000000 > > RDX: ffff8885f3d13f00 RSI: 0000000000000246 RDI: ffff888620e38c00 > > RBP: 0000000000000000 R08: 0000000000000000 R09: ffff888620e38c98 > > R10: ffffffff810f9177 R11: 0000000000000000 R12: ffffc90007237d48 > > R13: ffffc90007237d48 R14: 00000000ffffffc3 R15: 00000000000d9e3a > > FS: 0000000000000000(0000) GS:ffff8886279c0000(0000) knlGS:0000000000000000 > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > CR2: 00000000081ceff0 CR3: 000000000200a004 CR4: 00000000000226e0 > > Call Trace: > > dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > process_cell+0x2a3/0x550 [dm_thin_pool] > > ? mempool_alloc+0x6f/0x180 > > ? sort+0x17b/0x270 > > ? u32_swap+0x10/0x10 > > process_deferred_bios+0x1af/0x870 [dm_thin_pool] > > do_worker+0x94/0xe0 [dm_thin_pool] > > process_one_work+0x171/0x370 > > worker_thread+0x49/0x3f0 > > kthread+0xf8/0x130 > > ? max_active_store+0x80/0x80 > > ? kthread_bind+0x10/0x10 > > ret_from_fork+0x1f/0x40 > > > > > > > > [199391.677689] ------------[ cut here ]------------ > > > [199391.678437] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! > > > [199391.679183] invalid opcode: 0000 [#1] SMP NOPTI > > > [199391.679941] CPU: 4 PID: 31359 Comm: kworker/u16:4 Not tainted 4.19.75 #1 > > > [199391.680683] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > > > [199391.681446] Workqueue: dm-thin do_worker [dm_thin_pool] > > > [199391.682187] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > > [199391.682929] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > > [199391.684432] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > > [199391.685186] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > > [199391.685936] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > > [199391.686659] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > > [199391.687379] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > > [199391.688120] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > > [199391.688843] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > > [199391.689571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [199391.690253] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > > [199391.690984] Call Trace: > > > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > > > [199391.693852] ? sort+0x17b/0x270 > > > [199391.694527] ? u32_swap+0x10/0x10 > > > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > > > [199391.695890] process_one_work+0x171/0x370 > > > [199391.696640] worker_thread+0x49/0x3f0 > > > [199391.697332] kthread+0xf8/0x130 > > > [199391.697988] ? max_active_store+0x80/0x80 > > > [199391.698659] ? kthread_bind+0x10/0x10 > > > [199391.699281] ret_from_fork+0x1f/0x40 > > > [199391.699930] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_bit drm_kms_helper > > > [199391.705631] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > > > [199391.708083] ---[ end trace c31536d98046e8ec ]--- > > > [199391.866776] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > > [199391.867960] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > > [199391.870379] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > > [199391.871524] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > > [199391.872364] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > > [199391.873173] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > > [199391.873871] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > > [199391.874550] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > > [199391.875231] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > > [199391.875941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [199391.876633] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > > [199391.877317] Kernel panic - not syncing: Fatal exception > > > [199391.878006] Kernel Offset: disabled > > > [199392.032304] ---[ end Kernel panic - not syncing: Fatal exception ]--- > > > [199392.032962] unchecked MSR access error: WRMSR to 0x83f (tried to write 0x00000000000000f6) at rIP: 0xffffffff81067af4 (native_write_msr+0x4/0x20) > > > [199392.034277] Call Trace: > > > [199392.034929] <IRQ> > > > [199392.035576] native_apic_msr_write+0x2e/0x40 > > > [199392.036228] arch_irq_work_raise+0x28/0x40 > > > [199392.036877] irq_work_queue_on+0x83/0xa0 > > > [199392.037518] irq_work_run_list+0x4c/0x70 > > > [199392.038149] irq_work_run+0x14/0x40 > > > [199392.038771] smp_call_function_single_interrupt+0x3a/0xd0 > > > [199392.039393] call_function_single_interrupt+0xf/0x20 > > > [199392.040011] </IRQ> > > > [199392.040624] RIP: 0010:panic+0x209/0x25c > > > [199392.041234] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > > > [199392.042518] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 > > > [199392.043174] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > > > [199392.043833] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > > > [199392.044493] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > > > [199392.045155] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > > > [199392.045820] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > > > [199392.046486] oops_end+0xc1/0xd0 > > > [199392.047149] do_trap+0x13d/0x150 > > > [199392.047795] do_error_trap+0xd5/0x130 > > > [199392.048427] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > > [199392.049048] invalid_op+0x14/0x20 > > > [199392.049650] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > > [199392.050245] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > > [199392.051434] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > > [199392.052010] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > > [199392.052580] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > > [199392.053150] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > > [199392.053715] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > > [199392.054266] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > > [199392.054807] ? __wake_up_common_lock+0x87/0xc0 > > > [199392.055342] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > > > [199392.055877] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > > [199392.056410] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > > [199392.056947] process_cell+0x2a3/0x550 [dm_thin_pool] > > > [199392.057484] ? sort+0x17b/0x270 > > > [199392.058016] ? u32_swap+0x10/0x10 > > > [199392.058538] do_worker+0x268/0x9a0 [dm_thin_pool] > > > [199392.059060] process_one_work+0x171/0x370 > > > [199392.059576] worker_thread+0x49/0x3f0 > > > [199392.060083] kthread+0xf8/0x130 > > > [199392.060587] ? max_active_store+0x80/0x80 > > > [199392.061086] ? kthread_bind+0x10/0x10 > > > [199392.061569] ret_from_fork+0x1f/0x40 > > > [199392.062038] ------------[ cut here ]------------ > > > [199392.062508] sched: Unexpected reschedule of offline CPU#1! > > > [199392.062989] WARNING: CPU: 4 PID: 31359 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40 > > > [199392.063485] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_bit drm_kms_helper > > > [199392.067463] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > > > [199392.068729] CPU: 4 PID: 31359 Comm: kworker/u16:4 Tainted: G D 4.19.75 #1 > > > [199392.069356] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > > > [199392.069982] Workqueue: dm-thin do_worker [dm_thin_pool] > > > [199392.070604] RIP: 0010:native_smp_send_reschedule+0x39/0x40 > > > [199392.071229] Code: 0f 92 c0 84 c0 74 15 48 8b 05 93 f1 eb 00 be fd 00 00 00 48 8b 40 30 e9 85 f4 9a 00 89 fe 48 c7 c7 08 23 e5 81 e8 c7 9a 05 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48 > > > [199392.072528] RSP: 0018:ffff88880fb03dc0 EFLAGS: 00010082 > > > [199392.073186] RAX: 0000000000000000 RBX: ffff88880fa625c0 RCX: 0000000000000006 > > > [199392.073847] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff88880fb168b0 > > > [199392.074512] RBP: ffff88880fa625c0 R08: 0000000000000001 R09: 00000000000174e5 > > > [199392.075182] R10: ffff8881f2512300 R11: 0000000000000000 R12: ffff888804541640 > > > [199392.075849] R13: ffff88880fb03e08 R14: 0000000000000000 R15: 0000000000000001 > > > [199392.076512] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > > [199392.077179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [199392.077833] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > > [199392.078481] Call Trace: > > > [199392.079117] <IRQ> > > > [199392.079745] check_preempt_curr+0x6b/0x90 > > > [199392.080373] ttwu_do_wakeup+0x19/0x130 > > > [199392.080999] try_to_wake_up+0x1e2/0x460 > > > [199392.081623] __wake_up_common+0x8f/0x160 > > > [199392.082246] ep_poll_callback+0x1af/0x300 > > > [199392.082860] __wake_up_common+0x8f/0x160 > > > [199392.083470] __wake_up_common_lock+0x7a/0xc0 > > > [199392.084074] irq_work_run_list+0x4c/0x70 > > > [199392.084675] smp_call_function_single_interrupt+0x3a/0xd0 > > > [199392.085277] call_function_single_interrupt+0xf/0x20 > > > [199392.085879] </IRQ> > > > [199392.086477] RIP: 0010:panic+0x209/0x25c > > > [199392.087079] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > > > [199392.088341] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 > > > [199392.088988] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > > > [199392.089639] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > > > [199392.090291] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > > > [199392.090947] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > > > [199392.091601] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > > > [199392.092255] oops_end+0xc1/0xd0 > > > [199392.092894] do_trap+0x13d/0x150 > > > [199392.093516] do_error_trap+0xd5/0x130 > > > [199392.094122] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > > [199392.094718] invalid_op+0x14/0x20 > > > [199392.095296] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > > [199392.095866] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > > [199392.097006] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > > [199392.097558] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > > [199392.098103] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > > [199392.098644] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > > [199392.099176] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > > [199392.099703] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > > [199392.100229] ? __wake_up_common_lock+0x87/0xc0 > > > [199392.100744] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > > > [199392.101251] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > > [199392.101752] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > > [199392.102252] process_cell+0x2a3/0x550 [dm_thin_pool] > > > [199392.102750] ? sort+0x17b/0x270 > > > [199392.103242] ? u32_swap+0x10/0x10 > > > [199392.103733] do_worker+0x268/0x9a0 [dm_thin_pool] > > > [199392.104228] process_one_work+0x171/0x370 > > > [199392.104714] worker_thread+0x49/0x3f0 > > > [199392.105193] kthread+0xf8/0x130 > > > [199392.105665] ? max_active_store+0x80/0x80 > > > [199392.106132] ? kthread_bind+0x10/0x10 > > > [199392.106601] ret_from_fork+0x1f/0x40 > > > [199392.107069] ---[ end trace c31536d98046e8ed ]--- > > > [199392.107544] ------------[ cut here ]------------ > > > [199392.108017] sched: Unexpected reschedule of offline CPU#7! > > > [199392.108497] WARNING: CPU: 4 PID: 31359 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40 > > > [199392.108996] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c_algo_bit drm_kms_helper > > > [199392.112967] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > > > [199392.114203] CPU: 4 PID: 31359 Comm: kworker/u16:4 Tainted: G D W 4.19.75 #1 > > > [199392.114819] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > > > [199392.115439] Workqueue: dm-thin do_worker [dm_thin_pool] > > > [199392.116061] RIP: 0010:native_smp_send_reschedule+0x39/0x40 > > > [199392.116683] Code: 0f 92 c0 84 c0 74 15 48 8b 05 93 f1 eb 00 be fd 00 00 00 48 8b 40 30 e9 85 f4 9a 00 89 fe 48 c7 c7 08 23 e5 81 e8 c7 9a 05 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48 > > > [199392.117982] RSP: 0018:ffff88880fb03ee0 EFLAGS: 00010082 > > > [199392.118632] RAX: 0000000000000000 RBX: ffff8887d093ac80 RCX: 0000000000000006 > > > [199392.119295] RDX: 0000000000000000 RSI: 0000000000000092 RDI: ffff88880fb168b0 > > > [199392.119961] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000017529 > > > [199392.120623] R10: ffff8881438ba800 R11: 0000000000000000 R12: ffffc9000a147988 > > > [199392.121283] R13: ffffffff8113f9a0 R14: 0000000000000002 R15: ffff88880fb1cff8 > > > [199392.121938] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > > [199392.122589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [199392.123229] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > > [199392.123867] Call Trace: > > > [199392.124495] <IRQ> > > > [199392.125116] update_process_times+0x40/0x50 > > > [199392.125742] tick_sched_handle+0x25/0x60 > > > [199392.126367] tick_sched_timer+0x37/0x70 > > > [199392.126987] __hrtimer_run_queues+0xfb/0x270 > > > [199392.127601] hrtimer_interrupt+0x122/0x270 > > > [199392.128211] smp_apic_timer_interrupt+0x6a/0x140 > > > [199392.128819] apic_timer_interrupt+0xf/0x20 > > > [199392.129423] </IRQ> > > > [199392.130020] RIP: 0010:panic+0x209/0x25c > > > [199392.130619] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > > > [199392.131883] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 > > > [199392.132530] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > > > [199392.133180] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > > > [199392.133830] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > > > [199392.134482] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > > > [199392.135136] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > > > [199392.135792] oops_end+0xc1/0xd0 > > > [199392.136444] do_trap+0x13d/0x150 > > > [199392.137094] do_error_trap+0xd5/0x130 > > > [199392.137740] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > > [199392.138383] invalid_op+0x14/0x20 > > > [199392.139010] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > > [199392.139630] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > > [199392.140870] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > > [199392.141472] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > > [199392.142066] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > > [199392.142647] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > > [199392.143211] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > > [199392.143759] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > > [199392.144301] ? __wake_up_common_lock+0x87/0xc0 > > > [199392.144838] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > > > [199392.145374] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > > [199392.145909] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > > [199392.146435] process_cell+0x2a3/0x550 [dm_thin_pool] > > > [199392.146949] ? sort+0x17b/0x270 > > > [199392.147450] ? u32_swap+0x10/0x10 > > > [199392.147944] do_worker+0x268/0x9a0 [dm_thin_pool] > > > [199392.148441] process_one_work+0x171/0x370 > > > [199392.148937] worker_thread+0x49/0x3f0 > > > [199392.149430] kthread+0xf8/0x130 > > > [199392.149922] ? max_active_store+0x80/0x80 > > > [199392.150406] ? kthread_bind+0x10/0x10 > > > [199392.150883] ret_from_fork+0x1f/0x40 > > > [199392.151353] ---[ end trace c31536d98046e8ee ]--- > > > > > > > > > -- > > > Eric Wheeler > > > > > > -- > > > dm-devel mailing list > > > dm-devel at redhat.com > > > https://www.redhat.com/mailman/listinfo/dm-devel > > > > > > > > > -- > > dm-devel mailing list > > dm-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/dm-devel > > > > > > > -- > dm-devel mailing list > dm-devel at redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel > > ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 @ 2019-12-28 2:13 ` Eric Wheeler 0 siblings, 0 replies; 43+ messages in thread From: Eric Wheeler @ 2019-12-28 2:13 UTC (permalink / raw) To: lvm-devel Cc: Mike Snitzer, markus.schade, ejt, linux-block, dm-devel, joe.thornber On Fri, 27 Dec 2019, Eric Wheeler wrote: > On Fri, 20 Dec 2019, Eric Wheeler wrote: > > On Wed, 25 Sep 2019, Eric Wheeler wrote: > > > We are using the 4.19.75 stable tree with dm-thin and multi-queue scsi. > > > We have been using the 4.19 branch for months without issue; we just > > > switched to MQ and we seem to have hit this BUG_ON. Whether or not MQ is > > > related to the issue, I don't know, maybe coincidence: > > > > > > static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > > > { > > > int r; > > > enum allocation_event ev; > > > struct sm_disk *smd = container_of(sm, struct sm_disk, sm); > > > > > > /* FIXME: we should loop round a couple of times */ > > > r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); > > > if (r) > > > return r; > > > > > > smd->begin = *b + 1; > > > r = sm_ll_inc(&smd->ll, *b, &ev); > > > if (!r) { > > > BUG_ON(ev != SM_ALLOC); <<<<<<<<<<<<<<<<<<<<<<<<<<<<< > > > smd->nr_allocated_this_transaction++; > > > } > > > > Hello all, > > > > We hit this BUG_ON again, this time with 4.19.86 with > > scsi_mod.use_blk_mq=y, and it is known to be present as of 5.1.2 as > > additionally reported by Markus Schade: > > > > https://www.redhat.com/archives/dm-devel/2019-June/msg00116.html > > and > > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1777398 > > > > In our case, the most latest trace (below) is from a different system that > > has been stable for years on Linux 4.1 with tmeta direct on the SSD. > > We updated to 4.19.86 a few weeks ago and just hit this, what Mike > > Snitzer explains to be an allocator race: > > > > On Wed, 25 Sep 2019, Mike Snitzer wrote: > > > > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > > > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > > > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > > > > [199391.693852] ? sort+0x17b/0x270 > > > > [199391.694527] ? u32_swap+0x10/0x10 > > > > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > > > > [199391.695890] process_one_work+0x171/0x370 > > > > [199391.696640] worker_thread+0x49/0x3f0 > > > > [199391.697332] kthread+0xf8/0x130 > > > > [199391.697988] ? max_active_store+0x80/0x80 > > > > [199391.698659] ? kthread_bind+0x10/0x10 > > > > [199391.699281] ret_from_fork+0x1f/0x40 > > > > > > The stack shows the call to sm_disk_new_block() is due to > > > dm_pool_alloc_data_block(). > > > > > > sm_disk_new_block()'s BUG_ON(ev != SM_ALLOC) indicates that somehow it is > > > getting called without the passed 'ev' being set to SM_ALLOC. Only > > > drivers/md/persistent-dat/dm-space-map-common.c:sm_ll_mutate() sets > > > SM_ALLOC. sm_disk_new_block() is indirectly calling sm_ll_mutate() > > > > > > sm_ll_mutate() will only return 0 if ll->save_ie() does, the ll_disk *ll > > > should be ll_disk, and so disk_ll_save_ie()'s call to dm_btree_insert() > > > returns 0 -- which simply means success. And on success > > > sm_disk_new_block() assumes ev was set to SM_ALLOC (by sm_ll_mutate). > > > > > > sm_ll_mutate() decided to _not_ set SM_ALLOC because either: > > > 1) ref_count wasn't set > > > or > > > 2) old was identified > > > > > > So all said: somehow a new data block was found to already be in use. > > > _WHY_ that is the case isn't clear from this stack... > > > > > > But it does speak to the possibility of data block allocation racing > > > with other operations to the same block. Which implies missing locking. > > > > Where would you look to add locking do you think? > > > > > But that's all I've got so far... I'll review past dm-thinp changes with > > > all this in mind and see what turns up. But Joe Thornber (ejt) likely > > > needs to have a look at this too. > > > > > > But could it be that bcache is the source of the data device race (same > > > block used concurrently)? And DM thinp is acting as the canary in the > > > coal mine? > > > > As Marcus has shown, this bug triggers without bcache. > > > > > > Other questions: > > > > 1) Can sm_disk_new_block() fail more gracefully than BUG_ON? For example: > > > > + spin_lock(&lock); /* protect smd->begin */ > > smd->begin = *b + 1; > > r = sm_ll_inc(&smd->ll, *b, &ev); > > if (!r) { > > - BUG_ON(ev != SM_ALLOC); > > smd->nr_allocated_this_transaction++; > > } > > + else { > > + r = -ENOSPC; > > + smd->begin = *b - 1; > > + } > > + spin_unlock(&lock); > > Just hit the bug again without mq-scsi (scsi_mod.use_blk_mq=n), all other > times MQ has been turned on. > > I'm trying the -ENOSPC hack which will flag the pool as being out of space > so I can recover more gracefully than a BUG_ON. Here's a first-draft > patch, maybe the spinlock will even prevent the issue. > > Compile tested, I'll try on a real system tomorrow. > > Comments welcome: > > diff --git a/drivers/md/persistent-data/dm-space-map-disk.c b/drivers/md/persistent-data/dm-space-map-disk.c > index 32adf6b..cb27a20 100644 > --- a/drivers/md/persistent-data/dm-space-map-disk.c > +++ b/drivers/md/persistent-data/dm-space-map-disk.c > @@ -161,6 +161,7 @@ static int sm_disk_dec_block(struct dm_space_map *sm, dm_block_t b) > return r; > } > > +static DEFINE_SPINLOCK(smd_lock); > static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > { > int r; > @@ -168,17 +169,30 @@ static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > struct sm_disk *smd = container_of(sm, struct sm_disk, sm); > > /* FIXME: we should loop round a couple of times */ > + spin_lock(&smd_lock); > r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); > - if (r) > + if (r) { > + spin_unlock(&smd_lock); > return r; > + } > > smd->begin = *b + 1; > r = sm_ll_inc(&smd->ll, *b, &ev); > if (!r) { > - BUG_ON(ev != SM_ALLOC); > - smd->nr_allocated_this_transaction++; > + if (ev == SM_ALLOC) > + smd->nr_allocated_this_transaction++; > + else { > + /* Not actually out of space, this is a bug: > + * https://lore.kernel.org/linux-block/20190925200138.GA20584@redhat.com/ > + */ > + WARN(ev != SM_ALLOC, "Pool metadata allocation race, marking pool out-of-space."); > + r = -ENOSPC; > + smd->begin = *b - 1; > + } > } > > + spin_unlock(&smd_lock); > + > return r; > } So far, so good. There are 3 systems running with the patch (1 w/ MQ=y, 2 w/ MQ=n), time will tell if we hit the WARN or if the spinlock prevents the race. -- Eric Wheeler > > > > -- > Eric Wheeler > > > > > > The lock might protect smd->begin, but I'm not sure how &smd->ll might > > have been modified by sm_ll_inc(). However, since ll->save_ie() failed in > > sm_ll_mutate() then perhaps this is safe. What do you think? > > > > Putting the pool into PM_OUT_OF_DATA_SPACE isn't ideal since it isn't out > > of space, but I would take it over a BUG_ON. > > > > 2) If example #1 above returned -EAGAIN, how might alloc_data_block be > > taught retry? This bug shows up weeks or months apart, even on heavily > > loaded systems with ~100 live thin volumes, so retrying would be fine > > IMHO. > > > > 3) In the thread from June, Marcus says: > > "Judging from the call trace, my guess is that there is somewhere > > a race condition, when a new block needs to be allocated which has > > still to be discarded." > > > > Is this discard sitation possible? Wouldn't the bio prison prevent this? > > > > -- > > Eric Wheeler > > www.datawall.us > > > > > > > > Here is the new trace, old trace below: > > > > kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! > > invalid opcode: 0000 [#1] SMP NOPTI > > CPU: 11 PID: 22939 Comm: kworker/u48:1 Not tainted 4.19.86 #1 > > Hardware name: Supermicro SYS-2026T-6RFT+/X8DTU-6+, BIOS 2.1c 11/30/2012 > > Workqueue: dm-thin do_worker [dm_thin_pool] > > RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 b0 a5 a9 e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 > > RSP: 0018:ffffc90007237c78 EFLAGS: 00010297 > > RAX: 0000000000000000 RBX: ffff88861d7ac000 RCX: 0000000000000000 > > RDX: ffff8885f3d13f00 RSI: 0000000000000246 RDI: ffff888620e38c00 > > RBP: 0000000000000000 R08: 0000000000000000 R09: ffff888620e38c98 > > R10: ffffffff810f9177 R11: 0000000000000000 R12: ffffc90007237d48 > > R13: ffffc90007237d48 R14: 00000000ffffffc3 R15: 00000000000d9e3a > > FS: 0000000000000000(0000) GS:ffff8886279c0000(0000) knlGS:0000000000000000 > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > CR2: 00000000081ceff0 CR3: 000000000200a004 CR4: 00000000000226e0 > > Call Trace: > > dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > process_cell+0x2a3/0x550 [dm_thin_pool] > > ? mempool_alloc+0x6f/0x180 > > ? sort+0x17b/0x270 > > ? u32_swap+0x10/0x10 > > process_deferred_bios+0x1af/0x870 [dm_thin_pool] > > do_worker+0x94/0xe0 [dm_thin_pool] > > process_one_work+0x171/0x370 > > worker_thread+0x49/0x3f0 > > kthread+0xf8/0x130 > > ? max_active_store+0x80/0x80 > > ? kthread_bind+0x10/0x10 > > ret_from_fork+0x1f/0x40 > > > > > > > > [199391.677689] ------------[ cut here ]------------ > > > [199391.678437] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178! > > > [199391.679183] invalid opcode: 0000 [#1] SMP NOPTI > > > [199391.679941] CPU: 4 PID: 31359 Comm: kworker/u16:4 Not tainted 4.19.75 #1 > > > [199391.680683] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > > > [199391.681446] Workqueue: dm-thin do_worker [dm_thin_pool] > > > [199391.682187] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > > [199391.682929] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > > [199391.684432] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > > [199391.685186] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > > [199391.685936] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > > [199391.686659] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > > [199391.687379] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > > [199391.688120] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > > [199391.688843] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > > [199391.689571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [199391.690253] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > > [199391.690984] Call Trace: > > > [199391.691714] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > > [199391.692411] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > > [199391.693142] process_cell+0x2a3/0x550 [dm_thin_pool] > > > [199391.693852] ? sort+0x17b/0x270 > > > [199391.694527] ? u32_swap+0x10/0x10 > > > [199391.695192] do_worker+0x268/0x9a0 [dm_thin_pool] > > > [199391.695890] process_one_work+0x171/0x370 > > > [199391.696640] worker_thread+0x49/0x3f0 > > > [199391.697332] kthread+0xf8/0x130 > > > [199391.697988] ? max_active_store+0x80/0x80 > > > [199391.698659] ? kthread_bind+0x10/0x10 > > > [199391.699281] ret_from_fork+0x1f/0x40 > > > [199391.699930] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c _algo_bit drm_kms_helper > > > [199391.705631] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > > > [199391.708083] ---[ end trace c31536d98046e8ec ]--- > > > [199391.866776] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > > [199391.867960] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > > [199391.870379] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > > [199391.871524] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > > [199391.872364] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > > [199391.873173] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > > [199391.873871] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > > [199391.874550] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > > [199391.875231] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > > [199391.875941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [199391.876633] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > > [199391.877317] Kernel panic - not syncing: Fatal exception > > > [199391.878006] Kernel Offset: disabled > > > [199392.032304] ---[ end Kernel panic - not syncing: Fatal exception ]--- > > > [199392.032962] unchecked MSR access error: WRMSR to 0x83f (tried to write 0x00000000000000f6) at rIP: 0xffffffff81067af4 (native_write_msr+0x4/0x20) > > > [199392.034277] Call Trace: > > > [199392.034929] <IRQ> > > > [199392.035576] native_apic_msr_write+0x2e/0x40 > > > [199392.036228] arch_irq_work_raise+0x28/0x40 > > > [199392.036877] irq_work_queue_on+0x83/0xa0 > > > [199392.037518] irq_work_run_list+0x4c/0x70 > > > [199392.038149] irq_work_run+0x14/0x40 > > > [199392.038771] smp_call_function_single_interrupt+0x3a/0xd0 > > > [199392.039393] call_function_single_interrupt+0xf/0x20 > > > [199392.040011] </IRQ> > > > [199392.040624] RIP: 0010:panic+0x209/0x25c > > > [199392.041234] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > > > [199392.042518] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 > > > [199392.043174] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > > > [199392.043833] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > > > [199392.044493] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > > > [199392.045155] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > > > [199392.045820] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > > > [199392.046486] oops_end+0xc1/0xd0 > > > [199392.047149] do_trap+0x13d/0x150 > > > [199392.047795] do_error_trap+0xd5/0x130 > > > [199392.048427] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > > [199392.049048] invalid_op+0x14/0x20 > > > [199392.049650] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > > [199392.050245] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > > [199392.051434] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > > [199392.052010] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > > [199392.052580] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > > [199392.053150] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > > [199392.053715] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > > [199392.054266] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > > [199392.054807] ? __wake_up_common_lock+0x87/0xc0 > > > [199392.055342] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > > > [199392.055877] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > > [199392.056410] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > > [199392.056947] process_cell+0x2a3/0x550 [dm_thin_pool] > > > [199392.057484] ? sort+0x17b/0x270 > > > [199392.058016] ? u32_swap+0x10/0x10 > > > [199392.058538] do_worker+0x268/0x9a0 [dm_thin_pool] > > > [199392.059060] process_one_work+0x171/0x370 > > > [199392.059576] worker_thread+0x49/0x3f0 > > > [199392.060083] kthread+0xf8/0x130 > > > [199392.060587] ? max_active_store+0x80/0x80 > > > [199392.061086] ? kthread_bind+0x10/0x10 > > > [199392.061569] ret_from_fork+0x1f/0x40 > > > [199392.062038] ------------[ cut here ]------------ > > > [199392.062508] sched: Unexpected reschedule of offline CPU#1! > > > [199392.062989] WARNING: CPU: 4 PID: 31359 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40 > > > [199392.063485] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c _algo_bit drm_kms_helper > > > [199392.067463] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > > > [199392.068729] CPU: 4 PID: 31359 Comm: kworker/u16:4 Tainted: G D 4.19.75 #1 > > > [199392.069356] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > > > [199392.069982] Workqueue: dm-thin do_worker [dm_thin_pool] > > > [199392.070604] RIP: 0010:native_smp_send_reschedule+0x39/0x40 > > > [199392.071229] Code: 0f 92 c0 84 c0 74 15 48 8b 05 93 f1 eb 00 be fd 00 00 00 48 8b 40 30 e9 85 f4 9a 00 89 fe 48 c7 c7 08 23 e5 81 e8 c7 9a 05 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48 > > > [199392.072528] RSP: 0018:ffff88880fb03dc0 EFLAGS: 00010082 > > > [199392.073186] RAX: 0000000000000000 RBX: ffff88880fa625c0 RCX: 0000000000000006 > > > [199392.073847] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff88880fb168b0 > > > [199392.074512] RBP: ffff88880fa625c0 R08: 0000000000000001 R09: 00000000000174e5 > > > [199392.075182] R10: ffff8881f2512300 R11: 0000000000000000 R12: ffff888804541640 > > > [199392.075849] R13: ffff88880fb03e08 R14: 0000000000000000 R15: 0000000000000001 > > > [199392.076512] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > > [199392.077179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [199392.077833] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > > [199392.078481] Call Trace: > > > [199392.079117] <IRQ> > > > [199392.079745] check_preempt_curr+0x6b/0x90 > > > [199392.080373] ttwu_do_wakeup+0x19/0x130 > > > [199392.080999] try_to_wake_up+0x1e2/0x460 > > > [199392.081623] __wake_up_common+0x8f/0x160 > > > [199392.082246] ep_poll_callback+0x1af/0x300 > > > [199392.082860] __wake_up_common+0x8f/0x160 > > > [199392.083470] __wake_up_common_lock+0x7a/0xc0 > > > [199392.084074] irq_work_run_list+0x4c/0x70 > > > [199392.084675] smp_call_function_single_interrupt+0x3a/0xd0 > > > [199392.085277] call_function_single_interrupt+0xf/0x20 > > > [199392.085879] </IRQ> > > > [199392.086477] RIP: 0010:panic+0x209/0x25c > > > [199392.087079] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > > > [199392.088341] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 > > > [199392.088988] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > > > [199392.089639] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > > > [199392.090291] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > > > [199392.090947] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > > > [199392.091601] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > > > [199392.092255] oops_end+0xc1/0xd0 > > > [199392.092894] do_trap+0x13d/0x150 > > > [199392.093516] do_error_trap+0xd5/0x130 > > > [199392.094122] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > > [199392.094718] invalid_op+0x14/0x20 > > > [199392.095296] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > > [199392.095866] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > > [199392.097006] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > > [199392.097558] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > > [199392.098103] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > > [199392.098644] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > > [199392.099176] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > > [199392.099703] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > > [199392.100229] ? __wake_up_common_lock+0x87/0xc0 > > > [199392.100744] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > > > [199392.101251] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > > [199392.101752] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > > [199392.102252] process_cell+0x2a3/0x550 [dm_thin_pool] > > > [199392.102750] ? sort+0x17b/0x270 > > > [199392.103242] ? u32_swap+0x10/0x10 > > > [199392.103733] do_worker+0x268/0x9a0 [dm_thin_pool] > > > [199392.104228] process_one_work+0x171/0x370 > > > [199392.104714] worker_thread+0x49/0x3f0 > > > [199392.105193] kthread+0xf8/0x130 > > > [199392.105665] ? max_active_store+0x80/0x80 > > > [199392.106132] ? kthread_bind+0x10/0x10 > > > [199392.106601] ret_from_fork+0x1f/0x40 > > > [199392.107069] ---[ end trace c31536d98046e8ed ]--- > > > [199392.107544] ------------[ cut here ]------------ > > > [199392.108017] sched: Unexpected reschedule of offline CPU#7! > > > [199392.108497] WARNING: CPU: 4 PID: 31359 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40 > > > [199392.108996] Modules linked in: dm_snapshot btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio drbd lru_cache xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables binfmt_misc ip6table_filter ip6_tables bcache xt_comment crc64 iptable_filter netconsole bridge 8021q garp stp mrp llc lz4 lz4_compress zram sunrpc x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul pcc_cpufreq ghash_clmulni_intel pcspkr sg ipmi_si ipmi_devintf lpc_ich ipmi_msghandler video i2c_i801 mfd_core ip_tables ext4 mbcache jbd2 mgag200 i2c _algo_bit drm_kms_helper > > > [199392.112967] syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel i2c_core ahci libahci ixgbe libata e1000e arcmsr mdio dca dm_mirror dm_region_hash dm_log dm_mod > > > [199392.114203] CPU: 4 PID: 31359 Comm: kworker/u16:4 Tainted: G D W 4.19.75 #1 > > > [199392.114819] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 > > > [199392.115439] Workqueue: dm-thin do_worker [dm_thin_pool] > > > [199392.116061] RIP: 0010:native_smp_send_reschedule+0x39/0x40 > > > [199392.116683] Code: 0f 92 c0 84 c0 74 15 48 8b 05 93 f1 eb 00 be fd 00 00 00 48 8b 40 30 e9 85 f4 9a 00 89 fe 48 c7 c7 08 23 e5 81 e8 c7 9a 05 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48 > > > [199392.117982] RSP: 0018:ffff88880fb03ee0 EFLAGS: 00010082 > > > [199392.118632] RAX: 0000000000000000 RBX: ffff8887d093ac80 RCX: 0000000000000006 > > > [199392.119295] RDX: 0000000000000000 RSI: 0000000000000092 RDI: ffff88880fb168b0 > > > [199392.119961] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000017529 > > > [199392.120623] R10: ffff8881438ba800 R11: 0000000000000000 R12: ffffc9000a147988 > > > [199392.121283] R13: ffffffff8113f9a0 R14: 0000000000000002 R15: ffff88880fb1cff8 > > > [199392.121938] FS: 0000000000000000(0000) GS:ffff88880fb00000(0000) knlGS:0000000000000000 > > > [199392.122589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [199392.123229] CR2: 00007f5ae49a1000 CR3: 000000000200a003 CR4: 00000000001626e0 > > > [199392.123867] Call Trace: > > > [199392.124495] <IRQ> > > > [199392.125116] update_process_times+0x40/0x50 > > > [199392.125742] tick_sched_handle+0x25/0x60 > > > [199392.126367] tick_sched_timer+0x37/0x70 > > > [199392.126987] __hrtimer_run_queues+0xfb/0x270 > > > [199392.127601] hrtimer_interrupt+0x122/0x270 > > > [199392.128211] smp_apic_timer_interrupt+0x6a/0x140 > > > [199392.128819] apic_timer_interrupt+0xf/0x20 > > > [199392.129423] </IRQ> > > > [199392.130020] RIP: 0010:panic+0x209/0x25c > > > [199392.130619] Code: 83 3d 86 de 75 01 00 74 05 e8 ff 5d 02 00 48 c7 c6 e0 bc 80 82 48 c7 c7 10 cd e5 81 31 c0 e8 fe 2b 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 eb 2e 0d 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 2b de > > > [199392.131883] RSP: 0018:ffffc9000a147a30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 > > > [199392.132530] RAX: 0000000000000039 RBX: 0000000000000200 RCX: 0000000000000006 > > > [199392.133180] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88880fb168b0 > > > [199392.133830] RBP: ffffc9000a147aa0 R08: 0000000000000001 R09: 00000000000174b5 > > > [199392.134482] R10: ffff8883d9753f00 R11: 0000000000000001 R12: ffffffff81e4a15c > > > [199392.135136] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81e49951 > > > [199392.135792] oops_end+0xc1/0xd0 > > > [199392.136444] do_trap+0x13d/0x150 > > > [199392.137094] do_error_trap+0xd5/0x130 > > > [199392.137740] ? sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > > [199392.138383] invalid_op+0x14/0x20 > > > [199392.139010] RIP: 0010:sm_disk_new_block+0xa0/0xb0 [dm_persistent_data] > > > [199392.139630] Code: 22 00 00 49 8b 34 24 e8 4e f9 ff ff 85 c0 75 11 83 7c 24 04 01 75 13 48 83 83 28 22 00 00 01 eb af 89 c5 eb ab e8 e0 95 9b e0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 > > > [199392.140870] RSP: 0018:ffffc9000a147c88 EFLAGS: 00010297 > > > [199392.141472] RAX: 0000000000000000 RBX: ffff8887ceed8000 RCX: 0000000000000000 > > > [199392.142066] RDX: ffff8887d093ac80 RSI: 0000000000000246 RDI: ffff8887faab0a00 > > > [199392.142647] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8887faab0a98 > > > [199392.143211] R10: ffffffff810f8077 R11: 0000000000000000 R12: ffffc9000a147d58 > > > [199392.143759] R13: ffffc9000a147d58 R14: 00000000ffffffc3 R15: 000000000014bbc0 > > > [199392.144301] ? __wake_up_common_lock+0x87/0xc0 > > > [199392.144838] ? sm_disk_new_block+0x82/0xb0 [dm_persistent_data] > > > [199392.145374] dm_pool_alloc_data_block+0x3f/0x60 [dm_thin_pool] > > > [199392.145909] alloc_data_block.isra.52+0x6d/0x1e0 [dm_thin_pool] > > > [199392.146435] process_cell+0x2a3/0x550 [dm_thin_pool] > > > [199392.146949] ? sort+0x17b/0x270 > > > [199392.147450] ? u32_swap+0x10/0x10 > > > [199392.147944] do_worker+0x268/0x9a0 [dm_thin_pool] > > > [199392.148441] process_one_work+0x171/0x370 > > > [199392.148937] worker_thread+0x49/0x3f0 > > > [199392.149430] kthread+0xf8/0x130 > > > [199392.149922] ? max_active_store+0x80/0x80 > > > [199392.150406] ? kthread_bind+0x10/0x10 > > > [199392.150883] ret_from_fork+0x1f/0x40 > > > [199392.151353] ---[ end trace c31536d98046e8ee ]--- > > > > > > > > > -- > > > Eric Wheeler > > > > > > -- > > > dm-devel mailing list > > > dm-devel@redhat.com > > > https://www.redhat.com/mailman/listinfo/dm-devel > > > > > > > > > -- > > dm-devel mailing list > > dm-devel@redhat.com > > https://www.redhat.com/mailman/listinfo/dm-devel > > > > > > > -- > dm-devel mailing list > dm-devel@redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel > > -- lvm-devel mailing list lvm-devel@redhat.com https://www.redhat.com/mailman/listinfo/lvm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [lvm-devel] [dm-devel] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 2019-12-28 2:13 ` Eric Wheeler (?) @ 2020-01-07 10:35 ` Joe Thornber -1 siblings, 0 replies; 43+ messages in thread From: Joe Thornber @ 2020-01-07 10:35 UTC (permalink / raw) To: LVM2 development Cc: Mike Snitzer, markus.schade, ejt, linux-block, dm-devel, joe.thornber On Sat, Dec 28, 2019 at 02:13:07AM +0000, Eric Wheeler wrote: > On Fri, 27 Dec 2019, Eric Wheeler wrote: > > Just hit the bug again without mq-scsi (scsi_mod.use_blk_mq=n), all other > > times MQ has been turned on. > > > > I'm trying the -ENOSPC hack which will flag the pool as being out of space > > so I can recover more gracefully than a BUG_ON. Here's a first-draft > > patch, maybe the spinlock will even prevent the issue. > > > > Compile tested, I'll try on a real system tomorrow. > > > > Comments welcome: Both sm_ll_find_free_block() and sm_ll_inc() can trigger synchronous IO. So you absolutely cannot use a spin lock. dm_pool_alloc_data_block() holds a big rw semaphore which should prevent anything else trying to allocate at the same time. - Joe ^ permalink raw reply [flat|nested] 43+ messages in thread
* [dm-devel] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 @ 2020-01-07 10:35 ` Joe Thornber 0 siblings, 0 replies; 43+ messages in thread From: Joe Thornber @ 2020-01-07 10:35 UTC (permalink / raw) To: lvm-devel On Sat, Dec 28, 2019 at 02:13:07AM +0000, Eric Wheeler wrote: > On Fri, 27 Dec 2019, Eric Wheeler wrote: > > Just hit the bug again without mq-scsi (scsi_mod.use_blk_mq=n), all other > > times MQ has been turned on. > > > > I'm trying the -ENOSPC hack which will flag the pool as being out of space > > so I can recover more gracefully than a BUG_ON. Here's a first-draft > > patch, maybe the spinlock will even prevent the issue. > > > > Compile tested, I'll try on a real system tomorrow. > > > > Comments welcome: Both sm_ll_find_free_block() and sm_ll_inc() can trigger synchronous IO. So you absolutely cannot use a spin lock. dm_pool_alloc_data_block() holds a big rw semaphore which should prevent anything else trying to allocate at the same time. - Joe ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 @ 2020-01-07 10:35 ` Joe Thornber 0 siblings, 0 replies; 43+ messages in thread From: Joe Thornber @ 2020-01-07 10:35 UTC (permalink / raw) To: LVM2 development Cc: Mike Snitzer, markus.schade, dm-devel, linux-block, ejt, joe.thornber On Sat, Dec 28, 2019 at 02:13:07AM +0000, Eric Wheeler wrote: > On Fri, 27 Dec 2019, Eric Wheeler wrote: > > Just hit the bug again without mq-scsi (scsi_mod.use_blk_mq=n), all other > > times MQ has been turned on. > > > > I'm trying the -ENOSPC hack which will flag the pool as being out of space > > so I can recover more gracefully than a BUG_ON. Here's a first-draft > > patch, maybe the spinlock will even prevent the issue. > > > > Compile tested, I'll try on a real system tomorrow. > > > > Comments welcome: Both sm_ll_find_free_block() and sm_ll_inc() can trigger synchronous IO. So you absolutely cannot use a spin lock. dm_pool_alloc_data_block() holds a big rw semaphore which should prevent anything else trying to allocate at the same time. - Joe -- lvm-devel mailing list lvm-devel@redhat.com https://www.redhat.com/mailman/listinfo/lvm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [lvm-devel] [dm-devel] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 2020-01-07 10:35 ` Joe Thornber (?) @ 2020-01-07 10:46 ` Joe Thornber -1 siblings, 0 replies; 43+ messages in thread From: Joe Thornber @ 2020-01-07 10:46 UTC (permalink / raw) To: LVM2 development, Mike Snitzer, markus.schade, ejt, linux-block, dm-devel, joe.thornber On Tue, Jan 07, 2020 at 10:35:46AM +0000, Joe Thornber wrote: > On Sat, Dec 28, 2019 at 02:13:07AM +0000, Eric Wheeler wrote: > > On Fri, 27 Dec 2019, Eric Wheeler wrote: > > > > Just hit the bug again without mq-scsi (scsi_mod.use_blk_mq=n), all other > > > times MQ has been turned on. > > > > > > I'm trying the -ENOSPC hack which will flag the pool as being out of space > > > so I can recover more gracefully than a BUG_ON. Here's a first-draft > > > patch, maybe the spinlock will even prevent the issue. > > > > > > Compile tested, I'll try on a real system tomorrow. > > > > > > Comments welcome: > > Both sm_ll_find_free_block() and sm_ll_inc() can trigger synchronous IO. So you > absolutely cannot use a spin lock. > > dm_pool_alloc_data_block() holds a big rw semaphore which should prevent anything > else trying to allocate at the same time. I suspect the problem is to do with the way we search for the new block in the space map for the previous transaction (sm->old_ll), and then increment in the current transaction (sm->ll). We keep old_ll around so we can ensure we never (re) allocate a block that's used in the previous transaction. This gives us our crash resistance, since if anything goes wrong we effectively rollback to the previous transaction. What I think we should be doing is running find_free on the old_ll, then double checking it's not actually used in the current transaction. ATM we're relying on smd->begin being set properly everywhere, and I suspect this isn't the case. A quick look shows sm_disk_inc_block() doesn't adjust it. sm_disk_inc_block() can be called when breaking sharing of a neighbouring entry in a leaf btree node ... and we know you use snapshots very heavily. I'll get a patch to you later today. - Joe ^ permalink raw reply [flat|nested] 43+ messages in thread
* [dm-devel] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 @ 2020-01-07 10:46 ` Joe Thornber 0 siblings, 0 replies; 43+ messages in thread From: Joe Thornber @ 2020-01-07 10:46 UTC (permalink / raw) To: lvm-devel On Tue, Jan 07, 2020 at 10:35:46AM +0000, Joe Thornber wrote: > On Sat, Dec 28, 2019 at 02:13:07AM +0000, Eric Wheeler wrote: > > On Fri, 27 Dec 2019, Eric Wheeler wrote: > > > > Just hit the bug again without mq-scsi (scsi_mod.use_blk_mq=n), all other > > > times MQ has been turned on. > > > > > > I'm trying the -ENOSPC hack which will flag the pool as being out of space > > > so I can recover more gracefully than a BUG_ON. Here's a first-draft > > > patch, maybe the spinlock will even prevent the issue. > > > > > > Compile tested, I'll try on a real system tomorrow. > > > > > > Comments welcome: > > Both sm_ll_find_free_block() and sm_ll_inc() can trigger synchronous IO. So you > absolutely cannot use a spin lock. > > dm_pool_alloc_data_block() holds a big rw semaphore which should prevent anything > else trying to allocate at the same time. I suspect the problem is to do with the way we search for the new block in the space map for the previous transaction (sm->old_ll), and then increment in the current transaction (sm->ll). We keep old_ll around so we can ensure we never (re) allocate a block that's used in the previous transaction. This gives us our crash resistance, since if anything goes wrong we effectively rollback to the previous transaction. What I think we should be doing is running find_free on the old_ll, then double checking it's not actually used in the current transaction. ATM we're relying on smd->begin being set properly everywhere, and I suspect this isn't the case. A quick look shows sm_disk_inc_block() doesn't adjust it. sm_disk_inc_block() can be called when breaking sharing of a neighbouring entry in a leaf btree node ... and we know you use snapshots very heavily. I'll get a patch to you later today. - Joe ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 @ 2020-01-07 10:46 ` Joe Thornber 0 siblings, 0 replies; 43+ messages in thread From: Joe Thornber @ 2020-01-07 10:46 UTC (permalink / raw) To: LVM2 development, Mike Snitzer, markus.schade, ejt, linux-block, dm-devel, joe.thornber On Tue, Jan 07, 2020 at 10:35:46AM +0000, Joe Thornber wrote: > On Sat, Dec 28, 2019 at 02:13:07AM +0000, Eric Wheeler wrote: > > On Fri, 27 Dec 2019, Eric Wheeler wrote: > > > > Just hit the bug again without mq-scsi (scsi_mod.use_blk_mq=n), all other > > > times MQ has been turned on. > > > > > > I'm trying the -ENOSPC hack which will flag the pool as being out of space > > > so I can recover more gracefully than a BUG_ON. Here's a first-draft > > > patch, maybe the spinlock will even prevent the issue. > > > > > > Compile tested, I'll try on a real system tomorrow. > > > > > > Comments welcome: > > Both sm_ll_find_free_block() and sm_ll_inc() can trigger synchronous IO. So you > absolutely cannot use a spin lock. > > dm_pool_alloc_data_block() holds a big rw semaphore which should prevent anything > else trying to allocate at the same time. I suspect the problem is to do with the way we search for the new block in the space map for the previous transaction (sm->old_ll), and then increment in the current transaction (sm->ll). We keep old_ll around so we can ensure we never (re) allocate a block that's used in the previous transaction. This gives us our crash resistance, since if anything goes wrong we effectively rollback to the previous transaction. What I think we should be doing is running find_free on the old_ll, then double checking it's not actually used in the current transaction. ATM we're relying on smd->begin being set properly everywhere, and I suspect this isn't the case. A quick look shows sm_disk_inc_block() doesn't adjust it. sm_disk_inc_block() can be called when breaking sharing of a neighbouring entry in a leaf btree node ... and we know you use snapshots very heavily. I'll get a patch to you later today. - Joe -- lvm-devel mailing list lvm-devel@redhat.com https://www.redhat.com/mailman/listinfo/lvm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [lvm-devel] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 2020-01-07 10:46 ` Joe Thornber (?) @ 2020-01-07 12:28 ` Joe Thornber -1 siblings, 0 replies; 43+ messages in thread From: Joe Thornber @ 2020-01-07 12:28 UTC (permalink / raw) To: LVM2 development, Mike Snitzer, markus.schade, ejt, linux-block, dm-devel, joe.thornber On Tue, Jan 07, 2020 at 10:46:27AM +0000, Joe Thornber wrote: > I'll get a patch to you later today. Eric, Patch below. I've run it through a bunch of tests in the dm test suite. But obviously I have never hit your issue. Will do more testing today. - Joe Author: Joe Thornber <ejt@redhat.com> Date: Tue Jan 7 11:58:42 2020 +0000 [dm-thin, dm-cache] Fix bug in space-maps. The space-maps track the reference counts for disk blocks. There are variants for tracking metadata blocks, and data blocks. We implement transactionality by never touching blocks from the previous transaction, so we can rollback in the event of a crash. When allocating a new block we need to ensure the block is free (has reference count of 0) in both the current and previous transaction. Prior to this patch we were doing this by searching for a free block in the previous transaction, and relying on a 'begin' counter to track where the last allocation in the current transaction was. This 'begin' field was not being updated in all code paths (eg, increment of a data block reference count due to breaking sharing of a neighbour block in the same btree leaf). This patch keeps the 'begin' field, but now it's just a hint to speed up the search. Instead we search the current transaction for a free block, and then double check it's free in the old transaction. Much simpler. diff --git a/drivers/md/persistent-data/dm-space-map-common.c b/drivers/md/persistent-data/dm-space-map-common.c index bd68f6fef694..b4983e4022e6 100644 --- a/drivers/md/persistent-data/dm-space-map-common.c +++ b/drivers/md/persistent-data/dm-space-map-common.c @@ -380,6 +380,34 @@ int sm_ll_find_free_block(struct ll_disk *ll, dm_block_t begin, return -ENOSPC; } +int sm_ll_find_common_free_block(struct ll_disk *old_ll, struct ll_disk *new_ll, + dm_block_t begin, dm_block_t end, dm_block_t *b) +{ + int r; + uint32_t count; + + do { + r = sm_ll_find_free_block(new_ll, begin, new_ll->nr_blocks, b); + if (r) + break; + + /* double check this block wasn't used in the old transaction */ + if (*b >= old_ll->nr_blocks) + count = 0; + + else { + r = sm_ll_lookup(old_ll, *b, &count); + if (r) + break; + + if (count) + begin = *b + 1; + } + } while (count); + + return r; +} + static int sm_ll_mutate(struct ll_disk *ll, dm_block_t b, int (*mutator)(void *context, uint32_t old, uint32_t *new), void *context, enum allocation_event *ev) diff --git a/drivers/md/persistent-data/dm-space-map-common.h b/drivers/md/persistent-data/dm-space-map-common.h index b3078d5eda0c..8de63ce39bdd 100644 --- a/drivers/md/persistent-data/dm-space-map-common.h +++ b/drivers/md/persistent-data/dm-space-map-common.h @@ -109,6 +109,8 @@ int sm_ll_lookup_bitmap(struct ll_disk *ll, dm_block_t b, uint32_t *result); int sm_ll_lookup(struct ll_disk *ll, dm_block_t b, uint32_t *result); int sm_ll_find_free_block(struct ll_disk *ll, dm_block_t begin, dm_block_t end, dm_block_t *result); +int sm_ll_find_common_free_block(struct ll_disk *old_ll, struct ll_disk *new_ll, + dm_block_t begin, dm_block_t end, dm_block_t *result); int sm_ll_insert(struct ll_disk *ll, dm_block_t b, uint32_t ref_count, enum allocation_event *ev); int sm_ll_inc(struct ll_disk *ll, dm_block_t b, enum allocation_event *ev); int sm_ll_dec(struct ll_disk *ll, dm_block_t b, enum allocation_event *ev); diff --git a/drivers/md/persistent-data/dm-space-map-disk.c b/drivers/md/persistent-data/dm-space-map-disk.c index 32adf6b4a9c7..bf4c5e2ccb6f 100644 --- a/drivers/md/persistent-data/dm-space-map-disk.c +++ b/drivers/md/persistent-data/dm-space-map-disk.c @@ -167,8 +167,10 @@ static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) enum allocation_event ev; struct sm_disk *smd = container_of(sm, struct sm_disk, sm); - /* FIXME: we should loop round a couple of times */ - r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); + /* + * Any block we allocate has to be free in both the old and current ll. + */ + r = sm_ll_find_common_free_block(&smd->old_ll, &smd->ll, smd->begin, smd->ll.nr_blocks, b); if (r) return r; diff --git a/drivers/md/persistent-data/dm-space-map-metadata.c b/drivers/md/persistent-data/dm-space-map-metadata.c index 25328582cc48..9e3c64ec2026 100644 --- a/drivers/md/persistent-data/dm-space-map-metadata.c +++ b/drivers/md/persistent-data/dm-space-map-metadata.c @@ -448,7 +448,10 @@ static int sm_metadata_new_block_(struct dm_space_map *sm, dm_block_t *b) enum allocation_event ev; struct sm_metadata *smm = container_of(sm, struct sm_metadata, sm); - r = sm_ll_find_free_block(&smm->old_ll, smm->begin, smm->old_ll.nr_blocks, b); + /* + * Any block we allocate has to be free in both the old and current ll. + */ + r = sm_ll_find_common_free_block(&smm->old_ll, &smm->ll, smm->begin, smm->ll.nr_blocks, b); if (r) return r; ^ permalink raw reply related [flat|nested] 43+ messages in thread
* [dm-devel] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 @ 2020-01-07 12:28 ` Joe Thornber 0 siblings, 0 replies; 43+ messages in thread From: Joe Thornber @ 2020-01-07 12:28 UTC (permalink / raw) To: lvm-devel On Tue, Jan 07, 2020 at 10:46:27AM +0000, Joe Thornber wrote: > I'll get a patch to you later today. Eric, Patch below. I've run it through a bunch of tests in the dm test suite. But obviously I have never hit your issue. Will do more testing today. - Joe Author: Joe Thornber <ejt@redhat.com> Date: Tue Jan 7 11:58:42 2020 +0000 [dm-thin, dm-cache] Fix bug in space-maps. The space-maps track the reference counts for disk blocks. There are variants for tracking metadata blocks, and data blocks. We implement transactionality by never touching blocks from the previous transaction, so we can rollback in the event of a crash. When allocating a new block we need to ensure the block is free (has reference count of 0) in both the current and previous transaction. Prior to this patch we were doing this by searching for a free block in the previous transaction, and relying on a 'begin' counter to track where the last allocation in the current transaction was. This 'begin' field was not being updated in all code paths (eg, increment of a data block reference count due to breaking sharing of a neighbour block in the same btree leaf). This patch keeps the 'begin' field, but now it's just a hint to speed up the search. Instead we search the current transaction for a free block, and then double check it's free in the old transaction. Much simpler. diff --git a/drivers/md/persistent-data/dm-space-map-common.c b/drivers/md/persistent-data/dm-space-map-common.c index bd68f6fef694..b4983e4022e6 100644 --- a/drivers/md/persistent-data/dm-space-map-common.c +++ b/drivers/md/persistent-data/dm-space-map-common.c @@ -380,6 +380,34 @@ int sm_ll_find_free_block(struct ll_disk *ll, dm_block_t begin, return -ENOSPC; } +int sm_ll_find_common_free_block(struct ll_disk *old_ll, struct ll_disk *new_ll, + dm_block_t begin, dm_block_t end, dm_block_t *b) +{ + int r; + uint32_t count; + + do { + r = sm_ll_find_free_block(new_ll, begin, new_ll->nr_blocks, b); + if (r) + break; + + /* double check this block wasn't used in the old transaction */ + if (*b >= old_ll->nr_blocks) + count = 0; + + else { + r = sm_ll_lookup(old_ll, *b, &count); + if (r) + break; + + if (count) + begin = *b + 1; + } + } while (count); + + return r; +} + static int sm_ll_mutate(struct ll_disk *ll, dm_block_t b, int (*mutator)(void *context, uint32_t old, uint32_t *new), void *context, enum allocation_event *ev) diff --git a/drivers/md/persistent-data/dm-space-map-common.h b/drivers/md/persistent-data/dm-space-map-common.h index b3078d5eda0c..8de63ce39bdd 100644 --- a/drivers/md/persistent-data/dm-space-map-common.h +++ b/drivers/md/persistent-data/dm-space-map-common.h @@ -109,6 +109,8 @@ int sm_ll_lookup_bitmap(struct ll_disk *ll, dm_block_t b, uint32_t *result); int sm_ll_lookup(struct ll_disk *ll, dm_block_t b, uint32_t *result); int sm_ll_find_free_block(struct ll_disk *ll, dm_block_t begin, dm_block_t end, dm_block_t *result); +int sm_ll_find_common_free_block(struct ll_disk *old_ll, struct ll_disk *new_ll, + dm_block_t begin, dm_block_t end, dm_block_t *result); int sm_ll_insert(struct ll_disk *ll, dm_block_t b, uint32_t ref_count, enum allocation_event *ev); int sm_ll_inc(struct ll_disk *ll, dm_block_t b, enum allocation_event *ev); int sm_ll_dec(struct ll_disk *ll, dm_block_t b, enum allocation_event *ev); diff --git a/drivers/md/persistent-data/dm-space-map-disk.c b/drivers/md/persistent-data/dm-space-map-disk.c index 32adf6b4a9c7..bf4c5e2ccb6f 100644 --- a/drivers/md/persistent-data/dm-space-map-disk.c +++ b/drivers/md/persistent-data/dm-space-map-disk.c @@ -167,8 +167,10 @@ static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) enum allocation_event ev; struct sm_disk *smd = container_of(sm, struct sm_disk, sm); - /* FIXME: we should loop round a couple of times */ - r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); + /* + * Any block we allocate has to be free in both the old and current ll. + */ + r = sm_ll_find_common_free_block(&smd->old_ll, &smd->ll, smd->begin, smd->ll.nr_blocks, b); if (r) return r; diff --git a/drivers/md/persistent-data/dm-space-map-metadata.c b/drivers/md/persistent-data/dm-space-map-metadata.c index 25328582cc48..9e3c64ec2026 100644 --- a/drivers/md/persistent-data/dm-space-map-metadata.c +++ b/drivers/md/persistent-data/dm-space-map-metadata.c @@ -448,7 +448,10 @@ static int sm_metadata_new_block_(struct dm_space_map *sm, dm_block_t *b) enum allocation_event ev; struct sm_metadata *smm = container_of(sm, struct sm_metadata, sm); - r = sm_ll_find_free_block(&smm->old_ll, smm->begin, smm->old_ll.nr_blocks, b); + /* + * Any block we allocate has to be free in both the old and current ll. + */ + r = sm_ll_find_common_free_block(&smm->old_ll, &smm->ll, smm->begin, smm->ll.nr_blocks, b); if (r) return r; ^ permalink raw reply related [flat|nested] 43+ messages in thread
* Re: [lvm-devel] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 @ 2020-01-07 12:28 ` Joe Thornber 0 siblings, 0 replies; 43+ messages in thread From: Joe Thornber @ 2020-01-07 12:28 UTC (permalink / raw) To: LVM2 development, Mike Snitzer, markus.schade, ejt, linux-block, dm-devel, joe.thornber On Tue, Jan 07, 2020 at 10:46:27AM +0000, Joe Thornber wrote: > I'll get a patch to you later today. Eric, Patch below. I've run it through a bunch of tests in the dm test suite. But obviously I have never hit your issue. Will do more testing today. - Joe Author: Joe Thornber <ejt@redhat.com> Date: Tue Jan 7 11:58:42 2020 +0000 [dm-thin, dm-cache] Fix bug in space-maps. The space-maps track the reference counts for disk blocks. There are variants for tracking metadata blocks, and data blocks. We implement transactionality by never touching blocks from the previous transaction, so we can rollback in the event of a crash. When allocating a new block we need to ensure the block is free (has reference count of 0) in both the current and previous transaction. Prior to this patch we were doing this by searching for a free block in the previous transaction, and relying on a 'begin' counter to track where the last allocation in the current transaction was. This 'begin' field was not being updated in all code paths (eg, increment of a data block reference count due to breaking sharing of a neighbour block in the same btree leaf). This patch keeps the 'begin' field, but now it's just a hint to speed up the search. Instead we search the current transaction for a free block, and then double check it's free in the old transaction. Much simpler. diff --git a/drivers/md/persistent-data/dm-space-map-common.c b/drivers/md/persistent-data/dm-space-map-common.c index bd68f6fef694..b4983e4022e6 100644 --- a/drivers/md/persistent-data/dm-space-map-common.c +++ b/drivers/md/persistent-data/dm-space-map-common.c @@ -380,6 +380,34 @@ int sm_ll_find_free_block(struct ll_disk *ll, dm_block_t begin, return -ENOSPC; } +int sm_ll_find_common_free_block(struct ll_disk *old_ll, struct ll_disk *new_ll, + dm_block_t begin, dm_block_t end, dm_block_t *b) +{ + int r; + uint32_t count; + + do { + r = sm_ll_find_free_block(new_ll, begin, new_ll->nr_blocks, b); + if (r) + break; + + /* double check this block wasn't used in the old transaction */ + if (*b >= old_ll->nr_blocks) + count = 0; + + else { + r = sm_ll_lookup(old_ll, *b, &count); + if (r) + break; + + if (count) + begin = *b + 1; + } + } while (count); + + return r; +} + static int sm_ll_mutate(struct ll_disk *ll, dm_block_t b, int (*mutator)(void *context, uint32_t old, uint32_t *new), void *context, enum allocation_event *ev) diff --git a/drivers/md/persistent-data/dm-space-map-common.h b/drivers/md/persistent-data/dm-space-map-common.h index b3078d5eda0c..8de63ce39bdd 100644 --- a/drivers/md/persistent-data/dm-space-map-common.h +++ b/drivers/md/persistent-data/dm-space-map-common.h @@ -109,6 +109,8 @@ int sm_ll_lookup_bitmap(struct ll_disk *ll, dm_block_t b, uint32_t *result); int sm_ll_lookup(struct ll_disk *ll, dm_block_t b, uint32_t *result); int sm_ll_find_free_block(struct ll_disk *ll, dm_block_t begin, dm_block_t end, dm_block_t *result); +int sm_ll_find_common_free_block(struct ll_disk *old_ll, struct ll_disk *new_ll, + dm_block_t begin, dm_block_t end, dm_block_t *result); int sm_ll_insert(struct ll_disk *ll, dm_block_t b, uint32_t ref_count, enum allocation_event *ev); int sm_ll_inc(struct ll_disk *ll, dm_block_t b, enum allocation_event *ev); int sm_ll_dec(struct ll_disk *ll, dm_block_t b, enum allocation_event *ev); diff --git a/drivers/md/persistent-data/dm-space-map-disk.c b/drivers/md/persistent-data/dm-space-map-disk.c index 32adf6b4a9c7..bf4c5e2ccb6f 100644 --- a/drivers/md/persistent-data/dm-space-map-disk.c +++ b/drivers/md/persistent-data/dm-space-map-disk.c @@ -167,8 +167,10 @@ static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) enum allocation_event ev; struct sm_disk *smd = container_of(sm, struct sm_disk, sm); - /* FIXME: we should loop round a couple of times */ - r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); + /* + * Any block we allocate has to be free in both the old and current ll. + */ + r = sm_ll_find_common_free_block(&smd->old_ll, &smd->ll, smd->begin, smd->ll.nr_blocks, b); if (r) return r; diff --git a/drivers/md/persistent-data/dm-space-map-metadata.c b/drivers/md/persistent-data/dm-space-map-metadata.c index 25328582cc48..9e3c64ec2026 100644 --- a/drivers/md/persistent-data/dm-space-map-metadata.c +++ b/drivers/md/persistent-data/dm-space-map-metadata.c @@ -448,7 +448,10 @@ static int sm_metadata_new_block_(struct dm_space_map *sm, dm_block_t *b) enum allocation_event ev; struct sm_metadata *smm = container_of(sm, struct sm_metadata, sm); - r = sm_ll_find_free_block(&smm->old_ll, smm->begin, smm->old_ll.nr_blocks, b); + /* + * Any block we allocate has to be free in both the old and current ll. + */ + r = sm_ll_find_common_free_block(&smm->old_ll, &smm->ll, smm->begin, smm->ll.nr_blocks, b); if (r) return r; ^ permalink raw reply related [flat|nested] 43+ messages in thread
* Re: [dm-devel] [lvm-devel] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 2020-01-07 12:28 ` [lvm-devel] " Joe Thornber (?) @ 2020-01-07 18:47 ` Eric Wheeler -1 siblings, 0 replies; 43+ messages in thread From: Eric Wheeler @ 2020-01-07 18:47 UTC (permalink / raw) To: Joe Thornber Cc: LVM2 development, Mike Snitzer, markus.schade, ejt, linux-block, dm-devel, joe.thornber On Tue, 7 Jan 2020, Joe Thornber wrote: > On Tue, Jan 07, 2020 at 10:46:27AM +0000, Joe Thornber wrote: > > I'll get a patch to you later today. > > Eric, > > Patch below. I've run it through a bunch of tests in the dm test suite. But > obviously I have never hit your issue. Will do more testing today. Thank you Joe, I'll apply the patch and pull out the spinlock. I'm not familiar with how sync IO prevents a spinlock. Can you give a brief explaination or point me at documentation? -- Eric Wheeler > > - Joe > > > > Author: Joe Thornber <ejt@redhat.com> > Date: Tue Jan 7 11:58:42 2020 +0000 > > [dm-thin, dm-cache] Fix bug in space-maps. > > The space-maps track the reference counts for disk blocks. There are variants > for tracking metadata blocks, and data blocks. > > We implement transactionality by never touching blocks from the previous > transaction, so we can rollback in the event of a crash. > > When allocating a new block we need to ensure the block is free (has reference > count of 0) in both the current and previous transaction. Prior to this patch we > were doing this by searching for a free block in the previous transaction, and > relying on a 'begin' counter to track where the last allocation in the current > transaction was. This 'begin' field was not being updated in all code paths (eg, > increment of a data block reference count due to breaking sharing of a neighbour > block in the same btree leaf). > > This patch keeps the 'begin' field, but now it's just a hint to speed up the search. > Instead we search the current transaction for a free block, and then double check > it's free in the old transaction. Much simpler. > > diff --git a/drivers/md/persistent-data/dm-space-map-common.c b/drivers/md/persistent-data/dm-space-map-common.c > index bd68f6fef694..b4983e4022e6 100644 > --- a/drivers/md/persistent-data/dm-space-map-common.c > +++ b/drivers/md/persistent-data/dm-space-map-common.c > @@ -380,6 +380,34 @@ int sm_ll_find_free_block(struct ll_disk *ll, dm_block_t begin, > return -ENOSPC; > } > > +int sm_ll_find_common_free_block(struct ll_disk *old_ll, struct ll_disk *new_ll, > + dm_block_t begin, dm_block_t end, dm_block_t *b) > +{ > + int r; > + uint32_t count; > + > + do { > + r = sm_ll_find_free_block(new_ll, begin, new_ll->nr_blocks, b); > + if (r) > + break; > + > + /* double check this block wasn't used in the old transaction */ > + if (*b >= old_ll->nr_blocks) > + count = 0; > + > + else { > + r = sm_ll_lookup(old_ll, *b, &count); > + if (r) > + break; > + > + if (count) > + begin = *b + 1; > + } > + } while (count); > + > + return r; > +} > + > static int sm_ll_mutate(struct ll_disk *ll, dm_block_t b, > int (*mutator)(void *context, uint32_t old, uint32_t *new), > void *context, enum allocation_event *ev) > diff --git a/drivers/md/persistent-data/dm-space-map-common.h b/drivers/md/persistent-data/dm-space-map-common.h > index b3078d5eda0c..8de63ce39bdd 100644 > --- a/drivers/md/persistent-data/dm-space-map-common.h > +++ b/drivers/md/persistent-data/dm-space-map-common.h > @@ -109,6 +109,8 @@ int sm_ll_lookup_bitmap(struct ll_disk *ll, dm_block_t b, uint32_t *result); > int sm_ll_lookup(struct ll_disk *ll, dm_block_t b, uint32_t *result); > int sm_ll_find_free_block(struct ll_disk *ll, dm_block_t begin, > dm_block_t end, dm_block_t *result); > +int sm_ll_find_common_free_block(struct ll_disk *old_ll, struct ll_disk *new_ll, > + dm_block_t begin, dm_block_t end, dm_block_t *result); > int sm_ll_insert(struct ll_disk *ll, dm_block_t b, uint32_t ref_count, enum allocation_event *ev); > int sm_ll_inc(struct ll_disk *ll, dm_block_t b, enum allocation_event *ev); > int sm_ll_dec(struct ll_disk *ll, dm_block_t b, enum allocation_event *ev); > diff --git a/drivers/md/persistent-data/dm-space-map-disk.c b/drivers/md/persistent-data/dm-space-map-disk.c > index 32adf6b4a9c7..bf4c5e2ccb6f 100644 > --- a/drivers/md/persistent-data/dm-space-map-disk.c > +++ b/drivers/md/persistent-data/dm-space-map-disk.c > @@ -167,8 +167,10 @@ static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > enum allocation_event ev; > struct sm_disk *smd = container_of(sm, struct sm_disk, sm); > > - /* FIXME: we should loop round a couple of times */ > - r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); > + /* > + * Any block we allocate has to be free in both the old and current ll. > + */ > + r = sm_ll_find_common_free_block(&smd->old_ll, &smd->ll, smd->begin, smd->ll.nr_blocks, b); > if (r) > return r; > > diff --git a/drivers/md/persistent-data/dm-space-map-metadata.c b/drivers/md/persistent-data/dm-space-map-metadata.c > index 25328582cc48..9e3c64ec2026 100644 > --- a/drivers/md/persistent-data/dm-space-map-metadata.c > +++ b/drivers/md/persistent-data/dm-space-map-metadata.c > @@ -448,7 +448,10 @@ static int sm_metadata_new_block_(struct dm_space_map *sm, dm_block_t *b) > enum allocation_event ev; > struct sm_metadata *smm = container_of(sm, struct sm_metadata, sm); > > - r = sm_ll_find_free_block(&smm->old_ll, smm->begin, smm->old_ll.nr_blocks, b); > + /* > + * Any block we allocate has to be free in both the old and current ll. > + */ > + r = sm_ll_find_common_free_block(&smm->old_ll, &smm->ll, smm->begin, smm->ll.nr_blocks, b); > if (r) > return r; > > > -- > dm-devel mailing list > dm-devel@redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel > > ^ permalink raw reply [flat|nested] 43+ messages in thread
* [dm-devel] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 @ 2020-01-07 18:47 ` Eric Wheeler 0 siblings, 0 replies; 43+ messages in thread From: Eric Wheeler @ 2020-01-07 18:47 UTC (permalink / raw) To: lvm-devel On Tue, 7 Jan 2020, Joe Thornber wrote: > On Tue, Jan 07, 2020 at 10:46:27AM +0000, Joe Thornber wrote: > > I'll get a patch to you later today. > > Eric, > > Patch below. I've run it through a bunch of tests in the dm test suite. But > obviously I have never hit your issue. Will do more testing today. Thank you Joe, I'll apply the patch and pull out the spinlock. I'm not familiar with how sync IO prevents a spinlock. Can you give a brief explaination or point me at documentation? -- Eric Wheeler > > - Joe > > > > Author: Joe Thornber <ejt@redhat.com> > Date: Tue Jan 7 11:58:42 2020 +0000 > > [dm-thin, dm-cache] Fix bug in space-maps. > > The space-maps track the reference counts for disk blocks. There are variants > for tracking metadata blocks, and data blocks. > > We implement transactionality by never touching blocks from the previous > transaction, so we can rollback in the event of a crash. > > When allocating a new block we need to ensure the block is free (has reference > count of 0) in both the current and previous transaction. Prior to this patch we > were doing this by searching for a free block in the previous transaction, and > relying on a 'begin' counter to track where the last allocation in the current > transaction was. This 'begin' field was not being updated in all code paths (eg, > increment of a data block reference count due to breaking sharing of a neighbour > block in the same btree leaf). > > This patch keeps the 'begin' field, but now it's just a hint to speed up the search. > Instead we search the current transaction for a free block, and then double check > it's free in the old transaction. Much simpler. > > diff --git a/drivers/md/persistent-data/dm-space-map-common.c b/drivers/md/persistent-data/dm-space-map-common.c > index bd68f6fef694..b4983e4022e6 100644 > --- a/drivers/md/persistent-data/dm-space-map-common.c > +++ b/drivers/md/persistent-data/dm-space-map-common.c > @@ -380,6 +380,34 @@ int sm_ll_find_free_block(struct ll_disk *ll, dm_block_t begin, > return -ENOSPC; > } > > +int sm_ll_find_common_free_block(struct ll_disk *old_ll, struct ll_disk *new_ll, > + dm_block_t begin, dm_block_t end, dm_block_t *b) > +{ > + int r; > + uint32_t count; > + > + do { > + r = sm_ll_find_free_block(new_ll, begin, new_ll->nr_blocks, b); > + if (r) > + break; > + > + /* double check this block wasn't used in the old transaction */ > + if (*b >= old_ll->nr_blocks) > + count = 0; > + > + else { > + r = sm_ll_lookup(old_ll, *b, &count); > + if (r) > + break; > + > + if (count) > + begin = *b + 1; > + } > + } while (count); > + > + return r; > +} > + > static int sm_ll_mutate(struct ll_disk *ll, dm_block_t b, > int (*mutator)(void *context, uint32_t old, uint32_t *new), > void *context, enum allocation_event *ev) > diff --git a/drivers/md/persistent-data/dm-space-map-common.h b/drivers/md/persistent-data/dm-space-map-common.h > index b3078d5eda0c..8de63ce39bdd 100644 > --- a/drivers/md/persistent-data/dm-space-map-common.h > +++ b/drivers/md/persistent-data/dm-space-map-common.h > @@ -109,6 +109,8 @@ int sm_ll_lookup_bitmap(struct ll_disk *ll, dm_block_t b, uint32_t *result); > int sm_ll_lookup(struct ll_disk *ll, dm_block_t b, uint32_t *result); > int sm_ll_find_free_block(struct ll_disk *ll, dm_block_t begin, > dm_block_t end, dm_block_t *result); > +int sm_ll_find_common_free_block(struct ll_disk *old_ll, struct ll_disk *new_ll, > + dm_block_t begin, dm_block_t end, dm_block_t *result); > int sm_ll_insert(struct ll_disk *ll, dm_block_t b, uint32_t ref_count, enum allocation_event *ev); > int sm_ll_inc(struct ll_disk *ll, dm_block_t b, enum allocation_event *ev); > int sm_ll_dec(struct ll_disk *ll, dm_block_t b, enum allocation_event *ev); > diff --git a/drivers/md/persistent-data/dm-space-map-disk.c b/drivers/md/persistent-data/dm-space-map-disk.c > index 32adf6b4a9c7..bf4c5e2ccb6f 100644 > --- a/drivers/md/persistent-data/dm-space-map-disk.c > +++ b/drivers/md/persistent-data/dm-space-map-disk.c > @@ -167,8 +167,10 @@ static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > enum allocation_event ev; > struct sm_disk *smd = container_of(sm, struct sm_disk, sm); > > - /* FIXME: we should loop round a couple of times */ > - r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); > + /* > + * Any block we allocate has to be free in both the old and current ll. > + */ > + r = sm_ll_find_common_free_block(&smd->old_ll, &smd->ll, smd->begin, smd->ll.nr_blocks, b); > if (r) > return r; > > diff --git a/drivers/md/persistent-data/dm-space-map-metadata.c b/drivers/md/persistent-data/dm-space-map-metadata.c > index 25328582cc48..9e3c64ec2026 100644 > --- a/drivers/md/persistent-data/dm-space-map-metadata.c > +++ b/drivers/md/persistent-data/dm-space-map-metadata.c > @@ -448,7 +448,10 @@ static int sm_metadata_new_block_(struct dm_space_map *sm, dm_block_t *b) > enum allocation_event ev; > struct sm_metadata *smm = container_of(sm, struct sm_metadata, sm); > > - r = sm_ll_find_free_block(&smm->old_ll, smm->begin, smm->old_ll.nr_blocks, b); > + /* > + * Any block we allocate has to be free in both the old and current ll. > + */ > + r = sm_ll_find_common_free_block(&smm->old_ll, &smm->ll, smm->begin, smm->ll.nr_blocks, b); > if (r) > return r; > > > -- > dm-devel mailing list > dm-devel at redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel > > ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 @ 2020-01-07 18:47 ` Eric Wheeler 0 siblings, 0 replies; 43+ messages in thread From: Eric Wheeler @ 2020-01-07 18:47 UTC (permalink / raw) To: Joe Thornber Cc: Mike Snitzer, markus.schade, dm-devel, LVM2 development, linux-block, ejt, joe.thornber On Tue, 7 Jan 2020, Joe Thornber wrote: > On Tue, Jan 07, 2020 at 10:46:27AM +0000, Joe Thornber wrote: > > I'll get a patch to you later today. > > Eric, > > Patch below. I've run it through a bunch of tests in the dm test suite. But > obviously I have never hit your issue. Will do more testing today. Thank you Joe, I'll apply the patch and pull out the spinlock. I'm not familiar with how sync IO prevents a spinlock. Can you give a brief explaination or point me at documentation? -- Eric Wheeler > > - Joe > > > > Author: Joe Thornber <ejt@redhat.com> > Date: Tue Jan 7 11:58:42 2020 +0000 > > [dm-thin, dm-cache] Fix bug in space-maps. > > The space-maps track the reference counts for disk blocks. There are variants > for tracking metadata blocks, and data blocks. > > We implement transactionality by never touching blocks from the previous > transaction, so we can rollback in the event of a crash. > > When allocating a new block we need to ensure the block is free (has reference > count of 0) in both the current and previous transaction. Prior to this patch we > were doing this by searching for a free block in the previous transaction, and > relying on a 'begin' counter to track where the last allocation in the current > transaction was. This 'begin' field was not being updated in all code paths (eg, > increment of a data block reference count due to breaking sharing of a neighbour > block in the same btree leaf). > > This patch keeps the 'begin' field, but now it's just a hint to speed up the search. > Instead we search the current transaction for a free block, and then double check > it's free in the old transaction. Much simpler. > > diff --git a/drivers/md/persistent-data/dm-space-map-common.c b/drivers/md/persistent-data/dm-space-map-common.c > index bd68f6fef694..b4983e4022e6 100644 > --- a/drivers/md/persistent-data/dm-space-map-common.c > +++ b/drivers/md/persistent-data/dm-space-map-common.c > @@ -380,6 +380,34 @@ int sm_ll_find_free_block(struct ll_disk *ll, dm_block_t begin, > return -ENOSPC; > } > > +int sm_ll_find_common_free_block(struct ll_disk *old_ll, struct ll_disk *new_ll, > + dm_block_t begin, dm_block_t end, dm_block_t *b) > +{ > + int r; > + uint32_t count; > + > + do { > + r = sm_ll_find_free_block(new_ll, begin, new_ll->nr_blocks, b); > + if (r) > + break; > + > + /* double check this block wasn't used in the old transaction */ > + if (*b >= old_ll->nr_blocks) > + count = 0; > + > + else { > + r = sm_ll_lookup(old_ll, *b, &count); > + if (r) > + break; > + > + if (count) > + begin = *b + 1; > + } > + } while (count); > + > + return r; > +} > + > static int sm_ll_mutate(struct ll_disk *ll, dm_block_t b, > int (*mutator)(void *context, uint32_t old, uint32_t *new), > void *context, enum allocation_event *ev) > diff --git a/drivers/md/persistent-data/dm-space-map-common.h b/drivers/md/persistent-data/dm-space-map-common.h > index b3078d5eda0c..8de63ce39bdd 100644 > --- a/drivers/md/persistent-data/dm-space-map-common.h > +++ b/drivers/md/persistent-data/dm-space-map-common.h > @@ -109,6 +109,8 @@ int sm_ll_lookup_bitmap(struct ll_disk *ll, dm_block_t b, uint32_t *result); > int sm_ll_lookup(struct ll_disk *ll, dm_block_t b, uint32_t *result); > int sm_ll_find_free_block(struct ll_disk *ll, dm_block_t begin, > dm_block_t end, dm_block_t *result); > +int sm_ll_find_common_free_block(struct ll_disk *old_ll, struct ll_disk *new_ll, > + dm_block_t begin, dm_block_t end, dm_block_t *result); > int sm_ll_insert(struct ll_disk *ll, dm_block_t b, uint32_t ref_count, enum allocation_event *ev); > int sm_ll_inc(struct ll_disk *ll, dm_block_t b, enum allocation_event *ev); > int sm_ll_dec(struct ll_disk *ll, dm_block_t b, enum allocation_event *ev); > diff --git a/drivers/md/persistent-data/dm-space-map-disk.c b/drivers/md/persistent-data/dm-space-map-disk.c > index 32adf6b4a9c7..bf4c5e2ccb6f 100644 > --- a/drivers/md/persistent-data/dm-space-map-disk.c > +++ b/drivers/md/persistent-data/dm-space-map-disk.c > @@ -167,8 +167,10 @@ static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b) > enum allocation_event ev; > struct sm_disk *smd = container_of(sm, struct sm_disk, sm); > > - /* FIXME: we should loop round a couple of times */ > - r = sm_ll_find_free_block(&smd->old_ll, smd->begin, smd->old_ll.nr_blocks, b); > + /* > + * Any block we allocate has to be free in both the old and current ll. > + */ > + r = sm_ll_find_common_free_block(&smd->old_ll, &smd->ll, smd->begin, smd->ll.nr_blocks, b); > if (r) > return r; > > diff --git a/drivers/md/persistent-data/dm-space-map-metadata.c b/drivers/md/persistent-data/dm-space-map-metadata.c > index 25328582cc48..9e3c64ec2026 100644 > --- a/drivers/md/persistent-data/dm-space-map-metadata.c > +++ b/drivers/md/persistent-data/dm-space-map-metadata.c > @@ -448,7 +448,10 @@ static int sm_metadata_new_block_(struct dm_space_map *sm, dm_block_t *b) > enum allocation_event ev; > struct sm_metadata *smm = container_of(sm, struct sm_metadata, sm); > > - r = sm_ll_find_free_block(&smm->old_ll, smm->begin, smm->old_ll.nr_blocks, b); > + /* > + * Any block we allocate has to be free in both the old and current ll. > + */ > + r = sm_ll_find_common_free_block(&smm->old_ll, &smm->ll, smm->begin, smm->ll.nr_blocks, b); > if (r) > return r; > > > -- > dm-devel mailing list > dm-devel@redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel > > -- lvm-devel mailing list lvm-devel@redhat.com https://www.redhat.com/mailman/listinfo/lvm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 2020-01-07 12:28 ` [lvm-devel] " Joe Thornber @ 2020-01-14 21:52 ` Eric Biggers -1 siblings, 0 replies; 43+ messages in thread From: Eric Biggers @ 2020-01-14 21:52 UTC (permalink / raw) To: Mike Snitzer Cc: LVM2 development, markus.schade, ejt, linux-block, dm-devel, joe.thornber, dm-devel On Tue, Jan 07, 2020 at 12:28:25PM +0000, Joe Thornber wrote: > On Tue, Jan 07, 2020 at 10:46:27AM +0000, Joe Thornber wrote: > > I'll get a patch to you later today. > > Eric, > > Patch below. I've run it through a bunch of tests in the dm test suite. But > obviously I have never hit your issue. Will do more testing today. > > - Joe > > > > Author: Joe Thornber <ejt@redhat.com> > Date: Tue Jan 7 11:58:42 2020 +0000 > > [dm-thin, dm-cache] Fix bug in space-maps. > > The space-maps track the reference counts for disk blocks. There are variants > for tracking metadata blocks, and data blocks. > > We implement transactionality by never touching blocks from the previous > transaction, so we can rollback in the event of a crash. > > When allocating a new block we need to ensure the block is free (has reference > count of 0) in both the current and previous transaction. Prior to this patch we > were doing this by searching for a free block in the previous transaction, and > relying on a 'begin' counter to track where the last allocation in the current > transaction was. This 'begin' field was not being updated in all code paths (eg, > increment of a data block reference count due to breaking sharing of a neighbour > block in the same btree leaf). > > This patch keeps the 'begin' field, but now it's just a hint to speed up the search. > Instead we search the current transaction for a free block, and then double check > it's free in the old transaction. Much simpler. > I happened to notice this patch is on the linux-dm/for-next branch (https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=2137c0dcc04b24efb4c38d4b46b7194575718dd5) and it has: Reported-by: Eric Biggers <ebiggers@google.com> This is wrong, I didn't report this. I think you meant to put: Reported-by: Eric Wheeler <dm-devel@lists.ewheeler.net> - Eric (the other one) ^ permalink raw reply [flat|nested] 43+ messages in thread
* kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 @ 2020-01-14 21:52 ` Eric Biggers 0 siblings, 0 replies; 43+ messages in thread From: Eric Biggers @ 2020-01-14 21:52 UTC (permalink / raw) To: lvm-devel On Tue, Jan 07, 2020 at 12:28:25PM +0000, Joe Thornber wrote: > On Tue, Jan 07, 2020 at 10:46:27AM +0000, Joe Thornber wrote: > > I'll get a patch to you later today. > > Eric, > > Patch below. I've run it through a bunch of tests in the dm test suite. But > obviously I have never hit your issue. Will do more testing today. > > - Joe > > > > Author: Joe Thornber <ejt@redhat.com> > Date: Tue Jan 7 11:58:42 2020 +0000 > > [dm-thin, dm-cache] Fix bug in space-maps. > > The space-maps track the reference counts for disk blocks. There are variants > for tracking metadata blocks, and data blocks. > > We implement transactionality by never touching blocks from the previous > transaction, so we can rollback in the event of a crash. > > When allocating a new block we need to ensure the block is free (has reference > count of 0) in both the current and previous transaction. Prior to this patch we > were doing this by searching for a free block in the previous transaction, and > relying on a 'begin' counter to track where the last allocation in the current > transaction was. This 'begin' field was not being updated in all code paths (eg, > increment of a data block reference count due to breaking sharing of a neighbour > block in the same btree leaf). > > This patch keeps the 'begin' field, but now it's just a hint to speed up the search. > Instead we search the current transaction for a free block, and then double check > it's free in the old transaction. Much simpler. > I happened to notice this patch is on the linux-dm/for-next branch (https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=2137c0dcc04b24efb4c38d4b46b7194575718dd5) and it has: Reported-by: Eric Biggers <ebiggers@google.com> This is wrong, I didn't report this. I think you meant to put: Reported-by: Eric Wheeler <dm-devel@lists.ewheeler.net> - Eric (the other one) ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 2020-01-14 21:52 ` Eric Biggers @ 2020-01-15 1:22 ` Mike Snitzer -1 siblings, 0 replies; 43+ messages in thread From: Mike Snitzer @ 2020-01-15 1:22 UTC (permalink / raw) To: Eric Biggers Cc: LVM2 development, markus.schade, Joe Thornber, linux-block, device-mapper development, Joe Thornber, Eric Wheeler On Tue, Jan 14, 2020 at 4:53 PM Eric Biggers <ebiggers@kernel.org> wrote: > > On Tue, Jan 07, 2020 at 12:28:25PM +0000, Joe Thornber wrote: > > On Tue, Jan 07, 2020 at 10:46:27AM +0000, Joe Thornber wrote: > > > I'll get a patch to you later today. > > > > Eric, > > > > Patch below. I've run it through a bunch of tests in the dm test suite. But > > obviously I have never hit your issue. Will do more testing today. > > > > - Joe > > > > > > > > Author: Joe Thornber <ejt@redhat.com> > > Date: Tue Jan 7 11:58:42 2020 +0000 > > > > [dm-thin, dm-cache] Fix bug in space-maps. > > > > The space-maps track the reference counts for disk blocks. There are variants > > for tracking metadata blocks, and data blocks. > > > > We implement transactionality by never touching blocks from the previous > > transaction, so we can rollback in the event of a crash. > > > > When allocating a new block we need to ensure the block is free (has reference > > count of 0) in both the current and previous transaction. Prior to this patch we > > were doing this by searching for a free block in the previous transaction, and > > relying on a 'begin' counter to track where the last allocation in the current > > transaction was. This 'begin' field was not being updated in all code paths (eg, > > increment of a data block reference count due to breaking sharing of a neighbour > > block in the same btree leaf). > > > > This patch keeps the 'begin' field, but now it's just a hint to speed up the search. > > Instead we search the current transaction for a free block, and then double check > > it's free in the old transaction. Much simpler. > > > > I happened to notice this patch is on the linux-dm/for-next branch > (https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=2137c0dcc04b24efb4c38d4b46b7194575718dd5) > and it has: > > Reported-by: Eric Biggers <ebiggers@google.com> > > This is wrong, I didn't report this. I think you meant to put: > > Reported-by: Eric Wheeler <dm-devel@lists.ewheeler.net> > > - Eric (the other one) Fixed it up, not sure how that happened, sorry about that! ^ permalink raw reply [flat|nested] 43+ messages in thread
* kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 @ 2020-01-15 1:22 ` Mike Snitzer 0 siblings, 0 replies; 43+ messages in thread From: Mike Snitzer @ 2020-01-15 1:22 UTC (permalink / raw) To: lvm-devel On Tue, Jan 14, 2020 at 4:53 PM Eric Biggers <ebiggers@kernel.org> wrote: > > On Tue, Jan 07, 2020 at 12:28:25PM +0000, Joe Thornber wrote: > > On Tue, Jan 07, 2020 at 10:46:27AM +0000, Joe Thornber wrote: > > > I'll get a patch to you later today. > > > > Eric, > > > > Patch below. I've run it through a bunch of tests in the dm test suite. But > > obviously I have never hit your issue. Will do more testing today. > > > > - Joe > > > > > > > > Author: Joe Thornber <ejt@redhat.com> > > Date: Tue Jan 7 11:58:42 2020 +0000 > > > > [dm-thin, dm-cache] Fix bug in space-maps. > > > > The space-maps track the reference counts for disk blocks. There are variants > > for tracking metadata blocks, and data blocks. > > > > We implement transactionality by never touching blocks from the previous > > transaction, so we can rollback in the event of a crash. > > > > When allocating a new block we need to ensure the block is free (has reference > > count of 0) in both the current and previous transaction. Prior to this patch we > > were doing this by searching for a free block in the previous transaction, and > > relying on a 'begin' counter to track where the last allocation in the current > > transaction was. This 'begin' field was not being updated in all code paths (eg, > > increment of a data block reference count due to breaking sharing of a neighbour > > block in the same btree leaf). > > > > This patch keeps the 'begin' field, but now it's just a hint to speed up the search. > > Instead we search the current transaction for a free block, and then double check > > it's free in the old transaction. Much simpler. > > > > I happened to notice this patch is on the linux-dm/for-next branch > (https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=2137c0dcc04b24efb4c38d4b46b7194575718dd5) > and it has: > > Reported-by: Eric Biggers <ebiggers@google.com> > > This is wrong, I didn't report this. I think you meant to put: > > Reported-by: Eric Wheeler <dm-devel@lists.ewheeler.net> > > - Eric (the other one) Fixed it up, not sure how that happened, sorry about that! ^ permalink raw reply [flat|nested] 43+ messages in thread
end of thread, other threads:[~2020-01-15 1:22 UTC | newest] Thread overview: 43+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-09-25 18:40 kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 with scsi_mod.use_blk_mq=y Eric Wheeler 2019-09-25 18:40 ` Eric Wheeler 2019-09-25 18:40 ` Eric Wheeler 2019-09-25 20:01 ` Mike Snitzer 2019-09-25 20:01 ` Mike Snitzer 2019-09-25 20:01 ` Mike Snitzer 2019-09-25 20:33 ` Eric Wheeler 2019-09-25 20:33 ` Eric Wheeler 2019-09-25 20:33 ` Eric Wheeler 2019-09-26 18:27 ` Eric Wheeler 2019-09-26 18:27 ` Eric Wheeler 2019-09-26 18:27 ` Eric Wheeler 2019-09-27 8:32 ` Joe Thornber 2019-09-27 8:32 ` Joe Thornber 2019-09-27 8:32 ` Joe Thornber 2019-09-27 18:45 ` Eric Wheeler 2019-09-27 18:45 ` Eric Wheeler 2019-09-27 18:45 ` Eric Wheeler 2019-12-20 19:54 ` [dm-devel] " Eric Wheeler 2019-12-20 19:54 ` Eric Wheeler 2019-12-20 19:54 ` Eric Wheeler 2019-12-27 1:47 ` [dm-devel] kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178 Eric Wheeler 2019-12-27 1:47 ` Eric Wheeler 2019-12-27 1:47 ` Eric Wheeler 2019-12-28 2:13 ` Eric Wheeler 2019-12-28 2:13 ` Eric Wheeler 2019-12-28 2:13 ` Eric Wheeler 2020-01-07 10:35 ` [lvm-devel] " Joe Thornber 2020-01-07 10:35 ` Joe Thornber 2020-01-07 10:35 ` Joe Thornber 2020-01-07 10:46 ` [lvm-devel] " Joe Thornber 2020-01-07 10:46 ` Joe Thornber 2020-01-07 10:46 ` Joe Thornber 2020-01-07 12:28 ` [dm-devel] [lvm-devel] " Joe Thornber 2020-01-07 12:28 ` [dm-devel] " Joe Thornber 2020-01-07 12:28 ` [lvm-devel] " Joe Thornber 2020-01-07 18:47 ` [dm-devel] " Eric Wheeler 2020-01-07 18:47 ` [dm-devel] " Eric Wheeler 2020-01-07 18:47 ` Eric Wheeler 2020-01-14 21:52 ` Eric Biggers 2020-01-14 21:52 ` Eric Biggers 2020-01-15 1:22 ` Mike Snitzer 2020-01-15 1:22 ` Mike Snitzer
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.